466: Attacked by a Goose on the Way to the Office
Transcript from 466: Attacked by a Goose on the Way to the Office with Ralph Hempel, Christopher White, and Elecia White.
EW (00:00:07):
Welcome to Embedded. I am Elecia White, alongside Christopher White. Our guest this week is Ralph Hempel, and we are going to talk about engineering and toys.
CW (00:00:17):
Hi Ralph. Welcome.
RH (00:00:19):
Hi, how are you guys doing?
EW (00:00:20):
Good.Good
CW (00:00:21):
Better now that the computers are behaving.
RH (00:00:23):
Oh, I know.
EW (00:00:25):
Could you tell us about yourself as if we met at, I do not know, electronica?
RH (00:00:31):
Sure. Hi, my name is Ralph Hempel. I have been doing embedded systems for about 40 years now. I have programmed on everything from punchcard based computers, to HP calculators, to micros, to embedded Linux systems. But my favorite thing to program is M4 and lower class MCUs.
(00:00:47):
I led the firmware development team at the Lego Group for the last seven years. We developed the latest generation of Lego Mindstorms, as well as all the powered up components. So that is all the recent Technic and train electronic elements.
(00:01:06):
Now I am back in North America at Edwards Fire and Safety, as a staff firmware engineer. So it has been a fun ride.
EW (00:01:13):
I want to ask about your career and your time at Lego, and all kinds of questions. But first we are going to do lightning round. Are you ready?
RH (00:01:21):
I am ready.
CW (00:01:22):
Most underappreciated Lego piece?
RH (00:01:26):
Oh, here we go <laugh>. Most underappreciated Lego piece? I would say... My goodness. Now you are really stuck with me now here, because that was not on the list, you guys.
EW (00:01:42):
<laugh>
RH (00:01:45):
There are so many parts!
EW (00:01:45):
"Question may include, but are not limited to."
CW (00:01:45):
<laugh>
RH (00:01:45):
But are not limited to. Oh, okay. Thanks. I did not read the fine print, obviously. Most underappreciated Lego piece. I would have to say the Hassenpin. It is not actually called the "Hassenpin," but it is actually a right-angled Lego Technic connector-
CW (00:02:01):
Okay, yeah.
RH (00:02:01):
That was used for joining beams together. Now of course the Lego group has actual square frames or rectangular frames that make that a whole lot easier. But there is a really fun fan story about how that part came together, which we might get into later on in the call.
EW (00:02:18):
When someone finds out what you do, what question do they always ask you?
RH (00:02:23):
"Oh, that must be so cool. Can I get some Lego?"
EW (00:02:25):
<laugh>
CW (00:02:25):
<laugh>
RH (00:02:25):
Yeah, that is it. No, but it is actually kind of fun at parties, is like "What? You worked at the Lego group?" "Yeah, yeah." "Oh, that is cool. Where in the States were you?" I am like, "No, it is a Danish company. I was based in Billund in Denmark." It is always kind of fun to just get around that.
(00:02:43):
That is much more interesting than saying, "I program embedded systems. That is computers you cannot see, like in the microwave oven or in your engine controller unit."
CW (00:02:53):
Yeah. "We write software for things that are not computers."
EW (00:02:54):
<laugh>
CW (00:02:54):
Yeah, that is my usual answer.
RH (00:02:58):
Yeah, that is a real buzzkill. Lego Group. That is awesome, yeah.
CW (00:03:03):
Favorite course you took in your academic career?
RH (00:03:06):
Favorite course? Actually, that is a great question. So in high school- It actually was not a university course, it was a high school course. In Canada we have an algebra and functions and relations in calculus course, in what was grade 13. We do not do that anymore here either.
(00:03:20):
At our school at Jarvis Collegiate in Toronto, we had two excellent math teachers and they taught a course called "Triple Math." So instead of having all of those three courses with three different teachers, they actually made their own textbook, and we had all of those courses with one teacher for the whole morning, with usually a one course break in the morning.
(00:03:44):
That sounds really brutal, but it was the best preparation I had for university. I did not see anything new in math until the second half of second year. So I am really grateful for those teachers.
EW (00:03:58):
Speaking of university-
CW (00:03:59):
Oh God.
EW (00:04:00):
Have you gotten bitten by a goose? Are you still afraid of geese? And are the geese afraid of you?
RH (00:04:07):
Ah, you have done a little bit of background work on the University of Waterloo. Yeah, there are a lot of geese. We have geese in our yard here, up in Ontario as well. I have not been bitten by a goose, but I have been chased by the swan that lives on our river. Not really afraid of geese.
(00:04:25):
But my son, one of my boys, works at the Bruce nuclear plant, and he was attacked by a goose on his way into the office.
CW (00:04:30):
<laugh>
RH (00:04:30):
It is a thing. Yeah.
CW (00:04:35):
Best language to learn programming? And which one did you learn first?
RH (00:04:41):
The first language I learned was HP keystroke programming, for Hewlett-Packard calculators.
CW (00:04:46):
Oh, yeah.
RH (00:04:46):
That is really what got me hooked on it. The first language I learned, I think was assembler, 6502 assembler on a Commodore PET.
CW (00:04:56):
Yay! Sorry. Makes me happy.
RH (00:04:58):
<laugh> So that was kind of cool. I mean, the tools that we had available then, are so much different than what we have now. And we will talk about KIM-1s and all those other cool retro computers that are coming around again. But yeah, that is one.
(00:05:15):
But if I had to learn a first language today, I would probably pick Python, as a getting started language. Or Scratch, if you are-
EW (00:05:25):
Younger.
RH (00:05:25):
Under 12 years old.
EW (00:05:28):
Do you have a favorite fictional robot?
RH (00:05:31):
I do, and it is Chappie.
EW (00:05:33):
Cool.
RH (00:05:34):
From the movie, "Chappie."
EW (00:05:35):
Hm-hmm.
RH (00:05:35):
Have we seen "Chappie"?
(00:05:38):
Have you seen "District 9"?
CW (00:05:40):
I have not seen "District 9." I know of it. I know of "District 9."
RH (00:05:44):
Okay. Chappie is a cool robot, because he is supposed to be a police and surveillance control robot, but he gets kidnapped and is put to work in interesting ways. It is actually a fun movie. Well, it is not actually a fun movie. It is kind of a depressing movie, but it is a-
CW (00:06:04):
You are changing your story around.
RH (00:06:06):
The character Chappie suffers some real human things. I think he is probably the most human robot that I can think of.
EW (00:06:18):
<music> I would like to thank our sponsor this week. Nordic Semiconductor specializes in ultra-low-power wireless communication. They have a wide technology portfolio, including Bluetooth Low Energy, Low Energy Audio, Bluetooth Mesh, Thread, Matter, cellular IoT, and Wi-Fi.
(00:06:38):
They have thousands of customers worldwide, with 40% market share in Bluetooth Low Energy. They have nearly two million System-on-a-Chips produced every day. Sounds like a system that you cannot go wrong with. So please check out nordicsemi.com.
(00:06:57):
If you would like to win one of the Power Profiling Kit IIs that our sponsor is so generously handing out, please send us an email, that is show@embedded.fm and tell us what your favorite PPK2 feature or spec is. Again, thank you to Nordic for sponsoring this week's show. <music>
(00:07:24):
Okay. Moving out of lightning round. I remember Lego Mindstorms coming out like 20 years ago?
CW (00:07:33):
More, sorry.
EW (00:07:33):
25?
RH (00:07:35):
25. This year would have been the 25th anniversary.
CW (00:07:38):
I remember getting the first version of it.
EW (00:07:39):
It was really cool. We were out of school, so we- You mostly played with it, I think.
CW (00:07:46):
Yes, but I also worked with computers for a living.
EW (00:07:48):
You played with it a lot.
CW (00:07:48):
So coming home to work with computers for fun was not working out.
EW (00:07:52):
But I was so fascinated by it. And then it seemed to go away. Was that just my attention, or?
CW (00:08:00):
<laugh>
RH (00:08:03):
I think maybe real life caught up with you. I think that is true. Your attention. No, actually the Lego Mindstorms brand has been up and running for the last 25 years. This year, I think the Lego Group retired it or officially discontinued Mindstorms.
(00:08:16):
You would have been familiar with the RCX. That is the yellow brick, way back when. That was where I really started hacking on Lego. So I am going to go back and tell a little story about Usenet. Do you remember the actual Usenet system?
CW (00:08:31):
Oh yeah. I was Usenet administrator for our college, so, yes.
RH (00:08:34):
Oh, okay. So Usenet had- This is long before social media, as you all know, and there was a group called rec.toys.lego. On that group, we had a whole bunch of discussions on, "Hey, we should- If we are electronic geeks or embedded systems geeks, we should make some sort of a controller for Lego creations." Some of us that have actually made commercial products realize that, hey, that is just not too practical. Or at least it was not 25 years ago.
(00:09:04):
Then the Lego Group brought out the RCX. A group of four of us, Kekoa Proudfoot, Mike Gasperi, myself and Dave Baum. We were the original hackers on the Lego RCX. So this is the reverse engineering story. Do we have time for a cool reverse story?
EW (00:09:22):
Oh yeah.
CW (00:09:22):
Yep. Got time for anything.
RH (00:09:22):
Cool. So Kekoa, he was a grad student, I think at Stanford. He came up with the idea- The original communication with the RCX was over a USB- Sorry, it was not USB. A serial light link, or a little infrared tower. He sniffed on the serial wires, and realized what the protocol looked like, and then was able to- Because you had to download the firmware to the brick every time, initially. It was only 30K or something like that, so it did not take very long.
(00:09:56):
In the course of sniffing that out, he was able to figure out what kind of processor it was, because you could figure out what the assembly language was. Then he wrote a little dumper that would then replay the entire contents of the ROM back on the serial port.
(00:10:13):
From there we were able to reverse- Or he was mostly able to reverse-engineer the assembly code, and make an annotated listing of it. We were all- That was really cool. So then I wrote a Forth interpreter for the RCX. This got the attention of the Lego Group, the work that we had done on the reverse engineering of it.
(00:10:38):
They invited us down to MIT for the very first Mindfest celebration. There we got to meet the engineers that made the RCX. They looked at our disassembled code and they were like, "Hey, this looks actually better than our source code."
EW (00:10:53):
<laugh>
CW (00:10:53):
<laugh>
RH (00:10:53):
Because it actually had comments and everything. That was my first introduction to the engineering team at the Lego Group. I stayed in touch with them for years and years afterwards.
(00:11:05):
Getting back to what your original question was, Elecia, the RCX was the first generation. About every seven or eight years, the Lego Group would make a new generation. So then after that came the NXT, which was mostly a whitish grayish brick, with orange colored highlights. That was using an AT micro SAM21 or something like that.
(00:11:29):
The next generation was the NXT. That was the full-blown Linux. So a little- That was a TI AM335, I think, based unit. And then the most recent generation is we are back to STM32F413. That is the most recent Mindstorms, that was released I think in late 2018 or early 2019.
EW (00:11:55):
When you were doing this disassembly, was that when you were also working at a company for safety and fire suppressant?
RH (00:12:04):
Yes. Yep.
EW (00:12:06):
How was it different to be thinking about toys versus important products?
RH (00:12:13):
Versus important products? Yeah, actually it is an interesting leap between the two, because in some ways they are extremely different. And in other ways they are not different at all, because they are both highly regulated industries.
(00:12:26):
The Lego Group is always really super concerned about child safety, and more recently with security and online safety and things like that. The product itself had to be intrinsically safe and not disassemblable, because of certain toy safety standards around the world. So the whole openness of the product, you know, for right to repair and all those things that are coming out in the next few years, those are going to be really difficult to reconcile with toy safety.
(00:13:01):
The Lego Group made Mindstorms in quantities of hundreds of thousands every year. Every part was individually tested, so the motors, the sensors, they all went through separate functional test fixtures. It is really not that different.
(00:13:17):
The main difference with fire safety and safety equipment is that it just sits in the wall waiting for the big day, right? Waiting for somebody to pull the fire alarm or something to happen, something interesting, please. But it has to sit there and work for months, years, and in some cases decades, without being power cycled. And that is a real challenge.
EW (00:13:40):
That is pretty different from toys, which if something goes wrong, you just power cycle it. And nobody really- I mean, sure, there is the safety issue of making it large enough so that a child cannot swallow it, and making sure that it does not electrocute you very badly.
CW (00:13:53):
Electrocute you very badly. Oh, okay. I did not know there were varying levels of electrocution.
RH (00:13:57):
"Do not electrocute the kids very badly." That is our motto.
EW (00:14:03):
<laugh>
RH (00:14:03):
No. The key thing there though is that, I mean, you would not think about it, but there are things like light intensity. The LEDs, they can only be so bright. You have to deal with flashing for other light sensitivity. You have got to deal with heat on the batteries, when you are driving a lot of motors in stall mode, so that the batteries do not get overheated. So there is a lot of stuff that you would not think would be relevant, but it really is.
EW (00:14:32):
You listed processors that are off-the-shelf processors. You did not do custom chips?
RH (00:14:39):
Earlier generations would have done custom chips, so they would be ROM masked. The RCX was an H8/300 from Hitachi, I think, and that was a masked ROM part. After that, the parts were mostly firmware updatable. But yeah, no custom parts. Those are generally way more expensive to produce. And the quantities that were- The overall lifetime quantities, that just did not justify the cost or the risk.
EW (00:15:08):
I worked at LeapFrog, where we made custom chips. But we used the same chip for multiple product lines, and that worked out well. Yes, masked ROM. There was a time when masked ROM was a little cheaper, and then all of these chips came out with flash, and firmware update was just so nice.
RH (00:15:31):
Oh, I know. I will tell you a funny story about firmware updates. When I first started in the fire alarm industry, the signature panel that the Edwards Group was making was- You had masked, not masked, EEPROMs in them. So you remember the old windowed EPROMs that you had to stick into an eraser? Or the sun, if you did not have an eraser with you.
CW (00:15:53):
<laugh> Yeah.
EW (00:15:55):
<laugh> Or you were just by accident trying to erase your system without knowing it.
RH (00:15:59):
Yeah, that happens too. So the trick there is that you cannot just- You could send a technician there with- This is literally 35, 40 years ago. You could send a technician onto the site with the files and everything. But on a 300 baud modem, it would take forever to send the files down. So at some points, we were actually putting people on airplanes with a sleeve full of EPROM devices, that you could literally pull the old ones out of the fire alarm and put the new ones in.
(00:16:26):
So the whole idea of field upgradable, or even over-the-air updates as are much more common now, or more common now than they were before, or DFU updates, that stuff just- We did not even think that that would happen. But it is here now and we should really take advantage of it. It has changed everything really.
EW (00:16:47):
With Lego, I have a question from Svec about what processing did you have inside the Mario Lego sets? These were the ones that you could drive around and pretend to be in Mario Kart?
CW (00:17:02):
Maybe. There was that one, but there was also- Yeah, I am not sure which one he is asking about.
RH (00:17:06):
Yeah, so there was a line of products that the Lego Group did in partnership with Nintendo, and that was a small Mario character. So there was a Mario character, a Luigi, and I think a Princess Peach came out after that.
(00:17:19):
It had a little accelerometer and a gyroscope in it.
CW (00:17:23):
Oh, okay.
RH (00:17:24):
You could sort of bounce the character around and collect points. And it had a color sensor, so it would tell if it is on a red tile, or a- And there was actually some specially printed barcoded tiles, that would give him different points or different actions. That is the same basic- Oh no, that was a different part. That was a TI part with Bluetooth Low Energy enabled in it.
EW (00:17:49):
Let me see. TI CC2640? 50? 30?
RH (00:17:57):
One of those.
EW (00:17:57):
<laugh>
RH (00:17:57):
And then there was a bunch of sound files that went with that. Yeah, it was a cool little product. But I think some enterprising fans actually put it into a Mindstorms or a BOOST. BOOST was another robotics line that we had years ago. You could put the Mario figure in it, and then you could drive around.
(00:18:21):
That is usually fan type stuff, where the fans take our product- I say our product, I do not currently work there anymore. But they would take the Lego products, and really extend them in ways that we really did not expect. And that is cool.
EW (00:18:37):
Yeah, I mean that is not something you are going to get with fire alarms, for sure.
CW (00:18:42):
Yeah. What you do not want. "Look what I hacked my fire alarm to do," is not something you want to hear.
RH (00:18:47):
No.
EW (00:18:49):
Do you have any good fan hacking stories?
RH (00:18:53):
Well, I am a good fan hacking story.
EW (00:18:55):
Yes.
RH (00:18:58):
I have made firmware for every generation of Mindstorms, up until the one that I was actually hired to make. So everything from pbForth on the RCX, to a Lua interpreter on the NXT. Lua was- It is actually a pretty cool little language. And then we put the full Debian Linux on the EV3. So those are kind of cool hacks.
(00:19:24):
I think I sent you some links that I think will be showing up in the show notes. I am always so impressed by what fans were able to do with things, that we just really never imagined. I think there should be a link to somebody who has made a log house maker out of cucumbers. So he actually takes a cucumber and slices it into little log size pieces with notches and everything and makes a log house out of cucumber pieces. Yeah, you do not have to look far to find some crazy fans to do some amazing things.
CW (00:19:57):
When you were initially reverse engineering the first one, it sounds like it was all external. That you and your friends looked at the serial communications and figured everything. Was there- Were you tempted to open it up and start poking at things, or was it just easier to go from the outside? Because when I am thinking about reversing things, sometimes I want to open it up and see, okay, what parts are these?
RH (00:20:22):
Well, yeah, a combination of things there, Chris. That is a good question. So we started with- I think it was again mostly Kekoa and Dave Baum that did the reverse engineering. But you start with the serial protocol, all the affordances that are external that are easy to get at, you do that first. But then of course those elements were fairly easy to open up without destroying the package.
(00:20:44):
You could see that it was an H8/300, and you could figure out where the external flash and the external RAM was. You could figure out where all the parts are, what the motor drivers were and where the pins went. So we did a fair bit of wringing out of this PCBA. But it is really just a case of slowly reverse engineering, building up your knowledge of the product bit by bit.
(00:21:12):
Then finally the big day comes and you are able to do a complete dump. And then you send that binary dump into a reverse disassembler, and wow, there it is. It is all the source code. Cool. Now we can start commenting it, and figuring out what goes where. It was surprisingly complete. It was pretty cool.
(00:21:32):
The story again that the Lego engineers were like, "Oh, how are we going to encode this serial stream?" So you have to think about it from the other side. "First of all, is anyone even going to bother trying to reverse engineering?" "No, not really."
EW (00:21:44):
<laugh>
RH (00:21:44):
"How are we going to protect the stream?" "Well, we are going to balance the data out, by sending out the inverse of every byte." So every byte really gets sent twice, once in the positive, once in negative, just to balance out the energy. And, "Ah, they are never going to reverse engineer that, because if they did, they would have to- Sure, it is binary code, but now you would have to reverse disassemble it. Who is going to do that?"
EW (00:22:11):
Who would bother? It is just a toy.
RH (00:22:12):
"Who would bother? It is just a toy." That whole philosophy, we showed that that was, "Hey, it is just a toy. But there is a lot of people out there that are really interested in it, and motivated to figure out how this thing works." That can be taken for good or for bad. But fortunately, hacking on Lego was always a good thing.
EW (00:22:31):
Yeah, I mean when we think about hackers, it is like scary people who steal our identities. But the truth is sometimes you just want to see how something works. And Lego is such a good place to play with that.
RH (00:22:48):
And I think if you sampled many of your listeners or members of your Patreon group, they would say- Or any engineers that you know, most of them will have had Lego in their hands at some point in their lives. It is not that playing with Lego makes you an engineer, but most engineers, if not- Yeah, most engineers in my experience have touched Lego and it is a great prototyping tool.
(00:23:11):
I have used it- I mean the pop filter from my microphone here, that is actually a Lego frame with some-
CW (00:23:20):
<laugh>
RH (00:23:20):
3M sanding paper in between it, just to even out the airburst. We use it for all kinds of things around the house.
EW (00:23:29):
It is kind of like a 3D printer, but with blocks instead of filament.
CW (00:23:32):
And you are the printer.
RH (00:23:33):
Exactly.
EW (00:23:33):
And you are the printer. <laugh>
RH (00:23:34):
Well, yeah, exactly. And then of course when you really complete the loop, like I just did, we actually 3D print things like mounts for Raspberry Pis, so that they have Lego compatible pins. Then you can build your Raspberry Pi into a Lego creation. So yeah, we go all the way.
EW (00:23:53):
You worked with Damien of MicroPython. How did that come about, and where did it go?
RH (00:24:00):
Well, that was a really happy accident. So when I joined the Lego Group to work on this latest generation of Mindstorms, I looked around and surveyed what could possibly be put on the brick. We had a rough idea of the processor that we needed, and some of the features, but not all of the mechanical and visual or UI features. We will talk about those in a bit.
(00:24:21):
But I have always tried to put an interpreted language right on the brick. So that is why I put a Forth and a Lua interpreter on the brick. My goal is even if there is no app, so let us say the app support goes away, you can always program the device with a PC and a terminal, right? A terminal emulator.
EW (00:24:41):
How come I did not know that? Sorry, go ahead.
RH (00:24:45):
The official stuff cannot do it. That is just my way of thinking. I would love to be able to put an interpreter or a REPL loop on pretty much anything. Once you have a REPL loop, you can hack into the system and do more.
(00:24:57):
But my goal with the latest generation of Mindstorms was to put some sort of an interpreted language- We looked at Lua and it was not all that popular. And then I poked around and I saw Damien's MicroPython was getting quite a bit of traction, also with the micro:bit. The micro:bit has inspired a few things about Mindstorms as well.
(00:25:19):
So I checked out MicroPython on the micro:bit and a few other processors and like, "Hey, this is actually pretty good, and it might work." It totally fit on the processor we were using. We had some room left for drivers, so that we could talk to our motors and sensors and the display. It just was a natural fit.
(00:25:40):
But there was quite a bit of resistance, because previously the Lego programming app was a LabVIEW type app.
EW (00:25:48):
Hm-hmm.
RH (00:25:48):
Yeah. Hm-hmm. The idea of Python was not all that popular. And then we also had to think about it from the app development side. LabVIEW, it is an okay system. Personally, I never really liked it, but there are a lot of fans. A lot of kids have done a lot of amazing things with it. So that is kind of cool.
(00:26:09):
So we looked at Scratch and Blockly specifically. We did a proof of concept with the EV3 using Blockly. Blockly translated- At the time there was a thing that you could take your Blockly blocks and generate Python with those. So as a proof of concept, can we assemble a series of blocks, spit out some Python that we could interpret on a brick?
(00:26:34):
Once we had that proof of concept done, it was like, "Let us go full steam ahead." Python was then the language of choice. And then I talked to Damien just to make sure how would he feel about that being in our product. He was really cool with it.
(00:26:52):
He is probably, I do not know exactly how to say this, but he is probably one of the cleverest developers that was not a developer, right? I mean you know he is a physicist, and he has done some just amazing things with memory management. Over the years he has modularized and isolated the modules, the support modules for all the different MicroPython variants, really well. That codebase has grown.
(00:27:24):
I have learned a lot about how to write drivers and how to write Python code properly. Also how to write C code properly, from looking at his examples and the way he structures his code. I hope that that relationship has gone both ways. But he is just an awesome guy to work with. I just wish there were more people like him in the world.
EW (00:27:48):
I do like MicroPython. The more I use it, the more impressed I am by it. It is a lot of code, but it is good code.
RH (00:27:56):
It is, yeah.
EW (00:27:58):
So MicroPython for Mindstorms led to Pybricks?
RH (00:28:04):
Yes. The Pybricks is a fan created product. So that is David Lechner and- Sorry, mainly Laurens Valk and David Lechner together. They established Pybricks a few years ago. One of the other things that you may not know about Lego Mindstorms, is that currently all of the Lego Mindstorms products, you could re-flash them with your own firmware. There is very few people, you could count them pretty much on one hand, that actually made replacement firmware for those bricks. But Laurens and David were one of them, along with myself.
(00:28:40):
The Pybricks guys, they have made a full new firmware, and an app that is a downloadable app. It is a web app essentially, that you can put on your Android tablet or your PC. You can program the bricks now, all of the bricks, all the powered up bricks. So that is the Technic brick, the BOOST brick, the train brick, and of course the Mindstorms bricks.
(00:29:04):
You can program them all either with a block-based language that they have got developed as well, or with MicroPython. And that is part of the open firmware updatable product philosophy, that I championed at the Lego Group.
EW (00:29:21):
How much did you have to push for that? How much were they against basically what you did?
RH (00:29:27):
Well, how do I put this nicely? I do not think they were fully aware of the openness that was there. The product has always been open. We replaced firmware on the RCX, the NXT, EV3. They all had a way to put new firmware on it, so it was not really that much of a struggle to say, "Hey, we should open this up."
(00:29:48):
The harder challenge is getting the documentation and everything out there. That is something that we did not do all that well, frankly. But the Pybricks team has really done a great job with documentation and making the app available. So it carries on.
(00:30:04):
The nice thing about that open philosophy is that even if the manufacturer drops support, which Lego rarely does, but eventually they cannot support things forever. So that is where fans like Pybricks have really kept those products alive.
EW (00:30:20):
Switching questions or switching topics a little bit, this one is from a listener. Sahil asks, "How did you get into test-driven development?"
RH (00:30:31):
That is a great question. I was probably kicking and screaming.
EW (00:30:35):
<laugh>
RH (00:30:39):
In some ways. I had read about it. I read James Grenning's book, "Test-Driven Development for Embedded Systems" probably eight years ago or even more than that. There are a couple of- It took me three tries. I do not know if this is a typical story or your experience as well. Maybe we can talk a little bit about how you two feel about test-driven development.
(00:31:03):
For me, the first time I tried it was, "Hey, we need some unit tests around this code. Write some unit tests." And when you write unit tests for code that is already there, that is not very modularized or not very decoupled, you write really brittle tests. You write to get code coverage, and it is not really test-driven development.
(00:31:26):
The second time I did it, a few years later I tried to pick it up again, and it was kind of the same story, but I could sense that there was something there, it was getting better.
(00:31:38):
Then I rewrote- I have a memory manager that I wrote called umm_malloc and it is basically a malloc replacement for embedded systems without a memory management unit. That is when I wrote the memory manager and I wrote unit tests as I was going along, "the way you are supposed to do it" in quotes. I finished it and it was like, "Hey, wait a minute. I did not actually have to really touch the debugger while I was doing this, because I wrote tests to match functionality."
(00:32:08):
After that it was like, "I am never going back. I am never doing this any other way from now on." When I was at the LEGO Group, I wrote a little proof of concept for a byte code interpreter, again using TDD. Once we integrated it with one of our products, it just worked the first time, out of the box. So I am fully convinced that TDD is the way to go. But how do you guys see and experience TDD?
EW (00:32:41):
I love the theory, but I do not like the implementation. The whole CppUTest, the way the mocks work, it does not really work for me. When I am bringing up a flash driver, none of those things help me. It is the code on top of that. And that I tend to do more in a sandbox way, where I do not necessarily have the CppU driving it, I just have a series of tests I can run and I do it in a less formal manner.
(00:33:15):
The thing that I really, really, really like about TDD is the idea that even before you start writing the feature, even before you really think about how you are going to implement the feature, you need to think about how you are going to test it. So for me, it is a thought exercise because the implementation does not work the way I want it to.
(00:33:35):
But I think as a thought exercise of "How am I going to test this?" before "How am I going to implement it?" is just the way it should be. Waiting to think about how you are going to test it, until after you have implemented it, is going to lead to bugs.
RH (00:33:52):
Yes, absolutely. Chris, do you have any thoughts on this as well?
CW (00:33:57):
I have different thoughts. One is just practical. We are both consultants, and so we tend to be hired guns to come in and do something for some company. Usually they are very small companies, usually they are in a hurry, usually they do not enjoy paying a lot of money for things.
EW (00:34:16):
And they do not like to see things they cannot use right away.
CW (00:34:19):
Or their code is already in some state, or-
EW (00:34:22):
Interesting state.
CW (00:34:23):
So it is very difficult for us to come in and apply TDD principles in those situations. I am thinking of a project I am doing right now. They want a demo sometime early next year, and they want some certain firmware features to work. And I am building on top of a vendor's chip library with their drivers and things.
(00:34:46):
For me to sit down and say, "Okay, I am going to do this in a TDD way," I do not actually even know what that- I know what TDD is, I have read the book. But I do not know what that means in the context of a project like this, because I am not writing isolated code that I can test easily.
EW (00:35:03):
And you are bringing up a driver of hardware.
CW (00:35:05):
Right. I do not want to write mocks for their hardware, because I cannot even see that deeply necessarily into their- It is just a lot of complexity. But when you say things like, "Okay, I have this memory manager," or "I have a module or a piece of functionality," that is where I think it makes a lot of sense and you can do that.
(00:35:23):
But a lot of times I am not in that situation anymore. I am grinding low level firmware stuff, that is mixed in with ST's HAL, or some random company's implementation of CMSIS. Yeah.
EW (00:35:41):
Or optimizing. Like, DMA is not something that is that interesting in TDD, because it is a chip function.
CW (00:35:51):
Yeah.
RH (00:35:51):
Right.
CW (00:35:51):
So I just get confused by "How do I apply this?" and I end up going back to my old ways. But when I do have isolated functionality, and this was some of the stuff I did at Fitbit, where I was like, "Okay, this is the graphics library. It does mathematical transformations of stuff. I can put tests around this." and I did do that. That is very helpful.
(00:36:12):
But I do not think it is universally applicable. At least given traditional embedded development, and how chip vendors architect their stuff.
EW (00:36:24):
<laugh> 97 layers.
RH (00:36:24):
Yeah. We can talk about that at length, I am sure. One of the things that I do is as a bit of a shortcut is, first of all, if I take a vendor library or any third party code, I am just going to assume it works. Especially if it has test cases already, great. But I am not going to bother testing a HAL, like the STM32 HAL. So we stop there.
(00:36:48):
The second thing is that we do not mock out their HAL either, because that is just not- It is just a waste of time.
CW (00:36:58):
That would take you a year just to do that. <laugh>
RH (00:37:01):
Exactly. So the third piece that has worked for me anyways is, do not test on the target. So assume all your testing is happening off target, on your PC. That frees you up from having to bite off too much of a chunk. When you are doing that-
(00:37:22):
You said something really interesting, Elecia, which is "I think about how to test it, before I write the code." One of the tricks I use is, I turn that around and say, "How is this supposed to work? What is this actually supposed to do? And then, how do I test that that gets done?"
(00:37:42):
That way I am actually writing the code incrementally. So I do not really think about "What are all the tests that I need to do, to make this thing work?" It is more, "What specifically does this have to do, and how am I going to test that it did that thing?"
(00:37:56):
We probably should not talk a whole lot more about that, but it is an interesting topic. Once you get your brain around making sure that you can test off target, and then leave all the stuff that thousands of developers use and just assume that it works, that frees you up to work on the interesting part.
(00:38:13):
But Chris, when you are talking about a codebase that is already there, and your client does not want to spend a lot of money, and they do not want to have a lot of additional tests, then yeah, you are kind of stuck to do it that way.
CW (00:38:29):
Yeah. And a lot of this stuff, if I take it off target, there is not anything.
EW (00:38:32):
<laugh> Yeah.
CW (00:38:32):
Sometimes it is a driver, and I am moving a buffer from this driver to another driver, and it is okay-
EW (00:38:39):
The only question is how fast can you do that.
CW (00:38:40):
It is a DMA and eight lines of code, once you figure out how to do it. The tough part is figuring out how to do it, not that you are writing a lot of code.
RH (00:38:46):
Exactly. Yeah. So I think we are agreed that TDD is not for the whole project and it works in certain areas. One other thing is that once you have a codebase with TDD already baked in, then your newer developers are more confident making changes. Because they can just literally run the green test, and hopefully everything works.
CW (00:39:07):
Those should be tests that are run on check-ins, and things like that.
RH (00:39:09):
Yep.
EW (00:39:10):
But then if you do not have somebody champion it, then when that person leaves the project, they become failures.
CW (00:39:16):
Stale tests, and people stop paying attention. Yes, I have seen all that. <laugh> "Turn that unit test off! It always fails!" "Well, wait."
RH (00:39:25):
"Oh wait, when was the last time we ran this test?" "Oh yeah." So that is the other part of the whole TDD thing, is that it does not really work unless you have a pretty decent CI system.
(00:39:35):
One of the things that I brought in at the Lego Group was- When I joined, people were still bench-testing it line by line, or running motor tests manually. And then I said, "Hey, wait a minute, what if we automate some of this testing?" "Well, we cannot automate all of it, so what is the point?" "Well, okay. But what if we could automate 80 or 90% of it? Would that help?" "Yeah, okay. Yeah. But that is so hard to do."
CW (00:40:01):
<laugh> That does not make any sense.
RH (00:40:02):
So eventually we do little pieces. You just have to bite little chunks off, right? We ended up with a farm of 30 or 40 Raspberry Pis connected to our Mindstorms bricks, just continuously exercising code as we checked it in.
(00:40:20):
And if you ask people now, "Hey, would you want to go back?" "No." But it does take a CI- You have to have one or two people there that are dedicated, and love getting a CI/CD pipeline up and running.
(00:40:35):
And again, use the tools that are out there. GitLab has a great series of GitLab runners and tooling that works in that realm. And then when we had one or two Raspberry Pis, yeah, you can set them up manually. Once you get to 30, yeah, you are going to want to use Ansible or some other provisioning system to get your farm up and running.
CW (00:40:58):
It is like saying you do not want to a dishwasher, because it will not reach into the sink and put them away in the dishwasher for you.
RH (00:41:03):
Exactly.
EW (00:41:03):
But then, I do not want to babysit 30 Raspberry Pis, and figure out what went wrong with them.
CW (00:41:09):
Well, yeah.
EW (00:41:11):
I am not the person to set up your CI/CD system. I really respect people that are, but when we work for little companies, it is sometimes hard to convince them...
CW (00:41:22):
You need another person.
EW (00:41:23):
You need a whole another person just to do tool stuff, because-
CW (00:41:28):
Every company needs a tool person.
EW (00:41:28):
Many engineers are not geared to enjoying that.
RH (00:41:33):
It depends on the size, and there are some engineers that love it. So the trick really when you have a big team or even a small team, is to not insist that everybody does all the things. Because they are going to hate- Most people are going to hate doing 80% of the stuff.
(00:41:47):
But the trick is finding a nice cross section of developers, where somebody really has a bee in their bonnet about TDD or about CI/CD or clean code or whatever. And make sure that they are the evangelists for that aspect of programming.
EW (00:42:06):
So TDD is often paired with Agile, and Agile has a fraught history with actual hardware. I mean, some people love it and some people think, "Wait a minute. Some of this stuff takes time. You cannot just change it willy-nilly."
CW (00:42:23):
Some people think all kinds of things about Agile. Some of them are in this room.
EW (00:42:26):
<laugh>
RH (00:42:30):
<laugh> Well, that would be my first question to you. In your world, what does Agile mean? Does it mean just running in sprints? Or? What is Agile in your mind?
EW (00:42:40):
That is the main problem for me. It is like programming languages without actual specs. It could be anything, because its-
CW (00:42:48):
It is Agile. <laugh> Sorry.
EW (00:42:49):
Definition changes. Is it standups?
CW (00:42:52):
Is it scrum?
EW (00:42:52):
Is it points?
CW (00:42:53):
Every time I have done quote "Agile" at a company, it has been mostly scrum and standups and retros and all of that stuff.
EW (00:43:01):
I can tell you I do not want daily meetings.
CW (00:43:03):
Point poker and a bunch of stuff that frankly does not help.
RH (00:43:07):
Does not work.
EW (00:43:07):
I do want to interface with the users early, and make sure that their needs are taken care of.
CW (00:43:14):
Yes. But at every company I have worked at, the user was just my manager.
RH (00:43:17):
Yep.
EW (00:43:18):
So many times.
RH (00:43:20):
We have a whole product owner team that deals with the customers, and you are not going to get to in front of those guys. Yeah, it is a challenge. So how does Agile work with TDD? So first of all, I do not think that- Well, in my mind, I have not seen Agile work really well in a lot of places.
(00:43:39):
I have been in one project, my last project that I worked at the Lego Group where we had- It might have just been an amazing team where things just kind of clicked. I had a project manager that was really good with the upward communication, and with all that stakeholder management stuff, because I tend to say "No" a lot and I am not good at it. And I handled the technical side of things.
(00:44:06):
We had a complete cross-functional team. So we had some electronics, we had board layout, we had BOM specialists for- When I say "BOM," bill of materials, not actual bomb. And then we had the whole procurement interface, and then the firmware and production equipment. It was really cross-functional.
(00:44:25):
As you know, in the life of an embedded product, you are going to be heavy upfront with electronics and board layout and bill of materials and procurement. And then we have firmware. Once that is all up and running, you might get your dev boards in and you have to integrate. Eventually, you have to put it on production equipment.
(00:44:46):
We took advantage of the cyclical nature of that, so that while the hardware guys were sorting stuff out, we were writing the firmware, off target without an actual HAL. So the whole lifecycle of boot time and heartbeats and task management and all that stuff, that happened off target.
(00:45:08):
So that when we got hardware, we just had to make sure that- Oh, and we used the Nucleo development boards, of course. You can get most of your stuff working on a dev board. When the real hardware came in, it generally took a day, maybe two days sometimes, to get it up and running. We had exactly one flipped wire- One pair of wires that were the wrong way. RX and TX, of course, on a UART.
CW (00:45:33):
Oh. You never get that right the first time, so that is okay.
RH (00:45:35):
No. Well, fortunately, out of I think six boards, we only got it wrong on one board. It was actually- It was not RX/TX, it was the wrong UART port. But anyways, we got that sorted out. And so you take advantage of those cycles.
(00:45:49):
How does this tie in with Agile? Well, over the course of the six or eight months we worked on this, we could come together for a two week sprint and say, "Hey, what is the most important thing that we have to accomplish as a team, in this two week sprint?" Setting sprint goals-
(00:46:05):
Maarten Dalmijn has a great book on Agile development using sprint goals. He feels that sprint goals are one of the most important things. If you cannot state what your most important thing to do is as a team, then it is really hard to use Agile effectively.
(00:46:23):
But again, that worked for me and my team on that project. But it is not everywhere, especially if you are trying to push Agile into a waterfall world. Because then you are just doing Agile ceremonies. It is not really-
EW (00:46:43):
Agile ceremonies.
RH (00:46:43):
"Agile circus," I guess you could call it. It is not really Agile. Agile is not "Let us make all the changes at the end." It is not that either. There is definitely some planning that has to happen, but what Maartin says is, "Make humble plans. Plan more. Plan better later." Does that make sense?
CW (00:47:02):
Yeah, that makes sense. And I agree with all of that. Certainly it is difficult to not have visibility into short-term goals. It is easier to work on small pieces of things. It is good to have, and this is kind of anti-Agile in a way, but it is good to have requirements and things you are working towards that guide all of that. I do not disagree with any of that. And the best teams I have worked on, have been small teams that just did all that kind of thing naturally.
EW (00:47:31):
And that communicate. Agile is about communication, and not going off into your closed-door office and doing something that you never-
CW (00:47:38):
Closed-door office! It is 2023!
EW (00:47:40):
Ish. Dream okay.
CW (00:47:42):
First of all, we do not go to an office. And second of all, if we did, it would be basically football bleachers.
EW (00:47:47):
But I have been at companies where somebody went off into their closed-door office, and came back with a grand thing that did not integrate at all, and that is no good.
CW (00:47:58):
I know, I know. Yeah. Or the manager decides they are going to try their hand at code, because they have not done it in a while. Sorry <laugh>.
RH (00:48:03):
That is always a good- Yeah. Well, that was my downfall as a manager, is that when the team is shorthanded-
CW (00:48:09):
Oh, me too, yeah.
RH (00:48:10):
Just apply, "I got a spare pair of hands and a keyboard. I can help." And I did. It is something we did not help and do things. It is just- That is the other thing that is hard for developers as they transition into management, is keeping your hands off the keyboard and keeping your eyes on the big picture.
(00:48:30):
But usually when the team is shorthanded or stuff happens, you are going to want to help and try to help and pitch in. So it is hard to manage those things. Yeah.
EW (00:48:42):
Were you already a manager when you went to Lego Group?
RH (00:48:45):
No, I was not. No. I have been around managers. I have been a lead developer and heading up root causing teams, for when problems really hit the fan, and customers are ready to pull the panel out of the wall. That is when I would go in and do root cause analysis. So I am aware, and I knew how to handle a reasonably sized team, but this was the first time I had been in a formal organization with leadership teams and all of that.
(00:49:16):
But the surprise for me was, "Hey, senior manager at Lego." You walk into the office the first day, so "Where is my office?" And "Right here, next to the PA." It is just everything was open. So there are very, very few closed offices at the Lego Group anywhere.
(00:49:33):
I know people have strong opinions about that. In some environments it works, and there are other days when we just worked from home, because you needed to get work done. So that was a little bit of a-
(00:49:46):
COVID was an interesting time for us, for sure. You were able to get quite a bit of work done at home. And then there was some, "Hey, wait a minute. Why are people also effective at home, as they are in the office?" I am sure there is a lot of debate going on around that today.
EW (00:50:06):
Yeah, it did show some things we thought were true, or thought were- Some assumptions were not correct.
CW (00:50:15):
Yeah. Well, people have a lot of investment in real estate, so.
RH (00:50:18):
Yes, they do.
EW (00:50:22):
What advice would you want to go back and give yourself, as you became a manager from a tech lead?
CW (00:50:28):
Oh, yeah.
RH (00:50:32):
I did not really appreciate how difficult that first level of management is. So the interface between your team or an extended team, and the director level and higher levels, is you are really stuck. Your team is counting on you to run an interference for them and get resources. But in most corporate structures, you have also got fixed budgets and allocation cycles and all of that stuff. So it is a tough job.
(00:51:07):
I think the advice would be to listen to folks like Michael Lopp, or Rands. Just listen to as many manager type podcasts as you can, and figure out "Is this really what I want?" I wish more companies had a ladder for technically competent people, rather than folks that want to get into management. Both are needed, right?
CW (00:51:33):
Yeah. The bigger companies tend to have those now. Smaller companies still I think are like, "Oh, you have reached your limit. It is time to become a manager." It is like, "What?" <laugh>
EW (00:51:45):
I think that in order to become an individual contributor at the next level, having some management experience is probably good.
CW (00:51:52):
Yeah, but you do not want to inflict that on a poor team. It is like, "Here, go figure out a few things." That is not in the best interest of the company either. So it is a balance there.
RH (00:52:02):
No, you have really got to- It is a completely different way of thinking about projects and resources and all of that stuff. Honestly, I am quite happy as a staff engineer working on certain kinds of problems. Also trying to advocate for not being the person that gets all the hard and interesting problems. Trying to bring on juniors while you are doing your root cause analysis, so that they can learn from it.
(00:52:30):
Because it is no good as a tech lead to be an additive effect. You want to be a multiplier. So that means getting your knowledge and your wisdom and any mentoring that you can do, out into your organization, rather than- Because nobody reads documentation. There is no sense writing down your after actions, because nobody reads that stuff.
(00:52:51):
It is much, much more effective to be working shoulder to shoulder with somebody who knows what they are doing. And also to ask other people for, "Hey, what would you do in this situation? What is your next step?" If I was working together with somebody who is less experienced, it is "Okay, here is what we know. Here is where we are. What is your next step? What do you think is the next thing to do?" That is an effective way of learning, I find.
(00:53:21):
For better or worse, Uncle Bob of- We all know-
EW (00:53:24):
Bob Martin.
RH (00:53:25):
If I say "Uncle Bob," we all know who we are talking about?
EW (00:53:27):
Yeah.
RH (00:53:27):
Bob Martin. He had an interesting thing at the beginning of one of his talks. That is that the number of programmers in the world doubles about every five years. Right? And if you think about that from the other side, that means about half the developers have less than five years of experience. That is kind of scary.
EW (00:53:49):
I have thought that the role of senior engineers should be primarily to make more senior engineers.
RH (00:53:55):
Yep.
EW (00:53:55):
Our role is to help people figure out if this is the career they want, and then to grow in the career. Not necessarily to get things done, which is hard because I really enjoy getting things done.
RH (00:54:11):
Yep.
CW (00:54:13):
But it is also not our particular careers. That does not work, because-
EW (00:54:19):
It works some.
CW (00:54:20):
Okay. Well, you do outside mentoring and stuff.
EW (00:54:23):
I do outside mentoring. And even in some of the gigs I have, I do manage to structure it, so I have somebody to play with.
CW (00:54:30):
But they are not necessarily people who are becoming senior engineers. They are people who are learning about embedded systems, on the side. But it is fine. Yeah.
EW (00:54:39):
Some of them.
CW (00:54:39):
But it is challenging as consultants to do some of these things.
EW (00:54:43):
Yeah. It is one of the things that I miss about regular work, full-time work, is being able to do a little bit more mentoring outside. I mean, I have formal mentoring relationships with folks, but they are all like, I cannot help them in their careers directly. I can only advise.
RH (00:55:03):
Right.
EW (00:55:05):
You said you are 40 years into your career. We are about 30, almost, and sometimes it is exhausting. How do you stay engaged? Well, we have both been fighting burnout.
CW (00:55:22):
<laugh> I just thought you were asking the question for me. <laugh>
EW (00:55:26):
No, not just for you.
CW (00:55:27):
Okay. <laugh>
RH (00:55:28):
Well, no, that is a great question. If you are going to be in this- I should not call it a lifestyle. In this career for any length of time, yeah, you are going to go through cycles. It is not all going to be a nice, happy, happy journey.
(00:55:42):
Try to keep learning. So one of your previous guests- Try to learn in your adjacent field. So if you are interested in how things work, you maybe want to get a 3D printer and try to figure out how can you optimize that kind of thing. So try to always be continuously learning.
(00:56:07):
Watch what the cool kids are doing. I learned on vi, so I have all those vi keystrokes in my fingertips, and I was very reluctant to give it up. And then I saw somebody VSCode, and it is like, "Hey, wow. This is pretty cool." Before that even there was like, "Oh, we got to get into Docker, because that is really awesome." I am like, "I do not need Docker. I have got VirtualBox." Or some other VM systems that we were using.
(00:56:37):
And now, for embedded systems, I use Docker and VSCode. So I can just spin up a Docker instance with GCC and Sphinx and Python and all the stuff that I need for embedded development. I put the host GDB interface, so the ST-LINK if I am using that, or SEGGER or whatever, run that on the host machine and then connect from the Docker machine to the debugger, to the GDB backend, over a port. And "Hey, bingo. All of a sudden I am using VSCode as an IDE." That is just cool.
(00:57:14):
So I guess the real way to fight burnout, is to keep doing interesting things, find cool people to talk to, and always hang around the younger developers. Because they are bringing in new ideas all the time. It is just a cool way to stay engaged.
EW (00:57:33):
What are your preferred ways to learn things?
RH (00:57:36):
That is a question that I had for you as well. Would you prefer to learn at a conference? Do you want to do individual learning, or learning as a group with a team? For me, it is when I need to learn how to do something, I go into it shallow depth, and just get things done. So a little bit more cookbook style. I will find a decent web tutorial and follow that for a while.
(00:58:05):
At a certain point you have to decide to either dive in or just stay on the surface. And when I do want to dive in, then I will just take the time. I will book out a half a day or something like that, just so you do not get overly burned out. And just really deep dive and learn the stuff that I need to learn.
(00:58:25):
But before I do that, I write out a list of, here are the things that I want to accomplish with this new tool or with this new idea, and then try to apply it right away. Does that make sense?
CW (00:58:40):
Yeah.
EW (00:58:43):
The way that I prefer to learn is to snooker people to come onto the podcast, and talk to me about what it is I want to learn about. So that is been pretty nice actually.
RH (00:58:53):
You have had some awesome guests in the past. I have really enjoyed listening to some of the back issues of the Embedded podcast.
EW (00:59:01):
Thank you. It depends on what I want to learn. I have been in a multi-year obsession with origami. That continues and I go in different directions, and then I go back to the curves that I like. I do not have a purpose for that. That sometimes is the hardest part about learning, is when I do not have a goal.
(00:59:26):
My problem with the work you did with reverse engineering is that I would have gotten partway into it, and then I would have looked around and said, "Oh wait, I cannot bill for this, and it is not helping anyone." And then I would talk myself out of it.
(00:59:42):
I know I need to do silly things. Silly things are fun, silly things keep you engaged. But I am not great at giving myself that permission. With origami, I can say, "Well, it is mathy. And you are using computers. And you can use your hands. And it keeps you away from screens." Which if you look at all of those together, that cannot all be true. But yeah.
RH (01:00:07):
So can you actually simulate? You mentioned using computers to work out your origami work. So can you actually simulate some of that on the computer, and then go from there? Or how does that work?
EW (01:00:20):
Amanda Ghassaei has this amazing web simulator, that simulates origami. There is a whole bunch of theory about developable surfaces, and whether or not things collide, and how the pliability of your material. Her web simulator is just fantastic. I think it is origamisimulator.com.
(01:00:42):
When I do math, it is usually in Python, making patterns that I can then repeat, usually slightly changing. So if I want to get an ocean wave, I can do a big sea and then a sea that is a little less and then a sea that has less, but a few more sine waves at the end. And then I can end up with a 3D curvy wave that is singular. Boy, is that not a great description!
RH (01:01:17):
Well, I hope we will add that to the show notes then. I think that sounds like an interesting thing to look at. An interesting rabbit hole to go down for a little while.
EW (01:01:26):
It is. I have learned quite a bit about Python and SVG and plotting things. And how snails actually evolved. Yeah, it has been fun.
RH (01:01:42):
But you cannot bill for that.
EW (01:01:44):
But I cannot bill for that. No.
RH (01:01:47):
<laugh> The trick there I think is just disconnecting from the billable hours, and making sure that your hobby/obsession does not go crazy. If I could share a last story, when I was a kid, I was into model trains or my dad was anyways. I always looked at this Märklin, which is a German brand of trains, a Z scale, which is like 1:220. Tiny, tiny little trains.
(01:02:15):
Anyways, I finally found one and bought a done table that was all- It was years and years old and I needed to upgrade it. So then I thought, "Hey, this is a good opportunity to learn about Arduinos.". [I] built a little Arduino controller for it, little pulse-width modulators for the different track sections, so I could run the trains at different speeds.
(01:02:37):
Totally not billable. But today I will be working on a little Arduino tool that I can use for testing certain aspects of a fire alarm system. So you never know, things just come around and I think learning adjacent, just a little bit off of where your day-to-day work is. Sometimes that sneaks back in and makes you more productive, or gets you something that you can bill for. You just never know.
EW (01:03:04):
It is never really wasted. Learning five years ago about snail shells, I cannot believe that actually affected my origami. So yeah, it is all really connected in interesting ways. And Chris is staring off into space. I think, because otherwise I will make him talk about learning more music stuff.
CW (01:03:28):
Well, the answer for me is I hate computers, and doing computers outside of work makes me itch.
EW (01:03:36):
And it is funny, sitting in this room filled with computers.
CW (01:03:39):
Full of computers. Well, I know. They surround me.
EW (01:03:42):
That you have put together.
CW (01:03:43):
That is because it is the only way to do anything anymore. I am not going to get a tape deck.
EW (01:03:47):
Mental note, get Christopher a tape deck for Christmas.
RH (01:03:49):
Uh oh.
CW (01:03:50):
No, this stuff is still fun. Yeah, I have a lot of things I like to learn. It has been trouble for me. I go in and out of having technical projects I enjoy. I am working on some ham radio stuff that is not exactly computers. It is electronics and things.
EW (01:04:07):
It counts. That siren is useful.
CW (01:04:09):
It counts. But-
RH (01:04:09):
It counts.
CW (01:04:11):
Yeah, last few years, the more computers I use, the less I just enjoy being around them.
EW (01:04:20):
But as soon as you need to work on an instrument project, you are there.
CW (01:04:22):
An instrument project?
EW (01:04:23):
Yeah.
CW (01:04:24):
What do you mean?
EW (01:04:24):
Somebody comes and wants to build a different arpeggiator.
CW (01:04:28):
Never happened.
EW (01:04:29):
We have had people talk to us about instruments.
CW (01:04:31):
But not they have asked me to help.
EW (01:04:33):
No, but someday they will. Your clients will come if you build it.
CW (01:04:38):
I do not know.
RH (01:04:39):
Well, just as a fun side project, once I built a simulator on an STM32. I built a simulator of the Game Boy sound synthesizer.
CW (01:04:49):
Oh yeah. <laugh>
RH (01:04:50):
Believe it or not. So you do not have to tell me about the crazy stuff that you should not do. But you know what is great? I think you folks are in Santa Cruz area, right?
EW (01:04:58):
Hm-hmm.
CW (01:04:59):
Yeah.
RH (01:04:59):
So you are in an amazingly advantaged place, because you can just sneak up over the hill and head over to Big Basin, and take your dog for a nice walk, and all kinds of really amazing wildlife. So I envy you your location. It is a great spot. And the beach is awesome too.
EW (01:05:19):
The beach, our local forests, our lakes. Yes, we are-
CW (01:05:23):
Lakes?
EW (01:05:25):
There are ducks, and ponds are lakes.
CW (01:05:26):
Ponds. Ponds.
RH (01:05:27):
They are ponds. Yeah.
CW (01:05:28):
<laugh>
EW (01:05:30):
Ralph, it has been really good to talk to you. It is fun to talk to folks who have done things.
CW (01:05:36):
<laugh>
EW (01:05:36):
I mean, that have done lots of things. I feel like we could talk for another hour and a half and still not really get to being bored. But we should let you go back to your regular life. Do you have any thoughts you would like to leave us with?
RH (01:05:54):
Back to our regularly scheduled programming, is what I usually call it. Yeah, actually, I guess the final thought is, 99% of the smartest people in the world do not work for your company. And code is a lot harder to read than it is to write.
(01:06:10):
So when you are faced with third party code or something else, just trust that it works or take the time to understand it. Do not just blindly rewrite stuff because you do not understand how this person wrote it. Basically, always look around first, instead of writing more code. Try not to write more code. That is what I would like to leave you with.
CW (01:06:37):
Can I get really, really mad for 15, 20 minutes, before I accept that maybe I should look and understand their code?
RH (01:06:44):
Yep.
CW (01:06:44):
Okay, good.
RH (01:06:45):
Absolutely! No, get frustrated and then time box rewriting, but just do not rewrite-
CW (01:06:51):
<laugh> I am not going to rewrite anything.
RH (01:06:53):
Do not succumb to the temptation of rewriting code. That is my biggest advice right now.
CW (01:06:58):
Want more comments. That is all. Just more comments. <laugh>
EW (01:07:01):
Our guest has been Ralph Hempel, formerly Senior Manager of Firmware for Product Technology at Lego Group, currently Firmware Staff Engineer at Edwards Safety.
CW (01:07:13):
Thanks, Ralph.
RH (01:07:14):
Thanks for having me. It has been a real pleasure talking to you.
EW (01:07:18):
Thank you to Christopher for producing and co-hosting. Thank you to our Patreon listener Slack group for their questions. Thank you to Nordic for sponsoring the show. And thank you for listening. You can always contact us at show@embedded.fm or hit the contact link on embedded.fm.
(01:07:34):
Our quote this week is from James May. "Someone once told me that I was 12 inside. The only thing 12 year olds crave is more Lego. Lego is fun. It is therapeutic. It is a beautiful sensation when you click the pieces together."
(01:07:51):
<pause>
(01:08:02):
How does Agile work for <nonsense words>.
CW (01:08:09):
Am I supposed to cut that? <laugh>
EW (01:08:10):
I hope so.
RH (01:08:11):
Yeah, cut that. <laugh>
EW (01:08:14):
But knowing you, you will put it back in at the end or something.
CW (01:08:16):
Now you have told me.