211: 4 Weeks, 3 Days

Transcript from 211: 4 Weeks, 3 Days with Dennis Jackson, Elecia White, and Christopher White.

EW (00:00:07):

Welcome to Embedded. I'm Elecia White, alongside Christopher White. We get asked quite a bit about how to get into embedded systems and how to get a job in embedded systems. I'm pleased to have Dennis Jackson on the show to discuss it with us. Before we get started though, the dogs are doing well. Thank you for asking. The thing we didn't get to talk to Alan about was the Hackaday SuperCon in Pasadena in November. He'll be there. He may be speaking and you can be there too. Tickets are on sale now.

CW (00:00:40):

Hi Dennis. Welcome back.

DJ (00:00:43):

Hey Chris. Hey Elecia. Thanks for having me back on.

EW (00:00:46):

It's funny for me to have to ask you to tell us about yourself, 'cause I feel like I know you pretty well, but that's kind of not fair to the people listening. So could you tell us about yourself?

DJ (00:00:56):

I can and things have changed a little bit since last time I was on, but, I'm currently the Director of Firmware at Element Science, a medical device company here in San Francisco. One of the few in San Francisco, which is pretty awesome...For those paying attention from the last episode, my math was a little off, but I've been doing this for two decades now. And, it just keeps, I enjoy every aspect of it.

DJ (00:01:22):

But I've been working in medical devices, with a little stint in drones, for a few years, for...all those 20 years. And my goals are always, make sure the software going out is really good, really solid, and make the world a little bit better, every time a release goes out, or every time a new product goes out. I like to stay focused on medical devices because of the safety critical aspect of it. And also just the helping of people. This is the thing that I started out, I got lucky, and I got started in medical devices and stuck with it.

EW (00:01:57):

And you are on show number 94: Don't Be Clever, which was a great show that starts out with a lot of discussion about drones and Airware. And it was before the FAA had gotten those, that certification process.

CW (00:02:13):

Which is now null and void.

DJ (00:02:14):

Which is now null and void.

EW (00:02:14):

But we spent a while discussing what it might be.

DJ (00:02:18):

Yeah.

EW (00:02:18):

Only to find out it was never going to be.

DJ (00:02:22):

Yeah.

EW (00:02:22):

So I'm looking forward to talking more about the code side of things. Before that though, we have lightning round and we have a lot of lightning round questions, so we're going to need you to be very fast.

DJ (00:02:33):

Alright. And I've never done lightning round, so I'm actually a little nervous. [Laughter].

EW (00:02:38):

You've listened to the show. You know what's coming.

DJ (00:02:40):

I know.

CW (00:02:42):

Okay. Favorite, we were just talking about this, favorite movie or book, which you encountered for the first time in the last year. [Laughter].

DJ (00:02:50):

Well, it's been a bad year for reading. Favorite movie? Hmm.

EW (00:02:55):

Any fiction.

DJ (00:02:57):

Any fiction. You know, I actually read Handmaiden's Tale and I know a lot of people didn't like it, but I thought it was a really interesting view of the way things can go south really quick in a society. So I think that's going to be it.

EW (00:03:10):

Cool.

EW (00:03:12):

Favorite tool, but not software. I want a physical tool.

DJ (00:03:16):

Hammer is a really nice tool.

EW (00:03:20):

[Laughter]. Especially for fixing hardware.

DJ (00:03:24):

[Affirmative].

CW (00:03:24):

Best perk you've ever had at a job.

DJ (00:03:27):

You know, just flexibility, being able to take the time to do things with family or, you know, I need some time off to take care of something and I'll make it up another time.

EW (00:03:41):

If you woke up tomorrow as an animal, what animal would it be?

DJ (00:03:45):

Well, a funny thing, I was thinking there might be an animal question.

EW (00:03:50):

[Laughter].

DJ (00:03:50):

I know these aren't fast. Rhino.

EW (00:03:51):

That's a good one.

CW (00:03:54):

[Laughter]. Preferred voltage.

DJ (00:03:56):

3 point, no, yeah, 3.3.

EW (00:03:59):

Favorite airplane.

DJ (00:04:01):

Ooh, the SR-71. Although the...Piper Arrow is pretty nice. I've flown those before, so.

CW (00:04:10):

If someone gave you $10,000, but you had to spend it today, what would you spend it on? [Laughter].

DJ (00:04:15):

I'd probably buy a vacation.

EW (00:04:20):

[Laughter]. Okay. Technical tip. You think everyone should know.

DJ (00:04:23):

Learn make!

EW (00:04:25):

That's a good one. Chris, you had one more.

CW (00:04:28):

Favorite office arrangement.

DJ (00:04:31):

Not open.

EW (00:04:34):

[Laughter]. We knew he was going to say that. Okay. So we got in a discussion, not too long ago, about people wanting to get into embedded systems. And once they're kind of there, how do they get a job in embedded systems, if all they have on their resume is computer science or software in their past. And I want to talk more about that with you, because I know it's something that you come across pretty often.

DJ (00:05:02):

Yeah. Yeah. It happens quite a bit actually. Either current coworkers, former coworkers, people who have been trying to, you know, submit resumes to where I'm working, or friends of friends who say, "Hey, I hear this is what you do and how do I get into that?"

EW (00:05:18):

So what do you suggest? What's the first pithy, easy thing they can try?

DJ (00:05:23):

So it's been a huge change in the world, in software development, especially embedded development in the last 20 years, being able to access some of this stuff. You know, Raspberry Pis, Discovery boards, all these things just didn't really exist 20 years ago. You can get one of those today and start writing something, try and make something that...you want to see...Usually what I say is, you know, "Is there something in the real world that you want to see interacting with? Do you just want to turn lights on? Do you want to measure temperature in your house?"

DJ (00:06:00):

Yeah, you can go out and buy a thermometer. You can go out and buy, you know, a light, but this is something, if you want to try it, and you want to write some code for it, this is something that can get you started with something very simple.

EW (00:06:14):

We totally agree. Of course, if you want to get into embedded systems, the first step is to try it.

DJ (00:06:22):

And these systems are super cheap. I haven't used the Raspberry PI, but I've used Discovery boards and you can find them for $20. That's, you know, an easy, cheap way to get started. The compilers are free. The tools are free. There's a lot of resources out there.

CW (00:06:39):

I think a Pi Zero's $5. So yeah. Cost is not really an issue anymore.

DJ (00:06:42):

Yeah.

CW (00:06:42):

As long as you have the computer. Which is probably the higher cost.

DJ (00:06:45):

Yeah, yeah.

EW (00:06:46):

So the Discovery boards, one of the hard things with them is that it's a big opaque system. I mean, we talk about Arduino and I love Arduino, but the Discovery boards are a little closer to what you might use in a professional environment. And yet, people get pretty stuck on 'em. What do you suggest for compiler, and just, development environment?

DJ (00:07:14):

So for those ones, the vendors have tools that you can start with. So with the discovery boards,...you can use ARM's Keil compiler, and that comes, there's a free version of it. It's a decent compiler and decent IDE. It comes with sample code. So you can, you know, turn the lights on. They have accelerometers on there, so you can do basic things with accelerometer. They have buttons. So you can do basic interaction like that. And they have samples of all this.

DJ (00:07:44):

So you get that board, you...download the compiler, and you can at the beginning, just compile and run it. Make sure your computer is set up, make sure your compiler is set up. The connection to the board. You have power to the board. You usually power to a USB. So once you have that on, you know you're gonna be able to talk to it. And then once you do that, then, you know...you've got the start of it. And their sample code is good enough to give you an idea of what you should be doing. Now, you're not going to turn that into production code, but it's good enough to start out and start learning how to play with the board.

EW (00:08:22):

I would add Mbed to that. Mbed is an online compiler. You don't have to install much. Maybe some drivers, and many of the discovery boards are Mbed enabled.

DJ (00:08:34):

Yeah. I've heard about that. I haven't tried it.

EW (00:08:36):

It's kind of nice, because in an Arduino-like fashion, they have huge numbers of libraries and example code. So you don't have to write everything from scratch. It's also very C++-oriented. So it's not as much just straight C that you get in the tutorials from the discovery board.

CW (00:08:55):

Yeah. And the Mbed boards are more capable than Arduino. They have networking, a lot of them, on it. So you can do IoT things if you like.

DJ (00:09:05):

IoT.

CW (00:09:06):

Why...are you making a face?

EW (00:09:09):

Oh, because the Arduino boards are so big now that -

CW (00:09:11):

The default Arduinos, you need to get a shield and stuff.

EW (00:09:13):

The Arduino Unos are kind of brain dead, but that covers so much now. And the Mbeds cover the same things.

CW (00:09:21):

Okay.

EW (00:09:21):

Maybe more.

DJ (00:09:21):

It's funny. It's funny because I know these things exist, but I haven't used them. Because typically I'm working on a project that already...you know, we're using usually an ARM core somewhere in there. And we have the, you know, industry tools on there and usually there's an EE on the team or someone else on the team that says, "Hey, we want to try this new sensor out and by the way...I got an Arduino and wrote something to try it out." I'm like, "Oh,...you did that yesterday. We talked about the sensor yesterday and you already wrote it?" I'm like, "That's pretty awesome."

EW (00:09:56):

Arduino gives you a lot of speed. That initial prototyping is really useful. But then we go to Discovery and other things that make it faster and cheaper.

DJ (00:10:12):

Yeah.

CW (00:10:12):

Faster to run, not faster to produce.

EW (00:10:15):

Right. Okay. So our new college grad in CS, has a day job doing software that he doesn't really like, and he wants to do embedded software. And he got a project, or she, got a project and has decided to blink some lights. Cool. I mean, that's a tutorial. There are lots of examples on how to do that out there. What next? I mean, they can't put that on their resume and say, hire me.

DJ (00:10:42):

Yeah. So this is an interesting story. A couple months ago I was hiring from where I am now and I had a resume come in and the resume looked really good. Definitely junior level, but really good. And I talked to him on the phone and did a phone screen and answered all my phone screen questions really well...This is a really good potential candidate. So let's bring him in. Brought him in.

DJ (00:11:07):

And as we started talking, I was the first interviewer of the day and the whole schedule lined up, and we started talking and just, I don't know. I don't remember. I unfortunately don't remember exactly what I asked, but I asked something and the way he responded, I thought, "Hmm, that's not in line with everything else I heard." So I said, "Let's just write a little bit of stuff up on the board."

DJ (00:11:32):

And we started doing a little bit on the board and within 30 seconds, I knew, "Okay, this guy has potential, but he's not going to be the guy I'm going to hire." And I said, "Hey, hang on a second here, as I stop you...You have a lot of potential, but this isn't gonna work out. But...we're gonna talk for a while here." So I went out and told the rest of the interview team, this isn't happening... We stopped the interview, and I went back in and talked to him for another hour and a half. And I just basically broke down "Here's where you are. Here's where you need to go." And he was in the fortunate position of being at a company that was doing embedded work. He just wasn't on that team.

DJ (00:12:15):

He was helping tests and he would help, you know, find bugs. But as we were talking, I said, "Well, you know, you guys have a version out. You're probably going to have a new version coming out." And he said, "Yeah, you know, we're actually, the process we're using is coming end-of-life. We're looking at new ones. You know, we've heard a lot about ARM so we're thinking about going to that."

DJ (00:12:35):

And I said, "This is the perfect opportunity for you. Go buy a Discovery board, you know, get the company to pay for it. It's 20 bucks, but worst case, you buy it yourself. And you already have a list of the sensors that you're using. You can get samples of those. You probably have them in house. You can not talk to the EE and say, 'Hey, can I get one of those? And you know, I don't want it on the board, but I want to be able to hook up some leads to it and hook it up to this board.' "

DJ (00:13:00):

"And that is the, you can take that, get the sample program and write, you know, either look at sample drivers or start writing a driver that talks to it. Get the data sheet. And you can do that as your starting point for developing yourself. Because now you're looking at new hardware, you're doing the embedded work that you want to be doing, that you already have some experience in, you just...don't have enough experience with the writing of the software part of it. And you can use that as your starting point. So to, generalize that it's another thing you can do is say, 'Hey, you know, is there a project that worked that is using embedded software that you want to get into?' "

DJ (00:13:40):

Or if you're not at a company that has embedded software, but that's the role you want to switch into, is get some hardware, get a couple sensors. You know, like we said, like you get the Arduino, you get any other parts like that. And you can download drivers and make 'em work right away. And that's fantastic for rapid prototyping.

DJ (00:14:03):

But when you go to that STM part, which you'll more likely be using on a production system, you're not necessarily going to have the driver for, you know, the current limiting chip or the current reading chip or the accelerometer, or you know, the temperature sensor or any other sensors on there. But get something that you can play with, and use, and look at that datasheet and start and write a very simple driver for it.

DJ (00:14:27):

And now you don't have to, you know, everyone says, "Well, you know, you're working in embedded. I've got to worry about power usage. I have to worry about size limits. I have to worry about putting the processor to sleep. It's like, you know what, that board's going to be sitting on your desk. It's not going into a product. It's not going to be running off...you know, a coin cell battery that has to last, you know, six to 12 months or longer. It's going to be plugged into your computer, running, you know, whatever speed that you set that thing up for. You just run it at max speed, run it at one of the low speeds. It doesn't really matter, but get it running. Get that driver written and make sure you can read it.

DJ (00:15:04):

You have the data sheet there, and usually...if it's I2C, if it's SPI, even serial, there'll be usually a, you know, an identification byte that you ask...especially in I-squared-C and SPI, you say, you send a command, and it sends back to you this is my ID. Get that command working first. Then you know you're talking to that piece of hardware. You know, if you're using the accelerometer, get back X, Y, and Z, and now you can pick that board up and you can start moving it around. You write a little something that, you write something on your processor that continually reads that. And, you know, do you have printf, do you have, are you using debug?

EW (00:15:48):

An LED?

DJ (00:15:49):

An LED? Yeah. Some way of getting output back on to say, "Hey, this thing is moving and it's reading. Now, the other thing is, "Oh, I have to worry about interrupts. I have to worry about all this other stuff." Just pull it. That's, you know, that's a very, very simple way to get started. Do the simplest thing possible. Then you can graduate into those other, you know, other methods to make it more efficient. And more likely the sample programs are going to have examples of interrupts and DMA transfers...That's intermediate and advanced level. Just get it talking.

CW (00:16:28):

So I would say, just to add to that, if you're going to be playing with SPI and I2C, you probably want to have some sort of logic analyzer.

DJ (00:16:36):

Yes.

CW (00:16:36):

Because even the simplest devices that you think are just going to work, sometimes they don't, for some reason, or your driver isn't quite right for some reason.

EW (00:16:43):

Or you wired it wrong.

CW (00:16:45):

Right. And if you're just doing it blind, it's impossible to figure out what's wrong.

DJ (00:16:51):

That should have been my favorite tool, a Saleae. That thing is amazing.

EW (00:16:55):

Okay, so we've relaxed to the constraints. We have decided we are not going to try to build the most power efficient, most RAM efficient, most cycle efficient system. We're just going to build a system and we're going to use the example code as much as we can, and we're going to learn to use these tools. 'Cause I mean, we all can use oscilloscopes and logic analyzers, because we've been doing this...since the dinosaurs roamed, or since 8051s were in the wild. No, they're still in the wild. And yet those are all skills.

EW (00:17:32):

It's like, you're making an embedded system and you're not doing all these things that people say, "Oh, embedded systems are hard." What you're doing is you're learning the whole thing. Cross-compiling is something that you don't know natively and using a debugger on a system that you aren't running on is, it's different. So, yes, I agree with your approach that you don't worry about everything at the start. You just try to make it work. And at that point you will realize there are things around you that you're working on.

CW (00:18:04):

Well, nobody learns to drive on a race track.

DJ (00:18:06):

Oh, you didn't? [Laughter].

EW (00:18:11):

It might have been easier.

DJ (00:18:11):

Yeah. Yeah, I know, and that's exactly the right thing is, you're going to read, you're going to, especially if...this is where you're going to go, you're going to read about all these different methods to keep things tight and efficient. And that's not where you're going for when you're learning. You're learning, you're going for, "Can I compile it? Can I load it on the board? Can I turn the lights on?"

DJ (00:18:32):

You get those down and then everything else, you'll start approaching all those other things like, "Oh, you know, I can time this" and say, "Oh, it takes a really long time to run this. And it should be a lot faster. How do I make it faster?" Then you start reading like, "Oh, this is how I do interrupts. Oh, this is how I reduce the amount of code that goes on it...you know, to initialize, or to read those values, or do those computations."

EW (00:18:57):

And the reason I know that printfs take a long time is because it has bitten me, and the way that people will learn how to do these interrupts and callbacks and changing how they do printf, is because at some point, you'll realize, "Oh, that's the problem."

DJ (00:19:15):

Yeah.

EW (00:19:15):

But how long does it take? Okay. So I'm in a software company, I have found a hardware project. Maybe we don't do embedded. Maybe I've decided to do some physical monitoring of our servers or something that will get me a project, some project I can do as a 10% project at my work. And I've done that, but -

DJ (00:19:37):

That's actually a really good one. I'd forgotten that one...So when I was at Airware, we had a DevOps guy who wanted to do some embedded stuff and what he did, I'm pretty sure he used an Arduino. I'm not one hundred percent sure. But he wrote this little device.

DJ (00:19:53):

It must have been, 'cause I know, I'm pretty sure he was using Mbed, but he wrote this little device that sat on our network, that got messages from our continuous integration server when...builds finished and it would monitor those and then keep track of it over time, create a little summary and then post to Slack. Now, if that is not an integration of a lot of technologies, I don't know what it is. It was doing network stuff. It was network. It was processing internally.

DJ (00:20:25):

Now it wasn't using, oh, it actually did, it did blink lights. So we knew the status of things. So it was just, it was an amazing little thing. And it just, it was a little board, probably two inches long, and it just sat plugged in next to his desk and we would interact with it via Slack. And just, you know, we'd ask it questions,...basic questions of "What's the build status?" And it would return back a list of, you know, pass and fails...It was one of the coolest little embedded projects I'd seen.

CW (00:20:53):

Well, that's a good idea too, because that's normally something somebody would just do on a server somewhere.

DJ (00:20:58):

Yeah, yeah.

CW (00:20:58):

And there's no reason you can't do any kind of software on an embedded device. If you have to find an excuse for a project, take some agent that you've got already running that is doing testing or monitoring and put it on some little device and -

DJ (00:21:11):

Yeah, yeah.

CW (00:21:11):

Yeah, and the Mbed, I said it had networking, but, it does have networking, but it also has an RTOS, which makes a lot of that stuff easier.

EW (00:21:19):

Mbed is like a class of boards.

DJ (00:21:21):

Yeah.

EW (00:21:21):

It's kind of like saying ARM.

CW (00:21:23):

Well, but it's also the framework and the library they provide for it.

EW (00:21:26):

Yes, and the compiler. And so it's not just one, it's sort of like saying Arduino is not just one board.

CW (00:21:31):

Right.

DJ (00:21:31):

Right.

EW (00:21:31):

But yeah. Okay. How long do you have to do these projects before you can get a job?

CW (00:21:43):

Well, let me ask that a different way. It's sort of been rattling around my brain 'cause I haven't done interviewing in recent years, at least of new people, like junior people. So I haven't encountered people coming in with, "Oh, well I don't have embedded work experience, but here's some projects I've done." Have you seen that, Dennis? And what do you think about it and how do you evaluate that? I guess that goes to her question of how long does it take, in a way asking more...how much are you looking for in personal projects to decide, "Okay, this person can do something?"

DJ (00:22:18):

I have, because especially at Airware, we were hiring interns. And now it's a little bit different than hiring a junior developer, fresh out of school, looking for a full-time job. It's just a little bit different. But the thing that surprised me was how much people had,...how many projects people had their resume. And I was just thinking back when I was in college,...I wasn't going for, you know, embedded software. I was actually a physics major, and I just happened into this industry. But when I looked at these resumes, I'm like, "Whoa, that's a lot, they've done a lot,... they have a lot of experience, I mean, classroom experience, but they have running projects. So that's pretty awesome.

DJ (00:23:07):

For someone just coming out of school,...if that's where you want to go, if you're in school and that's where you want to go, make sure your classes are helping you create these things. You know, create projects, or -

EW (00:23:21):

Join the robotics team.

DJ (00:23:23):

Yeah. Or do projects, or do things outside of class that are, that require some kind of embedded work...And do personal projects like we started the whole conversation with, "I want to make something that does this thing." And you don't have to, here's the other thing too, you don't have to think about it in terms of, "I want to sell this as a someday so it has to be perfect, and there isn't a product on the market that already does this thing." Just do something simple.

DJ (00:23:52):

But coming into, it really depends on, you know, the companies that you're looking at. Now, if you're going for an early stage startup that has...you know, five to 20 people, they're not, I'm sorry, but they're not looking for someone who's...looking to gain all that experience on the job 'cause they want someone who can help turn things around quickly.

DJ (00:24:15):

You know,...if they're in that stage and they're looking for interns, that could be a thing. But if you think, I really have a hard time believing a startup is going to say, "Hey, you've never done embedded work. You've done some projects. We're going to hire you to make our first product." That's going to be a hard sell. The bigger companies, those are going to be easier to get into, and are a good way to get a lot of experience quickly.

DJ (00:24:42):

My first job, I had never done embedded development. It was about 130 people at the company, but I had done a lot of software. And I got lucky that, there's a lot of luck in interviewing, but I got lucky that, you know,...the folks hiring saw a lot of potential. And I remember that first week on the job and they're like, "Okay, here's the, we're working on this pump, a fluid pump. And here's your compiler...Here's how you talk to it." And I remember the first time I just, I typed something in and I saw, I heard things clicking and moving. I'm like, "Okay, this is what I'm doing the rest of my life." [Laughter].

EW (00:25:24):

Actually I want to stop you there for a second. Sorry, if you can hold that train of thought. Chris, how did you get from, well you had a math degree, but you went into software, you went into, well, you went to Cisco and routers, but for a long time you didn't consider that embedded.

CW (00:25:41):

'Cause I didn't know the word. I don't think anybody was using that word, really.

EW (00:25:45):

Yeah, we were using firmware.

CW (00:25:45):

Back then Cisco, I mean, Cisco, by most people's definition would be embedded. You know, I was cross-compiling. I was downloading images to a device that was not a computer. We were on some weird, you know, MIPS processor, and with some purpose, you know, home-built operating system with, yeah. So it was embedded. And I actually, you know, thinking back, learned some of the things we're talking about, remote debugging and cross-compiling, and I got exposed to those through Cisco and it didn't even seem weird to me.

CW (00:26:15):

Because that was just, "Oh, I thought this is the way people do things." You know, I'd done software development and of course on normal computers before, but it didn't seem that different. The first time I encountered moving things, like real embedded where, you know, you're doing SPI and I2C and things like that was two jobs later? And that was a medical device company and that was, I had already, you know, watched your career. So I knew about all of these things. So it wasn't, it's not exactly fair. I wasn't really just dumped into it and shocked by a lot.

EW (00:26:54):

For me, I went to HP and did servers. They made big servers and wanted monitoring and I kept getting lower and lower in the hardware. Lower, I wrote drivers for all the different operating systems, you know? OS/2, Windows NT -

CW (00:27:13):

OS/2? [Laughter].

DJ (00:27:15):

OS/2? That's awesome.

EW (00:27:16):

Novell Netware

CW (00:27:16):

You wrote an OS/2 driver?

EW (00:27:18):

I did. I've written several.

CW (00:27:19):

You should just, that should be your entire resume.

DJ (00:27:22):

Yeah. I would hire you right on that.

EW (00:27:26):

And so I kept writing drivers and getting lower and lower and then blinking lights on the hardware. And then when I went to go look for a new job, they were like, "Why are you applying for software? You should be doing embedded" or firmware, was what they were saying. And I said, "I don't know what you mean." And they said, "Do you really have a signals background? Can you do PID loops?" And I said, "Oh yeah, sure." And they said, "Well, we'll teach you everything in between." And then I actually got to do the cross-compiling. Before that I didn't know about remote debugging and cross-compiling. So yeah, I went from software to embedded it in large part because of luck, because somebody said, "Oh, you have all the skills. So let's just put them together."

CW (00:28:14):

Yeah. And seeing output over a serial port. That's your only interface to things, I mean, there are all these little bits of "this is different" that you have to pick up.

EW (00:28:24):

It's hard, we're telling people to do projects, but none of us did it. We just kind of lucked into it.

CW (00:28:29):

Well, I did. I kind of did, maybe?

DJ (00:28:31):

Yeah. I mean,...there's going to be a lot when you're learning this..., I mean, when you're in the real world, you're going to be doing all kinds of crazy things. You're going to, it's going to be this circuitous path, but you gotta get started somewhere.

EW (00:28:44):

True.

DJ (00:28:44):

And maybe you get a job at a company that does have embedded stuff. And that's what you start working on...I was, so my first job was at DEKA Research and they were an embedded company. The products all had processors somewhere inside of 'em. Then I went to St. Jude Medical and I went there to work on the firmware team, but we had other, and this is, we had other people that were not embedded. They didn't work on firmware. They did not work on embedded software.

DJ (00:29:14):

They worked on application software, but they had said, "Hey -." I wasn't involved at this time. But they, whoever is in charge at that time, they said, "Hey, you know, we really want to get involved in the embedded side." And I remember there'd be conversations about, "Oh, you know, so-and-so, he's only ever, you know, he or she, he, it's actually a man and a woman. They have only written application software for the desktop. They're not going to be able to do it."

DJ (00:29:39):

And you know, you just show 'em like, "Hey, you don't have, you know, a gigabyte of memory to use. You don't have, you know, this infinite hard drive space to save stuff to. You have, actually the device I think had 250, 256k of memory, you have this amount." And they just said, "Oh, okay. These are the new constraints." And you know, they learned some techniques and things to look out for, and they became super valuable members of the team.

DJ (00:30:08):

...So you touched on a couple of topics in that part of the conversation. I've almost never used the word firmware. Like I'm a firmware engineer except for where I am now, because that's just the way they've divided it up. I've always said embedded software. And the reason I say it that way is to me, people have asked me, "What's the difference between firmware and software?" I'm like "Just a couple of letters."

CW (00:30:32):

It used to have meaning.

DJ (00:30:32):

Yeah.

EW (00:30:34):

It used to have more meaning.

CW (00:30:34):

I think it's lost it, yeah.

DJ (00:30:36):

Yeah. I mean, if you think about, "Okay, you know, I'm writing the control, the software that controls this motor, that lives on a PIC, that lives in something, okay, maybe you can call it firmware. But you know, at the end of the day, it's taking inputs, taking output, it's doing something, it's taking input, it's doing something and outputting something. You know what software does? It takes inputs. It does something with them and does outputs. It's all software.

DJ (00:30:59):

So to me,...I don't distinguish between them. The distinguishing thing is the constraints, you know, do you have lots of memory? Do you have little memory? Do you have power requirements? Do you not have power requirements? But it's still software at the end of the day. I personally just don't like distinguishing between 'em, because I think it does a disservice, because now it's like, "Oh, I write firmware."

DJ (00:31:23):

I don't have to follow those rules for software development. Like, what?

EW (00:31:26):

[Laughter].

DJ (00:31:26):

You have to follow those rules. You know why? Because your software, your firmware, is running on something that has to work. So I mean, rules are, you know, good naming, good code, unit testing, documentation, like what does this thing do? Why does it do the way it does? Now you don't have to write, you know, pages and pages of documentation to say, you know, this motor controller takes an input and, you know, computes, you know, does computation and then outputs a PWM signal to something.

DJ (00:31:58):

You know, I don't need pages. One page is fine, but you have to write down what it does. So when you leave that job, when you get sick and you're out, and someone else has to do something with it, or when you go onto another project and someone else takes over, or you come back, 6 or 12 months later to fix something, you understand what the heck is going on there. So that to me is why I don't distinguish between them. Because at the end of the day, you still have to write, you have to be...you're a software person. Even if it's, you're flipping bits on something at the lowest level, you're still writing software.

CW (00:32:34):

I do think you're getting at something there that I haven't really realized, is that that naming of roles has some power because people do say, "Oh, you know, I'm an embedded software engineer, or I'm a firmware engineer." And they have some understanding of what that means, and that might be colored by their, you know, a decade of career. And it might not apply really anymore because, you know, if you're releasing a quarterly update, that firmware updates your device, it's hard to call that firmware anymore.

DJ (00:33:05):

Yeah. [Laughter].

CW (00:33:05):

Right? And it's hard to say this isn't an application even, and the techniques that you...go through to secure something, that can be updated frequently, are far different than if you're burning a ROM, right, that you're never going to touch again.

DJ (00:33:24):

Yeah.

CW (00:33:24):

And I think people need to realize that we're all kind of moving to a space where we're converging with normal software, "normal" software development -

DJ (00:33:32):

[Laughter].

CW (00:33:32):

- and all of those techniques that came from even web development and now apply to us, except Agile.

EW (00:33:37):

[Laughter].

DJ (00:33:37):

Except Agile. I concur.

EW (00:33:42):

It used to be more Wild West. It used to be more "Just make it work. Once it works as well as it's ever going to work, we burn a million ROMs and are done and we drop the code and we go on - ."

DJ (00:33:53):

Yeah.

CW (00:33:53):

Yeah.

EW (00:33:53):

" - because whatever we're going to build next is going to be on a different processor, it's going to be totally different. There's no reuse or continuity. Testing is external to the code." And that's not true anymore. We write software. We write software primarily for humans. And the fact that the computer or the processor does what we want is sort of a by-product. We should be more aware that the humans are the people we're writing for.

CW (00:34:20):

Yeah.

DJ (00:34:21):

Yeah.

CW (00:34:22):

And I think a lot of this went out the window, as soon as the first radio module was put on a device.

EW (00:34:28):

Firmware update. I mean, it means that our firmware, it means that our software, lives.

CW (00:34:31):

...Well, it means it can be worse than we want it to be the first time we ship it. But it also means it's much more insecure. So those are two strikes.

DJ (00:34:43):

Depends on what you're working on, depends on what you're -

CW (00:34:45):

Right, right, right. [Laughter].

DJ (00:34:46):

So, you touched on another thing...It was a lot of times just, "Get it working," and that's still the norm, is like, "Okay, we have all this, we have all these pieces. We need to make them work. So let's pull something together quickly and make sure everything works together." Now, the scariest thing is having a working demo because once you have a working demo, everyone was like, "Oh -"

EW (00:35:10):

[Laughter]. "Let's ship it."

DJ (00:35:10):

"- we're going to ship this in two weeks. We just gotta put the packaging around it." And I'm like, "Whoa, no, you know, we just got it working."

CW (00:35:17):

We just gotta take this down to South America and, nevermind.

DJ (00:35:20):

Yeah. [Laughter]. Yeah.

EW (00:35:25):

We got it working once.

DJ (00:35:26):

Yeah. We have it working once. Yeah, that's the big risk...And that's where it comes in on the team. And not just not just to lead, but it's the team who says, "...You know, we can show that it works here in the lab. We can put one on someone's desk and they can use it, but here's all the other work that has to happen to make sure that can be done, not just one time, but you know, 10, a hundred, a thousand, a million times. And it works reliably."

EW (00:35:57):

Like secure firmware update.

DJ (00:36:00):

Yeah.

EW (00:36:01):

Okay. Back to the question of, "How long do I have to do projects before I start actually getting a job?" Which is not a real question, because how long doesn't make any sense.

DJ (00:36:13):

It's actually four weeks and three days.

EW (00:36:14):

[Laughter]. Oh, I see. I didn't know.

DJ (00:36:17):

No, I mean,...again, it depends on the...type of company you're targeting and you can get your foot in the door. So for me, you know, especially when I was talking to interns, or if I'm looking for a junior developer,...I want to see,...you know, have you done something? You know, they haven't shipped anything, but have they done something and been able to demonstrate it to someone else and it works. You know, if you have something you can bring to an interview, that's really cool.

EW (00:36:50):

That is such a great way to control an interview because now you are going to be talking about something, you know really well.

CW (00:36:57):

To control it from the interviewee perspective.

DJ (00:36:59):

Yes.

EW (00:37:01):

Oh, yes.

DJ (00:37:01):

Yeah.

EW (00:37:01):

I mean, if you're holding this thing, of course, they're going to ask you about it. And now, instead of asking you about their product, which you don't know anything about, they're asking you about your thing, would you can talk endlessly about.

DJ (00:37:12):

Yes...you're right. It's a pretty open-ended answer. But, so when I'm looking at these developers, I'm looking at, so I always open my interviews like, "We're going to do some, we're going to write some code on the wall at some point." And I'm not looking to see if you can put semicolons in the right place or your parentheses are balanced and all that. I'm not, the compiler will tell you if you've done that.

DJ (00:37:41):

If you do that, I'm like, "Hey, I'm noticing that, but I'm not going to go, you know, this guy is not going or this person's not going to work out because they didn't put the semicolons in the right place." That's just stupid. But I'm looking like, "How do...you think about the problem?...You know, if I asked you to write a UART driver to, you know, talk to another device, how would you approach that? What are the things a UART has to do?"

DJ (00:38:06):

And this is actually a really good question. A really good thing to think about if you want to get into embedded, "How do I talk to a couple of different devices? How do I, what do I have to do in order to talk to UART? Well, I have to be able to open it, I have to be able to close it. I have to be able to set baud rates and bit rates." Parity, if I'm talking to I2C, "Okay, how do I address those? Is there...you know, this address, is there a command response? If it's SPI, you know, how do I different things? How do I write the SPI, how do I read from SPI?"

DJ (00:38:37):

There's addresses that you use that all SPI devices use. Now you don't have to know those like, "Oh, it's this address. And this is exactly how I do it." Having that, we can have a conversation, but I'm also looking at just general software development. That's why I'm not thinking, "Oh, you're an embedded person. You don't have to follow the rules of, you know, asking questions, asking about a spec. What does it have to, what kind of data rates do I need it to be able to handle? How often is it going to be called? Do I have to put in interrupt? You know, do I just put it in a, do I just pull it whenever I need that information?"

DJ (00:39:14):

So those are the kinds of questions I'm going to ask. And just, if anyone ever comes to interview with me, I'm going to leave holes in the questions, it's going to be vague. I want you to ask those questions.

EW (00:39:27):

That is an interview thing that I don't think I understood right away. That the question may actually be impossible as stated, and what they are trying to find is, can you ask the questions or are you just going to bash yourself against those rocks until you realize you can't, and then you're going to leave angry because our jobs are about asking the questions.

CW (00:39:50):

You're describing my last four weeks.

DJ (00:39:55):

[Laughter]. It's just addition and subtraction, man.

CW (00:39:58):

[Laughter]. No, no, there's other stuff.

DJ (00:39:59):

Maybe some multiplies.

EW (00:40:02):

Okay. So you said I2C and SPI and UART, and do they need to know all of these terms? And if so, if they've only done a couple of projects, how do they know these things? They're not easy.

DJ (00:40:16):

I don't think you have to know 'em. You don't have to know 'em inside and out. So that was another thing when you guys were talking about first jobs. I was thinking back over my career when you guys had the, you know, past people episode, unfortunately I missed it, but I was thinking about "When is the first time I really, truly wrote a low-level driver?" I was like, "Oh, it had to be back at DEKA, my first job." And I was thinking, I was like, "No, nope. I think it was at Airware. I got hired on there - "

EW (00:40:43):

[Laughter]. That was not that long ago.

DJ (00:40:45):

I know, that's the funny thing. So, I've been doing embedded work for 20 years, but the first time I really wrote some low-level stuff, I think was at Airware. That was 17, 16, 17 years into my career. But I was writing embedded applications. I was writing, you know, I was working on control algorithms. I was working, you know, I knew I had memory restraints. I had power restraints.

DJ (00:41:14):

I had all of this stuff, but I was like "huh"...I blinked some lights early on. I definitely wrote some stuff that turned displays on and wrote some stuff to display, but the display driver was already there, someone else on the team had already written it. And I was like, "Oh, I was just putting stuff on the screen. I guess I did respond to some button clicks. Oh." But someone else wrote the thing that listened to the button, told me the button was clicked or not. I'm like, "Wow. Yeah. It was pretty far into my career before I wrote some real, quote unquote, true embedded."

CW (00:41:42):

Yeah. I think, I'm trying to think, it must've been 10 or 11 years into my career, nine, somewhere between 9 and 11 years before I did that too.

EW (00:41:51):

And it was much earlier for me.

CW (00:41:53):

Yeah, of course. [Laughter].

EW (00:41:53):

That was because I actually went to embedded.

CW (00:41:57):

Yeah.

DJ (00:41:58):

So I guess this is throwing a little kink in all the stuff we've been talking about, because we're have to get through all this stuff.

CW (00:42:01):

'Cause we're all liars. [Laughter].

DJ (00:42:02):

Yeah. We're all liars. It definitely gives you a leg up, especially, for me, if I'm hiring someone to do embedded, I just want, I want some exposure to that, but really if you are, if you can understand and write software and a whiteboard question is a very, it's a very limited scope. It's very time constrained and...it's what we do as an industry. It's not maybe the best thing to do. But take-home coding tests, I've done those once. And I actually hated it. I thought that was really annoying. But it's a way to show some knowledge. And really, my interviews are a lot more conversational.

DJ (00:42:51):

I'm not gonna throw you up on the wall and say, "Okay, write this thing." I definitely have done that. And I know I did that a few times at Airware, but I like to have a conversation about, "How do you approach this? How do you do that?" One of my questions is always, unfortunately, the answer is almost always the same thing..."Do you do unit testing?" And it's almost always no, get some experience with that. There's a lot of great frameworks out there.

DJ (00:43:18):

You know, you don't even have to do unit testing of embedded code, just get some experience with it. If you walk into an interview with me and you say, "Yeah,...I do unit testing and this is my framework." I'm like, "You already went up 10 notches on my scale." Because that means you're concerned about the quality of the code. Not just "Does it work, does it work when,...you know, I write it and throw it on the processor." You're concerned about writing code that can be tested, and that is going to be solid, that I know it was like, "We're going to write that and we're probably not going to touch it ever again."

EW (00:43:52):

And you know, I don't ask in interviews about I2C, or SPI, or UART. I hope that they have used them, but I don't care if they've written a driver for them.

DJ (00:44:05):

Yeah.

EW (00:44:06):

I want to know more that you can understand how a system isn't all spaghetti, that you can make boxes. I'm really impressed by people who can make block diagrams of their system, because they understand that these are separate pieces of code that potentially have reuse, that can be tested on their own. I care a lot about testing. I don't always, you know, unit test is a word pair that has particular meaning, I'm okay with understanding more about how do you test it? How do you make it testable -

DJ (00:44:39):

Yeah.

EW (00:44:39):

- without saying unit testing? But yeah, we want you to know what all of these little protocols are, but you don't have to have implemented every single one of 'em. If you've implemented one and can talk about the difference and talk about how logic analyzers work and why it was important to know that in order to debug a SPI driver, that is awesome.

DJ (00:45:03):

Yeah.

EW (00:45:04):

More, I want you to be able to say, "Oh, I had an accelerometer, I had an LED, I used a little bit of code to make it so that the LED color changed based on the accelerometer. And this is why I did it. And this is how I kept them separate. And this is how I changed the accelerometer to something else." I mean, it's the separation and the testing that are a bigger deal.

DJ (00:45:26):

So, I mean, tying that all back together is when we talked before, and when I tell people things they can read to become better developers, I don't send them, I do have some embedded software articles, like "read these," but one of my favorite ones is Joel Spolsky's lists. And they have nothing to do with embedded software. They are software in general. If you're writing web servers, web applications, embedded applications, things that sit on desktops, These are articles that, you read those, and apply what you can. Almost all of it applies. You know why? Because firmware's software.

CW (00:46:04):

That leads me to a question of what is, what are we assuming the minimum experience of the person is who is trying to get into embedded? What background are we saying is, "Okay, you need this background before you even attempt to play with embedded and attempt to get an embedded job?" Because there's plenty of people, I think who listen to our show who maybe don't have a software engineering background who play around with Arduino and are learning some other stuff.

DJ (00:46:29):

I mean, most people I've worked with have not had computer science or software engineering backgrounds. They've been mathematicians. They've have been physicists. They've been electrical engineers. You know, they took some software classes in school. I mean, they've done those and they had a, you know, basically had a knack for, and like, "Oh, I like writing software." And that's where, I mean, that's how I ended up in it.

DJ (00:46:52):

When I was in college, people were like, "Oh, you must be a computer science person." I'm like, "No, I'm physics." But...I'm always in the lab writing software because that's kinda what I do. But I'm writing software for physics stuff. So, just the way it worked out...I mean, so software is a craft. It's not just...you are going to, if you like doing it and you want to do it, you need to do it.

DJ (00:47:22):

You can't just say, wake up one day, and say, "I'm going to be a software engineer. I want to be a software person." And just say, "Okay, now I'm going to go get that job." You have to be practicing it. You have to do it at home, you know, or in your current job. You...have to get that experience of working with a computer.

DJ (00:47:43):

You know, everyone, anyone who doesn't write software thinks, "Oh,...I want the computer to this one little thing. It should be really simple for you, right?"

EW (00:47:50):

[Laughter].

DJ (00:47:50):

And the fact that you're laughing is because, you know, that's never the case.

EW (00:47:54):

That's never the case.

DJ (00:47:56):

Never the case. There's always a million little details that you have to get just right. I mean, the compiler is brutal. If you don't tell the exactly what you want it to do, it won't do it. So, you know, having that experience, so, you know -

EW (00:48:11):

And shifting between the detail-oriented "Where do I put my semicolon, and how do I talk to a register on a peripheral?" To the high-level of "How do I keep my code separated and clean and testable and architected well?" And there's this, you have to slide back and forth, in mental shifts...that orders of magnitude difference.

DJ (00:48:34):

Yeah. [Laughter]. Yeah.

EW (00:48:36):

You mentioned resources and I agree, joelonsoftware.com, he's got reading lists on that front page that are just fantastic. It's funny when I go back and read them, how many things I must have read 15 years ago, and internalized so much -

DJ (00:48:52):

Yeah.

EW (00:48:52):

- that now it feels like he's plagiarizing from my head. So I agree with that. You have other books you suggest.

DJ (00:49:02):

Other articles, other books. Well, everyone knows I'm a big fan of unit testing, and test-driven development. There's James Grenning's book. That's geared towards embedded C, but there's a lot of other TDD books out there. Now, I always get asked this, "Oh, you're a real advocate for TDD. You must do that every time." And I'm like, "Actually I don't."...I mix it, it's kind of like the whole Agile thing. I'm not a fan of all the stuff with Agile, but I like some of the stuff from it. So I've taken the stuff I think is appropriate. Some other ones, actually is your book, "Making Embedded Systems."

EW (00:49:41):

Yay!

DJ (00:49:41):

...When someone asks me, I'm like..."You don't have to read this thing cover to cover, but read through it. 'Cause these are the kind of things you need to know about." Because you have everything in there from, you know, the high-level, the block diagrams, how's this system going to go together to, you know, low-level debugging. So it's a good place to start. So other articles are things like, if you're using C...this article's probably, it's gotta be 25, 30 years old now, "Rules for defensive C programming."...It's really, I think there's 15 tips in there. 10, 15 tips.

DJ (00:50:23):

I's just naming, you know, putting units on your names, don't use #defines because that can lead to other problems. And I found out from a guy I used to work with at DEKA, 20 years ago, I was gone from there a few years, and he said, "Yeah, we found this problem in the released version of the software and it all came down to a #define being incorrect." And it had gone through all the testing, it had gone through everything and it'd been on the field for awhile, but...I went back to the article and reread it and I was like, "Yeah, right there."

DJ (00:50:59):

Some other things...I know a lot of the embedded world uses C and C is a great language. It's also, it's very powerful and...it'll let you cut your own foot off. But the article is, "Why are you still using C" and it's advocating for using C++. And I've got a mix, back and forth, between C and C++. My current project is all in C, but it's funny. 'Cause...you know, I've been there seven months now, and as I'm looking at it, I'm like, "Oh, you guys should just use C++."

DJ (00:51:33):

Because the code is written to try and do that. All the things that C++ gives you for free -

CW (00:51:39):

Yeah.

DJ (00:51:39):

- they're trying to do in C...You know, 20 years ago the arguments were well, C++ has all this extra overhead, all these things you can do. And you know what C++ has gotten a lot bigger.

CW (00:51:51):

You make spaghetti code.

DJ (00:51:52):

You don't have to -

CW (00:51:53):

Says the person in a bowl of spaghetti.

DJ (00:51:54):

Yeah, yeah, yeah.

EW (00:51:57):

[Laughter].

DJ (00:51:57):

...You don't have to use all those features. You can use a subset of it, but you know, take a look at it. I do remember a story from way, way back that, when I started at DEKA, they were using C++, and they had an interview candidate come in and say, "You absolutely cannot use C++ in embedded software." That guy didn't make it very far in the interview process.

CW (00:52:19):

There's a robot. Escorted him out.

DJ (00:52:21):

[Laughter]. Yeah. Then, there's another set of articles that were in, I think, EE Times way back in the day, "30 Pitfalls for Real Time Systems Development." And that's another one...you know, I have this set of articles. I keep 'em, I have 'em on, I have PDFs of 'em. And whenever someone starts, I'm like "read these," or actually, I usually do it when, you know, we've made them an offer and they've accepted and they haven't started yet.

DJ (00:52:51):

I'm like "Here's some stuff to start out," especially,...and interns too. I send them the articles. "These are good things to read." I'm not going to quiz them on day one, but you know, it does come up in conversations and I can usually tell if they've read them or not. You know, and in that time, it's like when you get a new device, you want to just start playing with it and seeing how it works and you put the manual to the side.

DJ (00:53:14):

But if someone were to hand you the manual before you ever had the device, you would actually read it. You might not read it cover to cover, but you're going to look through it. So I like to give these articles when they're excited about the job, but they haven't come in and seen, "Okay, now I have to make all this stuff work." So it's that little window of opportunity to sneak these things into 'em. And they usually give them the TDD, I buy the TDD book as well and send that, either send it to 'em or it's sitting on their desk when they start. But those are -

EW (00:53:40):

Just the TDD book? Sorry.

CW (00:53:43):

[Laughter].

DJ (00:53:43):

I have the "Making Embedded Systems" as well. Maybe I should add it to my list. I will, I will, especially the more junior developers, but yeah.

EW (00:53:54):

But none of these are new.

DJ (00:53:57):

No.

EW (00:53:57):

None of these came out in the last six months.

DJ (00:53:59):

No, no.

CW (00:54:01):

That goes back to software being software.

DJ (00:54:02):

Yeah.

EW (00:54:02):

Yes.

DJ (00:54:04):

Well, that's why my favorite tool is Make. There's another, there's a great article that someone had given me when I, just a few years ago, that now I pass around when people say, "What? Make? Make is 40 years old." I think it just turned 40, this year, maybe last year. And my favorite thing is like, "Oh, we can't use whatever tool it is 'cause it hasn't been updated in 6 months, 12 months." I'm like, "You know, Make hasn't been updated in a long time and it's based on a 30 year, 40-year-old plan...But you know what, it works really, really well." It's got some quirks. It's got some things that you gotta watch out for.

EW (00:54:42):

A few. [Laughter].

DJ (00:54:42):

But it works really well. And I see all these other systems, people are like, "Wow, we've created, we've replaced Make with this new system."

CW (00:54:52):

That sits on top of Make.

DJ (00:54:53):

That sits on top of Make, either sits on top of Make or recreates it poorly...and "Oh, it handles this one thing that Make doesn't do." Yeah, but it doesn't do the other 40 things that Make does really, really well. And you don't do 'em at all. Like "Oh, but that's not that important." And you see them every week, they have a new release that does another little feature. I'm like, yeah, I'm sticking with Make. I'm old school like that.

EW (00:55:18):

Sometimes you find tools that work. And even if they're quirky, you keep using 'em because they're effective.

DJ (00:55:26):

Oh, so another question you didn't ask in the lightning round was, I think you asked it, recently, last week or recently, was IDE versus command line. And I use both, you know, I use the IDE to debug, I use the IDE to do simple, whatever project I'm working on to do the build right then. 'Cause I'm just, I'm in the groove, but every project I work on can build from the command line.

DJ (00:55:55):

It's a Make target and I, you know, you say Make, either Make and it's the all target, the first one, or a special package. The reason you do this is because, when you go to release this, you don't want to say, "Hey, we're doing a release today. You know, is Joe here? Is Sue here? She's up for the release." You don't release from your desktop. You release from your continuous integration server. This is another software, software in general, best practice, is having a computer that is not a development machine that can do all your builds and...you know, your IDE is not going to, you know, your build machine is not going to open your IDE and click build and save off the packages.

DJ (00:56:40):

All that stuff has to be scripted. It all has to be driven from a command line. That thing sits in a closet and you know, it either runs every day, every night, every branch, every commit, you know, you can set this, you can set whatever you want. Mine runs on every commit by the way. Commit to the main repo. But that thing, when you go to do the release, all you do is you kick off the release branch, you update the version number, any other little pieces of documentation that you want to have on there, you push it to the server, and depending on how long your build takes, five minutes later, you have your release candidate.

DJ (00:57:22):

Now you just go off and do your formal test. Oh, you know what, all your unit tests ran on the release server as well. You know, all your unit tests pass. If you are really far along, you've run basic smoke tests on hardware from your integration server. There's probably still some stuff that you have to do manually. Someone on your team, someone in QA, they pull that release binary, they run the tests, and you know, they give the thumbs up, thumbs down. You merge the release back in, you tag it, "Hey, look...your release is done." But you know where that started from? Being able to build from the command line, being able to build the entire project from the command line.

DJ (00:58:02):

That's actually one of the things from...Joel's list as well. Can you build it in one step? And it's amazing how many people don't do that. They say, "Oh, we're going to release, go pull down the latest branch or the latest, you know, whatever develop is or master, however you do your release process, change the version number and hit compile. Okay. Here's the thing that I'm going to copy into the network drive." I'm like, "Okay, did you know...you know, your tools updated, you updated your tools, and now you're on a different tool set? Now, you just broke something for everyone else."

DJ (00:58:37):

"Oh, you didn't have whatever the special version of some key, or whatever you had to have in place. And you didn't do that. Now that version is completely different than all the ones we built before." There's so many things that will go wrong if you're doing it from your desktop. But if you have your integration server doing it, it's going to be the same every single time. It's just, that's what you should be focused, that's what you should be aiming on. Now that has nothing to do with getting into embedded software, but that's further down the line, but just do that, and you'll be happy.

CW (00:59:10):

Being able to say that you understand those things is useful.

DJ (00:59:13):

Yeah.

EW (00:59:14):

Well, and the software engineer, you may have an advantage with that. You may be able to say, "I know how to do these things. I may not know how to write an I2C driver, but I can make a test system that will actually work so that you're no longer dependent on one computer that may go up in flames at any time."

DJ (00:59:34):

Yeah. Yeah. And that's...you know, if you come in to me and say, "Hey, I want to be in embedded. I have a little bit of experience, but I also do, I've seen all this other stuff." I'm like, "Oh, you've used those. Okay. Let's let's start talking."

EW (00:59:48):

Yeah. Your software knowledge is not wasted.

DJ (00:59:50):

Yes.

EW (00:59:51):

How do you balance being confident in both the theory, like all of the Spolsky stuff and all of the TDD stuff, and the tactical information, like processors and specific peripherals? It seems like it's so much knowledge to get all at once. What do you look for for them?

DJ (01:00:12):

Yeah, that's a great question. I'm always thinking of myself like, "Okay, I know this and I know that and I don't know anything."

EW (01:00:22):

Yes. [Laughter]. I feel that way a lot.

DJ (01:00:26):

I think that the key is to realize you're part of a team and this is my little kumbaya moment, but you're part of a team. You don't have to know everything. Especially as I've, you know, been in on the management side of things. My current job, I have the compiler, I have hardware. I can do some of it, but unfortunately, I don't touch the hardware all that often as much anymore. I'm doing a lot of other things and especially setting up build stuff for making the team work more efficient.

DJ (01:00:59):

But, you know, I rely on the fact that I have this team and you know, this person knows a lot more about the language, the intricacies of the language. And this person knows a lot more about processors. This person knows about sensors...It's not just on the firmware embedded team. It's also the EEs. You know, if you look, if you have the current product, that's what you have, you're gonna make it, you gotta make it work. But when you start talking about that next generation one, you don't have to know everything. Everyone's going to bring something to the table.

EW (01:01:35):

I keep working on teams where I'm the only person.

DJ (01:01:37):

That's the flip side, yeah.

EW (01:01:39):

Then you do have to know everything.

DJ (01:01:40):

Then you do have to know everything. Or you know people who know other things and you reach out to them. I've been in the industry long enough that I have friends on both coasts. I have people I've worked with. I have medical, non-medical. And actually, when I started at Airware, I was like, "Hey, we need, I need some of these, you know, I was looking at static analysis tools and, you know, embedded software." That's not the first thing on your list is, "I need to know about embedded and static analysis tools."

DJ (01:02:14):

But I reached out to old coworkers and colleagues and friends and said, "What are you guys using?" and just took a sample. And, you know, I've poked around on the web and seen people are using. But being able to talk to people is huge. And it wasn't, you know, hour-long conversations. It was, you know, through an email or phone call, and say "Hey, you got five minutes?" You know, you're going to build those resources up over your career. Make use of 'em.

EW (01:02:42):

And how do you convince management?

DJ (01:02:47):

I am management. [Laughter].

EW (01:02:48):

I know, I know.

DJ (01:02:50):

No, but there's always a -

EW (01:02:50):

But you haven't always been.

DJ (01:02:50):

There's always a layer above you. There is a layer above you...That's the hard one. That, you know, people will talk about, writing software is really hard. Convincing people is even harder. People aren't compilers, you know, they're not a piece of hardware. You just, if you get it just right, they're going to go your way. A lot of it is demonstration.

DJ (01:03:17):

You know, if you need to lay out, you know, thousands and thousands of dollars, you're not going to just go get that tool and start using it, but, you know, maybe you can demonstrate some of this, like a CI server, a continuous integration server. If you go to...if you don't currently have that, if you go to someone and say, "Hey, you know, I need a server, I need a computer, it needs to live on the network. It has to have these kinds of resources, and I need to have this software," that's going to be a hard sell.

DJ (01:03:47):

Depending on the size, you know, depending on the company you're at. But you know what you can do? You can find, "Hey, is there a computer that just, you know, an older computer that's, no one's using anymore an older laptop. Can I have it, can I use it for a few weeks? Is there a lab machine that I can use that no one's, that is just sitting there gathering cobwebs?" You don't need the most powerful, you know, latest machine. You just need something that runs. There are free CI servers out there. There are free tools. Jenkins is what you know is a very popular one.

EW (01:04:20):

A lot of people use that.

DJ (01:04:21):

You know, you're going to have to learn. I'm in the business. I'm learning it right now. I've got it set up, but there's so many things about it that I just don't know. But get that set up, have it sitting on your desk and, you know,...especially, if your boss or your team leads,...you know, "We gotta get this thing out" and you don't have time to work on these other things. You know, lunch hour is a great time, you know, grab your food, eat it quick and get back. After hours is -

EW (01:04:54):

And baby steps.

DJ (01:04:54):

Yeah. Baby steps, you know, you're not going to get it in the first step. But get it set up. Can it read from your repository? You know, you might have to create some, depending on where things are, you might have to create an account for it, but you know what, until you get that all rolled out, use your account...

DJ (01:05:13):

If it's on a remote, if you're using GitHub or Bitbucket or any remote code repository, this isn't your production level. This is your "I want to prove that this works" and get it set up so it runs once a night and you can come in in the morning and say, "Look, it did this." Or it runs, you know, at lunchtime, so you can see the results right away. So a lot of times, that's one example, but it's, a lot of times it's just setting up...an example of why, why you want to go somewhere. Now if you weren't using embedded tool chains that have really expensive compilers, you're not going to be able to do that, but maybe you can run it on your machine.

DJ (01:05:52):

Now you are working on your machine, so you don't want to distract, you know,...you don't want that running while you're doing things, but you can set it up. If you don't take your machine home, have it run at seven o'clock in the morning or six o'clock in the morning before you come in or have it run at the lunch hour. So you can sit there and watch it and make sure everything's working.

DJ (01:06:11):

That's for a CI server. And for other tools, I've seen this before too, is,...you know, a static analysis tool on Linux. You" have Lint, that's a really, it's again, it's an old school tool, but it works. It gives you a lot of information about your software and, you know, you can't just, maybe you can't go to your project lead and say, "We now must use Lint" or, you know, whatever static analysis tool you want to use. That's the only way we can do really, really good software. Run it on your code, run it, you know, run it on the stuff that you're working on.

DJ (01:06:48):

You know, make it...part of your development process. And you know, in your code reviews, hopefully you're doing code reviews, you know, people start saying, "Huh," you know,...QA is not coming back saying, "Hey, we found these problems" or you know, your tests are always passing or that you can start demonstrating, the code is getting better. Or you can just look at the code and say, look it, as you're reviewing other people's codes or...as you're making modifications to other parts of the code base, you can be making these improvements.

DJ (01:07:25):

And when people say, "Hey, why did you add this check in here? Why did you know, why'd you change this?," I was like, "Well, that's a common error that causes, you know, that gets hidden by this other thing." "And then how did you find that? How did you know about that?" "Well, use this tool called Lint, or I use this other -

EW (01:07:44):

Clang.

DJ (01:07:44):

Clang or Cppcheck. That's a free tool, same thing. You know, I use this tool,...I've been running it over the code and identifying, you know, potential problem areas. And they're not a hundred percent guaranteed to be hot problems, but these are things that could cause a problem down the line. Or yeah, I mean, if you have a GCC-based project or you have a project that can be built in Clang, run it with Clang and see if it's, you know, the compiler is really good at spitting out a lot of other warnings about what could go wrong. Your tools, use your tools.

EW (01:08:21):

They're there to help you.

DJ (01:08:22):

Yeah. So I mean, and that gets back to, you can do things within your own little sphere and then maybe, you know, the person sitting next to you says, "Hey, I noticed you've been doing that. Can you show me it?" And like, "Oh yeah...I have it built in here." And now they start using it. It's a little bit of guerrilla warfare. But, sometimes that's what you gotta do. Especially if you don't have a lead as like, "Hey," you know, if the lead's like, "We just gotta get this out," maybe you can help change it from the bottom up.

EW (01:08:52):

And that does work, but going in and putting your foot down and saying, "We should do this, and you have 25,000 errors in your code." It's not, it doesn't work. And you bash your head against the wall and it's no fun for anybody.

DJ (01:09:06):

Yeah.

EW (01:09:06):

And yet, if you can start being the person that everybody goes to, because your code works -

DJ (01:09:11):

Yeah.

EW (01:09:11):

And your code works because you use a tool that tells you, you use a tool, you use unit testing, and suddenly you're the person everybody wants to go to. It takes awhile.

DJ (01:09:25):

It does.

EW (01:09:26):

It's baby steps. And it's a lot of learning the tools.

DJ (01:09:29):

Yep.

EW (01:09:29):

'Cause you don't necessarily want all of those warnings, but -

DJ (01:09:33):

That's actually a good one.

EW (01:09:35):

You need to start going.

DJ (01:09:36):

Yeah.

EW (01:09:36):

You need to start doing it.

DJ (01:09:37):

Yeah.

EW (01:09:37):

And yes, it's hard to start.

DJ (01:09:39):

Another really, really simple one is if you don't have warnings as errors turned on in your build -

CW (01:09:46):

Right.

DJ (01:09:47):

Start eliminating those errors and then turn that on. And now other people on the team are going to, you know, turn on your build, commit it to the repo. Talk to people, say, "Hey, I've eliminated all the warnings." One, they should notice it. If they're not noticing the error, the warning count go down, start wondering about your coworkers.

EW (01:10:07):

Yeah.

DJ (01:10:07):

But, once you get it down, if you can eliminate all those, talk to the lead of the project, say, "Hey, I, you know,...I took some time." You know, "I'm delivering," make sure you're delivering the things you promised to deliver on, but say "I took some time and I've eliminated the warnings. Can we turn this on so it's now an error? So that we don't get any more?" And, you know, just say, you know,...the warning, my favorite thing is leaving warnings on and people say, "Oh, it's just a warning. Don't worry about it."

DJ (01:10:37):

A warning is a compiler telling you something, you know, you're doing something a little weird. I'm not sure that's really what you want to do. And maybe it is. And if it is, there're compiler flags that you can go in and say, ignore it for this one piece of code, because we're doing something really, really clever. Don't do clever things, and then you'll be better. But, you can turn it off for very specific cases. That's the better thing to do. And then just say, "Oh, just ignore the warnings." 'Cause those warnings are going to come back and bite you at the worst possible time. You're going to be doing a demo in front of your CEO and turn to the company in front of a customer. Or that thing is going to be shipping and it's in the field. That's when that warning is going to break.

EW (01:11:16):

So we have this Slack channel that is private and Dennis is on it and a couple other people, Andrei and Svec, and some other experienced engineers, who are all nominally supposed to be writing for the Embedded FM blog, but I'm noticing some deficiencies there.

DJ (01:11:39):

[Laughter].

EW (01:11:39):

Anyway, we have this channel and we talk on it. And Dennis often mentions unit testing and continuous integration servers and things he's working on at work like that. And everyone on the Slack channel wants to work at Dennis's company. I don't think we place enough understanding, that not only are these making better pieces of code, better software, better devices, it's making a better environment to work in. And I wish that the management and leads who don't support test-driven development, unit testing, better lint checkers, all of these things. I wish they understood. That yes, it's a little hard to get started, but once you're there, it's so much better and experienced people know that.

DJ (01:12:31):

That's the funny, that's actually really interesting because, folks on my team here, when I came in, they were doing, you know, pull on it, you know, here's the branch, hit build in the compiler, here's a release. There were a lot of things that were not in place. And, you know, you talk about how do you bring this into a place? When I started at Airware, I came on really hot and I definitely tweaked some noses when I did it.

DJ (01:13:00):

And so I scaled back and then just started building things within, the stuff I was, I did the whole process that we said before. I just kind of built it out from there. They already were doing unit testing. So that was great, but I just kinda introduced more things to it. And they had CI servers set up. So I learned a lot of amazing techniques from the folks at Airware. But eventually, all of a sudden I was like, "Okay," now I had the team, I had more of the team now I'm in charge of all the embedded development.

DJ (01:13:35):

And so when I came to this company, one, I came in at that lead position, but I also knew that there was an established team there. And I can't, you just can't come in and say, "You guys are doing this all wrong." And they weren't doing it all wrong. They were doing, they had functional code. They had, you know, they had things working. So, you know, the greatest thing at the end is, does the device work. If it works, then you did something, right.

DJ (01:14:00):

The technique to get there may have been, you know, questionable at some points. But, they had a functional device. Now taking that to production, there's things you can do to improve, but especially in those first, in that first week or two, first couple of weeks, just looking at how people were developing. I'm like, "Okay, this could be better. And here's what we're going to start. You know, we're going to start adding on how we do things."

DJ (01:14:26):

And you know, getting that CI server in place so it was always building. There were no more broken, you know, develop. There was no more broken trunk. We were using SVN originally. When I first got there, we switched to git. But you know, we started, people started noticing like, "Oh, I don't have to worry about such and such thing happening 'cause, you know, it tells me there's issues coming up."

DJ (01:14:48):

We just, unfortunately it took a lot longer to get in, but that was my fault. But we got unit testing going and across the team, every single person has said, "Wow, I just found a problem that I would not have found or would have been impossible to find on the hardware. But I wrote a unit test and it failed. And I was not expecting that. And now I put the fix in and now it's better." And, or, you know, "We have this code that has low-level driver stuff and application stuff kind of mixed together." And they're like, "It's going to be impossible to unit test. So I need to refactor it and pull out a driver-level piece of it and an application piece of it." I'm like, "Alright, good, go for it." And they come back and say, "Okay, we've separated it out."

DJ (01:15:29):

And you go look at the code. And it's so much more readable. And the whole team was like, "This is really good." ...From above me, people were still like, "Okay, software needs to go faster and faster," but I look at what we're doing and we're working so much faster than we were before. When I started in the first couple months, we had someone who recently started on the team and they're like, "Wow, you guys work really fast here. I'm like, "We're still jogging. We haven't started running yet. We still have a ways to go, but we're getting there."

DJ (01:16:01):

So these tools and techniques, and just looking at how things work, and making them work better, helps everyone. My favorite, actually, had this conversation just just yesterday. For some reason, our build is recompiling everything every single time, instead of doing partial builds. And I didn't notice it because I hadn't been using it. And someone said, "Yeah, it's been doing this for this for a little while now." I'm like, "Okay. When things like this happen, we have to fix them right away." Because if you compile, if it takes three minutes to compile the entire project and you do that 10 times a day, that's half an hour of your day just sitting there looking at the compiler.

DJ (01:16:40):

But if it's doing partials and it takes three seconds, that's 30 seconds of your day. Which would you rather use, 30 seconds, would you rather be spinning that other 29 and a half minutes, thinking about the code, working on stuff, or would you rather spend that time watching the compiler go "compiling file, compiling file, compiling file." So all those little things, they add up really quick over the course of a day, over the course of a week, over the course of a project.

EW (01:17:05):

I feel like it's redundant now to ask you for a thought to leave us with. [Laughter].

DJ (01:17:09):

Know your tools.

EW (01:17:12):

And I think that's where we are in the show.

DJ (01:17:14):

Okay, yeah.

EW (01:17:14):

So "know your tools," or do you actually want -

DJ (01:17:20):

I mean, I think that's it, is really understand your tools and it goes back to what I said earlier, software is a craft. You know, you're not going to go to a, you know, a woodworker who's going to have, you know, a dull ax, a dull knife. They're going to keep 'em sharp because that's their trade. It's the same with software. Your tools are your trade. You don't have to know every single option that they can do. Every single thing they can do, but the ones that you need to use, make sure you understand how they work and use them. And if things start looking like, you know, pay attention, how long things take. If things start taking later, what changed? Start taking longer, what changed? But just really know your tools, understand them and keep them sharp.

EW (01:18:10):

Thank you. That is good advice. And I think you've had lots of good advice through the show. So thank you very much for being with us, Dennis.

DJ (01:18:16):

Oh, it was a great, it's always a great fun talking to you guys.

EW (01:18:19):

Our guest has been Dennis Jackson, Director of Firmware Engineering at Element Science. Dennis sounds familiar because he was on episode 94: Don't Be Clever, when he was at Airware, making a drone operating system. I would like to say thank you to Christopher for producing and for co-hosting. And of course, thank you for listening. Special thanks to Patreon supporters who let me send Dennis a microphone.

EW (01:18:48):

I am looking for a new gig, ideally something mathy or robotics-ish. If you know someone needing a strange set of embedded systems, architecture, prototyping, code implementation, and personal communication skills, please keep me in mind. I should get a job at some point.

EW (01:19:07):

And a quote to leave you with, this time from Voltaire, "The perfect is the enemy of the good."

EW (01:19:14):

Embedded is an independently produced radio show that focuses on the many aspects of engineering. It is a production of Logical Elegance, an embedded software consulting company in California. If there are advertisements in the show, we did not put them there and do not receive money from them. At this time, our sponsors are Logical Elegance and listeners like you.