524: This Isn't a Movie
Transcript from 524: This Isn't a Movie with Nathan Jones, Christopher White, and Elecia White.
EW (00:00:06):
Welcome to Embedded. I am Elecia White, here with Christopher White. This week we will be talking to Nathan Jones about teaching, embedded systems, RTOSs, and potentially mental health. I do not really know, because I did not make an outline.
CW (00:00:24):
Hi, Nathan. Welcome back to the show.
NJ (00:00:27):
What is up you two? It is so good to be here.
EW (00:00:30):
Could you tell us about yourself, as if we met at a real table at the Embedded Online Conference, which has no real tables?
CW (00:00:38):
Very confusing, but okay.
NJ (00:00:40):
Maybe virtual tables?
CW (00:00:41):
Mm-hmm.
EW (00:00:41):
Yeah.
NJ (00:00:41):
Yeah, I would say I am just super stoked to be back at the Conference. I get to present again this year, on... I am giving a two hour workshop on security stuff.
(00:00:53):
I spent 15 years in the military, but sort of stumbled my way into embedded systems. Now I get to spend all my time writing articles for DigiKey, and figuring out how to teach people cool stuff, which is just a blast.
EW (00:01:09):
All right. It appears I prepared no... <laugh>
CW (00:01:15):
Look, it has been a rough couple of weeks, folks. I am sorry. This is what you are going to get.
NJ (00:01:18):
This is going to be a great episode.
CW (00:01:20):
This is what you are going to get. Nathan is going to carry it. Lightning round is on vacation.
NJ (00:01:23):
<laugh>
EW (00:01:26):
Lightning round is on vacation. Yes!
NJ (00:01:28):
Okay, but I have got a tip.
EW (00:01:28):
Okay.
CW (00:01:29):
Go for it. Good. Yeah.
NJ (00:01:31):
Again, I had to think long and hard about this, but I settled on the tip that everyone once a day should go out for a hot girl walk.
CW (00:01:41):
Excuse me?
NJ (00:01:41):
Or as I like to call it, a "hot dad walk."
CW (00:01:43):
Okay.
NJ (00:01:45):
Which for me- If you go outside, man, every time it is just beautiful, it is awesome. You step away from a screen for a bit. Do like one little lap around your block and your mind is refreshed.
(00:01:56):
But for me, the other key is to pop in the earbuds and play a YouTube video where a guy did a bunch of funk covers of Linkin Park's greatest hits. I am going to tell you, amazing. I am like dancing and sliding my way around my loop every time. It brings you joy.
EW (00:02:16):
So part of it is the being silly part?
NJ (00:02:20):
Yeah.
EW (00:02:22):
I think that part is the part I miss sometimes.
CW (00:02:24):
We have the dog. The dog adds a plus three to silliness, without us doing anything.
NJ (00:02:31):
How is your beagle doing?
CW (00:02:32):
Oh no, she is not a beagle.
EW (00:02:34):
Oh yeah, we are going to- Okay.
CW (00:02:36):
So, misconception.
EW (00:02:39):
And my fault. Zoe was a beagle. We got her decades ago.
CW (00:02:45):
2004.
EW (00:02:46):
She passed away, what feels like recently-
CW (00:02:49):
2020.
EW (00:02:49):
But probably is not. The new dog is not a beagle. The new dog is a pound pup.
CW (00:02:57):
She might be a Miniature Schnauzer. Miniature- I cannot say that. Miniature Schnauzer something.
EW (00:03:04):
Yes. If we gave her a beard, she would look hilarious.
NJ (00:03:05):
<laugh>
EW (00:03:05):
But when we got her, she was- It was pretty much do not have any loud noises or kids. Or pets or cats, dogs, butterflies, everything. She was just very scared all the time. But now she is almost a normal dog.
(00:03:26):
No, we do not have a beagle anymore. Beagles are awesome. But the thing that they say in the books about beagles is that they are merrily stubborn.
(00:03:36):
Which means they are having a good time, but they think you are stupid. I cannot tell you how many times my dog looked at me- How many times Zoe looked at me and was just like, "I do not understand why you are not rolling in this with me. You must be completely out of your mind."
CW (00:03:55):
Why are you trying to stop me from this obviously fun thing I am doing, eating poison mushrooms?
EW (00:03:59):
Yes. JoJo is, while she is small, like 12 pounds, she is on the Labrador scale.
NJ (00:04:07):
Oh, okay.
CW (00:04:09):
In terms of affection.
EW (00:04:11):
She loves us. Really, truly, honestly, happy to have us home. Would do anything that she could in order to-
CW (00:04:19):
She will climb on our heads, basically.
EW (00:04:22):
To get belly scratches. She loves us.
NJ (00:04:23):
That is great.
EW (00:04:23):
So we do not have a beagle anymore. We have a dog who actually likes us. Although I loved the beagle so much. And JoJo is great too. Anyway.
CW (00:04:31):
Dogs are great. Anyway.
EW (00:04:32):
Dogs are great.
CW (00:04:32):
Yeah.
EW (00:04:32):
It is a good way to walk around the neighborhood and say, "Hello," to everybody too. As opposed to being that weird guy who walks around the neighborhood and is constantly doing weird tip things.
CW (00:04:45):
Excuse me? Tip things?
EW (00:04:46):
Hip things.
CW (00:04:48):
Hip things.
EW (00:04:48):
He is sliding around to Linkin Park's funky hits.
CW (00:04:52):
Got it. Got it. Got it. Got it. Got it. We should probably move on to something else <laugh>.
NJ (00:04:56):
<laugh>
EW (00:05:00):
Yes! You said you have been writing for DigiKey.
NJ (00:05:04):
Yeah.
EW (00:05:05):
And also for EmbeddedRelated.
NJ (00:05:08):
Mm-hmm.
EW (00:05:08):
One of the things that I was wondering about was, who is your audience when you are writing? And how do you decide that?
NJ (00:05:21):
I do not go through a super rigorous process. I think my audience is me before I start the project that I just finished, to be able to explain it.
EW (00:05:30):
It is hard, because sometimes you need to explain early career things, like what is a race condition. And sometimes you need to be more mindful of, "Well, we have already been doing this for ten years. Let me give you the low down, without having to use all the words."
CW (00:05:51):
Mm-hmm. Because we are skipping to something more advanced?
EW (00:05:52):
Because we are skipping to something pretty advanced. And I cannot really tell you about this, unless you already understand that mutexes are a solution to this problem, and I am going to give you a different solution.
NJ (00:06:03):
Yeah. I try not to overthink it. I guess if the topic is a little niche, a little more niche, then I pay a little more attention to making sure there is a longer on ramp, before we get to the meat of the article. Otherwise, I try to just assume I am going to help folks take one or two steps, to lead into it.
(00:06:24):
I am playing around with things that I can do in the articles. Using little dropdowns or HTML will give you some hover texts, where it is like, "Okay, I do not need to explain this term, but there is clearly a link here." So if you hover over it or follow it to a footnote or something, it will give you a little more depth for those folks that want it or need it.
(00:06:48):
And to not necessarily clutter up things with folks that see that term and are like, "Yeah, I know what you are talking about."
EW (00:06:54):
How often do you write something, or are thinking about writing something, and then find another article that explains it well enough?
NJ (00:07:05):
Oh, it is the classic PhD conundrum, right? Like you hope no one scoops you on your PhD thesis or something. I would say, it is definitely a concern, a fear, unwarranted or not. I find lots of articles that I use for references, where things are really well explained.
(00:07:29):
But maybe it is just because I am a little on the spectrum or something, but none of them usually hit exactly the ordering that I want, or the way things are explained, or has enough pictures. So I always feel that my contribution, maybe I am not necessarily offering anything novel, but I am explaining it in a different way.
(00:07:57):
When I was an instructor at West Point, that is right. A lot of times as a student, you read the description of something in the textbook, and it is not inaccurate. It just does not quite make sense on the first pass. And you need to hear someone explain it two or three or four other ways, before that fourth one finally clicks and you go, "Oh, okay, I think I get it."
(00:08:18):
So I try to remind myself that even just adding to the corpus of information is valuable. Maybe there is someone where none of that made sense, but they read mine and they go, "Oh, I finally get it."
CW (00:08:31):
I think that is totally, totally valid. I find that myself reading about stuff, that I want to see not just, "Oh, this person explained it differently," but sometimes as a different voice, and I can-
(00:08:40):
Especially for a dry topic, like if it is in, I do not know, Zephyr documentation voice, it is a lot harder for me to get through something. Than somebody who is writing a walkthrough and they, "Well, here is my experience was," and throws in a little bit of humanity. It is easier to go through technical documents or technical tutorials and explanations that way.
(00:09:04):
I think at a minimum, there is nothing wrong with having people explain things a different way. That is why there are 50 different physics textbooks. We do not just have one physics textbook that everybody says, "This is the physics textbook. We wrote it in 1950, and that is the way it is."
(00:09:18):
Everyone has a different perspective, and I think that is quite useful.
NJ (00:09:20):
"Physics has not changed since then. Why do you need a new one?"
CW (00:09:23):
Right. <laugh>
NJ (00:09:24):
Yeah. I find that I personally too, I like a lot more graphics and demonstrations than are in a lot of texts.
(00:09:30):
Even I will go- Sometimes I will read articles that I wrote, I finished a couple months or a year ago, and I go, "Ah man! I should have put a diagram or something right here," because there is a little block, a paragraph, where I describe something that I feel like is clear. But I had a picture in my head and I was like, "Ah, I should have given that to the reader."
EW (00:09:49):
I am looking at your RTOS series of posts for EmbeddedRelated, which was two years ago, so I am not going to give you quizzes on it.
NJ (00:09:57):
Thank goodness.
CW (00:09:59):
What does the "R" in RTOS stand for? No.
NJ (00:10:04):
Reese’s [the candy]
(00:10:04):
I am really proud of that one, honestly. That was probably two or three years in the making, honestly.
EW (00:10:10):
It is tough, because there are books about RTOSs. You take the position that a super loop might be enough, that we do not necessarily need RTOSs all the time.
NJ (00:10:20):
Mm-hmm.
EW (00:10:20):
Why? Is an RTOS not simpler? Should we all not just be using Zephyr?
CW (00:10:25):
<groan>
NJ (00:10:25):
<laugh>
EW (00:10:25):
Anyone missing the subtext, is missing the subtext that Zephyr is pretty complicated.
CW (00:10:38):
It is- Yeah. I suffer all the time these days.
NJ (00:10:42):
Yeah. As a consultant, I am sure maybe at one point someone is going to pay me to have to do something with it. But I am going to try to avoid it right now, which is maybe unfair because I have not done too much with it. But it just feels like the Swiss Army knife that is supposed to solve a thousand problems, but when you try to do that, the code just gets big and complicated.
EW (00:11:05):
There is goodness to it. There is a lot of goodness to it.
NJ (00:11:08):
Yeah. No. So to your question about RTOSs, I think there are- If you want something like just off the shelf, you do not want to touch the code that manages multiple tasks, then pull on something like a FreeRTOS or a ThreadX and throwing it at your system, is totally fine. It probably will be the quickest way to just getting something going.
(00:11:35):
But I was really taken aback the more I looked into things, how hard it is to write code for a pre-emptive RTOS, like FreeRTOS or ThreadX, in a way that is safe, that is thread-safe.
(00:11:58):
I share a couple of examples in that article series of race conditions. Some of them are, they are super nefarious. They come down to sometimes your processor reordering instructions in a way that opens up your thread to an error condition, if something happens to interrupt it at exactly the wrong time.
(00:12:17):
So my stance at the end of this was like, "Okay. There are lots of tasks and applications that need the things that an RTOS gives it. But when you do so, you have to understand you open yourself up to a lot of really nefarious bugs, that might creep into your code through these race conditions. Unless you absolutely need the response time that you get from an RTOS, you are probably better off just avoiding it entirely."
(00:12:44):
I thought it was really cool in looking at the math actually, that there are in fact some sets of tasks where you can come up with some numbers, where on an RTOS some of the tasks miss their deadlines, but in a super loop they do not.
CW (00:13:00):
That is fair, if your main concern is real time response. I feel like that is actually not high on most people's lists these days, when deciding whether or not to use an RTOS. It is more like, "I do not want to write a bunch of boilerplate code, to do either tasks or synchronization or messaging or handling my ISR or scheduling events and that kind of thing."
(00:13:28):
I feel like most of the time working on a project, real time is not that interesting. It is more like, "This is too complicated," and as soon as you add Bluetooth or networking, things explode.
(00:13:39):
But yeah, if you are looking at the real time response, certainly it is much easier to control all of that if you are in a loop.
EW (00:13:49):
You added pre-emption in there. Could you explain what pre-emption is?
NJ (00:13:53):
Yeah. So the crux of the problem is we have an application that is trying to do multiple things. It is reading from a sensor at the same time it is writing to a display, at the same time it is calculating some filter coefficient or whatever. Even on a single-core processor, we can do that by just rapidly switching between tasks.
(00:14:14):
The simplest way to do it is with that super loop, where in the most basic form literally inside main, I reach a while (1) loop that does read sensor, update display, calculate new coefficient, and then loops back up to the top. The nice part about that primarily is the simplicity.
(00:14:36):
I can just throw my tasks in there and I know that they run kind of in order. But the problem comes when, I do not know, let us say I need to take sensor readings at a very specific rate. If I am updating my display and it takes a long time, well maybe that rate, there is jitter. Or maybe I miss a sample, because I am spending too much time updating the display.
(00:14:56):
So in the article series, I talk about a couple of ways where you can stay with the super loop and stay with that simplicity, and get out of certain tasks faster to go back to reading a sensor.
(00:15:08):
But all those are forms of what we might call "cooperative scheduling," where the task itself has to finish and the function has to return. It has to yield back to the scheduler, in order for another task to run.
(00:15:20):
In a pre-emptive scheduler like FreeRTOS, essentially at a minimum once every timer tick, or once a millisecond at a minimum, but also any time one of the functions requests the scheduler for a mutex or a message queue or something, it gives FreeRTOS a chance to say, "Hey, is there any other task that is asking for my attention?"
(00:15:43):
So those API calls, those timer interrupts or whatever, they can potentially at that point say, "Oh, hang on display task, the ADC needs to run and grab a sensor." It will manually shuffle things around in memory, to let that task execute. And then return to whatever task was processing, like the display thing.
(00:16:08):
That gives you the ability to assign tasks to be very high priority, and to know that they are going to run almost as soon as they are ready to go.
(00:16:16):
But you then also have to consider, "Okay, if these things are touching shared resources, shared data or something like that, what are the race conditions? What are the bad things that could potentially happen, if my display task between any two machine instructions, gets interrupted to do something else?"
EW (00:16:36):
On one hand it is much simpler, because you do not have to worry because the high priority tasks are going to happen when they need to happen. On the other hand you can dig yourself a deep hole with the shared resources, because the protection for them is gone. Because at any time, the high priority thing could happen.
CW (00:16:56):
Which is always true if you have interrupts.
EW (00:17:00):
It is. If you think about interrupts as pre-emptive tasks, yes. If pre-emptive tasks make more sense to you than interrupts do, then that totally makes sense. Interrupts, for me, were interrupts first. It is like older processors, you had special instructions to return from interrupt, and you had to handle the interrupts differently.
(00:17:28):
But yes, they can happen at any time. That is why interrupts need volatile variables and to be locked out. Or if you are changing a variable that might need multiple instructions, you need to do- I do not even know what I am saying anymore.
CW (00:17:47):
You need to be careful, because you are managing resources that could be in use in the other context.
EW (00:17:54):
Right. That.
CW (00:17:56):
That is true of tasks as well. To a slightly lesser degree, but still true.
EW (00:18:02):
It is easy to get- If we follow the rules with interrupts- Like we have all of these handed down rules. You do not put print after interrupts. You try to get out of an interrupt as soon as possible. You try not to do a lot of math in an interrupt. You take the data and you set it up for next time and you go away.
(00:18:22):
These are all because we do not want to deal with the pre-emption that the interrupts cause. But yeah, there are other ways of handling this form of asynchronous behavior.
(00:18:38):
If you think about the difference between the pre-emption of interrupts, and the pre-emption of tasks in RTOSs, you could have a blocking no interrupt system. As long as you polled often enough, it would be fine. It is kind of the same. When you have a non-pre-empting RTOS, you are polling tasks to see if they are ready yet.
(00:19:05):
Anyway, I am sure I was going somewhere with that. So how long do you think about- <laugh> What have you been working on lately for articles?
NJ (00:19:17):
The latest projects- The big one is I started a series on hardware security for DigiKey, that is based off of the MITRE Embedded Capture of the Flag competition.
(00:19:32):
Back in 2023, I tried to get a team at West Point to compete, and we did. We signed up. We started our little defense phase. But we never quite made it too far, because cadets' time is just so squeezed. I really wanted to go back and say, "What would we have done, had we been able to actually take this project to completion?"
(00:19:56):
So I am starting with that competition, which was the MITRE gave each team a car key fob system. With a really basic insecure implementation of being able to press a button on a dev board to simulate unlocking the car, and have that communicate over UART to a second board that functioned as the car. It had to do a couple of the things, but ultimately the teams had to protect these two systems.
(00:20:24):
The neat part about the eCTF was that when the teams moved into the attack phase, they could download other teams' code onto the boards that they had on their lab bench. So in addition to just all kinds of different logical attacks, they could attack the actual hardware to try to extract secrets, and cause it to do things it was not made to do.
(00:20:48):
So I am working on that. And then I got a presentation also on hardware security for the Embedded Online Conference. And I will be giving a two-day workshop on hardware security at CppCon.
EW (00:21:03):
Okay. Let us talk about the Embedded Online Conference that is coming up in May.
NJ (00:21:09):
Yeah.
EW (00:21:11):
When is it?
NJ (00:21:13):
Yes.
CW (00:21:17):
<laugh> Yes, May.
NJ (00:21:18):
I think like the second week or so, like the 10th or 11th or something.
EW (00:21:23):
That is coming up in May. It is the 11th through the 15th. People can sign up for that now. You have a code? Can we publicize your code?
NJ (00:21:33):
Absolutely. JONES100 will take $100 off registration.
EW (00:21:38):
Cool. Speakers get codes.
NJ (00:21:42):
Yeah!
EW (00:21:43):
You are going to give the workshop. It is called "Introduction to Hardware (In)security with the Chip Whisperer-Nano."
NJ (00:21:51):
Yeah. I was very proud of that title. Yeah. So this is, in some parts, an introduction to the ChipWhisper device that Colin O'Flynn makes over at NewAE Tech. But perhaps in a larger context, an introduction to the idea of power analysis attacks.
(00:22:12):
We are going to spend the majority of the two hours showing how, by monitoring the power trace of a device performing AES encryption, you can use that data to extract the AES key, the secret key that is being used on that device. Hopefully I will have enough time at the end also to demonstrate the voltage fault injection to like skip over a password check.
CW (00:22:36):
Cool.
EW (00:22:36):
So I should bring this device that I want to hack, and you will help me?
NJ (00:22:42):
That is highly encouraged, but I recognize that not everyone is going to have it with them. So I am also making sure that the workshop materials, I have simulated data to do all the exercises. So the only part you really miss out is using the device itself to like collect data in real time.
(00:23:02):
My hope is to also have like a third path, of where you get to do the same exercises on a much, much reduced subset of data. Like small enough that you can like print a little table on the worksheet, so you can like in Excel or by hand actually kind of follow along.
EW (00:23:21):
We have talked to Colin before, so I know a little bit about the ChipWhisperer. I even have one. I have never used it. As an engineer, I find it impossible for me to write code to make it safe against this attack.
CW (00:23:39):
But it has been a long time since this has been in the wild, so I assume something exists to harden things.
NJ (00:23:46):
Yeah. I think the weirdest part is just that it involves looking at your device from a perspective that is not well trained. The techniques to safeguard a system against something like this are just kind of non- Weird. They are non-standard. It is not a thing that I think comes naturally or is readily trained.
(00:24:12):
But with the EU Cyber Resiliency Act coming into force, I think in September, a lot of companies I think in the US are going to suddenly realize that they have to, or they cannot sell in the EU markets.
EW (00:24:28):
Because of attacks like this?
NJ (00:24:31):
Well, so the Cyber Resiliency Act establishes a baseline level of protectedness for devices being sold in the EU. It includes simple things like there has to be an SBOM, up to there have to be ways of pushing security updates to the device for like a minimum of five years. And it has to default to a safe mode.
(00:24:58):
It is not super technically oriented, so I do not know if this attack is listed in there as needing to be defended against. But for higher vulnerability devices like smart cards or secure keys or things like that, I think it is absolutely something that those companies need to be considering if they are not.
CW (00:25:26):
Is this something where, okay, I have this loop that does my AES calculation and it alters the power. Can you just insert, I do not know, random things that make the power do... Can I just fuzz the power usage to defeat this, basically?
NJ (00:25:43):
Yeah, in a sense. The basic idea is that we- So like every engineer that works on embedded knows I can monitor the power trace, to determine how much stuff my device is doing. And if I need to go to a lower power level, I can start turning off peripherals and turning off things.
(00:26:02):
So we already have this implicit understanding that how much current my device is drawing, is related to what is going on in the chip. When you do something called a "simple power analysis," you can actually start to align features in the current trace, to things that the microcontroller is doing.
(00:26:26):
One example is when you are doing like RSA encryption, based on each bit of the secret key, you do either a multiply, or a square and multiply. So even if the device itself is super secure, I can look at the power trace and I can- If I kind of know where in the power trace this encryption is occurring, all I need to do is look for a repeated section that is shorter than another repeated section.
(00:26:51):
That is essentially enough information to determine whether the thing is doing a square, or a square and multiply, and to recover the zeros and ones that make up the RSA key.
(00:27:02):
To recover an AES key, we have to go one step further, which is to make the sort of rough estimation that magnitude, if you will, of the current draw is actually related to the magnitude of the number of ones that are on in the MCU at that time.
(00:27:23):
That is not enough to necessarily resolve the value in a single register. But if I make an educated guess as to a bit in the microcontroller as it is doing an encryption, I can take my power traces and I can sort them into two groups.
(00:27:40):
And if my guess was correct, then the group where my guess assumed the bit was a one, will statistically have a higher power trace right at the point where that bit was on in the encryption process. If I average my two data sets and subtract them, that is the differential part of differential power analysis, then I can actually see that as a tiny peak in my graph.
(00:28:06):
Essentially what this does is it takes the process of brute forcing an AES key from two to the 256, into more like a two to the eighth. Because each bit in a byte, I just have to guess what value the byte had and sort these power traces and subtract them to see if I find this peak. Ultimately I can recover the AES key that way.
EW (00:28:29):
How much of this is turnkey? I just walk up to my ChipWhisper, wave it nearby, and it comes up with an answer for me.
NJ (00:28:40):
Actually no, none of it, from what I hear. <laugh>
EW (00:28:44):
<laugh> "This is not a movie."
CW (00:28:45):
Why not? Just write a Python script that does it.
NJ (00:28:48):
With the ChipWhisperer it is turnkey, because they have set it up that way. One of the biggest limitations is the fact that the current consumption on the device spikes right at the clockage, right? Because as soon as the clock transitions, that is when all these lines go high or low.
(00:29:07):
So on the ChipWhisperer, they can get away with not needing a ton of expensive hardware on there, because they clock the microcontroller on the device that you are sort of exploiting, they clock it from the same clock source that they run the ADC at. So they can ensure that those data samples align at the exact point in the clock cycle every time.
(00:29:27):
If you do not have that, or like on a device if it is running from an internal clock and I do not have access to that, then I essentially need a much, much more expensive oscilloscope, to sample at ten times or higher the clock rate with a much higher bandwidth to try to grab enough of the data to get what I need.
(00:29:51):
And there are a whole bunch of other parameters I need to control for. Well, if it is a real device, I need to be able to trigger encryption on my command, thousands and thousands of times a second, potentially hopefully to get all this data.
(00:30:06):
So one of the ways of thinking about your security in your device is, "Well, if that represents a typical operation, I need to hopefully shut that down in a way on a real device where I can only grab a couple of samples before it makes me wait." That would make it take much, much longer to collect enough traces to analyze.
(00:30:26):
I got a lot of my information from Colin's book that he wrote with Jasper van Woudenberg, who was also on the show, "The Hardware Hacking Handbook." They are pretty quick to say like, "Hey, sometimes we spend weeks setting up these experiments and taking data. Just cannot get the info we need."
CW (00:30:48):
I guess that is reassuring that in many cases it is very difficult still, and you have to be motivated. But there are a lot of motivated people out there who would like to, for better or for worse, crack into things.
NJ (00:31:00):
Yeah. Yeah, it comes down to just what do I feel like I need to protect in my system, and at what level do I want to protect it? At some point, if you hand your device to a nation state hacker with a government's resources, there is just nothing you can do to protect against that.
(00:31:20):
But for most people, that is not a scenario they need to really consider. So we just need to make it hardened enough that we can say we did our due diligence.
CW (00:31:32):
At some point, physical access is all access if somebody wants to decap the chip or do fancy things.
NJ (00:31:39):
Yep.
EW (00:31:39):
Nathan mentioned the Cyber Resilience Act for the EU. One of the things there is that companies need to conduct a cyber risk assessment before the product is put on market. That is in order to carry the CE marking, which is one of the basic ones. That alone-
(00:32:03):
All of the other stuff, you need to have security updates, you need to keep security separate from features. Those are all really interesting. I just am excited that people will have to think about security before they put it on the market.
(00:32:18):
Because that seems to be a loss for a lot of companies. It is just like, "It does not really matter. Who would want to break into our thing? We are only going to ship a thousand of them."
CW (00:32:30):
Or, "Our thing is not important." But maybe your thing being hacked gets them access to a cloud key. Which gets them access to your S3 bucket. Which gets them denial of service your entire thing. And then they ransomware you.
(00:32:42):
You have to think a few steps ahead from, "Oh, I am making this little wearable that just counts my dog steps and puts it in the cloud. How is that going to hurt anybody if it gets hacked?" Well, there are probably ways.
NJ (00:32:57):
Yeah. PII, right, person identifiable information can be enough to figure out who a exact person is, if I just have a couple of pieces of information. If the device is storing my user's login to the app and my company, if I can extract their login, then that is money- I can sell that to someone else, because there is a good chance they are using that same login on a different website.
(00:33:24):
So yeah, there are lots of different ways where extracting seemingly small piece of information can accrue. In the military intelligence, we used to call it- What was it? I am forgetting the term, but it is classification by... I am forgetting the term now.
(00:33:43):
But it is the idea that I can take a whole bunch of different pieces of information that are each individually unclassified, but if I have put them all together, it paints a picture that is suddenly at a secret or top secret level.
EW (00:33:54):
How are you staying up to date with, I guess, security, since you have been doing a lot of security? You are writing about it, but you are also learning about it. Where do you go to learn more?
NJ (00:34:09):
Oh man, you totally just like hit the kid in me that is like, "I do not know, man! I am trying to figure it out!"
EW (00:34:16):
<laugh> We all are. That is why we are asking you.
NJ (00:34:19):
<laugh> Security is an interesting field, especially hardware security, because there are lots of great people out there doing security research. Colin and Jasper. And Mark Omo was on the show a couple weeks ago. But I think in embedded, it is still on the newish side of things.
(00:34:39):
Going back to the idea of how do I write my articles to try to make sure that I am reaching a broad audience, these are articles or workshops that I think, "Okay, I need a longer tail." Because I think there are a lot of embedded engineers out there, that have not thought they needed to worry about it. Like, "Oh, it is not connecting to the internet. It is fine."
(00:35:02):
So at the moment, the biggest info has been that "Hardware Hacking Handbook." And then me working through, "Okay, how do I actually conduct these attacks?" Do not tell Chris, but I use ChatGPT and Claude a lot to bounce ideas off of and just say, "Hey, does this attack make sense? What are other attacks I need to consider?"
(00:35:34):
But hopefully soon I will get a chance to head also to the Hardwear.io Conference that is in Europe. That is a whole hardware security conference.
EW (00:35:45):
They usually have some in the US too.
NJ (00:35:46):
Oh, really?
EW (00:35:48):
Yeah. This is hardwear, H A R D W E A R.
NJ (00:35:52):
Yeah.
EW (00:35:53):
It is confusing to me, and it has nothing to do with wearables.
CW (00:35:57):
Or hard hats. <laugh>
NJ (00:35:59):
No. For the two of you, what has been your experience professionally with security or hardware security? Has it been usually an afterthought, or have you worked at places where it was really given kind of first rate?
EW (00:36:15):
Consumables. Anything that has a consumable suddenly has a lot more security around it, because the consumables can be spoofed.
CW (00:36:25):
Yeah.
EW (00:36:25):
I have had that, and you have had that.
CW (00:36:27):
Yeah. My deepest experience with it was being in charge of that, but it was a very long time ago. Principles are kind of the same. You have things that are protected by keys in the firmware. And things that are provisioned with keys that you are supposed to be not able to read out, and all this stuff. But it is a cat and mouse game.
(00:36:45):
Like I said earlier, physical access is physical access, and people figure things out. Yeah, it depends on how much of a target your thing is, and what the gain is for somebody to hack it. If it is, well, you are selling a $400 consumable, that we can manufacture in China for free and charge somebody for knockoffs, then that is a pretty big incentive. That is what happened at that place.
EW (00:37:14):
It is really hard because we make embedded systems. One of my definitions for what is an embedded system is something purpose built for its application. So you are making a microwave. It is to be a microwave. You are making a pedometer. It is to be a pedometer. You are making a sport watch. Now maybe you are making nine things at once.
(00:37:33):
But there is always too little space, not enough money. I like the challenge of being resource limited. It is a fun challenge. But then when you add security, and security for five or ten years in the future, I have to admit, I just throw up my hands because I cannot.
(00:37:56):
Given the security changes in the last five years, or in the last ten years, I cannot predict where that is going to go. Except that sure, you will be able to crack anything I do right now, if you want to.
CW (00:38:10):
Well, most of what I have been doing, and I think you have been doing prior to the current client you have, is Bluetooth- Or, connected stuff. So there are a lot of security issues with Bluetooth. There are a lot of security issues with securing firmware and firmware updates, which most of the vendors are doing for you now. You just have to plug and go.
EW (00:38:28):
Because if they are not doing it, nobody is doing a good job of it.
CW (00:38:31):
But you have to be careful, because you have signing keys and things that are part of your build, and you do not put them <laugh> in GitHub. So you have to be-
EW (00:38:41):
<laugh> Okay. Everybody who has put their signing keys in GitHub, just shake your head now and let it go.
CW (00:38:45):
Or on one guy's laptop that nobody has access to and, "Oops, I dropped it." There are those kinds of considerations, the kind of, "How do we conform to best practices?" Not, "Oh my God, somebody is doing a power-" That is just not on. I have never had that be on any client's radar yet, is "How do we secure our device from power monitoring hacks?"
NJ (00:39:11):
Yeah. Yeah. See, I feel like too, there are going to be lots of teams- I am speculating here. Lots of teams where even basic stuff like, "Oh, you did not turn off the debug port on the microcontroller," is coming.
CW (00:39:25):
Yep. Yep.
EW (00:39:25):
<sigh>
NJ (00:39:25):
Right? Because folks- Like Elecia you said, they are just not doing that threat modeling. They think, "Oh, it is an unlabeled pin header on my PCB that is enclosed in my plastic case. Why should I need to worry about that?" Except that anytime someone opens that up and go, "Oh, unlabeled pin header, I am going to see what this is." And "Oh, it turns out it is a SWD interface. Cool."
EW (00:39:46):
<laugh> Yes. "Unlabeled pin header. Which one of these is TX? I will just monitor them all."
CW (00:39:49):
But are you going to get rid of the JTAG connector? You need it for the factory, you need it for field debugging. So it is a very difficult calculation sometimes. But there are also other pins you can set that say, "Oh, you are not allowed to rewrite this firmware." But all of those things are an nrfjprog command line away, for the most part.
NJ (00:40:12):
Yeah. And there are some MCUs now that are a little more security focused, where the debug port is actually authenticated, so you have to have the proper key.
CW (00:40:21):
Sure. Yeah.
EW (00:40:23):
Which is great, until you are trying to develop for this thing, and now it takes ten extra minutes to get anything done.
NJ (00:40:30):
<laugh>
CW (00:40:33):
Or if you are trying to- I will not go into it, but I have been fighting with Bluetooth the last two weeks. One of the things I was trying to do was Wireshark with a sniffer, to see what the hell was going on with our throughput. I was not able to make it work, but it is-
NJ (00:40:49):
Bluetooth is-
EW (00:40:50):
I assure you it is possible.
CW (00:40:51):
I believe it is possible.
EW (00:40:53):
It just took me like five tries.
CW (00:40:54):
Well, I was on try eight or so, but I am sure- There are just things where, "Okay, Wireshark needs to- You need to turn off all security, but you cannot turn off all security." Because some things are still secured between an iPhone and a device.
(00:41:05):
So stuff still does a key exchange, and Wireshark has to watch the key exchange. But it has to be in the right order, and the other device cannot get in the way first. Yes, everything has to be perfect. And then maybe you can see what is going on.
(00:41:19):
I guess what I am getting at is during development, security is a pain in the butt. You turn it on too soon, and people get mad.
NJ (00:41:26):
Yep.
EW (00:41:29):
Fail to turn it on at all, and other people get <laugh> mad.
NJ (00:41:33):
This used to make me so mad in the military, because as the intelligence officer for a lot of units, I was in charge of physical security. I could not tell you times, the system has got this crazy badge access behind every door, and I am about to leave the building, and all the exterior doors are propped open with bricks. I am like, "You got to be kidding me."
CW (00:41:49):
Yeah. But there are a lot of bricks in firmware <laugh>. You can find them. It is tough to find the right time to turn all that stuff on. And then it is tough to make sure you did it right, and there is not a way to accidentally turn it back off and stuff like that.
(00:42:04):
Even like, "Okay, the release image does not have the UART." "Okay, we are moving to the release image." And now, "Well, we have this fault in the field." "Well, tell me what the UART says." "There is not a UART. This is a release image." And everybody is shaking their head. It is like, "Well, should we have done that that soon?" Anyway. Yes! Yes!
NJ (00:42:23):
Yeah. In my head in some ways it is similar to, I do not know, thinking about reliability as an engineer. You come out of school, and you know how to make stuff work. But to actually ship a product, you have to actually stamp out those one in a million little bugs and edge cases.
(00:42:39):
Thinking about security to me, is the same thing, but maybe just one step further. Where those one in a million edge cases, one in a million times are going to happen a million out of a million times, because it is in the hands of an attacker.
(00:42:50):
So I need to think a little bit more rigorously about what my device is doing. How certain assumptions that I am making as a developer, could be circumvented by someone with malicious intent. How do I contain that fallout just enough, that I can forestall what I do not want to happen just long enough.
EW (00:43:18):
Do you want to teach this, or do you want to do this? Engineer or instructor?
NJ (00:43:25):
I think I am always going to meet an instructor first. But the reason why this is fascinating to me is because- I went through the tutorials that Colin has for the ChipWhisperer. The first time you run your attack script, and a minute and a half later your Python script pops out the supposedly secret AES key that is on your device. It is like, "Whoa!"
(00:43:57):
I do not think I will ever full-time be hacking. But I think I am carving out a space for myself consultancy wide, as someone who can come and help figure out hardware security stuff.
CW (00:44:08):
And it is fun. From that aspect, when you are not under the gun trying to make- When you do not have the UI team saying, "We need to make this color different and more 3D." And at the same time, somebody is screaming at you that there is a security problem.
(00:44:24):
It is more fun to think about these problems and be outside of it and be helping people, rather than <laugh> the one under attack, I suppose.
NJ (00:44:36):
Totally, totally.
EW (00:44:38):
You said that this talk is for the Embedded Online Conference. You gave one last year as well, a presentation at the Embedded Online Conference.
NJ (00:44:47):
Yep.
EW (00:44:48):
Exception handling?
NJ (00:44:49):
Yep.
EW (00:44:49):
That is all you are going to tell me about it?
CW (00:44:52):
<laugh>
NJ (00:44:55):
I am a man of few words.
CW (00:44:56):
You did not ask a question. It was a yes or no question.
EW (00:44:59):
Could you please tell me about exception handling as a presentation?
NJ (00:45:04):
Yeah. So this started- I can.
EW (00:45:08):
<laugh>
NJ (00:45:08):
Yeah. So this started a couple years ago. I thought my skills in terms of building robust firmware, making sure my system can handle errors and exceptions, is lacking. So I am going to go out and do a bunch of research. What are the leading experts say on how to do all this? What seems to be best practices?
(00:45:30):
So I got a bunch of notes together, sifting through the differences between try/catch and return codes, and in arguments, out arguments, all this stuff. The way that you can let program know something bad has happened, and how you respond to it.
(00:45:44):
I wanted to be able to share those. So we did a two hour workshop where I presented a couple of different techniques and a little structure to think about, "Hey, how am I going to kind of harden this code against things that are not necessarily supposed to happen, so that the program can handle them gracefully?"
(00:46:04):
I feel like I just want to clarify that the exception handling was not meant in any way to mean like try/catch exceptions.
EW (00:46:14):
Oh. No. Hard fault.
CW (00:46:15):
Any fault, right?
NJ (00:46:16):
Ahh...
EW (00:46:18):
Processor exceptions.
CW (00:46:19):
Oh, the big ones. Okay.
NJ (00:46:21):
So there is not a ton of standardized language around this. But what I settled on was an error was a thing that absolutely positively should never happen. I get the square root of something and it gives me back an integer value which is negative. Or there is just no possible way that the system is working correctly under these conditions.
(00:46:41):
At that point, it could be a bit got flipped. It could be that I am under attack. It could be that my stack got corrupted. But I am essentially in a point where I cannot trust the execution of my code anymore.
CW (00:46:53):
Gotcha.
NJ (00:46:53):
For those, I need to assert, and I need to either restart the system, or fail safe to some default behavior. The exceptions are kind of one step above that. These are things that could possibly happen, but the program cannot continue along its merry way with this data.
(00:47:13):
Example might be, I am reading from a temperature sensor and I read 30 degrees, 31 degrees, 30 degrees, negative a million degrees. Okay, maybe that last one, I need to handle it differently than the other ones. I cannot necessarily tell my system that suddenly it dropped to subfreezing.
EW (00:47:31):
Do not just average it in with the others.
NJ (00:47:34):
Right.
CW (00:47:34):
Throw it in the bucket. It is all data.
NJ (00:47:36):
<laugh> Yeah. Yeah, so step number one is to just start thinking about your functions. You do not actually have to do anything crazy. You just have to look at them and think, "Okay, what are the possible things that could happen, that I do not want to just throw back into the system."
(00:47:52):
The easiest thing to do is just at least ignore them, right? You do not have to necessarily do anything crazy, big error reporting, logging process. I just need to not pass along garbage data. That is easy enough to put those little conditionals into your function, and change the function signature a little bit, so that eventually it might hand back an error code to the system, to start to close those off.
CW (00:48:21):
Okay.
EW (00:48:21):
The previous year, 2024, I gave a talk called "Creating Chaos and Hard Faults," which was how to create hard faults, and then how to get the information back to yourself after you have rebooted it.
NJ (00:48:35):
Yeah.
EW (00:48:35):
But you are saying that the hard faults and the exceptions are not the same?
NJ (00:48:43):
Not the way I am using the term. Unlike a Cortex system, "You have triggered a hard fault," that is it telling you that you tried to assign to read only memory, right? Or you-
EW (00:48:55):
Divide by zero.
NJ (00:48:56):
Right.
EW (00:48:57):
Yeah.
NJ (00:48:58):
Which I would put in that category like, "Okay, something is very wrong in the system." It could be a one in a million, or it could mean we are under attack, or my stack is corrupted. Or, yeah, I do not know. But I would put that in the category of like, "This is a restart to try to start from a clean slate. Or go to some sort of fail safe behavior type situation."
EW (00:49:18):
And that one, the "Exception Handling" is available on YouTube.
NJ (00:49:27):
Yes. Yeah, because I got the video because someone on the Slack channel requested it. It has been posted, so we can put the link in the show notes.
EW (00:49:35):
Speaking of the Slack channel, we have been doing a book club. You have heroically been carrying the rest of us, who are spending more time reading your summary of each chapter, than actually reading the book ourselves.
CW (00:49:51):
Yeah. You should do a worse job on the summaries, I think.
NJ (00:49:56):
I am an instructor. It is in my blood.
EW (00:49:59):
I have not been in the right headspace to read technical books very much lately. How do you...
CW (00:50:09):
<laugh> I am anxious to see where this is going.
NJ (00:50:12):
<laugh> You are right.
EW (00:50:12):
How do you force yourself to read technical books? Was that where you thought it was going to go?
CW (00:50:19):
Yeah, pretty much. Nathan, how do you make us better at our jobs?
NJ (00:50:22):
<laugh>
CW (00:50:22):
Nathan, please make me like computers again. Please. Sorry. Answer Elecia's question, not mine.
EW (00:50:32):
Is there a way to convince myself, or to convince my coworkers if I am not in whatever funk I am in, to pick up more books?
NJ (00:50:47):
If this is not too personal, tell me more. Because I think you made a great point before, Elecia, that if you are doing a lot of writing for work, then writing it in the evening is not relaxing, right? You have to balance each other out.
(00:50:59):
So is it that you feel like there is a lot of technical documentation you are already wading through, and so reading a technical book at night is not relaxing? Or is there something else?
CW (00:51:13):
Go ahead, but I could take a stab at this too.
EW (00:51:16):
I messed with my neurotransmitters from the bottle, and now I am just not quite functional. Work is good. Things are good. My brain is not.
CW (00:51:27):
It is raining.
EW (00:51:28):
It is temporary. But I went from, "Yes, let us read 'Pragmatic Programmer.' I am in. Let us read a few chapters," to I can barely read trashy novels. So that is me, and it is temporary, and I acknowledge that now.
(00:51:52):
Thank you for filling in by the way, Nathan. I know this was very last minute. That is because I bailed, flaked, was a complete jerk by flaking on the other guest we had for this weekend. Because there was no way I was going to manage to read his book in any amount of time. Even though I started talking to him weeks ago, I just could not do it.
(00:52:14):
I want to read "Pragmatic Programmer." It is not hard. I have read it before. It is very small chapters. I totally can read it. I am reading a lot for work, but I am okay with that. I just am like, as soon as my billing stops, I am just like I am checked out.
CW (00:52:33):
That was what I was going to say. Because we are consultants, reading technical stuff for fun, I just cannot do it because I am doing work for free. I cannot bill my client for reading. "Oh, I should get this Bluetooth book to understand it better." Maybe I should be billing them, but I find that difficult to do. I am not going to bill for lying in bed reading a book.
NJ (00:52:52):
Yeah.
EW (00:52:54):
It depends on how related it is to the work.
CW (00:52:57):
Right. Yeah. But it does make it a little more of a jump. I am not working. I am relaxing by reading this technical book. That is confusing.
EW (00:53:03):
It is not relaxing. It is investing in yourself. See, I can say all these words, but I am just not doing it.
CW (00:53:08):
I know you are just trying to be devil's advocate over there. Or angel's advocate? I do not know.
EW (00:53:15):
All right, Nathan, fix us.
NJ (00:53:16):
Right. Here we go.
CW (00:53:17):
"Stop working as consultants." Oh.
NJ (00:53:21):
Step one. Take all your computers, throw them out the window.
CW (00:53:24):
Hey, man. You do not know how close I am.
EW (00:53:29):
Not today. It is raining. Wait for a day when it is not raining.
CW (00:53:32):
No, that will be more permanent.
NJ (00:53:35):
I feel like those are both really great reasons to give yourselves the permission to not feel like this is a thing you need to do on your off time. To excel at a thing typically means you need to give yourself the space to rest and recover between bouts of really intense work.
(00:53:57):
I think in our country, our field maybe, our culture, it is easy to just think you just need to push and grind. But that just leads to eventually a whole lot of mediocre substandard work.
(00:54:14):
One of the things I am learning as I have transitioned into this weird part-time consultancy thing, is how do I work with the little ebbs and flows of my attention and my focus? Some days I feel like I could sit down and type out 4,000 words. Some days I write the same sentence a hundred times, and then call it a day.
(00:54:38):
I am trying to, in those moments, go, "Okay, well, if writing is not the task that is coming to me right now, then is there something else that I can work on? " Or maybe just that day I need to recognize that that is a sign that I had a tough weekend, lots of family stuff going on, I am worried about travel that we are doing in a week. So my brain is just not in the headspace to do deep focus work.
CW (00:54:57):
Fair.
EW (00:54:57):
I think that is a very important consultant note, is that there usually is three or four things that you are supposed to be doing. And if you can do the one that has the least friction, then you probably should, at least to the point where you are not overdoing it.
CW (00:55:16):
Hmm.
NJ (00:55:17):
Yeah. Yeah, yeah. Recognizing too that there might be tasks that line up with that, right? If after lunch, you are the kind of person where those last couple hours of work are just not-
(00:55:30):
You cannot do anything really creative, but you still got to get through your email inbox, and making plans for future whatevers. It is like you can say, "I know that this is going to be a good time for that, when I do not necessarily have to have my brain at full capacity."
CW (00:55:46):
This is when it is time to work on the CI flow.
EW (00:55:51):
When it is time to edit the document I wrote two days ago and have forgotten. But I am just at copy editing at this point.
NJ (00:55:58):
That is the best. Because then you are like in a different enough headspace from when you wrote it, that you can be like, "This does not make sense. Send it back."
CW (00:56:05):
Yeah. Yeah. After lunch is the best time to read your documents, because I feel at my stupidest. I cannot understand what I wrote. I have taken a minus 15 to intelligence.
EW (00:56:20):
Christopher has been playing more D&D.
CW (00:56:21):
Sorry, I do not know why all these plus threes and minus things are coming up.
NJ (00:56:25):
I love it. So what has been your- Elecia, I think you and Chris talked about it a little bit on the last episode. But from the first time you read it and the little bit that you have read now, what has been your impressions of the book?
EW (00:56:40):
It is a weird 20th anniversary edition. Anytime you are writing a book and you have a second edition, they want you to add more. They never want you to remove things. So if there is something in your book that everybody comes up to you and says, "I love that part," you are going to leave it in. Of course. You are not even going to change it.
(00:57:08):
But if everybody comes up and says, "I disagree about this, " you are going to start putting the arguments in the book. In the new edition, you are going to argue with the people who have been talking to you for the last 20 years. This edition to "Pragmatic Programmer," I can feel them arguing with people who are not me.
NJ (00:57:30):
Oh.
CW (00:57:33):
That is okay, because in the third edition, they are coming for you.
EW (00:57:42):
<laugh>
NJ (00:57:42):
<laugh>
EW (00:57:42):
Yeah. So it feels like they have added a lot of things that to me seem like cruft. I think I mentioned it last episode, the "do not repeat yourself" has always been a section that I have really liked, because it was so tactical. If you are repeating code, you need to think about why am I pushing control C, control V, over and over again? And then-
NJ (00:58:09):
Well, I think a great example too, even down to the line level or the printf formatter that I am giving to my- If these things are repeated, I should wrap those up. Because at a minimum, if I want to change it, I change it at one spot.
EW (00:58:22):
It has got a nice name, it is the DRY principle. Are you writing WET code? But then in this edition-
CW (00:58:29):
I do not like that, <laugh> just for the record.
NJ (00:58:31):
<laugh>
EW (00:58:34):
They have said, "It is not just about code. It is about everything. You should not be repeating data." Okay, that is true. Agree. But before, you had this nice tactical advice. Now, you have this architectural advice, that is still good advice, but-
CW (00:58:50):
It is harder to implement. It is harder to apply, figure out what matches it.
EW (00:58:54):
Yeah!
NJ (00:58:54):
Yeah. Yeah.
EW (00:58:56):
It is a harder piece to internalize. I feel like they have done that with some of the others, where they have expanded them, because that is what you do in a new edition. But they forgot the fact that they were writing for somebody who had not read their previous edition.
NJ (00:59:14):
Yeah, I would totally agree. I feel like that is one of my critiques of the book, is that there are lots of places where it could really benefit from even just a very specific example of, "Hey, this does not cover every case. But if we are talking about this term 'orthogonality,' or 'actors and processes,' here is more of a concrete example of what that looks like."
EW (00:59:42):
Actually, that is a good example in that I thought they should give more credit to design patterns, and then just shuffle them off to the side. Yes, they are super important. This is not the book for that. But they wanted to add so much more, so it felt like they added more producers and consumers and design patterns.
(01:00:11):
I wish they had not, because I already have a book about that. I wish they had just told me they existed, and then sent me off to go look at some other book. Which I would not have read either, but that is just right now.
CW (01:00:22):
For different reasons.
EW (01:00:24):
No, but I am enjoying watching people read it. I intend to declare bankruptcy and read this week's chapter. But I have said that for the last three weeks, so we will see.
CW (01:00:40):
You are coming at it with a different perspective, which you are admitting. If somebody is coming at this never having read it before, there are probably a lot of useful things to pull out.
EW (01:00:47):
Indeed. And there are some good useful reminders. There are not as many good useful reminders as I had hoped. But then maybe if I actually was reading it, I would get more. So we will see.
NJ (01:00:59):
Well, speaking as someone who is reading this for the first time, I will say with all the respect of the authors, there are points like that we just mentioned that maybe things could have been more tactical.
(01:01:12):
There were a couple other points in the book where I feel like things were a little unclear or even kind of misleading. There has been one or two points where I disagreed with what they had written.
(01:01:26):
Looking over the topic list, I feel like it is a great collection of topics. It is the kind of thing where if I was, I do not know, if I was hiring someone and they could talk as intelligently as the authors about these things, I would be like, "Wow, this is a knowledgeable person."
(01:01:39):
But in terms of being able to hand this to someone and say, "Hey, these things are important to us as a design team or development team, so you should be familiar with these things." I feel like it would be not the book that would get them to that point.
(01:01:56):
To me, I remember, and I keep comparing it to Philip Koopman's book, the "Better Embedded System Software." I felt like between the two, they tackle similar things, but I felt like Philip Koopman's book had a little more oomph to it, that could make it a little more implementable.
EW (01:02:15):
Kind of sad, but cool. Okay. One more thing about the new edition. There are many more languages and many more paradigms, ideas on how to implement code. They tried to address all of them, and they ended up addressing none of them.
(01:02:34):
Okay. Let us see. Your shift from full-time US Army instructor at West Point, to part-time embedded systems engineer and writer and conference speaker. How major was that shift? How often do you wake up at 6:00 AM wanting to do PT?
NJ (01:03:08):
<laugh> Oh, man. Thankfully, hardly ever. I think it came at a right time. I had a nice little wind down that started with the army thinking I was not major material. That made me think, "Oh, well, maybe this is not the 20 year career I had thought maybe it was going to be." I got to finish out at a great place, at West Point, having taught there for five years.
(01:03:40):
At that point in my career, there just was not a whole lot left I really wanted to do in the military. I was honestly just much more excited, having spent those couple of years at West Point, to do more presenting. I did a little bit while I was there, that is where I started to get into it. It felt right.
(01:04:03):
I really lucked out. There was a friend on the Slack channel that I spoke with while I was doing some job hunting, that helped me land where I am now in getting to more or less work full-time. It is not really full-time work, but just basically 100% of what I am doing just about is for DigiKey.
(01:04:26):
I think without that, it would have been a lot scarier to have thought like, "I got to find a job. And I am at a weird spot where I am technically an entry level engineer, but I am almost 40. So what do we do with that? Which job do I apply for?"
(01:04:43):
But I feel very blessed. Blessed that I had a great career in the military. Blessed that I had the opportunity to build up a small portfolio while I was there, of blogs and presentations and conferences that are going to help me land a- And a great job. And blessed that my wife still works full-time. So it is not a huge financial concern when the income fluctuates from month to month.
(01:05:11):
What about the two of you? In that transition to consultancy, what were those first six months to a year like? Was it really scary that thinking like, "Oh, I am not going to be able to make ends meet, if I do not follow up this client with another one"?
EW (01:05:28):
Well, the first time I did it, Chris was working full-time.
CW (01:05:30):
Yeah.
EW (01:05:30):
And the first time Chris did it, I was working full-time.
CW (01:05:33):
Yeah.
EW (01:05:33):
So that does leave- It gives you a steadiness that you do not get if you are a solo contractor working on your own.
CW (01:05:41):
When I went into... Well. See, it was a long time ago, so it is a little confusing. Transitioned in and out of full-time and consulting a lot, so the history gets a little messed up. But there was a time I was in grad school, so she was working full-time, and I was in grad school making nothing. So that arrangement of having one of us have a fluctuating income was not unusual.
(01:06:04):
I would say that the anxiety about clients is a recent phenomenon, in the last five to say eight years, because we had more steady long-term clients back then. I think in the last few years, there has been a little bit more, at least on my side, a little more client churn. So there have been longer periods where I have been seeking things out.
(01:06:30):
Been very lucky. Most of the time I have gotten things very quickly. But there have been some times where it has been a few months where that has been kind of a question. Yeah.
NJ (01:06:40):
Is part of that though, because you are points in your career where you can be more selective with who your clients are, do you think?
CW (01:06:49):
I would say the longest unintentional break was not that. It was, I was willing to accept more than I am now. Now I am very tight on which clients I accept, for various reasons. But yeah, it is not getting easier for me to find things I like doing.
EW (01:07:10):
You are getting way pickier, which is fair. And you have not necessarily been wanting to do the work, until you found something interesting.
CW (01:07:24):
That is true.
EW (01:07:25):
It is hard to advertise that you are available, when you also want to say, "But not for most people."
CW (01:07:32):
Yeah. Yeah, but that is true full-time as well, at this point. There are very few full-time companies I would go to. Anyway. Yeah. That is personal stuff.
(01:07:41):
You have been laughing at me, but there are definitely been times where you have been looking for clients or worried about getting one. And then you end up getting three after a month of angst or something, and trying to figure out how to juggle it.
EW (01:08:01):
Oh, yeah. No. No, it can be difficult to- It is usually the, "I am done with this client." I want to say I am going to take a few weeks off, but then about three days into those few weeks, I was like-
CW (01:08:13):
"I do not have a job."
EW (01:08:14):
I do not have a job. And that is, yeah. It is a little scary.
CW (01:08:22):
Yeah. It takes me two months to get to that point.
EW (01:08:24):
Yeah.
NJ (01:08:26):
My wife always complains because for her employer, they also go out looking for clients. And the best time to look for a client is right when you are neck deep in a project being like, "I cannot take on more work!" But there is this lead lag thing going on.
EW (01:08:42):
And when it rains, it pours. I end up getting too many clients and tiring myself out and then saying, "I need a break." And then two days into the break, I am like, "Oh no, where are the clients?" So yeah.
NJ (01:08:53):
How do you say, "No," Elecia? Teach me this.
EW (01:08:57):
"I do not have time right now to help you. Let me find you someone else."
CW (01:09:03):
"Who are you funded by?" That is my usual first go now.
NJ (01:09:06):
<laugh> That is a good one.
EW (01:09:09):
The assumption that I have a limited number of hours I can provide. If they want three hours a week, then yes. If they want 25, then no.
CW (01:09:23):
Yeah. I am booked.
EW (01:09:24):
I am booked. I am booked.
CW (01:09:24):
"I am booked," is the usual no. Occasionally there is like, "I do not know how to do that. So if you want me to do that, it is going to cost you a lot more and it might take you longer. You probably should find someone else."
EW (01:09:38):
Yeah. I am willing to say, "I am not an expert in that. I would love to help you, but it is not worth training me." Are you having to turn people away? Because it is scary.
CW (01:09:50):
It is better to turn people away than to be desperate.
NJ (01:09:54):
Well, I have just always had that problem, where I want to do more than I have time in the day. Like the stuff I am doing for DigiKey, and prepping for the Embedded Online Conference and CppCon has already taken more time than I wanted it to. But it is because I- To me, if I am not done with May's work in January, then I feel like I am behind. I am working on that.
(01:10:27):
I got an invitation recently to work on something. I was very proud that I did not immediately reply, "Yes, yes, yes. That sounds awesome." I might still, but it is harder to like think, "Oh man, do I have the hours in the day to do this? Is this a thing that I want to do? What does the future look like and what is the ideal, and does this fit into it?"
EW (01:10:56):
"Yes, I am excited about this opportunity. What is your timeline? Because I am partially booked at this point." is a decent set.
(01:11:06):
Because I have had clients that they gave me a long list of things and I am like, "Okay, that is four months of work, and for the next two that would be way overtime for me." But then when I go back and I am like, "Okay, what is your timeline?" They are like, "Oh, we want this done over a year." And I am like, "Okay, I could fit you in. That would be great."
(01:11:29):
Yeah. It is a balancing act and it is always hard to turn down, because it is like you do want them to come back to you. You are only busy right now. Come back to me next time. Which is why I end up sometimes trying to match make and say, "Oh, I am not available, but I know somebody who is."
NJ (01:11:50):
I like that.
EW (01:11:51):
On the assumption that next time they will toss me the work if I need it.
(01:11:54):
All right. I am going to wind this up. Nathan, do you have any thoughts you would like to leave us with?
NJ (01:12:04):
I think the one is to just say there is a quote- Hopefully I get it right. I think it is by Howard Thurman, that says, "Do not ask what the world needs. Ask what makes you come alive. Because what the world needs is people who are alive.”
(01:12:21):
Talking about where I am now professionally, like I said, I feel very, very blessed that the opportunities came. But I think I can also see how I studied really hard in school. At West Point, even though no one was asking me to do conferences, I asked to speak at conferences. I found a way to publish material, and it paid off.
(01:12:46):
I like to think that I kind of embody that quote a little bit. That I pursued the things that have brought me joy, and that made me feel like invigorated. Even when it did not necessarily have an immediate monetary payback. But the world wants people who are passionate and wants people who are alive, and I think it came around.
EW (01:13:16):
Our guest has been Nathan Jones, educator, embedded systems engineer, 15 year US Army veteran, and frequent contributor to EmbeddedRelated and DigiKey. He has also on the Embedded Patreon Slack, and is a contributor there as well.
CW (01:13:31):
Thanks, Nathan. It was good to talk to you today.
NJ (01:13:34):
Thanks so much, you two. It is always a pleasure.
EW (01:13:36):
Thank you to our Patreon supporters, and thank you for listening. If you would like to contact the show, it is show@embedded.fm or hit the contact link on embedded.fm. You can also click that support link if you want.
(01:13:48):
Now a quote to leave you with. I am going to go with one from "Ancestral Night" by Elizabeth Bear, because it has a really interesting mechanic around brain chemistry, and is somewhat related to this show. "There is value in work you enjoy, or that serves a need. There is no value in work for its own sake."