290: Rule of Thumbs

Transcript from 290: Rule of Thumbs with Phillip Johnston, Elecia White, and Christopher White.

EW (00:00:06):

Welcome to Embedded. I'm Elecia White. My co-host is Christopher White, and our guest this week is Phillip Johnston of Embedded Artistry.

CW (00:00:18):

Hi, Phillip. Welcome.

PJ (00:00:18):

Hi, guys. How are you?

EW (00:00:19):

Good, good. Could you tell us about yourself?

PJ (00:00:23):

As Elecia mentioned, my name's Phillip, and I'm the founder of Embedded Artistry, an embedded systems consulting firm based in San Francisco, California. I run the company with my wife, Rozi, and we also run a website by the same name, which is dedicated to embedded systems content.

PJ (00:00:39):

On that site you can find hundreds of articles and other resources targeted for embedded systems engineers of all varieties and skill levels. When I'm not working, I enjoy volunteering as a gardener at the San Francisco Japanese Tea Garden, playing music, cooking good food for my friends and family, and reading Latin to my four-month-old son.

CW (00:01:00):

Buried the lede there. The last part there.

EW (00:01:02):

The Latin or the gardening?

CW (00:01:05):

Yeah, reading Latin...What sort of things do you read in Latin?

PJ (00:01:10):

Poetry or a lot of Cicero, philosophy kind of material.

EW (00:01:16):

Yeah, because there's a lot of Latin poetry that shouldn't probably be read to kids.

CW (00:01:22):

He's four months old.

PJ (00:01:23):

Certainly true. Yeah. He doesn't understand.

CW (00:01:26):

It's Latin.

PJ (00:01:26):

He likes the rhythm.

EW (00:01:30):

Yes. Alright. We're mostly going to be talking about that website -

CW (00:01:37):

And Latin.

EW (00:01:37):

- with lots of [laughter] embedded material, and what's on it, and why and how. All the usual questions. But before we get into that. Lightning round, where we ask you short questions, and we want short answers, and we might say "How?" and "Why?" But we're not supposed to. But you don't have any rules. Are you ready to Phillip?

PJ (00:01:59):

I'm ready.

CW (00:02:01):

Favorite processor.

PJ (00:02:04):

Family or architecture?

CW (00:02:07):

Dealer's choice.

PJ (00:02:08):

Dealer's choice. I would have to go with the nRF52.

EW (00:02:13):

Favorite artist.

PJ (00:02:15):

Can I pass?

CW (00:02:16):

Favorite plant.

PJ (00:02:18):

Favorite plant would be the black pine.

EW (00:02:21):

Black pine?

PJ (00:02:23):

It's a Japanese pine tree.

EW (00:02:25):

Alright. Least favorite compiler.

PJ (00:02:28):

IAR.

EW (00:02:31):

Such an easy target.

PJ (00:02:32):

Yeah.

CW (00:02:33):

This week we are sponsored by IAR. No. What's the most important point when looking at contracts for consulting?

PJ (00:02:42):

I really look for equitability in the relationship and in the contract. So I tend to not trust things that are one-sided on either my end or their end, because I think that sets up how the relationship is going to go.

EW (00:02:57):

What is the most useful part of newlib?

PJ (00:03:00):

I actually don't use newlib very much.

CW (00:03:03):

Alright. What's the least useful part of newlib? [Laughter].

EW (00:03:03):

But I saw it on your site.

PJ (00:03:06):

I've done some explorations with newlib. But I actually run, I use my own libc. So I tend to avoid most of the newlib.

CW (00:03:13):

Your own libc. Well, we'll have to talk about that.

PJ (00:03:17):

For my least favorite part of newlib, it would be, memory allocation scheme, I think is just too heavy-handed for most embedded systems.

EW (00:03:27):

Yes, yes, so much yes. [Laughter].

CW (00:03:30):

Favorite Roman philosopher.

PJ (00:03:33):

I'll take the easy target and go with Seneca.

CW (00:03:37):

Alright.

EW (00:03:38):

Tip everyone should know.

PJ (00:03:40):

Get enough sleep, and it will really improve every aspect of your productivity and quality of your work output.

EW (00:03:47):

I totally agree with that. Do you want to do one more Christopher, or?

CW (00:03:51):

Oh, I thought you were looking at me to disagree. No, I like sleep. I'm just bad at it. No, no, I think we should move on to the actual podcast.

EW (00:04:02):

Okay. Let's do the contracting part. You have a company, I mean, kind of like I do, but you actually do things a little more formally than I do. So if I was coming to you,...I had a project, I did a napkin sketch of an idea. Let's say a light that is also a temperature that hooks into my Alexa. How would the process work with you?

PJ (00:04:35):

Usually, there's an initial conversation where we're discussing the goals of the project, not necessarily the requirements, but what's the value you're looking to provide? How are you differentiating yourself from other products? What's your timeline? What are your constraints?

PJ (00:04:57):

Things of that nature, trying to get a good overall view of what the project would entail. And that informs a lot of the follow-up work. There is always the open-ended, "We want to build this thing, and we have no idea how to do it, and we have no idea what chips we need or parts we need."

PJ (00:05:15):

And that ends up leading more towards, I guess, what you would call formally a discovery-type arrangement. Where we're working perhaps on an hourly basis, or even a fixed fee, if the work is bounded, to try to scope out how the system will behave, and what the different requirements are, and what parts we might want to use.

PJ (00:05:37):

If all of those questions are answered, which is pretty rare when a startup comes to us for help, then we can outline a development plan for how we think we should best tackle the project. And a lot of that work is actually done by my business partner, Rozi, who spent 12 years doing project management at Apple.

PJ (00:05:55):

So she's very good about getting engineering to come up with a plan, and sort of wrangling the various ideas, and figuring out how you're going to answer the open questions that are most pressing, and how that informs your schedule, and sort of ordering those events so it's most sane.

PJ (00:06:14):

And usually, depending on the contract, that's something that we can do over two or three phone calls with the client, or we need to have an actual month-long engagement to sort of architect a solution, and see how we're going to proceed.

EW (00:06:30):

Do you do hardware as well as the software?

PJ (00:06:33):

I personally don't do the hardware, but I've got a few business relationships with various design houses that can do hardware. But I find that most of the companies that come to us, and I don't know why this is the case, but they almost always have an in-house electrical engineer, and have just really struggled to hire firmware engineers. And so firmware tends to be the piece that's stalled out.

EW (00:06:58):

Firmware also does tend to be the piece that ends up doing the project management. Why is that?

PJ (00:07:05):

I think it's because it's very nebulous to most of the rest of the organization. And if you don't have somebody who's experienced with firmware requirements in-house, you tend to have a lot of assumptions that are made by the rest of the project team on what is or isn't possible in the firmware side of it.

PJ (00:07:30):

And because firmware starts last a lot of times, I think it's the big driver for the completion of the project. And I don't know if that really answers the question. Just rambling there.

EW (00:07:42):

Oh, it's fine. Okay. So do you work with little companies, or big companies, or medium companies? Do you have a preference?

PJ (00:07:54):

My preference is the medium company who perhaps has shipped one or two projects, and has discovered that the way they've been working isn't sustainable, and needs to revisit their approach. I would say that's about 25% of our clients. Most of our clients are very, very, very early-stage startups in the two to six employee range, who just have an idea, or just created their initial prototype.

CW (00:08:21):

When you say not sustainable, what kinds of things, what kinds of realizations are they having? Or what kinds of things are you finding for them, and saying, "Well, you can't do it this way."

PJ (00:08:31):

The usual realizations, I guess, are, they released their product, and then their customers are using the product, and there's all these bugs while they're also trying to work on a second version of the product, or get a second product in their suite to market.

PJ (00:08:47):

And they don't know how to effectively manage those different requirements, and the different priority levels. And a lot of times, we find that it could be traced back to software design flaws, or process steps that are just being skipped, that could be added, and even automated in a lot of ways to sort of squash some of the easy bugs before they get into the field.

CW (00:09:15):

Do you have a lot of clients, you mentioned that a lot of them seem to have double E's on staff, but not firmware engineers.

PJ (00:09:22):

[Affirmative].

CW (00:09:22):

Do you have a lot of clients coming to you with a prototype, or something that they think is product quality -

EW (00:09:32):

[Laughter]. But it's based on an Arduino?

CW (00:09:32):

- or it's something modular, or based on Arduino, or, is that a big part of your work? Like, "Oh yeah. Well, we need to take the step from prototype to real hardware, and I need help." Are you're happy to do that? Or is it mostly, "Okay. They have the hardware, and it's just a matter of getting the firmware to work."

PJ (00:09:51):

It's a good mix of the two. I would say it's probably 50-50 split. You have the companies who have a prototype, they've shown the prototype to investors. It's hacked together as an Arduino, with a off-the-shelf camera module, and something else. And then they want to transform that into the final product-type thing.

PJ (00:10:13):

And they may not know how to do that. And that's something that we can help guide them for. The other 50% is like you said, they've got a double E, they've got their form factor decided on, they've got their first version of the circuit board, and their electrical engineer or their software team doesn't know how to write the firmware for it.

PJ (00:10:31):

And so then we're just handed a hardware design, and usually we'll perform the initial bring-up, and get them started on features, or help them explore what those features should be.

EW (00:10:41):

This sounds not very agile. Do you have feelings about the agile development process?

PJ (00:10:49):

Do you mean that my process doesn't sound very agile, or the customer's process doesn't sound very agile?

EW (00:10:55):

Well, you mentioned architecting, and processes, and design. That doesn't sound very agile-y. This is not an insult.

CW (00:11:09):

This is not an accusation.

EW (00:11:09):

This is not bad.

PJ (00:11:11):

Yeah...So I started out as a defense contractor. So I come from a formal background that was very, very, very stifling, but also did open my eyes to some more of the upfront design aspects. And then I went to Apple,...and I've worked at various startups as the first firmware hire, where there was no upfront design and no process.

PJ (00:11:38):

So my goal is a blending of the two. I think that there is some amount of upfront design that can be done to answer big questions on paper before you even start coding...Drawing things with a pen, or a whiteboard marker, and connecting little boxes, can unveil a surprising number of problems before you get into weeks of coding and find out you've made a solution that's untenable.

PJ (00:12:09):

On the flip side, you could spend all your time in design and justify that as not starting. And I definitely see teams do that, and that's not good. Where I get the most phone calls from existing companies is usually along these lines. It's "Hi, Phillip. We had an initial design, whether it was a prototype, or version one of our product, and we decided we need to go to a new processor" for some reason. Power, better radio, different radio, whatever it might be.

PJ (00:12:43):

And then they have the uncomfortable realization that they have to rewrite all of their software because they tied everything to the SDK and the RTOS that vendor A was using. And their new chip set is a totally different SDK with a totally different RTOS. And none of their code can be moved over easily.

PJ (00:12:59):

And usually that's coupled by, "We have a build in a month, and we need all of our software up and running by then, and we have no tests, and we have no way to determine whether it's successful or not." And I get that phone call at least twice a month.

New Speaker (00:13:12):

So when I'm doing up-front design, I'm really trying to make it so we can understand what parts of the system are going to change, and how can we design the software in a way that isolates most of the system from that change.

PJ (00:13:26):

So if you need to make a big critical change, such as even with your processor, that should be two to four weeks, rather than, now you need a six month rewrite of everything. So that's sort of where I try to blend agility into the design. I think some design can inform future agility, but you have to invest in that.

CW (00:13:47):

I think that's an angle and a question that I see so rarely asked at the start of projects. Like "What things could we potentially change in the future or swap out, and how does that affect how we architect it, what modules we have, what the API layers? Where do we want to abstract things?"

CW (00:14:07):

And a lot of times people do abstraction just for abstraction's sake, which can lead to some really bad software.

PJ (00:14:17):

[Affirmative].

CW (00:14:17):

But doing it that way and saying, "Wait, we might change the OS. Okay. Then everything that talks to the OS, we might need a translation layer," or "We might change out the hardware. Then we should have a good driver model that allows us to swap that out."

CW (00:14:32):

A lot of times people just don't don't bother to ask that question. Then you end up with what you're talking about, a very brittle thing that you have to start over on.

EW (00:14:41):

And you mentioned testing, and I think you and I have similar feelings on testing. If it is something that is fixed, something that is important beyond the scope of the processor, or the RTOS, if it's the secret sauce, the goo -

CW (00:15:02):

The goo! [Laughter].

EW (00:15:02):

- it needs to be tested. It needs to have a way to test so that when you change the other things, it still works. Do you do a lot of that testing? I implement sandboxes for people so that their code will work on a PC or a Linux box, so they can test their algorithms. Do you do much of that?

PJ (00:15:23):

I do exactly that. I actually try as much as possible to be able to test my embedded code on the PC. And I really try to avoid on-target testing as much as I can. Although it happens, and it's valuable, and you check different things.

PJ (00:15:40):

And the hard part is always, where's the dividing line, because I don't necessarily think that "test everything" is the right answer, but certainly test your algorithms, test the things that make your software special, and where you're providing value, especially. And -

EW (00:15:59):

The goo.

PJ (00:15:59):

The goo, as you say, the magic goo.

EW (00:16:03):

Okay. So your website,...my business website is like, "Yeah, we do stuff. Contact us if you want to." And occasionally if I don't have work, I will make it nicer. But since I'm fully booked for quite a while, I don't really care. It's a placeholder, it's a spam suck, but your website goes way beyond that.

EW (00:16:31):

You have a blog that you work on often. You have different parts of the site that go to resources for beginners to embedded systems. It's a lot. Where should I get started if I'm just coming to your website?

PJ (00:16:49):

If you're just coming to the website, we have a welcome page, which is up at the top on our menu bar. And that provides some orientation to the site, with some of the articles that I'm particularly proud of, and some that are popular, as well as explaining some of the different areas of the site.

PJ (00:17:07):

Because as you say, there's the blog, and there's different resources, and we have a glossary of strange embedded terms. So it's definitely a lot. And that page is meant to help orient people to the website.

EW (00:17:19):

You have a list of development kits you like.

PJ (00:17:22):

That too. And I need to keep adding to that. I've used a lot of embedded development kits now that I haven't written about.

EW (00:17:30):

Wow, this glossary. Okay, Christopher, I'm going to quiz you now.

CW (00:17:37):

Keep in mind that I do not care.

EW (00:17:42):

What is a critical section?

CW (00:17:44):

It is a section that is very important.

PJ (00:17:49):

I'll allow it.

CW (00:17:49):

It's a section you don't want to be interrupted while it's running, because there's a resource or something that might be shared, and you don't want something else to run that could interrupt it, or change a resource out from under you.

EW (00:18:03):

What is COGS. C-O-G-S. All capital.

CW (00:18:07):

Cost of goods sold.

EW (00:18:09):

Yeah. See, I mean, there's a lot of stuff like that here where, yeah, you might know it, but every once in a while, you'll come up with something. People come up with terms, and this is nice. This is nice.

EW (00:18:19):

Okay. One of the things that you mentioned on your welcome page is you have a list of your favorite articles, and then you have a list of your most popular articles. Right now, those do not overlap at all. Does that irritate you?

PJ (00:18:37):

It doesn't Irritate me, but it makes me laugh. I've definitely, I've realized that. And it's a little disappointing, because the articles that I'm proud of are the ones that I spent the most time on, or they're problems that I really grappled with for a long time. And some of those in the most popular articles section I put together in 15 minutes, based on a problem that was really annoying me with a client project or a personal project.

PJ (00:19:02):

And I just published it because I needed something to publish that week. And then for two years, it's been in the top 10 articles every month. So that definitely is a surprise. And I just learned that I can't predict what people are going to find useful, and it changes over time as well, what's popular and what's not.

PJ (00:19:20):

It's interesting to watch that sort of evolve month to month, and see something as popular for a year, and then nobody cares about it for six months. And then six months after that, it's suddenly the number one article again.

CW (00:19:32):

I think that's true of all of us who put things out creatively. It's like, "Why do you like that one?"

EW (00:19:38):

Yeah.

CW (00:19:38):

"Okay. I guess I don't understand. But at least you like something."

PJ (00:19:44):

Three of the ten most popular articles are about Jenkins, on an embedded blog. And I do a lot of build server work, but I just wouldn't have guessed that 30% of my top articles are going to be on Jenkins.

EW (00:19:56):

Oh, and one of them is installing LLVM/Clang on OSX. Yeah. That's so embedded. [Laughter]. Or creating a circular buffer in C/C++...That looks like it's your most popular article.

PJ (00:20:12):

It is absolutely the most popular article. It is the most commented on. I get a lot of emails about improving it, and how to do things differently, and why did I design it that way? I get a lot of engagement out of that article.

EW (00:20:26):

And yet it's something, I mean, it's important, and yet it's not something,...it's something you think about once. And then once you have a really good version, you just stop thinking about it. Circular buffers are circular buffers. You don't need engagement on that.

PJ (00:20:45):

I agree with you, but I just can't question it. It's just the way that one goes.

EW (00:20:51):

Okay. So let's go back to your favorite articles instead of the popular ones. You have number one as Embedded Rules of Thumb. Is that like, "Do stuff?"

CW (00:21:05):

Never start a land war in Asia.

EW (00:21:07):

Yeah. What is the most important embedded rule of thumb?

PJ (00:21:11):

I would say it changes depending on what problem I'm facing, but my current favorite is a rule of thumb from Jack Ganssle. Which is, it's easier and cheaper for you to completely throw away and rewrite your 5% of problematic functions that you're always trying to fix the bugs in, than it is to try to go back and incrementally fix all the bugs as they come up.

PJ (00:21:34):

And I think that's an underestimated and underused tactic for cleaning up code that's bad, is just throw it away and start over if it's bad. You'll do it better the second time. Because you have the problem fixed in your mind and you have the flaws fixed in your mind as well.

PJ (00:21:50):

And you're just going to come up with a better design than the initial one, which, you weren't necessarily aware of all of the things you were trying to solve and account for.

CW (00:22:00):

I think that's great advice and also really tough to manage in large companies. "What are you going to do this week? Well, I'm rewriting this section. We need to." "No, you need to fix these bugs," or, yeah, it can be an uphill climb to refactor. But yeah, the sunk cost fallacy thing, right? It's like, "Oh, we've spent all this time on this code. We have to keep it. But it's also taking all our time.

PJ (00:22:22):

And I've been through it. I've spent two months trying to clean up a bad module of code where the entire team acknowledged it was bad. And we knew it was the source of all of our problems, but we just wouldn't commit to redoing it. Which would have taken a week instead of the two months it took us to squash all the gopher bugs that started popping up.

EW (00:22:43):

In this rules of thumb post, you have lots of references to other people. How do you find this information? How do you find people who give you good rules of thumbs. Thumbs, thumbs?

CW (00:22:59):

Rule of thumbs.

EW (00:23:00):

Yeah. Thumbs.

PJ (00:23:02):

Is that where the strongest thumb wins?

EW (00:23:03):

Yes. Never start a thumb war in Asia.

CW (00:23:07):

That's the iron law of thumbs.

PJ (00:23:09):

Oh, okay. I get them all mixed up. Well, I can answer your question in a broader scope, which is when I was trying to become an embedded systems engineer,...I learned on the job essentially, and...I didn't know where to look. So I was reading different papers, different articles, looking for other embedded engineers who were putting out quality content, which is hard to find.

PJ (00:23:38):

And I've read thousands of articles over the past 10 years, and some pieces of information keep coming up, over and over and over again. And it's a simple rule of thumb that you'll hear repeated everywhere, might be something like "Comments should not talk about what the code's doing. It should talk about why you're doing it that way."

PJ (00:23:57):

So some of those ideas start to stand out over time. And as I've gotten more experience, other ideas that I'm encountering just ring true, such as the "Throw out your bad functions and start over." I've spent the two months dealing with that. And so, as I encountered these situations, I just started noting things down.

PJ (00:24:19):

And mostly they were personal notes, but for the rules of thumb article in particular, when I had like, I don't know, 50, 100 rules of thumb in here, I was like, "Okay, maybe other people would find them either useful or amusing." Applying them is hard because a lot of them are derived from experience.

PJ (00:24:37):

And you'll probably only agree with them if you've had the experience that gave rise to that particular rule of thumb. So whether they're actually useful to people is, I think, somewhat questionable, but they're certainly very amusing to those of us who have been in the trenches.

EW (00:24:56):

Oh, you don't have one of my favorites on here. Every sensor is a temperature sensor, some sensors measure other things as well.

PJ (00:25:03):

That's a great rule of thumb. I'm going to add that.

EW (00:25:07):

Okay. So you have your rules of thumb, which I agree, a lot of these things you don't get until you have experienced it. And so it's only in retrospect that you can use that information. You can't pre-apply it. On the other hand, one of your other favorite articles is Improving Software With Five Processes You Can Adopt This Month.

EW (00:25:32):

And you go through Jenkins, and build, and let's see, let me actually do all five. Fix all your warnings. Yes. Set up static analysis support, like Clang or PC-lint. Measure and tackle complexity. That's hard. You're going to make me do that in one month?

PJ (00:25:55):

It's hard. You could start doing that in a week. You could find the problems within a day. You can get complexity analysis set up in a day. Are you going to fix it all in a day? No. But you also won't know there's a problem unless you're looking.

EW (00:26:11):

Create autoformatting rules. Why is formatting such a big deal?

PJ (00:26:16):

For me, I propose autoformatting, because I don't think it's a big deal, and I wish teams would not talk about it as much. And so if you have an autoformatting system, it's a way to avoid the formatting arguments. Because I've been a part of too many code reviews that don't focus on the actual implementation at hand, but only focus on the tabs versus white space argument, or the way the braces are indented.

PJ (00:26:42):

And so for me, this is a strategy to eliminate the bike-shedding that happens with formatting, and to help teams focus on the stuff that actually matters.

EW (00:26:52):

And you also suggest doing code reviews, which is another thing that I think probably takes longer than a month to even start. How do you do code reviews given that you kind of work by yourself?

PJ (00:27:07):

Code reviews are the thing that I miss the most about working in a company. I would say that that...really has been a major boost to just exposure to new ideas and learning different approaches to solving problems. So now that I'm a contractor, I try as much as possible to work with my clients on reviewing the code that I write, which is a surprisingly difficult task. Most of them -

EW (00:27:35):

It's like they don't care.

PJ (00:27:37):

It's like they don't care. And I've even had plenty of deliverables where I know they don't care, because then three months later they told me they didn't use it, which is always a sad thing.

EW (00:27:48):

Oh, that - yeah.

PJ (00:27:48):

But people pay a lot of money for code they won't use. And I will never understand that.

CW (00:27:53):

Yeah, I've been paid a lot for code that people resented. Like, "Oh, I could have done that." Like, "Okay, then you should have done it. I'm just a consultant. I did what you told me."

PJ (00:28:03):

Right.

EW (00:28:06):

I have thought sometimes that the consultants that I know should band together to do code reviews exchange, but then you have NDA problems and all of that.

CW (00:28:19):

Yeah, you can't do that.

EW (00:28:19):

You can't do that, but...I miss it. And it is so often that I will do a code review for a client and at the end they'll be just like, "Why did you do this? We could have done a PowerPoint in five minutes." And I'm like, "But you needed to see the code."

PJ (00:28:37):

[Affirmative].

CW (00:28:37):

I have to say in my current situation -

EW (00:28:41):

You have like 25 people -

CW (00:28:42):

- you might not miss it. If you were going through what I go through every week.

EW (00:28:49):

There is such a thing as too much code review.

PJ (00:28:53):

...Yeah, there's a sweet spot. And finding that sweet spot, I think is the challenge that takes more than a month.

CW (00:28:59):

Yeah, definitely.

PJ (00:28:59):

Especially.

CW (00:29:00):

And it takes discussion and understanding of what a code review means. What's expected of the reviewer. What's valuable and what's not valuable. What kinds of comments are going to hold up your code. Or what kinds of comments or things that should be tackled later.

CW (00:29:16):

And being really clear about all of that stuff, because when that's not clear, it's just a free for all. And people are just throwing food at your code, and you don't even know necessarily what things to address, and what things not to, and where the important parts are. So I don't have any experience recently with that.

EW (00:29:33):

You have some best practices for code review and some discussion of the social aspects of code review, which is often an adventure. My code is not me. Your code is not you. I'm not criticizing you when I criticize your code. I'm not saying you're stupid. I'm not saying any of those things. I'm just saying this code could have been better.

EW (00:29:58):

Why are you pointing at me?

CW (00:29:59):

I'm just having flashbacks. Yeah...There's ways of being neutral and still giving comments.

EW (00:30:11):

I actually started going to a writing group about a year ago. Not necessarily because I really need help with my writing, but because I wanted to learn how to give criticism better. And that was what the group focused on. And so I could watch people give better updates, better feedback.

EW (00:30:30):

And that has helped me with code reviews for other people. Or design reviews usually. And it's not just that I am naturally sort of blunt, or naturally sound angry to people, even when I'm not, it's just...we aren't cogs. We aren't replaceable with each other. We do have feelings, and code reviews are hard on feelings.

PJ (00:31:02):

What are some of the things that you learned from your writing groups about giving feedback?

EW (00:31:09):

You can't give too much. I mean, you can't...No, I phrased that weirdly.

CW (00:31:14):

Yeah you can.

PJ (00:31:16):

There is too much feedback.

EW (00:31:17):

If you give too much, none of it will be taken. So you don't necessarily want to give only three pieces of feedback, which is advice I've heard from other people. Because in code, you can't do that. I mean more than three pieces per line is too much, but sometimes you have a big file.

EW (00:31:40):

You have a bunch of comments, but you do have to figure out if you can, instead of pointing out each error, point out the thought process error. Instead of circling where all of the commas are wrong, you tell the person, this is how you use commas, and this is how I would like you to use commas.

EW (00:32:04):

And so you give them the tools to improve for the future. That may mean this piece, this code, this writing, doesn't get the benefit, but you're helping them become better as a whole. Instead of making them feel criticized for circling all these commas, and they don't even know why. They don't have a tool to use in the future.

EW (00:32:25):

And so that, that's the thing that I have been a lot more focused on with code reviews is, "Okay, there are things wrong here, but what can I say that will allow them in the future to not make these mistakes, as opposed to just fix these right now."

PJ (00:32:42):

And I think another way to compliment that approach, and this I think applies more toward code reviews than writing, is I don't think people ask enough why the person implemented it in that way. So if I think that there's a problem with the implementation, it would help me significantly to understand why it was implemented a certain way, because I may be criticizing something that I also don't understand, which isn't productive at all.

PJ (00:33:11):

And if I'm a senior engineer talking to a junior engineer, there's a inherent power and respect differential, that's going to play into that. And I think that we're not always as aware of that, and asking for a thought process can help slow us down and prevent some of these easy to have happen mistakes from actually happening.

EW (00:33:32):

And if somebody figures out their own issue before you have to point it out, then the criticism doesn't come from you. And so they don't have to get defensive at you.

CW (00:33:44):

Yeah. I would just say that's important, but also has to be phrased correctly.

EW (00:33:48):

Oh yeah.

CW (00:33:49):

"Could you explain your thought processes? Great. I'm confused why you did X. It's not great."

PJ (00:33:54):

Yeah, that's the famous consultant's line we receive a lot.

EW (00:34:01):

And actually that does apply to the writing group. We do that a lot. "I don't understand why your character does this now...Could you walk me through your thought process of making this stream of consciousness instead of some reasonable writing style?"

EW (00:34:17):

Of course, you only say the first half of that sentence...And do you prioritize your code comments? That's always been the big one for me, that I used to do for the writing group, and they don't care anymore.

PJ (00:34:32):

What do you mean by prioritize my code comments?

EW (00:34:35):

Well, I separate them into "nits," which are things that I need to tell you about, but you don't need to fix, maybe, which is things that I think may be wrong, but I'm not sure. So if you agree, please fix it. "Ought to," which is this is wrong. And if you're not going to fix it, tell me.

EW (00:34:58):

"Bug," which is, "This is very wrong. Stop right now. Don't proceed here. Either we need to talk about this, or we need to file a ticket or something." And then I also have "kudos," which is, I really liked this. And I really try to use "kudos" because we forget to say, I mean, it's a review. You do want to encourage the good behaviors too.

PJ (00:35:23):

I absolutely do that. I absolutely do that. And I learned that from my first manager at Apple, who I went on to work with at a startup following that. And he was the most excellent reviewer, particularly because he would give you the kudos, and particularly because he would ask you why you did something before he explained another approach.

PJ (00:35:46):

And usually...I think, his style was, "I think there would be a different way to consider this problem that you might try such as X" and then that's much less personal.

CW (00:35:57):

[Affirmative].

PJ (00:35:58):

And every team I've worked on sort of has different categorizations of comments, but I think everything you listed is certainly, I tend to bucket my review comments in that way. And I usually summarize at the end of a code review, just to be clear, the ones that I expect to actually be fixed before something should land.

EW (00:36:19):

Yeah. That helps, because it is so easy to just have little nits all over the place that nobody really cares about, but need to be documented, because it isn't following the style guide. I don't want to talk about the style guide, but we have one so use it. That sort of thing.

PJ (00:36:40):

Usually it's we have a style guide that nobody looks at.

EW (00:36:43):

Yes.

PJ (00:36:43):

But you should use it anyway.

EW (00:36:45):

Yes.

CW (00:36:46):

Two style guides.

EW (00:36:49):

Okay. So what are some of your other favorite posts?

PJ (00:36:53):

The other posts I enjoy a lot are the series on implementing dispatch queue with RTOSs, because I think it's a really interesting concept that can clean up a lot of threading problems that I see across embedded systems. I also am really proud of the Boeing article, which started as a one paragraph newsletter update and ended up as a 25-page-long harangue about that whole saga.

EW (00:37:25):

Okay. This is the Boeing 737 -

CW (00:37:27):

MAX.

EW (00:37:27):

- MAX thing, where there was a sensor, and you had to have software to turn off the alarm. You had to pay extra to turn off the alarm, and it crashed, and it was bad and.

CW (00:37:39):

- most of that's wrong, but okay.

PJ (00:37:41):

Yeah.

EW (00:37:41):

Okay. So tell me what I should have said about it. What is it about?

PJ (00:37:48):

There were two crashes involving the 737 MAX...As you said, seemed to be related to a software system that behaved in a way that the humans didn't understand and prevented them from controlling the plane in a way that they didn't understand, is the base problem. As the situation has unfolded, there's a lot of other contributing factors that have come to light, such as, Boeing and Airbus are competitors.

PJ (00:38:20):

And Airbus announced a new airframe where they were going for improved fuel efficiency, and Boeing needed to respond to that so they didn't lose business to Airbus. This competition leads to specific timelines, and cost targets, and certification requirements, to minimize the cost and training time for airlines.

PJ (00:38:40):

And a lot of corners appear to have been cut through that process, so Boeing could maintain their type certification, and not have to have pilots go through expensive training to fly this airplane. The goal was, this is your grandfather's 737. You'll be able to fly it the same way that you've flown every other 737.

PJ (00:39:02):

The way they achieved fuel efficiency increases was with bigger engines, which they had to mount differently on the airframe, which caused a different aerodynamic behaviors, especially with regards to stall angles. And so the software system was implemented to prevent pilots from entering into an angle which could cause the airplane to stall.

PJ (00:39:25):

That seems to be a big point of contention on whether that should have been allowed, whether there should have been a type rating change. And how Boeing implemented that specific software also appears to be problematic. There are two angle of attack sensors on the plane, one on each side, and the flight computer, which has this new software that prevents the pilot from entering too high of an angle, only reads from the currently active side of the plane.

PJ (00:39:58):

So if the copilot's flying the plane, the copilot's sensor's red. If the pilot's flying the plane, the pilot's sensor's red. And as you mentioned, there was an optional upgrade that would check both and let you know if the sensors disagreed. For both of the crashes, it appears that there was a faulty angle of attack sensor reading, off by dozens of degrees, which caused the plane to -

EW (00:40:21):

Wha?

CW (00:40:21):

Yeah.

PJ (00:40:21):

Dozens of degrees. And they're known to be off, or to be easily damaged, which is another ding against the implementation, I think, is if you have a sensor that is notorious for being off, and you need to disagree, you should probably check both.

PJ (00:40:39):

Or if you're a pilot, if my AOA sensor...is telling me it's 20 degrees off, and I look out the window, and I see that that's not true, I'm not going to change how the planes flying. But a computer getting one data point doesn't necessarily know what's happening, and is just going to make the decisions it's programmed to make.

EW (00:40:59):

Okay. I don't understand. Because when I worked on aircraft stuff, it was just for little planes, and we often had to have three sensors agree or they would vote. For things like horizon sensing, which is very similar to angle of attack. This -

CW (00:41:15):

You're not Boeing.

EW (00:41:18):

...They should never have been allowed to certify anything with a single sensor.

CW (00:41:25):

Well, yeah, the certification process is under the microscope too.

EW (00:41:29):

Oh, yeah.

PJ (00:41:30):

Yeah. And they appear to have been able to take this approach because of a specific, I don't know the correct aviation terms, so I'm probably going to get this wrong, but say the failure rating. So it wasn't rated as a catastrophic failure if the sensor is wrong, because the pilot could still theoretically control the plane.

EW (00:41:54):

Theoretically.

PJ (00:41:56):

Theoretically.

CW (00:41:56):

And they were allowed to self-certify some of this, if I recall correctly.

PJ (00:42:01):

Yeah. For the past 20, 30 years, the FAA has been increasingly outsourcing much of the certification work to the different manufacturers. And so Boeing is responsible for a lot of the certification work, as you mentioned. And there's also some scrutiny happening, because I remember reading a report that the FAA wasn't aware of some of the changes that Boeing made to how the MCAS, this is the software under scrutiny. The MCAS software, it was changed after the certification process happened.

PJ (00:42:38):

And so when Boeing released data after the second crash, the FAA was like, "Hey, we didn't actually know that the plane could be controlled to this degree by the system." So that's also another eye-opening problem that's I'm sure being looked at by multiple parties.

CW (00:42:59):

One of the things I liked about your article, in contrast to another article that was going around the internet around the same time from IEEE Spectrum, was that you seem to correctly characterize it as a multi-system, multi-organizational failure, and not just a, "The software engineers took a shortcut failure."

PJ (00:43:21):

I think...it's easy to blame the software. Part of that I think is, we have this idea that software is released quickly, and we don't always take everything into consideration, because we can just patch it after the fact. And it's an easy target as well, because it is so changeable.

PJ (00:43:42):

If you focus on the software, Boeing can say "We corrected the software. We fixed the problem." And a lot of the other systemic problems still exist. And the factors that led us to this still exist, right? So it almost seems like the software is being used as a distraction to the rest of the problems that have happened under the hood.

CW (00:44:03):

Yeah. They can fix this, and then they can go on to the next bad decision that does something similar.

PJ (00:44:09):

Right.

CW (00:44:09):

Yeah.

EW (00:44:10):

...As a software person, even if we don't blame the software, because I agree that this is a systemic business, and environment -

CW (00:44:23):

Yeah, that's correct

EW (00:44:23):

- and quality control, and management problem. But as a software engineer, I do feel like at some point you have to stand up and say, "Wait a minute. If this, this, and this happens, we could crash planes, and I'm not having that happen on my watch."

CW (00:44:43):

Is it realistic that you would even know in a system this complex? Would you be responsible for such a small area that you might not know how it's all interacting?

EW (00:44:52):

That's true, because it had to do with both the autopilot and the sensor system. And if I was the sensor system engineer, I would figure that the autopilot people had their crap together. And if I was an autopilot engineer, I might not know how broken the sensor system is.

PJ (00:45:06):

I see that a lot, even outside of Boeing, in that organizations I've been a part of, it doesn't seem like in most organizations, there's a steward of the system as a whole. And the difficult thing about complex systems is if you take all of their pieces, and you just look at your piece, you can't really extract the behavior of the system out from all of the pieces.

PJ (00:45:33):

You have to be thinking about the entire thing as one thing that behaves on its own and probably is going to behave in ways that you're not expecting. And in the case of the MCAS software, I do think that, yeah, if you're on that team, then it probably would have been the right idea to say, "Okay, well I need more than one reading to really verify this."

PJ (00:45:56):

On the other hand, maybe as the software developer, you don't actually know the inherent flaws in that particular AOA sensor or the problems that are seen in the field. I've certainly seen a lot of disconnect between what the engineering team believes about a product and what the support team believes about a product, how customers use it, and what the bugs are.

EW (00:46:17):

That is so true. And it isn't even just the support team. Sometimes it's the customer. You're like," No, we never meant for it to do that. I'm glad it was working for you, and now it's broken, but that was never the goal." What is the most important thing to learn from the saga?

EW (00:46:35):

I mean, is there one thing you can take away?...There should be a system architect or a systems engineer who has a view into the whole system. Are there other things you think we should take away from that?

PJ (00:46:51):

I don't know that I have answers that are satisfactory, but I think there are lessons that we can learn that should drive ways that we handle this situation in the future. We're starting to build systems that are so complex that whole organizations cannot accurately grapple with the consequences of the behavior of the system or their decisions with relation to the system.

PJ (00:47:19):

And that's not going to change. Everything we're building is getting more and more complicated. Somehow we're going to have to get a handle on that. And I don't necessarily know the right way to do that. And I think the other key takeaway is, it's not like Boeing's unique in the decisions that they made that led them to the outcome.

PJ (00:47:39):

They're unique in that the decisions that were made led to two crashes, which resulted in the deaths of 346 people. And I work with dozens of organizations that make very similar compromises. They make very similar mistakes. And sure you might ship your widget, and there's no real critical customer impact, but it's not like Boeing's doing something that every other company isn't doing in some way.

PJ (00:48:07):

And so it's easy to point fingers, and say, "Oh, look at Boeing and their bad behavior." But I think we should also take a look within our own organizations, and see how we're cutting corners, and what are the consequences of the things that we're focusing on. Because you're probably not thinking about that. And somebody probably should think about that.

CW (00:48:26):

Yeah. Risk and hazard analysis are two things that most people don't do. And just in terms of security with IoT devices and things, that should be something that every company does, because every time you build something like that, there's a potential for something catastrophic. You're not going to crash a plane, but you might expose hundreds of thousands of people's private data or whatever.

PJ (00:48:50):

Or even some of the alleged Nest hacking attempts.

CW (00:48:55):

[Affirmative].

PJ (00:48:55):

Where you have a baby in a room, and the heater's just turned on full blast. Whether or not that one is actually substantiated, it's definitely the kind of possibility, when you skip out on security for connected devices.

EW (00:49:09):

Both the Boeing and the security points. You can find these sorts of risk analysis questions to put on your risk list by saying, "Okay, let's pretend it's two years later. And whatever the worst possible thing could happen has happened. How did it happen?"

EW (00:49:35):

And for Boeing, that actually should be part of their process of, "Let's pretend that a plane crashed because of the software. How did this happen?" And work backwards, and then try to minimize the risk at each stage. That is not that hard of a process to do when you're working with software.

EW (00:49:57):

I mean, it doesn't have to be that critical. It...can just be, the worst thing that happened was we lost all our client data, or the worst thing that happened was someone got shocked. And so you work backwards. How would you get shocked from the light switch? "Okay. Well maybe we shouldn't make it out of metal and electrify it."

CW (00:50:22):

You're saying you need black hats for trying to crash planes?

EW (00:50:26):

You need black hats for -

CW (00:50:29):

Maybe. [Laughter].

EW (00:50:29):

- the risk assessment.

CW (00:50:31):

Yeah, yeah.

EW (00:50:31):

So many of us go forward.

CW (00:50:34):

Yeah.

EW (00:50:35):

I didn't think about reverse engineering for a long time. It was something somebody else did. But then I got into it with BB-8, and I realized it is the same thing I'm doing usually. It's just debugging something I didn't write, which I do a lot of anyway, but it's the same process of, it's backwards. It feels backwards.

EW (00:50:56):

And yet it is just as important as what feels naturally, and...with practice it doesn't feel backwards anymore. Did that make any sense?

CW (00:51:05):

Yeah.

EW (00:51:05):

Yeah? Phillip didn't say yeah, but I think we're going to go with it.

PJ (00:51:09):

My question for you is, whether you think we're running up against, just call it a fundamental limit of human psychology. And how do you train teams to think in a way that's sort of antithetical to the way that we all think by default. For the most part, we're all thinking of the happy case, right? We're all going to make it rich. And we're only going to pick the things that we need to pick to get to the part where we're going to be rich.

PJ (00:51:34):

And all the things that are going to cause us to not be rich, we'll deal with them when they come up. I think that's the human tendency. So I wonder if there's a way to work around that and instill that kind of culture, where you are looking at the risks, even though it's not natural and it's not fun. And it feels like a step back from getting you toward your end goal.

EW (00:51:54):

Yeah. Because sometimes that step back, you realize, "Oh my God, this path doesn't work...I can't do security on this chip". And it just becomes untenable and terrifying...I mean, and so you have to give people time. You can't just ask for the answer. You can't just ask for the happy path. You can't just ask for people to fill out all of the features. You also have to ask them to think about the system.

EW (00:52:26):

And that is not something we get a lot of time to do as developers. And so it's sort of time we have to take and say, "Look, this is part of my job. This is part of what I do to be a developer, to be an engineer, to be a programmer, whatever word you want to use."

EW (00:52:47):

Even as a hacker, I would like people to consider, "Okay, I have hacked the Nests to do bad things to people's houses. What's the worst thing that can happen to this. For me, it was just a fun thing. But if I put it online, what's the worst thing that happens?" And you do take some responsibility for the consequences of your actions.

EW (00:53:12):

Christopher's making faces.

CW (00:53:14):

No, no, because I'm thinking about QA, right? And engineers don't test their code well, and -

EW (00:53:19):

No, because we think of how it should work.

CW (00:53:21):

Well, we think of how it should work, but also we know how it works. And so there's an inherent blind spot there.

EW (00:53:27):

Yes.

CW (00:53:27):

And so whether or not you're consciously avoiding doing things that you know might break it, because you know how it works, or whether that's unconscious, or whether you're just not creative enough at creating something and breaking it -

EW (00:53:39):

Yeah.

CW (00:53:39):

- I think...it's not going to be super effective to have the engineers doing that. Because this is why we have QA teams and dedicated testers, because they're better at coming at something with a fresh mind and finding ways to break it. So I think the same thing applies to risk analysis is, you kind of need somebody who does that at least for a significant portion of their time. And isn't just an engineer trying to turn their brain upside down for a day a week.

CW (00:54:10):

I mean, this is the security model, right? There's people who are penetration testers, and people who consult on, "Here's what's wrong...Here's what I'm finding are holes with your system." And they're not the people necessarily writing the code.

CW (00:54:27):

And maybe that's what's needed in some organizations where the risk is high, is to have dedicated adversarial people. Not testing necessarily, but doing what we were talking about. Imagining the worst case scenarios and trying to construct a way to get there.

EW (00:54:45):

Okay. I have a new idea for business. We're going to call it "Agents of Chaos," and it's going to be all risk assessment.

CW (00:54:51):

Well, wasn't that Netflix's is Chaos Monkey thing, where they just deliberately injected automated breakages?

EW (00:54:57):

Chaos? Yeah.

CW (00:54:57):

Yeah.

EW (00:54:58):

And then they made sure that their system was robust, but they did that on production.

CW (00:55:01):

Yeah.

EW (00:55:01):

Chaos Monkey still runs.

CW (00:55:03):

Yeah. I can see that sometimes.

EW (00:55:08):

Oh, Phillip, I'm sorry. We do sometimes end up talking to ourselves.

PJ (00:55:13):

It's okay. It's fun to listen to.

EW (00:55:15):

Let me get back to one of the articles. Actually we have a question from a listener, CrustyAuklet. CrustyAuklet, really? I've met him.

CW (00:55:27):

Is he crusty or an auklet?

EW (00:55:29):

No. No. Okay. There are a few articles on Embedded Artistry, your site, about re-targeting modern C++ using FreeRTOS and ThreadX. CrustyAuklet is curious what you think the pain points are and/or the limits of gcc and newlib. For example, it seems like thread, standard thread, is not great since you can't tweak the stack size of threads. What? You can't? That's really important.

PJ (00:56:00):

You cannot.

EW (00:56:02):

Okay. That was a pretty long question. I will just leave you to answer it. Go ahead.

PJ (00:56:08):

So I'm going to start my answer by prefacing that all of my re-targeting C++ experiences using libc++, which is Clang's implementation. So I don't actually have any comments on the gcc C++ library. But overall it's very clear to me that the C++ standards committee implemented the threading library support simply based on how Pthread works.

PJ (00:56:35):

And there's almost a direct one-to-one mapping for the API set. Including, you don't need to set thread stack sizes in Pthread, because there's a pretty good default thread stack size. And you really need to know what you're doing if you're modifying that. So as the reader mentioned, that tends to be a problem. The other types actually work pretty well. Mutex is relatively straightforward to implement. Using it makes sense. There's a lock, there's an unlock.

PJ (00:57:04):

There's all sorts of useful helper types that you can lock a lock when you enter a function. And no matter where you returned from the function, it'll automatically unlock. Using that is wonderful. Implementing it, the only difficulty is, because of this Pthread C++ standard library dependency, the standard mutex constructor has to be a constant expression, which I've never actually seen apply for any RTOS or other OS that I've used.

PJ (00:57:36):

And so that tends to be the one tricky thing you have to check before you lock your lock, whether you've initialized this mutex and if not, call some initialization code. That's some unneeded overhead.

EW (00:57:46):

Yeah.

PJ (00:57:49):

But using it once that's done, and doing that is straightforward. Condition variable, also easy to use once it's implemented, but implementing a condition variable can be a tricky problem space if your OS doesn't have a primitive already for that. Because the obvious approaches to handling it lead to logical flaws.

PJ (00:58:09):

So there's a great Microsoft research white paper, which I'll send you the link to, and you can add to the show notes, which goes through an algorithm for actually implementing that. Where you need a semaphore and a queue, where you keep track of threads waiting on a condition variable, and the ability to directly suspend and resume threads. And so if you have those things, you can implement the condition variable in a straightforward way.

EW (00:58:33):

Wait a minute, let's go back. A condition variable tells you which threads are running? The state of the thread? No.

PJ (00:58:40):

Condition variable is a concept where I want to sleep my thread until a specific condition is true. And that might be waiting for a variable value, or some func door to return true, or any number of things. But essentially when whatever my predicate is returns true, I'm going to wake the thread and resume running.

EW (00:59:02):

Okay. So condition variable, for me, when I think about not using this sort of thing, it's the global variable I set in the interrupt that tells the loop to go deal with the data I just collected.

CW (00:59:14):

Well, that's more of a single flag. So a condition variable could be a set of flags or a value that you're waiting for, right?

PJ (00:59:23):

Yeah. I think it's more in line with an event group, where my thread is actually sleeping until some condition which I specified in the flag get call is true.

EW (00:59:36):

Okay. So it could be a series of flags, or it could be a series of states happening, things being ready. Okay. Let's see. So that's condition variables.

PJ (00:59:49):

So then for thread, like the reader mentioned, implementing the threading support is relatively straightforward, but the APIs are very limited. So if you need to create a thread, and you have a good system default stack size, and good system default priority, and you need some generic concept where just an average old thread will do, then it works well.

PJ (01:00:09):

But on embedded systems, you often want to control your thread priority, because I want my thread that's handling my interrupt callbacks to run before anything else. And you might have different priorities for different hardware component.

EW (01:00:24):

Heck, yeah.

PJ (01:00:24):

Managing threads and stack sizes are definitely variable. My LED blinking thread doesn't need a 128 kilobyte thread stack.

EW (01:00:35):

Yes. [Laughter].

PJ (01:00:35):

So we often want to tune those values. And my particular solution has been...I have, I guess, a separate set of C++ threading concept APIs which lets you tweak all of those things. And I'm currently working on validating those APIs by using them for Pthread, and ThreadX, and FreeRTOS.

PJ (01:00:57):

And I'll probably pick two other RTOSs to make sure the abstractions work, and I'll release those as open source interfaces if you wanted a good C++ RTOS abstraction layer. And then you can always go the route that I went, which is, I have STL support for their types based on these APIs I wrote.

PJ (01:01:17):

And so you sort of get the best of both worlds. If you have a C++ app developer who wants to use standard mutex in the app layer of your firmware, and doesn't actually care about setting all the other little OS bits that might come with that. Like priority inheritance settings. That really works well and is a huge time saver, and sort of opens up the possibilities of who can work on your system.

EW (01:01:41):

Do you have examples of where this might be used? As you write tests, are you writing little demo programs too?

PJ (01:01:50):

I have a general, I guess, framework demo thing that I'm working on which is sort of nebulous at the moment. I don't actually know how to talk about that to answer your question in good details.

EW (01:02:04):

Oh, that's fine...Some of the concepts you've talked about, I know what they are, and yet I'm still a little on the fence of, "Okay. So if I have FreeRTOS running, and I have C++, how do I glue these LEGO blocks together with your code?"

PJ (01:02:21):

The Clang libc++ has a nice external threading abstraction layer. So there's a set of functions that, if you wanted to use the standard C++ types, you just need to supply an implementation for those particular functions. And so if you wanted to use my code, I have a external threading header that maps directly to that.

PJ (01:02:44):

And I will be, in the future, releasing some demo apps that do show how to use that, and how to hook the different pieces together, but that is under development.

EW (01:02:56):

Okay. So I have more questions about various blog posts, and going more in depth, but we don't have that much time.

PJ (01:03:05):

Okay.

EW (01:03:06):

So we want to go back to the question of, you have this website full of pretty useful information. Why? I mean, you're a consultant,...you get paid by the hour, I assume, unless you do a lot of fixed bids, but most of us don't. So yeah, why? I mean, you could be making money.

PJ (01:03:31):

I actually started the website before I started my business. And I alluded to this earlier in the episode. When I was learning about embedded systems,...there wasn't a class that I could take. There wasn't a good book to turn to, that I could find. There weren't really good websites on it.

PJ (01:03:52):

And so most of the aspects of how to write firmware, and how to debug embedded systems, and do all of the various things I needed to do, I had to do a lot of research for. I had to bug senior engineers and try to figure out how they did it. I had to read through thousands of pages of datasheets to figure out how something worked. Read through the ARM processing architecture requirements to figure out how arguments were passed.

PJ (01:04:20):

All sorts of things like that. And you slowly piece together various aspects, and I'm a pretty prolific note taker. So I ended up with thousands of pages of notes. And one day it just hit me that I'm probably not the only one in the world struggling with all of this stuff.

PJ (01:04:34):

The internet today is a bit better in that you can Google things, and get a lot more answers, and there tends to be more embedded resources. But even those I think are geared heavily toward the hobbyist and maker space. And so I just started going through all of my old notes, and cleaning them up, and adding some examples, and having my wife edit them.

PJ (01:04:58):

She worked in publishing for four years. So that's a handy person to have on your team. And just slowly releasing those. And I mean over time, I think the first year, 200 people read my website for the entire year, and now we're hitting 50,000 people a month. So it really seems like there's a need.

PJ (01:05:16):

And I really derive a lot of enjoyment from taking what I learn, and the things that I'm struggling with, and that I'm researching, and just writing it down, and publishing them, and then seeing that other people find that useful and helpful.

EW (01:05:32):

I understand that feeling, and I still get a little confused about it. I mean it is time. It's time we could be doing other things. Why are we spending our time helping other people for free? Do you...worry about that or do you just enjoy the sharing enough?

PJ (01:05:57):

I worry about it, because I would like to find a way to make money doing that, because I enjoy that. I enjoy publishing the information and helping other developers learn a lot more than I enjoy doing consulting contracts, which quite honestly end up being the same thing with a different set of requirements maybe. But most of that work is the same, and that can get old after a while.

PJ (01:06:27):

And the exploration of the unknown is certainly more exciting, and helping people is certainly more exciting. So I have struggled on how to find a way to have more of my income come in from doing the website without relying on ads. Because I mean, you look at the big embedded in electrical engineering news sites. They're mostly product placement ads.

EW (01:06:50):

[Laughter]. Yes. Even the articles.

PJ (01:06:50):

And it's really disappointing to me. Even the articles. It's just super disappointing to see. And I don't want to go down that route. But I understand why they do because you get offers, you can make money. But it's just that part, it would be sad to go that route. So it bugs me a lot to answer your question, and something I think about a lot, how I could make more income from that, and be able to focus my daily work activities on it.

EW (01:07:22):

You already spend quite a bit of each day working on the posts. About how long does it take per week?

PJ (01:07:29):

I spend, I would say an hour to an hour and a half every morning researching or writing. Depending on the length of the post. Most of them take five to ten days plus four hours of editing. So let's call that 20 hours. 20 hours, I sink on a post of any significant length. So that, it certainly adds up, like you were saying. Time is money.

EW (01:07:58):

It is tough. It's tough to have that mental shift of giving things away, but it is sort of advertising, isn't it? Do you consider it advertising for your consulting services?

PJ (01:08:10):

I do. I've certainly received work from my website, and I would actually say the most rewarding projects that I've worked on came from my website. Because they were from clients who shared the same values that I shared and who cared about what I had to say, rather than just thinking that I was some generic firmware developer, contractor, who would sit in a desk for them, and type code at a specific number of hours a week.

PJ (01:08:35):

So it certainly has had, it's paid its dividends beyond just getting enjoyment from it. And it does land me work, and introductions, and I meet a lot of great engineers who reach out to me about the site. And there's all sorts of nice network effects that come from that.

EW (01:08:55):

Yeah. I understand that. Have you considered writing a book?

PJ (01:09:00):

I would love to write a book, but I'm not sure yet what I need to put in a book that I can't put into an essay. And for me, that's the bar. And I've had some publishers reach out to me about publishing a book on embedded systems or embedded C++.

PJ (01:09:14):

But they always seem to have strange, almost startup-esque, deadlines where it's like, "We need you to write this book in three months, and we're going to pay you $5,000 to do that." Which you know is a full-time job. And who knows if that could actually be done in three months, and $5,000 certainly isn't enough to live on for three months while you're doing that work. So that's also been presented, but I haven't found the right thing yet.

EW (01:09:39):

We should talk more offline.

PJ (01:09:41):

Would love to.

EW (01:09:42):

I do think I should let you go. We are about out of time. And now I get to ask you this final thoughts, questions, or last thoughts. People didn't like -

CW (01:09:55):

Last thoughts? That's even worse.

EW (01:09:57):

It is worse, isn't it?

PJ (01:09:57):

On my way to the grave.

EW (01:09:57):

Phillip, before you go, do you have any thoughts -

CW (01:10:08):

Do you have any last words?

EW (01:10:08):

[Laughter]. Do you have any thoughts you'd like to leave us with?

PJ (01:10:11):

Yes. I would like to say that I publish a monthly newsletter. It comes out on the first Monday of every month. If you're interested in keeping up with ongoing developments in the embedded systems industry, you can find sign up on our website at embeddedartistry.com/newsletter. A modified version of our Boeing essay is going to be published in the June edition of the Software Quality Professional journal. So keep an eye out for that.

PJ (01:10:37):

And I want to end on a serious note. So I'm going to get real for a second, related to the Boeing discussion we had. Which is, an idea that's just been sitting in my mind since I've been researching the Boeing 737 MAX crashes, and it's rooted in my studies in philosophy and psychology.

PJ (01:10:59):

And it's a simple idea that I think all of us really understand deep in our bones, which is, where we're aiming at is the most important factor in determining where we're going to end up. And I'm not going to say what the aim be. It's certainly up for debate, but the goals and the values that we select, and that we focus on, change how our brain sees the world, and how we make decisions.

PJ (01:11:25):

And so I just want to ask all of the listeners to take a few moments, and think about where you're aiming and where in your life, and where your organizations are aiming, and are you happy with where that's going to end up?

PJ (01:11:37):

And if you're not happy with the consequences, or even if you are, I hope that you'll take the time to pick a new aim, raise your aim a little bit higher. And I think we'll all be pleasantly surprised at the positive outcomes that that will have on our own lives and the lives of the people around us.

EW (01:11:55):

Our guest has been Phillip Johnston, founder and principal at Embedded Artistry, an embedded consulting firm in San Francisco, California. You really should check out his blog, embedded artistry.com/blog. Thank you for being with us Phillip.

PJ (01:12:13):

Thanks for having me. It was really a pleasure, and I hope we get to do this again sometime.

CW (01:12:17):

Thanks, Phillip.

EW (01:12:18):

Thank you also to Christopher for producing and co-hsting. And thank you all for your patience over the last few weeks, as we missed a few shows. Also thank you for listening. You can contact us once again at show@embedded.fm or hit the contact link on embedded.fm. Welcome back, embedded.fm.

EW (01:12:39):

Now a quote to leave you with. This I am getting from Phillip's Rules of Thumb post, and it's from Jack Ganssle. It's one of my favorite quotes from him. "Study after study shows that commercial code, in all of the realities of its undocumented chaos, costs $15 to $30 per line. A lousy 1000 lines of code - and it's hard to do much in a thousand lines of code - has a very real cost of perhaps $30,000. The old saw 'it's only a software change' is equivalent to 'it's only a brick of gold bullion.'"

EW (01:13:24):

Embedded is an independently produced radio show that focuses on the many aspects of engineering. It is a production of Logical Elegance, an embedded software consulting company in California. If there are advertisements in the show, we did not put them there, and do not receive money from them. At this time, our sponsors are Logical Elegance and listeners like you.