475: Stuffed Animal or Colleague

<laugh> I am speaking at the Embedded Online Conference, which is happening at the end of April, very beginning of May. You can have a discount coupon, EMBEDDEDFM all one word, that is. Also you can get a discount if your whole team is going. There are group discounts. I will put a link to where the group discounts are. But for EMBEDDEDFM, you have to use that.

CW (00:01:50):

What day are you speaking?

EW (00:01:52):

May 1st.

CW (00:01:52):

May 1st.

EW (00:01:52):

At 9:30 Pacific Time.

CW (00:01:55):

Okay.

EW (00:01:55):

Maybe at 10:00 Pacific Time. Maybe they invited me early, but my meeting invite was for 9:30.

CW (00:02:03):

And what are you talking about?

EW (00:02:04):

I am giving a talk about "Creating Chaos and Hard Faults." The downside of my "Memory Maps" talk becomes clear to me now.

CW (00:02:15):

The downside being?

EW (00:02:17):

That there is no way I can make a talk that cool ever again <laugh>. But the "Creating Chaos and Hard Faults" is fun. I start with talking about how to smash the stack, intentionally. Going through some really bad code. It was fun writing really bad code.

(00:02:36):

Then a little bit about debugging. And then I get into hard faults. I actually look at those in Cube, STM32CubeIDE. And talk about how they happen, how you can turn them off, why you should turn them on. How to debug them while you are looking at them, but also how to store them to a bit of RAM and look at them on boot.

(00:03:02):

And then I talk a little bit more about debugging. Mostly debugging difficult things. Not just how do you use GDB, which I do not go into at all. But how do you think about heisenbugs, and impossible bugs, and "Oh my God, I am never going to figure this out" bugs.

(00:03:19):

Which was kind of funny, because at the time I was writing my talk, I had a really strange- I mean it was I2S, which is- It is like SPI, but for audio. It is not a hard protocol, and I could see it on my logic analyzer, and it mostly worked. I was getting stereo to one spot, and I was getting mono to the other. But under some cases, which took me a long time to understand exactly which left-right on both sides, I was getting noise.

(00:03:57):

There was Bluetooth involved here, as well. So there were lots of channels where the noise could come from. The noise had particular characteristics, that were not there when I generated tones on the processor. So it must be my codec, or the communication to my codec. It was not my codec, because I could have my codec play things it was receiving, and all of those- Stupid wire! It was a wire! It had a jumper cable. They were not very long, but they were a jumper cable, and- <sigh> It was a wire.

CW (00:04:33):

It is a problem with those things like audio and camera, and cameras to video, where you have got a serial link that is delivering the data. With SPI, when you are talking to a device and you are reading stuff that has meaning, it is easy for the code to tell, "Oh, this value is out of range or stupid."

(00:04:53):

But if you are just streaming audio, it can 85% work, and the thing will receive it and decode it because does not know that it is wrong. It is just audio bite, it is just samples. The same thing happens with cameras. If there is a timing thing or some noise, okay, that shows up in the image, not in an error. <laugh>

EW (00:05:12):

Yeah, it becomes noise in the data.

CW (00:05:14):

The fact that you can have digital noise- I mean, it is analog noise at the bottom. But the fact that it manifests as digital noise, that can still be working but not quite, is frustrating.

EW (00:05:28):

Well, I had so many new pieces. The codec was a new piece of hardware. The processor. The two BLE processors I was talking to were new pieces of hardware to me. And the way I was using them was not standard from their devkits. I kept assuming the problem was processor timing or something, and how I set up the streams, and it did not...

CW (00:06:00):

So are you going to talk about this sort of thing in your talk <laugh>?

EW (00:06:05):

I will talk about the process I used, which is the process I usually use. Which is to reproduce the problem with the minimal set, and to keep figuring out what minimal is. Then to also in parallel, or back and forth in serial, talk about how to explain the bug.

(00:06:23):

It is weird how often we just assume that a bug is not possible, and how explaining the bug- We talk about rubber duck debugging, and just talking over your issue with the stuffed animal or whatever. Or a colleague. It depends on how much help you need. The stuffed animal of course providing more.

(00:06:48):

But it is not just the act of hearing yourself say what is supposed to happen, versus what is actually happening, which is good practice. There is also this, "If I wanted it to happen, how could I possibly create this bug?"

CW (00:07:06):

That is good, not just for thinking about causality, but if you have seen things. If you make- You are talking about smashing the stack, or the other things you are going to demonstrate. If you know what those look like...

EW (00:07:22):

They become a lot less frightening, if you have seen them before.

CW (00:07:25):

Or you know what might be happening, the next time you see it for real. Instead of, "Well this is not working. I do not know what that means."

EW (00:07:36):

Yeah. That goes back to explaining the difference between what you expect, and what you are getting. With bugs that happen intermittently or are clearly impossible, figuring out those steps necessary to reproduce them.

(00:07:52):

Just going back and forth between explaining and reproducing the bug. And realizing you are not going to fix it if you cannot measure it. That is why you have to reproduce it. But you are probably not going to fix it, if you do not understand it, which is why you have to explain it.

CW (00:08:07):

Can I explain- Well let me explain the bug I am working on.

EW (00:08:09):

Okay, go ahead.

CW (00:08:09):

Which you will not be able to-

EW (00:08:11):

Is this the IMU thing?

CW (00:08:11):

<laugh>

EW (00:08:11):

I helped with that. I did help with that this morning.

CW (00:08:18):

I do not know how much to say about it. It involves Zephyr, which we are going to talk about some more. But basically I have a sensor that I have added to our build. It has got a standard Zephyr driver that can be included, although it comes from ST's HAL module, but it is something you can grab.

(00:08:37):

So the code involved to add it, is updating a DTS and some configs. So there is no- I am not writing any driver. It comes up, and I can interrogate it over the shell and Zephyr.

EW (00:08:53):

And you can get the WHO_AM_I from it.

CW (00:08:55):

And I can get the WHO_AM_I, because I walked debug and saw that happen. But what happens is, when I try to read it- When it tries to read multi-byte things, which are...

EW (00:09:05):

Like the data?

CW (00:09:06):

Right, the data. What happens with that is the device is configured by default to, if you do a SPI read of one byte from an address, and then you do another read, it increments the address and gives you the next thing automatically, and you can just stream out the data.

(00:09:21):

What I was seeing is that I could read out one byte fine, but every other byte was subsequently not fine <laugh>. Either 0xFF or 0x00, depending on which way I stood on my head. Which is bad, because I need the other bytes <laugh>.

EW (00:09:36):

It is surprisingly how the first byte of accel X does you not as much good as you need.

CW (00:09:40):

So anyway, I was fighting with this, and went back and forth and all sorts of things. I cannot easily scope it right now. I am trying to fix that. But then somebody at the company said, "Hey, I ran your code and it works fine."

EW (00:09:54):

Woo-hoo!

CW (00:09:54):

Which is- I think I mentioned today somewhere else, that this is the inverse of the "It works for me" problem. It is a new variant. It does not work for me. It works for everybody else. But went back and forth, and we discovered that he was not quite running the same code I was running. He had added some stuff for a completely different peripheral, that has nothing-

EW (00:10:16):

On a completely different bus, doing completely different things.

CW (00:10:17):

Different bus! Different bus type! Different hardware on a different board even, within the system. Yeah, totally different. And that makes it work.

EW (00:10:28):

Woo-hoo! Works.

CW (00:10:28):

Worked for him. I took his code and made it work. So that is disturbing. We do not know why it does not work in some circumstances. I added the other thing we wanted to have, which I will not talk about too much, which is an additional config to this peripheral, and it stopped working again.

(00:10:46):

So with his code, the other item does not work, and it goes back to not working. It is something about power-up sequencing, or power, or something.

EW (00:11:01):

Timing?

CW (00:11:01):

Timing?

EW (00:11:01):

Interrupts being set or not set?

CW (00:11:04):

How would you like me to debug that in Zephyr? <laugh>

EW (00:11:07):

Okay, so I have more to say about the Embedded Online Conference.

CW (00:11:11):

Okay, let us finish that. Yeah.

EW (00:11:12):

But let us go to Zephyr.

CW (00:11:13):

Okay.

EW (00:11:13):

Because clearly we both want to. I was actually- I had this whole plan of how I was going to bring this up, and I was like, "So, have you been writing much code lately for your work?"

CW (00:11:27):

No. So for Zephyr, I did do one thing. I ported an ST driver that did not exist in Zephyr, and that actually went okay.

EW (00:11:34):

But did you actually write any code? Or did you just copy some files and modify?

CW (00:11:40):

I had to write some code to the platform dependent- There are a couple of functions I had to write to the I2C stuff for Zephyr.

EW (00:11:49):

But overall?

CW (00:11:51):

Overall, no. Mostly it was moving files around and editing the DTS and DTSI, which define the devices. And the config, which defines the config. And some other stuff, which defines some other stuff. I think the Kconfig. Is that the thing where I put sensor in? Yeah, maybe that one.

(00:12:07):

Oh no, I had to do some more, because I had to make it conform to the sensor API, which it did not. So that was the thing. But no, right, yes, I have not written a lot of code. I have stared at a lot of things and made small-

(00:12:21):

It is like the moment when you have got a beautiful sculpture and there are five more chisels you have to make it, to make it work. That is what I have been doing. Okay, I do not have a beautiful sculpture, but- The metaphor is bad-

EW (00:12:34):

You have a sculpture.

CW (00:12:34):

But to make the things I have been asked to do work so far, it has been mostly adding three lines of- Figuring out what three magic lines in a config need to happen.

EW (00:12:45):

That is the thing with Zephyr, every time I am seriously thinking about writing code, I realize I must be doing it wrong. I did write some code, I added a command so I could test this thing, and I mashed a couple of things together. But overall, if I am writing code, I really should be looking at how to do this differently. Because it is all there in Zephyr.

CW (00:13:11):

People have done a lot of work on Zephyr. That is the thing about Zephyr. So I think my-

EW (00:13:15):

Oh, wait a minute. Zephyr is a real-time operating system, and ecosystem.

CW (00:13:21):

It is a way of life. I do not know.

EW (00:13:23):

It really is kind of a way of life.

CW (00:13:25):

It has its own build system. It has its own-

EW (00:13:27):

West.

CW (00:13:27):

Well "west" is a command that drives the build system.

EW (00:13:32):

Ninja? CMake?

CW (00:13:33):

CMake and ninja are part of the build system. But there is a whole structure involved. It has its own config thing, which is-

EW (00:13:40):

West even does Git stuff.

CW (00:13:42):

Yeah, west will do Git stuff, because you will have to because submodules are an integral part of Zephyr, and west manages those.

EW (00:13:49):

See I am doing Zephyr and I have not had to use west for Git management yet. But I have been really careful not to edit-

CW (00:13:59):

Do you ever type "west update"?

EW (00:14:01):

I try not to.

CW (00:14:02):

So I had to add a module from somewhere else. That was adding a few lines in the config file, and then doing "west update," so that the config files have- They point to different repositories.

EW (00:14:14):

My goodness, so many different repositories.

CW (00:14:16):

Anyway, we are getting into details. It is very confusing. Zephyr is this big thing and people have put a lot of work into it. It is really neat. I mean it is really cool.

EW (00:14:27):

I do really appreciate it. Many of the choices they have made are the things that I would make. Like the whole macro thing, I probably would have done that.

CW (00:14:38):

Well, if you are going to use C, what else are you going to do?

EW (00:14:40):

And even though as a user I am sometimes baffled by why I cannot...

CW (00:14:46):

Yeah.

EW (00:14:49):

It is pretty amazing.

CW (00:14:50):

It has- The things that are good about it- Let me start with the things that are good about it, before I complain. Because I have some complaining to do.

EW (00:14:57):

Do you? It surprises me.

CW (00:14:58):

Things that are good about it is people have put a ton of work into this. There are board definitions for so many boards. There are drivers for so many devices. And if it does not have one of them, it is likely the vendor has another repo somewhere you can point at to get a Zephyr driver for it. All the basic peripherals are done, done, done. SPI, I2C, all that stuff.

(00:15:21):

Now that I kind of get it, I kind of like how you can define your board and the pin stuff, and how that is easy to change. You can even at the build time, you can say, "I want to build my code for this board." And then you can say, "Nope, I want to build my code for this board." And it should work on both boards, as long as everything is configured correctly.

EW (00:15:43):

That is super magical. The idea that you can go from a Nucleo-L4 board to a Raspberry Pi, and all you do is change your build file-

CW (00:15:53):

Or a simulator. It has QEMU, so you can just build for a simulator. That is really cool, and it is kind of the promise of, "Okay, now we are moving away from the platform dependence that we have really been stuck with with embedded, to something that has got a lot of power to abstract that all away." In a way that is not too terrible. The abstraction layer is there, but honestly it is easier to follow the links.

(00:16:18):

Like I was having to debug the SPI stuff. It was easier to follow than getting into ST's HAL, once I did it. But there are some points where there are quasi function pointers and things, and debugging gets a little tricky.

(00:16:30):

What else do I like? The build system is fine. You do not have to touch it much. If you feel like you need to know CMake, I have not touched- You probably should not be touching anything in CMake, when you are working with Zephyr. Because it is all supposed to be run through the config files and that stuff.

EW (00:16:51):

I added some files to CMake lists.

CW (00:16:55):

Did you? Oh yeah, that is the one you do have to add. Right. So the most you should have to do I think is CMake lists.

EW (00:17:00):

Which is just adding your file name in amongst all of the other files in your directory. It is not hard.

CW (00:17:05):

If you are trying something tricky, like add a library- There is other stuff where that gets tricky and you have to do things.

EW (00:17:10):

But the goal with Zephyr is to play in the sandbox.

CW (00:17:13):

Yeah.

EW (00:17:13):

Do not get out of the sandbox, unless you really have to. If you think you have to, reconsider about five times. Because there is a good chance you do not actually have to leave their world. Okay. Now...

CW (00:17:34):

There are some things I am not quite clear on. So let us start with that, before I complain. I mentioned the build system, and how it is integrated with Git. It pulls in repositories from places, as modules and stuff.

(00:17:44):

But there is also how you can structure your whole application. This is where it gets really tricky. I have read a lot about this, and I do not quite understand the right way to do things.

EW (00:17:54):

We are both working on Zephyr and we have different ways of doing it, because our vendors set us up with different ways of doing it, and we have different chip vendors.

CW (00:18:04):

Yeah. There is this thing called- The Zephyr repository is the OS repository, right? You probably do not want your application to be in there.

EW (00:18:14):

Right. Because I want to commit my application to my code.

CW (00:18:17):

It is giant. And you do not want to fork Zephyr and carry that around, although everybody does that anyway.

EW (00:18:21):

Everybody does, but we are trying not to.

CW (00:18:24):

So there is a bunch of ways to have your application be in another repository, and have both of them side by side. So you check out the Zephyr repository as part of west. It also checks out your repository.

(00:18:34):

And then magic happens in the build system that I do not understand. So you can just type "west build," and it knows to look in Zephyr for the Zephyr stuff, and your repository for your application plus any stuff that you are kind of adding on top of Zephyr.

EW (00:18:49):

Like if you want to replace something in Zephyr, like you want to have your own board file-

CW (00:18:53):

Not even replace. Well yes, not replace, but add. So if you have a custom board, you put that in your repo in the same kind of directory structure that the Zephyr one has. If you want to add a device driver, you do the same thing in the same structure. So your repo looks like the Zephyr repo, just with nothing much in it. That is super weird. I do not understand how that works.

(00:19:15):

But there are three different ways that Zephyr says you can do that. They call them- I think they called them "topologies" or something.

EW (00:19:22):

Yeah.

CW (00:19:24):

There is a hub and spoke kind of thing. I glossed over at that. That is confusing. If you asked me to make that work from scratch, I would probably do something that might work, but not really understand it, after reading the examples.

(00:19:39):

I am not wild about errors, like with the config system. So the config system as you mentioned is-

EW (00:19:50):

<laugh>

CW (00:19:50):

This series of files.There are ones called DTS, which are the device tree-

EW (00:19:54):

Specifications.

CW (00:19:54):

Specifications. That is where you put all your peripherals that you define. You can do the pin control, so you define what pins are for what functions, whether the GPIOs- How the GPIOs are configured, et cetera.

EW (00:20:07):

My SPI is on pin 0.09.

CW (00:20:11):

Yeah. And it is a whatever.

EW (00:20:12):

And if you want to go to sleep, it is on 0.09, and these are its sleep parameters.

CW (00:20:18):

There is all of that. It walks all of that, and builds the real config that it builds the code out of. The problem with that is if there is a problem in those files. Sometimes it is fine. It will give you an error at the line you did it. But sometimes- Like I had one-

EW (00:20:36):

It is like old C compilers that would barf at the end of the file, and your error was on line two.

CW (00:20:42):

Yeah, I had one I was fighting with for a while. It was not with the DTS. Maybe it was a consequence of the DTS. It was something where there was a config macro. So all the config macros in the config files end up being like config_SPI. If you want SPI in your system, you have to do config_-

EW (00:21:00):

That is going to be like Kconfigs and prj.confs.

CW (00:21:04):

Some of those end up requiring other configs to exist.

EW (00:21:09):

Yes! And you can have it all treed.

CW (00:21:10):

Yeah, that is great.

EW (00:21:12):

And "disable" and "default if" and "default else."

CW (00:21:15):

Sometimes it will decide you need one, but for some reason there is no default. So it just puts config whatever thing I need equals and then stops. And that is what the actual config file that it builds from has. Config equals blank.

EW (00:21:32):

Equals nothing.

CW (00:21:34):

It is a new line. Config equals, and then new line, the next config. The consequence of that when you build, and people might want to know this, is that everything will build until it gets to an assembler step. And then it will give you an error in the assembler.

EW (00:21:51):

He was so mad about this for days.

CW (00:21:53):

At a particular line in a temporary file, with a random string of letters as its name, that no longer exists because once the build fails it cleans up the temporary files. I ended up looking on the internet. I found somebody who was complaining about something sort of similar. I looked in that config file and saw the blank thing.

(00:22:12):

Yeah, that is what happens is it turns out it gets blank. It happily builds that, and puts in a line that says like "if equals nothing" somewhere. But the compiler does not give you an error. The assembler gives you an error. It does not tell you what it is. It just says "expression not complete."

EW (00:22:33):

I think it said "expression blank" or "expression not complete." Something that was really not helpful.

CW (00:22:38):

And it does not tell you what the expression was.

EW (00:22:40):

No, and it is deleted so you cannot go look it online.

CW (00:22:44):

Right. That was from the Kconfig. I feel like the build tool, the thing that is parsing the Kconfig files and turning it into the config file, should notice when something is not populated and maybe give you an error there. Does not seem like a difficult task.

(00:23:02):

But there are a lot of things like this where something goes wrong, and the system is so complex and it does not give you an error where the error actually occurred, that it is really hard to backtrack and figure out what is going on.

EW (00:23:18):

I was going for a different Zephyr subsystem, but it is the same thing. It is so complex, that the error had nothing to do with what I was doing and did not make sense. How do you- I would remove that line, to make sure it really was that line causing this. But then where do you go from there?

CW (00:23:45):

As I told you, one of my Zephyr debugging techniques, is to write the word "poop" in the middle of config files, just to make sure that the config file is being...

EW (00:23:55):

The rest of us just write "ASDW" or-

CW (00:23:58):

Is being actually looked at, because-

EW (00:24:00):

Some keyboard stuff. But he actually writes "poop."

CW (00:24:02):

There was a ton of times where something was not changing. I do not think I ever solved this. I was changing- Was it a DTS file? It might have been a DTS file, and it was having no effect. I do not know how I got around that, but-

(00:24:16):

Oh, no, it was I was trying to pull in an ST module, and trying to figure out how to do that. And I edited- I had a west that I YAML- Huh! Another thing we did not talk about. The top level thing is the west YAML, which defines where all your code comes from and stuff. But that had an import to another one, and I changed the contents of the imported one, but west did not ever notice it.

EW (00:24:42):

Because you needed to do a west update?

CW (00:24:44):

No, it is supposed to notice during west update. That is what I was doing.

EW (00:24:46):

Oh.

CW (00:24:48):

I wanted a west update.

EW (00:24:49):

I never do a west update. Never ever.

CW (00:24:51):

But you were not changing things that required a west update. That is why.

EW (00:24:54):

I do not know if I ever...

CW (00:24:56):

I was adding something <laugh>.

EW (00:24:58):

Okay.

CW (00:24:59):

So I had to. There was no choice. But anyway, that one is a mystery to me. It does not seem to recursively notice dependencies, if you do an import in west YAML.

(00:25:10):

So Zephyr is cool. It is a really great system. But I feel like it is the guts of a really great system, and that something smoother needs to be on top of it?

EW (00:25:25):

Somebody needs to put some buttercream on west, or on Zephyr.

CW (00:25:30):

That is a metaphor one could use. <laugh>

EW (00:25:34):

<laugh>

CW (00:25:34):

But it is like- Gosh, I do not know. I have only been using it a couple of months, so I am not an expert. Definitely not an expert. Maybe as I get used to it more, it will be less frustrating. But I do feel like some of the stuff is like, "Okay, this is the guts of the system, and you have to know more than you should."

(00:25:59):

Like the way they abstracted away the build a little bit, to the CMake and the ninja, so you do not really have to touch that so much. I want a bit more of that. Like the DTS, the Kconfig stuff-

EW (00:26:15):

There are GUIs for some of that. But then if you change something in your GUI, it goes someplace random, and you have to figure out how to put it back in your system, which is really fun.

CW (00:26:22):

Yeah. I do not know what the solution is. I do not even really know what I am asking for.

EW (00:26:26):

Something simpler. Something a little easier to understand.

CW (00:26:30):

Maybe I am just asking for better static analysis.

EW (00:26:36):

Yeah. I think that is coming, because the macrobatics, the macro stuff that is...

CW (00:26:43):

Macrobatics? Is that-

EW (00:26:44):

"Macrobatics." That was the term that Trond uses to describe-

CW (00:26:47):

Oh. Okay. <laugh>

EW (00:26:47):

There is a talk about it. It was really useful in debugging some of those bugs that come up because of the build system. But they are complicated and it is really overusing the C precompiler, in amazing and horrible ways.

CW (00:27:11):

Yeah. I do not know what- They probably thought about making a Medic compiler doing Yacc and Lex or something instead.

EW (00:27:19):

They have still got all these DTS and DTSI and YAML and defconfigs and Kconfigs. It is not like they are not using other compiler things.

CW (00:27:28):

The hierarchy stuff is both a blessing and a curse, I find.

EW (00:27:31):

Oh my goodness!

CW (00:27:31):

Because it is really hard to find things. It is so big. That is the thing. I have got it in VSCode, and if I am looking for stuff, there is the board directory and then there is the DTS directory, which is not the same as the DTSI-

EW (00:27:45):

I deleted all of my extra board files, so it is just the ones I am using.

CW (00:27:48):

I know, but I cannot really do that. It is not my repo.

EW (00:27:50):

Well, I am not going to commit it.

CW (00:27:52):

Oh, I see. Well, I will accidentally- Yeah. <laugh>

(00:27:56):

Okay, well, that is-

EW (00:27:57):

There is a Git sparse command that is supposed to help with that. I did not use it.

CW (00:27:59):

Yeah. So maybe that is helpful. But that would be nice if that was an option that you could do, like, with west. Like say, "Hey west, here are three boards from Zephyr that I am going to use."

EW (00:28:08):

"And blow away everything else, so that when I search for the things that I want-"

CW (00:28:11):

"And these drivers. And make the Zephyr repo sparse." That would be nice.

(00:28:15):

But just- Like the hierarchy- <sigh>

EW (00:28:24):

<laugh> We are just going to title this one "Sigh".

CW (00:28:25):

I understand this stuff, but on some level I do not. Some of the choices probably were difficult to make. Maybe they all came from Linux, where they have a long storied history. Which is fine.

(00:28:37):

Honestly, if I was going to start a new project for myself right now, there is nothing I would choose other than Zephyr. Having gone through the pain of the learning curve. It is a nice enough system, it is modern enough, it has got a lot of features and drivers and stuff. That if I was going to build a system, I would probably reach for that immediately.

(00:28:59):

Because I do not have to screw with Cube or any of that other stuff, or weird mobiles. Just pull up, do my pin definitions from some devkit and go. Because the RTOS itself is nice too. The actual API is pretty cool.

EW (00:29:21):

The kernel. What I would consider the kernel.

CW (00:29:23):

Yeah. The kernel.

EW (00:29:23):

The scheduler. The muxes.

CW (00:29:25):

It has got a lot of features.

EW (00:29:29):

The RTOS part. But Zephyr, when you talk about it, it is not just the RTOS part.

CW (00:29:33):

No. That is a mistake. Probably, yeah.

EW (00:29:35):

That is one of the most common mistakes, is to go in thinking, "Okay, it is just an RTOS."

CW (00:29:39):

Yeah. It is like thinking Linux is the kernel. As the GNU people would like to point out, it is GNU Linux because it includes all the GNU stuff around it. That is just like Zephyr, it includes a lot of stuff around it. People have put a lot of work putting hardware stuff in there, that you just do not get with other systems.

EW (00:29:58):

It is really nice.

CW (00:30:00):

I have complained on this podcast multiple times about having to write another damn SPI driver for an accel-

EW (00:30:07):

Everything.

CW (00:30:07):

Or whatever. Zephyr really is trying to make it so you do not have to do that.

EW (00:30:13):

Right.

CW (00:30:14):

Even the driver I did not have, I was able to find an open-source driver, and it was adaptable very quickly. So I think it is the right direction, and you have to suffer through a little bit of learning curve <laugh>.

EW (00:30:35):

It makes me think about- I read this book about Bluetooth a while ago. It was awful.

CW (00:30:42):

<laugh>

EW (00:30:42):

I am not going to call out what it was. Because after I learned about Bluetooth, after I got used to it, mostly through trying stuff, but also through reading documentation from Nordic and TI, but really getting into the whole "What is Bluetooth and how does it work?"

(00:31:02):

I ended up going back to this book for some reason. I think I was trying to find a book to recommend to someone else, and I did not realize I was picking up a book I already had. And it was a great book! It laid it all out very clearly. It made a lot of sense.

CW (00:31:18):

Now you knew what it was talking about.

EW (00:31:20):

So I think there are instruction materials for "Here is what you need to know going in." And instruction materials for "Here is the reference, and how to balance out what you do not know. You know the basics. Here is all the stuff that you also can do." But that book was terrible for a beginner, because I did not need the excessive information.

CW (00:31:50):

That is part of the problem with me, not with Zephyr, is at least for what I am doing with it, it was the worst possible scenario for learning. Because I was not taking an ST devkit and putting some sensors on it, and firing up the build system and writing a little example application. I was working with a-

EW (00:32:08):

Custom board.

CW (00:32:08):

I am working with a real system, which is a custom board, with many, many, many sensors and buses on it. With a custom- Not a custom with a...

EW (00:32:20):

Rare?

CW (00:32:22):

Rare MCU, which has its own Zephyr repo, because it is from the vendor and it is rare. So I had to do all the hard stuff with Zephyr, that you maybe normally do not encounter. And I had to do that first, which is a great way to learn and a terrible way to feel good about yourself.

EW (00:32:48):

<laugh>

CW (00:32:53):

Even with that, my impression is still pretty favorable. But man, trying to figure out, "Okay, I need to-" <laugh> Even stupid stuff like the driver for the- This is super weird. The driver for the IMU I was putting in, looked like it was in the main Zephyr repo. But it would not build. If I brought it in, it would not build because it said it was missing files.

(00:33:20):

Well the deal is, the way ST puts their drivers in- I guess. I do not know. They put half the stuff in there, the non part specific stuff. Kind of, sort of? But then they have another repo that they have as a Zephyr module, that has the other half of the drivers.

(00:33:38):

And if you do not put that into your system, if you do not pull that in your west YAML, then you do not have the complete driver. Because all the register stuff is off in ST's repo. But the driver is in the main Zephyr repo. Why, I do not know. Because they have a directory for every single one of the chips, where they could have just put the register files in.

(00:34:00):

But anyway, that was like, "What?" So my initial response was, "Well, they just did not finish." And I found the register files and I put them in the Zephyr repo. I was feeling bad about having to add that to our Zephyr repo. And everything worked. But then I realized, "No, that cannot be how this is supposed to work."

(00:34:18):

So stuff like that is really weird, and I do not understand it to this moment. But yeah, as soon as you get outside the bounds of just piecing stuff together. Anytime you have got custom things going on, when you are going to reach into the DTS and really make some modifications, then things get a little bit trickier for learning curve, I think.

(00:34:40):

Which I would have appreciated starting with the basics. Because I was trying to learn the basics, along with the advanced. That is where pain lives. Because you feel completely incompetent about everything. And question your understanding of the basics, because trying to move past that really quickly.

EW (00:35:00):

But I think that is really common. You get the basics working, and you do not necessarily understand them. But you immediately want to change something.

CW (00:35:10):

Yeah, but it is a small thing, right?

EW (00:35:13):

You went full all out. But it did not take me long before I needed to change something, and I- There was so much.

CW (00:35:21):

But yours is complicated too. You have a multi-

EW (00:35:24):

Mine has a bunch of algorithms that-

CW (00:35:26):

You have a multi parts system.

EW (00:35:26):

So many parts. I spent a lot of time just drawing- Going back to explaining the bug. I spent a lot of time just drawing out what the system was, and what I wanted it to be. Each one was a full eight and a half by 11 drawing, of which parts of the system went where. It was- It is horrifically complicated. Okay, so-

CW (00:35:55):

Zephyr has a shell. I just want to plug one more thing.

EW (00:35:58):

I love the shell.

CW (00:35:58):

It has a CLI.

EW (00:36:00):

It is so easy to add stuff to.

CW (00:36:02):

Never have to write another damn CLI if you use it.

EW (00:36:04):

Oh, yeah.

CW (00:36:04):

It is good. It has got history if you want it. It has got all this stuff.

EW (00:36:09):

Tab complete? Did you set yours up for tab complete?

CW (00:36:11):

I do not know if I did. It has got basic- You can query I2C bus if you want. And scan it. And do all this basic stuff just out of the box without doing anything. You can add commands to it, and it is not that hard.

EW (00:36:25):

Actually, Chris Gammell is doing a whole talk about "Zephyr tools to debug hardware" at Embedded Online Conference.

CW (00:36:31):

Oh, I should watch them.

(00:36:31):

Anyway, just ending on Zephyr. It is very promising. More than promising, I think. It is there. It is just learning curve is steep. <laugh> Especially if you are diving in at the deep end.

EW (00:36:50):

On the topic of intro versus reference materials- By the way, nobody likes your Doxygen docs. Okay?

CW (00:37:00):

Nobody even uses your Doxygen docs.

EW (00:37:02):

They are horrible. I do not care how much time you-

CW (00:37:05):

Doxygen is an entire system devoted to convincing people they have written documentation.

EW (00:37:12):

But there is actually- Writing code versus reading code. Are they different skills?

CW (00:37:19):

Oh yeah.

EW (00:37:21):

I feel like at the beginning of career-

CW (00:37:23):

<laugh> I have an analogy for this.

EW (00:37:23):

Okay, go ahead.

CW (00:37:24):

I have been learning Morse code.

EW (00:37:25):

Mm-hmm. Oh! Yeah!

CW (00:37:27):

I can send code with no problems. At 12, 15 words per minute. I cannot read code at more than about nine words per minute right now.

EW (00:37:39):

Well, that is because it has got weird things, like "I"s are two "E"s, and-

CW (00:37:42):

It does not matter.

EW (00:37:42):

"C"s are two "N"s.

CW (00:37:43):

But I can send that.

EW (00:37:44):

Well, yeah, but you do not have to parse that.

CW (00:37:48):

Exactly! You do not have to parse your own code. <laugh>

EW (00:37:54):

I think the thing with Zephyr for me- One of the things with Zephyr has really made it clear that, while I write code very quickly and usually do a decent job, I need more practice with reading code. I am not terrible at it, but it is a very different skill.

(00:38:21):

It is not even working the way it used to for me, which was you take a file, you list all of the functions in the file, you write a little note about what each function-

CW (00:38:31):

Because stuff is not structured that way anymore.

EW (00:38:33):

Then maybe you have a little bit more about what function calls what function.

CW (00:38:37):

Yeah.

EW (00:38:37):

It is not structured that way anymore. And it is not just reading the header files, or understanding the structures. There is-

CW (00:38:46):

That is how modules work <laugh>. You cannot do it that way if everything is going to plug together and link together in various ways. Yeah.

(00:38:54):

I think it is the dialect thing, right? There is no one code language that people write. Yes, people write C or C++, but everyone has their own dialect. Whether the way they choose variable names, whether- Even conforming to a coding standard, how they structure stuff, how much they put in a particular scope. All that stuff matters to how you read it.

(00:39:20):

Every time I come upon somebody else's code, that stuff is a little bit different, and it takes some time to get used to.

EW (00:39:27):

It takes some time to get used to. But I do tend to find that I write in their dialect pretty quickly.

CW (00:39:32):

Sure. Yeah.

EW (00:39:34):

But that may just be a consultant thing. That I am just used to being flexible with matching what is around.

CW (00:39:40):

I found reading Linux kernel code to be maddening. Linux kernel code is extremely difficult to read. There are no comments, usually. There are a lot of meaningless variable names, that are very short. And lots of logic doing stuff that is hard to understand.

(00:39:56):

I have not read a ton of the Zephyr source code, and I do not know if it conforms to that. But I did find Linux one of the most difficult codebases to read, when I was working on it.

EW (00:40:10):

I guess the code I have been reading is mostly Nordic's layers on top of Zephyr. I do not know that I have gotten down to the Zephyr much.

CW (00:40:20):

Vendor code tends to have a flavor, does it not?

EW (00:40:24):

Oh, each vendor has their own flavor. Absolutely.

CW (00:40:26):

But there is an overarching whiff of "vendor" flavor to stuff. I cannot describe it.

EW (00:40:34):

I think it is Arm flavor, actually.

CW (00:40:36):

Oh, maybe. Maybe.

EW (00:40:40):

While we were debugging a DTS problem that I had, you suggested I use ChatGPT.

CW (00:40:48):

<laugh>

EW (00:40:48):

Which was a shock to me! When you suggested I use ChatGPT. It had not even occurred to me.

CW (00:40:55):

I think I suggested using Gemini, but that is fine.

EW (00:40:57):

Well in the end we used Gemini, yes.

CW (00:40:58):

Yeah.

EW (00:41:00):

It was thoroughly unhelpful and very confusing.

CW (00:41:01):

Yes.

EW (00:41:01):

On the edge of useful though.

CW (00:41:05):

Yes. That has been my experience with trying to ask questions of it, about anything. Yes. It is almost right! And wrong.

EW (00:41:14):

But it was another instance of I really had to read it very carefully. In this case it was an instance of I had to read it very carefully, and then throw away everything I read. What I should have been doing was reading very carefully some other document that I do not know what it is.

(00:41:34):

Or what I was doing before you walked in the room, which was copying files and trying to figure out which ones I could delete, which ones I had to keep.

CW (00:41:45):

Yeah. The times I used ChatGPT or Gemini, are times of total desperation. I have run out of ideas. Fine. You might come up with something that will jog my memory or point me in a direction, but I do not believe you are going to get me the answer.

(00:42:05):

That is why I suggested it, is might as well see what the robot hallucinates and maybe it will point somewhere we have not looked.

EW (00:42:13):

It was interesting. I know a lot of people are using ChatGPT and other AI coding tools. It is not a tool I-

CW (00:42:23):

I do not use it.

EW (00:42:24):

It is not one I reach for. It is not one I would have ever thought of. As you probably realized, I solved the bug as soon as I stopped looking at that. And yet-

CW (00:42:37):

That is because I told you to put "poop" in the DTS file.

EW (00:42:40):

And then I realized it was not being compiled. Yes. Your "poop" is great poop.

CW (00:42:44):

ChatGPT will never tell you to put "poop" in a config file.

EW (00:42:48):

Let us just keep coming up with titles right now. Okay, let us see.

CW (00:42:53):

And sometimes that is what you need to do.

EW (00:42:56):

Put "poop" in your files. Let us just keep saying "poop" until it-

CW (00:43:01):

<laugh> Anyway, what is the next topic?

EW (00:43:03):

You know there are two nine-year-olds listening to the show right now. I am pretty sure of it. I am betting that at least one of them is running around saying, "Poop, poop, poop, poop. Poop, poop, poop, poop." And if they were not, then now they are. You are welcome.

(00:43:18):

More talks from the Embedded Online Conference. "Breaking Good: Why Virtual Hardware Prefers Rough Handling," by Uri Shaked. "Beyond Coding: Toward Software Development Expertise," by Marian Petre. And "The Power of a Look-up Table," by Nathan Jones.

CW (00:43:39):

Look-up tables. That is my number one go-to for all problems that require things to be fast. Math. Databases. Look-up table.

EW (00:43:51):

Look-up tables are cool.

CW (00:43:51):

I once milled <laugh> a look-up table for an entire screen. I had to dewarp- Did I have to dewarp or warp? It was something where I had line scan data coming in, but it corresponded to polar coordinates. I needed to display it very fast as a donut image, instead of the rectangular. This was like 2007? Computers were fast, but they were not 2024 fast.

(00:44:24):

I needed to do this at high resolution, at 30, 60 frames per second or something. I was doing it with trig functions and calculating each pixel. It was working but very slow. Not very slow, but too slow. And then it hit me. I could just, for every pixel- I have enough memory. I have whatever 2007 memory was. It was plenty. Just store for every pixel, whatever the transformed coordinate. It just went through and it did a look-up for each one and put it in the right pixel spot.

(00:44:57):

Then I somehow did bilinear filtering on top of that. But that was super fast. If you have enough memory, look-up tables can solve a lot of problems that you do with computation. Good talk, I am sure.

EW (00:45:11):

James Grenning and Jack Ganssle and Jacob Beningo will also all be giving talks. There is plenty of stuff there. It is good stuff. If you are interested, EMBEDDEDFM is a coupon you could use. I honestly do not know what the price will be after you put that in.

CW (00:45:31):

Maybe it will go up.

EW (00:45:31):

Maybe it will go up.

CW (00:45:32):

Yeah. <laugh>

EW (00:45:36):

More self-promotion. My book, "Making Embedded Systems, Second Edition," is really, really, really, really shipping from Amazon and other physical book sellers. You can get it there, or Bookshop.org or eBooks.com or Google Play, or wherever you want. It does seem to be getting good reviews.

CW (00:46:02):

At the Apple bookstore. Somebody wanted it on the Apple bookstore.

EW (00:46:04):

Should be there.

CW (00:46:06):

It was not there in their country, I think.

EW (00:46:09):

Ah, yes. I do not read the reviews. So if you actually want to tell me something from the book that you liked or hated, you will have to email the show, at show@embedded.fm.

CW (00:46:18):

I read the reviews.

EW (00:46:19):

Christopher reads the reviews. He occasionally tells me that somebody mentioned the chicken button, which of course I am very happy about.

(00:46:25):

Let us see. Nordic winners. Oh right. Nordic was sponsoring the show, for which we thank them very much. They are giving away three-

CW (00:46:38):

PPK2-

EW (00:46:39):

PPK2-

CW (00:46:41):

Power analyzers.

EW (00:46:41):

It was supposed to be one a month, but apparently I suck at that. So now we are just going to give them all away. Farzard, Julian and Adrian, you should all expect emails from me. Then I will say, "So you won. Can I give your email to Nordic?" Then Nordic will dropship you stuff. Because we all know I am terrible at shipping things.

(00:47:05):

Let us see. Remaining question from Guyrandy, "I recall an episode where you talked about installing an antenna to talk to your dad. How is that going?"

CW (00:47:20):

<laugh> It is going slowly.

EW (00:47:21):

But it is going.

CW (00:47:22):

I have assembled my transceiver, which was a much more involved kit than I expected.

EW (00:47:30):

Ask him about toroids.

CW (00:47:31):

That took several weekends. I had to wind more toroids than I would like, ever. I tested that with WSPR and transmitted WSPR with that. I did some scanning around for some CW on 40 meters and heard some stuff, with just a random wire I had stuck out of it. So that is good.

(00:47:54):

I have a dipole that I need to experiment with. I have antenna tuners. I am learning Morse code. Really the only hold-up now is getting Morse code to where I am feeling comfortable enough to try out some- See, I do not even know the code for talking to someone on CW. Anyway, yeah, it is going well.

(00:48:16):

My dad has got his radio and stuff. I need to help him with the GPS, because he wanted to try WSPR. The GPS assembly for the QRP kit is a little weird. It is flexible, but <laugh> it is unclear how to do some things without some extra parts. But yeah, no, it is happening.

(00:48:36):

Yeah. It was nice to- The kit was complicated enough that I was so shocked when it just worked the first time.

EW (00:48:45):

You were pretty shocked <laugh>.

CW (00:48:47):

I am still not sure it is actually really working. Because there is a lot of- There are three trimpots and a trimcap that you have to adjust for the filters and stuff. It has got internal test tools, so it has got a screen and stuff, and some knobs you turn.

(00:49:05):

Those have to be adjusted while transmitting into a dummy load. I guess I did it right, but I think I want to do it again.

EW (00:49:12):

You transmitted. You got pings from various people. You went on the WSPR map and you saw it got here and there.

CW (00:49:20):

I think a lot of them were for reception, so I might be able to improve the noise floor and stuff if I do that again. But yeah.

EW (00:49:28):

Cool. What else have you been working on? Drums, piano, Morse.

CW (00:49:34):

I have not played drums in a while.

EW (00:49:36):

Zelda.

CW (00:49:36):

Some piano.

EW (00:49:37):

The dog.

CW (00:49:37):

I am working on some piano stuff. I do not know. I am working on nothing.

EW (00:49:47):

That is so not true. You are always busy.

CW (00:49:49):

I know. I probably should stop doing that. I have two clients, too. One of them is going to ramp up soon.

EW (00:49:58):

Okay. Well I think that covers-

CW (00:50:01):

Did not anybody have any other questions?

EW (00:50:03):

Oh, right. I had a list. What is NP and P, and what do those even mean?

CW (00:50:13):

We answered that in the last show.

EW (00:50:15):

With Inna?

CW (00:50:15):

Yeah.

EW (00:50:15):

Did we really answer?

CW (00:50:17):

Yeah. Polynomial and non polynomial.

EW (00:50:20):

Polynomials are easier. It is kind of bounded, where non polynomials are not really bounded.

CW (00:50:27):

It is that you can express the- <sigh>

EW (00:50:30):

The time it takes to do this algorithm.

CW (00:50:32):

To do this algorithm in polynomial time, rather than e to the x.

EW (00:50:38):

Which is more unbounded than polynomial time.

CW (00:50:46):

Here is the- Yeah, the number of steps that are required is order of n to the k, where number of steps is n and k is some number. So k squared or n cubed, n to the fourth, n to the sixth. So non polynomial would be one where that does not hold true. Which means it could be exponential with the input, or worse, doubly exponential.

EW (00:51:12):

NP bad.

CW (00:51:15):

The idea of P equals NP would be a proof that says, "Well, all problems that we think are non polynomial are actually polynomial."

EW (00:51:24):

Which would break lots of things.

CW (00:51:25):

Which would mean there are solutions to lots of problems that are faster. I think I have got that right. If I do not, then I do not care.

EW (00:51:32):

Then we are revoking your CS degree, which you do not have, so you do not care.

CW (00:51:34):

No, I do not have a CS degree. Yeah.

EW (00:51:38):

Let us see. What is your favorite computer algorithm, and which algorithms do you use the most?

CW (00:51:45):

I cannot remember the last time I used an algorithm.

EW (00:51:48):

That is not true.

CW (00:51:50):

I can remember a time I used an algorithm. I am trying to think of the last time I used one.

EW (00:51:54):

I mean, you did a big deep dive into the architecture of a machine learning system.

CW (00:52:00):

Okay. Are those algorithms though?

EW (00:52:03):

<laugh>

CW (00:52:03):

That is just a bunch of linear algebra.

EW (00:52:08):

All right. My favorite is an FIR filter because I am super lazy.

CW (00:52:15):

Those are nice. One that I used and understood extensively, that is probably the most "CSey" algorithm I worked with- I did a lot of stuff when I was doing networking. Networking, there were a lot of more fun algorithm kind of problems. One of those was Dijkstra's algorithm, which is a method for finding the shortest path between two points in a graph, which you have to do for certain routing protocols.

(00:52:40):

So for OSPF, which is the Open Shortest Path First routing protocol, which I implemented an implementation of, you have to do that whenever something in the network changes. You have your view of the network, and you need to compute the shortest path between various points. That is a very efficient algorithm for finding the shortest path in a graph between two points. That one was fun to work with.

(00:53:07):

There was other fun stuff there, like ways of doing timers, millions of timers, on a system. Because there are a lot of timers associated with all sorts of entities and networks, you end up on a-

EW (00:53:22):

Packets timing out.

CW (00:53:22):

Packet timing out. But more usually routes. Because packet timeouts are just counters that they are in the packet. But stuff like, "Oh, I need to send an update." Or, "I need to time out this route entry and refresh it." But you might have 60 million of those or something. So you can imagine, how do you- You are certainly not going to use hardware timers <laugh> or a linked list or something like that.

(00:53:55):

There were a lot of fun algorithms for how do you manage 60 million timers that could time out in an efficient way, so you are not constantly scanning. So you have one master real timer that drives all that. That was cool. Yeah, I would have to think about it. Hash tables are another thing I use all the time.

EW (00:54:16):

Yeah, hash tables are fun. They are weirdly efficient. Laplace versus Fourier?

CW (00:54:24):

Fourier.

EW (00:54:24):

Yeah!

CW (00:54:26):

I never understood what Laplace was good for. Transfer functions, I guess. Okay, sure. I do not do a lot of transfer functions. I do a lot more signal processing.

EW (00:54:36):

Have you seen or read "Dune"? Do you have any thoughts about it?

CW (00:54:41):

How much time do we have?

EW (00:54:44):

I believe ten more minutes.

CW (00:54:46):

I have read "Dune" through "God Emperor of Dune."

EW (00:54:51):

That is one and two.

CW (00:54:52):

No, that is one, two, three and four.

EW (00:54:58):

Okay. But you recently reread one and two.

CW (00:54:58):

I recently reread "Dune" and "Dune Messiah," which are the first two. I quite like the "Dune" series. It has got a lot of thought-provoking stuff in it. It is of its time. So there are elements of it that...

EW (00:55:13):

Did not age well.

CW (00:55:14):

A little squishy. But it ages better than a lot of- In my opinion, it aged a lot better than a lot of sci-fi from that era. I had not read it in a while. I read it recently. Very recently, actually. I found it more literary than I remembered it. Compared to other sci-fi, it was less swashbuckling adventure and more kind of deep. So that was cool.

(00:55:38):

As a series, it gets very weird from each book. So the first book is the least weird. The second book is more weird. The third book is pretty weird. And the fourth book is what!? I kind of crashed after the fourth. I am not even sure I finished the fourth book. I know I started it. Then I think there is one or two more after that, and they are just out there. Then his son wrote some follow-ons, which I hear are not well regarded, so I have not bothered.

(00:56:08):

We have seen the first movie, that came out a couple years ago. We have not seen the second part yet. I quite liked it. It is pretty close to the book, but they cut- Their choices were to- Sometimes when you are adapting stuff, you change stuff or you can cut stuff. They mostly seem to have cut stuff. Which is fine. A little disappointing, but movies cannot be eight hours long.

(00:56:36):

But yeah, it is cool. Some of it is especially relevant to today. There is some stuff about AI <laugh> that is pretty opinionated, which I quite enjoyed. But without spoiling much, it is set in the year 10,000 something. This is a human society. Earth is mentioned, so it is not just a random part of the galaxy or something like "Star Wars." It clearly references back to earth history and things.

(00:57:08):

But there are very few computers. Because they had a war over computers <laugh>. Because the AIs started causing trouble. People started using AIs for trouble. I think they made that point very clear. It was not the AIs doing things, it was people.

(00:57:26):

If you watch the movie, it is not really well described in the movie at all. But a lot of the technology looks pretty basic. Like the control panel for the ornithopters and stuff has just switches and dials, and analog stuff that looks like it could be from World War II. There is a reason for all of that. It is not because they just wanted to look cool. It was because computers are basically banned.

(00:57:51):

Anyway. I have said things about "Dune." Did you have things to say about "Dune"?

EW (00:57:58):

Having read I think beyond four, but not much beyond four. It definitely had very diminishing returns.

CW (00:58:06):

Yes.

EW (00:58:08):

I have not read them lately. I thought that the last "Dune" movie was very pretty, and very engaging. I know there is another one out, we have not seen yet. I do not know anything about that.

CW (00:58:25):

That is the second half of the book.

EW (00:58:28):

I am looking forward to seeing it. But it is very much with both the first book and the movie. If you do not enjoy the first ten minutes, you can just leave.

CW (00:58:39):

Yes <laugh>, definitely. It is not a happy story.

EW (00:58:42):

It is not a happy story.

CW (00:58:44):

And the hero is not a hero. He tries to make that very clear, and I think people still somehow miss it.

EW (00:58:50):

I think in the movie they did not make it clear that-

CW (00:58:53):

Oh, they did.

EW (00:58:54):

Did they?

CW (00:58:55):

Yeah. From what I have read, in the second movie, they make it even more clear that he is not a hero.

EW (00:59:02):

Okay, so one more-

CW (00:59:04):

<laugh>

EW (00:59:06):

Sorry, there are two more questions, one of which I do not want to answer.

CW (00:59:08):

Okay.

EW (00:59:09):

But this one is from Chris Greenley, "When in your career, did you start feeling like senior engineer roles were the right fit for you?"

CW (00:59:18):

Five years after I was given them. I do not know. I was given the title of "Senior Engineer" four years after starting, which was a bit too soon. I did a lot of senior engineer things at that point, and had a lot of responsibility.

(00:59:35):

But I do not think I did- I do not think the work I was producing- I had not quite figured out how to write code at that level of competency yet. My code was not good. I think my designs were good, and my thoughts about systems were good, but I do not think my coding skills were that good.

EW (00:59:57):

Mine was about four years as well. But that was because I went to a startup, and I was the only firmware engineer.

CW (01:00:05):

Yeah. I also went to a startup. I was one of a few senior software engineers.

EW (01:00:10):

But the startup you went to, you were one of several people like you.

CW (01:00:16):

Yeah, a few.

EW (01:00:18):

The startup I went to-

CW (01:00:19):

You were the only person.

EW (01:00:20):

I was the only one like me, which was like coding without a net. I just remember being jealous of your co-workers. Not that mine were not great, mine were great. My EE and my algorithms guy were both fantastic. But neither one of them-

CW (01:00:37):

There was nobody to bounce ideas off of.

EW (01:00:38):

There was nobody to play with. After that job, I felt like I had definitely earned the senior title. Having put together whole systems-

CW (01:00:48):

That is where-

EW (01:00:50):

And having been responsible for a large subsystem, which you were too.

CW (01:00:53):

Yeah.

EW (01:00:53):

And knowing- And then having to clean up your own messes.

CW (01:00:59):

Yep. Make a lot of mistakes and figure out, "Oh no, you cannot just do it this way <laugh>."

EW (01:01:04):

And you can no longer curse the person who did this to you, because it was you.

CW (01:01:09):

Yeah. Right. When you are building code. That was the thing. It was going from maintenance, mostly maintenance with writing a little bit of new code to, "There is nothing here."

EW (01:01:19):

Yeah. Blank page.

CW (01:01:21):

"We need to do this." Like, "Okay." So I had the title after five years, and I think I probably felt like I could claim the title of in eight to ten.

EW (01:01:36):

I felt that I grew up fast when I was by myself.

CW (01:01:40):

I was close, but somewhere around there. It took a couple years. I went straight to management after that startup job. Anyway, so that was kind of weird.

EW (01:01:51):

You also took time off and got a master's degree.

CW (01:01:56):

I started it during that period, yeah.

EW (01:01:59):

Let us see. Okay, let us go to the question I did not really want to answer. From John C, "No actual experience with Zephyr, but I have seen a lot of progress with POSIX support and Linkable Loadable Extensions. If you ported a very simple terminal text editor and a very simple C compiler to Zephyr, provided some AI wrappers- API wrappers-"

CW (01:02:20):

Thank God! <laugh> I was worried about that for a second.

EW (01:02:22):

"For a standardized file system."

CW (01:02:24):

Yeah.

EW (01:02:24):

"You could bootstrap a single user, more traditional Unix-ish OS in Zephyr on a microcontroller."

CW (01:02:33):

I guess.

EW (01:02:34):

"Obviously the big question is, 'Why would you want to do that?'" Yes. That is the big question.

CW (01:02:38):

<laugh>

EW (01:02:38):

Yes, it is, in fact. "But it has been fun to think about. How feasible do you think this is?" John, you need to see Forth. "Would there be any actual applications of it?" All EEs will tell you that there are many applications of Forth.

CW (01:02:53):

Why are you talking about Forth? <laugh>

EW (01:02:55):

Because it is a microcontroller compiler on shell.

CW (01:02:58):

Yeah.

EW (01:03:00):

No, he is talking about POSIX and-

CW (01:03:04):

You could totally- The difference between modern microcontrollers now, and a general purpose CPU from the late nineties, early two thousands, that you ran Linux on, is the MMU.

EW (01:03:17):

So you just need a Cortex-M4 with an MMU.

CW (01:03:20):

No, they do not have them. They have MPUs.

EW (01:03:23):

Oh, right.

CW (01:03:25):

They do not have any virtual memory. You could run- But there are lots of minimal versions of Unix that ran without MMUs and did stuff. So you could totally do this. Yeah, sure. I am sorry, the answer is "Yes, you could do it. Now you have to figure out why."

EW (01:03:42):

"Obviously the big question is, 'Why you would want to do that?'"

CW (01:03:44):

Yeah.

EW (01:03:46):

Yeah. You could definitely do that. But there are other ways of getting compilers on your system. You said you would reach for Zephyr in the future. I kind of agree with that, but I might also reach for MicroPython in the future.

CW (01:04:00):

Yeah, it depends on what you are doing.

EW (01:04:01):

It depends on who your audience is.

CW (01:04:02):

It depends on how fast I want to get something done.

EW (01:04:05):

Well, MicroPython- If I am working with clients who are not quite sure what they want, MicroPython was really beautiful.

CW (01:04:13):

Yeah, it depends.

EW (01:04:15):

But I do not know why you would want to do what he is doing, but if he did-

CW (01:04:19):

It almost looks like that already, right? The build system looks like Linux. It has got a shell that is decent. I am sure you could port some compiler that ran as a task. I am sure there are file systems available.

EW (01:04:35):

Definitely.

CW (01:04:35):

<laugh> Everything else is available.

EW (01:04:39):

There are file systems in Zephyr.

CW (01:04:43):

Yeah, you can relive recompiling GCC on a 386 in 1990 whatever.

EW (01:04:50):

Wow, that sounds like something.

CW (01:04:52):

Hear your hard drive crying for dear life, as it disassembles itself <laugh>.

EW (01:05:00):

Oh, sorry. Before I just start reading "Winnie the Pooh," I probably should say-

CW (01:05:04):

The show is over.

EW (01:05:04):

The show is over. Thank you to Christopher for co-hosting and for producing. Thank you for listening. Thank you to our Patreon supporters for their support and their questions. If you would like to contact the show, embedded.fm has a contact link. Or you can email the show at show@embedded.fm.

(01:05:27):

All right, now let us see. [Winnie the Pooh excerpt]

(01:06:19):

Let us go with, "When in your career did you fart?" Start. Did you fart? <laugh> "When in your career did you start feeling like a senior engineer?" <laugh> I cannot even get through the question.