484: Collecting My Unhelpful Badge
Transcript from 484: Collecting My Unhelpful Badge with Christopher White and Elecia White.
EW (00:00:07):
Hello and welcome to Embedded. I am Elecia White, and I am here with Christopher White. We are going to chat about the things we chat about.
CW (00:00:17):
The things!
EW (00:00:18):
The things.
CW (00:00:19):
Okay.
EW (00:00:20):
First up, we had a guest recently, Colleen Lewis. We talked about pointers and stuffed animals. I have some follow-up emails about that.
CW (00:00:29):
Okay, cool.
EW (00:00:30):
First of all, the stuffed animals. Nathan, who has been on the show, said that he wanted to stock his physics classroom with stuffed animals, based on the show. But he suggested that they have good names like Sir Isaac NEWTon and AlBEARt Einstein and Stephen HAWKing. So I guess you need specialized stuffed animals for these.
CW (00:00:56):
Okay <laugh>.
EW (00:00:58):
Tom also had a good note on that one, on pointers. Did you see that one?
CW (00:01:05):
Think so, but you have to remind me.
EW (00:01:07):
Tom said, "My problem with pointers was not so much understanding them, as figuring out why they were used so often."
CW (00:01:14):
Oh yes, yes, yes.
EW (00:01:15):
"Took me a while to realize they are used to so much because C in 1979 was such a dumb language."
CW (00:01:23):
<sigh> All right.
EW (00:01:24):
"The first hint was scanf(). It would have been easier if someone in 1979 had confessed, 'These pointers are terrible, and will make endless trouble. But we are going all in, and we will be using them everywhere, whether we need them or not. So get used to it.'"
CW (00:01:37):
I just think people, maybe, I do not know. I remember computers in 1979 were not so great overall, so <laugh> in hindsight, yes, sure.
EW (00:01:49):
Tom continues, "'Pointers are like GOTO statements, but for memory. Forget reasoning about code, device drivers have too much I/O preventing optimization anyway. If you want optimized number crunching, stick with Fortran.'" I actually liked this, because it is true that we do not explain that we use pointers because of optimization.
CW (00:02:17):
Well, C was supposed to be a low level language that was just a step up over assembly. There were many higher level languages that people-
EW (00:02:25):
Already. Yes.
CW (00:02:25):
Already. Yeah. Pascal, and I think Modula-2 existed around that time. And Fortran of course, which had been around for a long time. I do not know what the state of Ada was, I think that came a little later. But there were a lot of higher level languages.
(00:02:38):
C was designed to be a system level programming language. Correct me if I am wrong, but that is my understanding of it. So it came along with being pretty close to riding on top of assembly. And assembly, guess what? Is all pointers, <laugh> all the time. Because it is all about addresses and things like that.
(00:03:00):
If there is anybody to fault, it is taking C and applying it in places maybe it was not intended to, because it was easy to learn to start with compared to- I do not know.
EW (00:03:11):
Have you been learning Rust?
CW (00:03:13):
No. If you are taking the side that C is stupid, then you should be taking the side that Rust is good, I guess. I do not know. Yeah, I think C is definitely long in the tooth, and is misapplied. I totally agree with that.
EW (00:03:35):
It was not so much a denigration of C. Okay, yes, the whole "C was such a dumb language," might have sounded like an insult-
CW (00:03:43):
<laugh>
EW (00:03:44):
But it was the idea that pointers are essentially GOTO-
CW (00:03:49):
All of C essentially a GOTO <laugh>.
EW (00:03:51):
<laugh>
CW (00:03:54):
It is a very unsafe abstraction on top of assembly, as far as I am concerned. Yeah. C++ was supposed to be the evolution of that along from C.
EW (00:04:09):
But C++99[C++98] still had all sorts of pointers.
CW (00:04:16):
99 was 25 years ago.
EW (00:04:18):
Yes, but that is when I learned it, so it cannot have changed much since then.
CW (00:04:22):
<laugh> It is like complaining about C in 1990 and 2005! 2005. Yeah. Anyway. Yes, I am in agreement with most of that stuff.
(00:04:35):
Except I think stuff- Languages and idioms, and other things, people, there is a lot of momentum behind it. Momentum is hard to deal with, compared to when you have the right argument. You might have the right argument, but the momentum of five million deployed applications makes it difficult to switch.
EW (00:04:59):
You are talking about languages, and I am still thinking about teaching pointers. I think Tom's comment-
CW (00:05:04):
Is do not teach pointers.
EW (00:05:05):
Has made me realize that when I teach pointers, I need to make it very clear that they are an optimization to avoid copying data. And that of themselves, they have no real intrinsic value. It is only understanding them as necessary to understand that optimization, which is super, super common.
CW (00:05:29):
I partially agree. I think understanding how memory works is important-
EW (00:05:35):
Yes.
CW (00:05:36):
In a computer, and pointers are how memory works.
EW (00:05:39):
But you could do that with variables, and then build up to pointers as an optimization against copying variable.
CW (00:05:46):
Sure, yeah. You could write C programs without pointers by copying a lot.
EW (00:05:54):
Yeah. Okay, so that was the- I think that is what we talked about before.
(00:06:01):
In my "Hard Faults" talk, which I put up a little while ago and gave at the...
CW (00:06:11):
Some conference.
EW (00:06:12):
Embedded Online Conference. Sorry, I just realized that I may have passed their call for proposals date without submitting a proposal, so I have to look that up. Anyway.
(00:06:25):
"Hard Faults" talk. I mentioned setting aside memory for later use, to put in core dump or stack trace when you are in a hard fault.
CW (00:06:36):
Yes.
EW (00:06:38):
How would you do that? Those are fine words, but.
CW (00:06:43):
I would usually set aside a section in the linker script, and then define the struct that contain the stuff I want that points to that section. Is that the right answer?
EW (00:06:57):
That is totally the right answer. Yes.
CW (00:06:59):
Okay, cool.
EW (00:06:59):
Continue.
CW (00:07:00):
That is it. That is the whole answer.
EW (00:07:02):
That section of the memory should be in its own section in the linker script. It should be part of RAM, but defined at the top as a separate memory section, so that it does not move around.
CW (00:07:15):
If you want to be fancy, you could do it in flash. If you have memory addressable flash.
EW (00:07:20):
You are in a hard fault. You do not want to be playing with flash at that point.
CW (00:07:23):
Okay, fine!
EW (00:07:26):
The other thing is that you need it to not initialize.
CW (00:07:28):
Yes, yes.
EW (00:07:30):
And if you have a bootloader, you do not want the bootloader to have access to that memory, so that it does not erase that memory through its stack or heap or zeroing of data.
CW (00:07:40):
Okay.
EW (00:07:40):
And so usually your noinit memory for this sort of thing is at the top of your RAM, or at the bottom.
CW (00:07:49):
Okay.
EW (00:07:52):
Where "top" or "bottom" could mean the highest address or the lowest address.
CW (00:07:55):
But you wanted a defined location. You are not going to let the linker figure out where to put it.
EW (00:07:59):
Right. Because you want to access it in your code, and you need to be able to do the whole, "Hey, linker. Fill in this variable address for me."
CW (00:08:09):
Yeah.
EW (00:08:11):
Okay. I bring that up because I was talking to someone implementing Memfault. They needed some space set aside for Memfault, which would not get initialized. And so talking through, "How do you set up a linker file?" and "How do you look at memory?"
(00:08:30):
That was the piece I thought was missing was, "Okay, you have 256K of memory and 512K of flash. How are you allocating each piece of this?" If you are just looking at like three linker files to try to figure out where the bootloader is and where everything is, it gets confusing.
(00:08:51):
So it is like, "Draw it out here. Here, put in Excel. It does not matter where you put it. Just make a list of what everything is. Then put them in order, so you can see if anything is overlapping." You did not do that part.
CW (00:09:05):
No, but I am not a linker script adept.
EW (00:09:07):
Ologist?
CW (00:09:08):
Yeah. I try not to go there if I can avoid it. But yes, if you have got- Those are multiple builds, right? You have got a bootloader build, and they all may have different or overlapping or conflicting linker scripts.
EW (00:09:22):
You really do not want them to have conflicting one. That is where it all goes bad.
CW (00:09:25):
Well, it depends, right? <laugh> Maybe you are switching into one thing, and it is going to take the memory that the bootloader was using for something else.
EW (00:09:32):
True.
(00:09:37):
<music> Thank you to Memfault for sponsoring this show. We appreciate their sponsorship and the work that they do.
(00:09:44):
Memfault is a leading embedded device observability platform, that empowers teams to build better IoT products faster. What that means is that if you have just realized that you are going to build five, ten, a hundred, 50,000 units and you need to keep track of them, they will let you create your own dashboard to observe how your system is doing in the field.
(00:10:07):
Memfault gives developers a more scalable and sustainable process. This accelerates the time to market and de-risks product launches. You can cut product costs and deliver more high quality software. Trusted by leading brands such as Bose, Lyft, Logitech, Panasonic, and Augury, Memfault improves the reliability of devices across consumer electronics. And across mission critical systems such as access control, point of sale, energy and healthcare.
(00:10:37):
Thanks again to Memfault for sponsoring this show. Check out memfault.com and the Interrupt blog, which is filled with incredible amounts of information. <music>
(00:10:52):
Okay. From our Embedded Slack- Which some of these other questions have been from. "How do you read code when you are just entering a codebase? Do you have any tips or tools for that?"
CW (00:11:04):
I answered this.
EW (00:11:05):
You did! Do you remember what your answer was?
CW (00:11:07):
I said, "Angrily."
EW (00:11:09):
Yes. "How do you read code when you are just entering a codebase?" And his answer was, "Angrily."
CW (00:11:16):
Yeah. It greatly depends on- Well, it kind of depends on the reasoning that you are going into it. Right? Are you going in to fix- You have never seen this codebase before, but you have got to fix a bug. Or you have never seen these codebase before, but now you own it.
EW (00:11:35):
That one. And you may have a hundred outstanding bugs, but you do not really understand those either.
CW (00:11:42):
Right. Yeah. What I actually do?
EW (00:11:51):
<laugh> No, no. Let us go with what you think other people should do first. <laugh> Ideal case.
CW (00:11:58):
It depends on what it is for. Do I have the right IDE, first of all. Do I have all the right tools to-
EW (00:12:04):
So you want to build it first?
CW (00:12:05):
I do not know that I want to build it first. But I want to see it the way that the person who wrote it saw it.
EW (00:12:11):
And if they are not using VSCode?
CW (00:12:13):
Use whatever they used, if there is a project structure.
EW (00:12:16):
Oh no, I would do that. But I would also have VSCode.
CW (00:12:19):
Sure, that is fine.
EW (00:12:20):
Up next to it.
CW (00:12:20):
Yeah, yeah. But I want to see if there is a layout or something that is only visible- If it is totally flat in VSCode, that is not helpful. If it is something structured that is in a project file format where things are divided up, that makes it a little easier to navigate the first time.
EW (00:12:35):
<sings> "It makes me crazy."
CW (00:12:37):
Because IDEs do that, and they do not think about it. Anyway. Yeah. Then I want to get a sense for how things are organized. You can sometimes look at main, but it depends on how the project has been written. Sometimes main is just nothing, and it goes and does something else. Or if it is a C++ project, main will instantiate a single object, and then nothing. <laugh> You cannot see where it starts.
EW (00:13:02):
Well, an RTOS's may just fire off a bunch of threads, and main does nothing.
CW (00:13:06):
Yeah, exactly. If nobody is giving you any documentation, first of all, that is the red flag.
EW (00:13:15):
Well, I do this with open source software sometimes, where I want to get an idea of what it is they are doing.
CW (00:13:22):
But what you said. If it is an RTOS, then first, A, figure out is this using an RTOS? Because that is going to change how you approach it, right? You are going to look for something that defines the tasks, and then maybe you are going to write out, "Okay, here are the five tasks that I see. Here is the order they are launched." And then you can go look into those, and see what they do.
(00:13:41):
But I do not have a real prescription because it is different for every project. Every project is so different that you cannot really just say, "Here is a prescription for how to break this code down," that is going to work every time.
EW (00:13:57):
But I have a prescription here.
CW (00:13:59):
Okay. All right. Sorry.
EW (00:14:03):
Although I guess I did change it later and say, "The first step, the zero step, is to figure out what it is the software does."
CW (00:14:14):
Okay, sure. But I would hope that...
EW (00:14:17):
If I started at a new company, there are times that I do not know what the software is supposed to do.
CW (00:14:25):
But you at least know what the product is supposed to do, or the thing. Right?
EW (00:14:30):
I know what the whole system is supposed to do, but I may not know what this is supposed to do.
CW (00:14:34):
All right, fair enough.
EW (00:14:36):
So I have three different strategies. For me, I just switch back and forth between them as they become too frustrating. I tend to use VSCode because it is easier to search and follow paths and go up and down. But I agree with you that you need to build it.
CW (00:14:54):
I never said you needed to build it.
EW (00:14:56):
Oh, that you need to be in the environment they are in. I do think you need to build it. You do not have to build it, but it sure makes it a lot easier. Anyway.
(00:15:03):
Treat the device as a black box. Or the firmware as a black box if you have the schematic. Make block diagrams for what must be there, based on the peripherals and displays and the actuators and how it works. So basically-
CW (00:15:17):
You are describing the system before looking at the code.
EW (00:15:22):
Yeah.
CW (00:15:22):
Mm-hmm. Okay.
EW (00:15:22):
And if you have a UI, make a little state machine, not necessarily the level of each character, but generally how it works.
CW (00:15:31):
Is this not just setting you up for disappointment though?
EW (00:15:35):
Um. It is funny because my next sentence here is, "Sketch out the idea, how you would put it together." "Now go see what you got wrong," was how I wrote it initially, and then I changed it, "Now go see what you got right." <laugh>
CW (00:15:47):
<laugh> All right. Glass is half full.
EW (00:15:49):
Right. But yeah, what should be there? Think about the system as it could be.
CW (00:15:55):
Yeah. I guess I do that, but I probably do it in my head.
EW (00:15:59):
There is some value to actually sketching it out, because it does give you the opportunity to think of more things.
CW (00:16:06):
Yeah. I guess I usually sketch it out as I go through the code, not before as a black box exercise. So no, that is a good idea.
EW (00:16:13):
I am switching back and forth.
CW (00:16:15):
Yeah. This is a good idea. No, I am just thinking out loud about how I approach things.
EW (00:16:19):
"Look at the file tree. Try to find interesting ones, like the algorithms and sensors. Try to avoid standard hardware abstraction layers."
CW (00:16:29):
Yes.
EW (00:16:30):
"Try to focus on the code your company has written, or is otherwise product specific. It is easy to get lost."
CW (00:16:37):
"Do not go down the Zephyr tree. Do not look in there." <laugh>
EW (00:16:41):
"Stay away from the Zephyr tree." And then the final one was, "Go to main. Skim the initialization, and focus on the while loop. Walk through, ideally even with a debugger, but maybe just mentally. Walk through what the device does. If it has threads, look at the loop in each thread. Again, initialization is really important, but if you want to understand the code, you are better off trying to figure out what the code as a whole is doing."
(00:17:06):
There were a lot of people who said that building is the actual first step. I can see that, because you do kind of want to be able to step through the code, but if you are new to a company, you do not always have all your tools ready.
CW (00:17:21):
Yeah. Or hardware and stuff. Those are all good. That is much better than mine. It is hard for me to answer this, because the number of times I have come in and had to own something has been just a couple, maybe three times. As opposed to coming into an established shipping thing and, like I said, fixing bugs or owning a small piece of it.
(00:17:53):
Having to understand all of it was probably a useful exercise, that I should have done more often. But sometimes that was not even practical. Like Cisco. I am not going to understand the entire Cisco codebase for a router. Because that is like understanding Microsoft Windows level of complexity and volume.
(00:18:11):
But for smaller things like a small embedded thing, that is more practical, where even if you are not responsible for everything, it is probably worth looking at everything.
(00:18:23):
But I remember one client we both worked on last year. Or two years ago? Sometime. I do not think I looked at all their code, because I was- They said, "Write these drivers." And so I hooked into what they had, and wrote some drivers. But I never bothered to see how the whole system did everything.
EW (00:18:44):
Yeah, I did some of that, and then regretted it. Then looked around and then kind of regretted that.
CW (00:18:49):
Well.
EW (00:18:51):
<laugh> It is all just regret.
CW (00:18:52):
Well, it is about efficiency too, if you know you are not going to need to understand the entire system. And that depends on a well architected system too.
EW (00:19:00):
Right. We trusted the architect for that.
CW (00:19:02):
So if the system is not well architected, then things that you might not think impact the part you are working on, might have their tendrils in it causing all kinds of problems.
(00:19:12):
But if it is well architected, if it is using an RTOS, and things are in tasks, and it is like, "This is your task," and you communicate with these other tasks via messages, then you do not necessarily need to comprehend the whole thing.
EW (00:19:24):
When I did look around, I went to the architect and said, "This seems odd." And he had great answers. So I was not really disappointed, more I just wanted more out of the system.
CW (00:19:38):
Sure. Yeah.
EW (00:19:38):
Tom Anderson recommended profiling.
CW (00:19:44):
That is something I do not do much.
EW (00:19:47):
Because the long spots might be the most interesting part.
CW (00:19:55):
Yeah. Especially statistical profile that just tells you, "These are the functions that are getting called." The top ten functions in the running system. Those are the ones to pay attention to.
EW (00:20:05):
"Draw block diagram. Annotate the hotspots from profiling."
CW (00:20:08):
And there are tools, right? Are there not tools that will...
EW (00:20:12):
Yeah, Benny suggested-
CW (00:20:14):
Please do not say ChatGPT.
EW (00:20:16):
"Sourcetrail and Source Insight for code review. OpenGrok can index codebases." I actually do not even know what that means.
CW (00:20:26):
I do not know.
EW (00:20:27):
"This was useful for large projects like the Linux kernel and Android." So that is large codebases to index. Okay.
CW (00:20:34):
Yeah, that is another example. If you are going to work on Linux, do not comprehend the Linux kernel. <laugh>
EW (00:20:40):
<laugh>
CW (00:20:40):
A, you cannot.
EW (00:20:42):
Well, you can comprehend the architecture of it, and then focus on your piece.
CW (00:20:45):
I do not know if that is true. Maybe. <laugh>
EW (00:20:48):
Depends on how high level you go.
CW (00:20:50):
Yeah, yeah.
EW (00:20:53):
Tom also suggests, "Removing unreachable code, normalizing the file headers, reformatting with tools-"
CW (00:20:59):
No, no. Now you are taking actions. So this is-
EW (00:21:01):
"Create an exquisitely detailed set of instructions for how to build."
CW (00:21:04):
This is actions. This is not your first foray in. This is when you are-
EW (00:21:08):
Well, he does say- He goes even to creating a release, but then he says, "Do not tell anyone what you did, unless you want that to be your job."
CW (00:21:16):
Yeah, okay. Well, hmm. Yeah.
EW (00:21:20):
"Build locally and start poking at things." "Remove or fix the lies in the comments." Yeah, that is satisfying, Bailey.
CW (00:21:31):
<laugh> These are things upon taking ownership, that you want to change. Yeah, yeah. That is all fine.
EW (00:21:37):
It is funny because there are two things I think about with that question. There is our problem, which is we see a lot of codebases all the time. Our goal is to be as effective as possible and to not try to get too far into what the client customer is doing, because they have hired us for a job.
CW (00:21:59):
Nobody is paying us to do that list of things that they just mentioned, which is why I push back on it. Because in my circumstance, if I were to go through a new client's codebase and correct all their comments or remove all the unreachable code or whatever, they would say, "What are you doing?"
EW (00:22:14):
Right.
CW (00:22:15):
"Why are you charging me for this? I did not tell you to do this."
EW (00:22:19):
But going through their code and creating a few documents and some block diagrams, yeah, that is actually something I do often because clients tend to enjoy that.
(00:22:29):
Okay, what is next in our list? Okay, another quiz for you, since you did very well on how to set aside memory.
CW (00:22:42):
I got a B minus, but that is fine. Go ahead.
EW (00:22:44):
What is "semihosting," and how do you set it up?
CW (00:22:47):
Semihosting is where you need to put a data center on a truck and drive it around, because you are afraid that somebody is going to hack into it physically. So you put all your data center stuff on the truck it with satellite uplinks, and you drive it around the country in an unmarked truck...
EW (00:23:08):
And then you have a roving car. In case there are murder mysteries, you can drive the car into the truck. And the car is AI, ChatGPT.
CW (00:23:20):
<laugh> You just ruined "Knight Rider" for me. He was a cool AI! He did not make dumb mistakes.
EW (00:23:27):
<laugh>
CW (00:23:27):
He had a cool British accent that instilled confidence, and a quiet determination.
(00:23:33):
Semihosting is a hacky method of getting data in and out of your system, if you are precluded from doing so by a UART or a network or Bluetooth or blinking lights or screeching ultrasonic tones, and all you have got-
EW (00:23:54):
The ultrasonic tones. How did I forget that?
CW (00:23:56):
All you have got is SWD or JTAG or your programmer. What semihostings does, far as I understand it- I have only set it up a couple of times. Is it puts basically breakpoints in your code, and the debugger says, "Hey, that is a breakpoint that says this." You can send data out during that breakpoint, and then it resumes communication.
(00:24:22):
So basically you can set up your code and you can replace printf and stuff with the semihosted version that outputs- That does this breakpoint dance with some assembly, and then spits the stuff out to your programmer. And your programmer outputs it in your debug session.
EW (00:24:38):
Like your GDB session?
CW (00:24:39):
Yeah. You can do some more sophisticated things, like get access to files and stuff like that. If you want to get really fancy, you can make a whole console interface that way. But it is cool.
EW (00:24:51):
Because it goes both ways.
CW (00:24:52):
Yeah. Yeah.
EW (00:24:53):
It is kind of like a UART in your JTAG, or a UART in your programmer.
CW (00:24:58):
Yeah.
EW (00:24:58):
But it does have this problem that you are doing a lot of breakpoints, and it can be slow for that reason. One way to use it is to put a bunch of data into a big block of RAM, and then after some event or some command you dump all of that, because it does have such an impact on your loop.
CW (00:25:22):
Yeah. But it is handy in a pinch, especially if you have got- Sometimes you have only one UART, and you need to do two things. Or you are debugging the UART <laugh> code. Or something is just not working.
(00:25:36):
That was when it was useful for me, was a couple of times where, gosh- I was actually setting up UART and making a console happen. Things were not working right and I could not figure out. It was useful to- There were other reasons I could not just breakpoint and debug there. I needed some output. But you can do it that way.
EW (00:25:56):
I set it up in STM32CubeIDE. I had to use OpenOCD, because it was not part of the ST-LINK GDB implementation.
CW (00:26:07):
Oh really?
EW (00:26:10):
It was actually really ugly, because I had to make code changes and I had to make configuration changes to the debugger. But because I knew that the speed was important, I ended up making a whole nother configuration for each.
CW (00:26:26):
Oh. Okay.
EW (00:26:27):
It was just awful. And going back and forth was...
CW (00:26:30):
For just printf, you can usually- When I am doing stuff I have a bunch of debug macros defined, and then you can swap the implementation with a ifdef. But you still need to...
EW (00:26:45):
This project had gone from a printf debug on an Arduino to an STM32. The client had never heard of semihosting, had put in a USB port that they did not use, and did not realize that they would have been much happier if they had put in a UART port and just used an FTDI cable.
(00:27:10):
So any printf was a good thing. Of course, then they said, "Can you make it run faster?" which is the code that is waiting for me upstairs. Which I say, "Not if you want new lines, because STM32 IDE and blah, blah-blah."
(00:27:25):
But as part of this I found an application note from ST, the "microcontroller debug toolbox." It talks about development tools like ST-LINK probes, and other debugger probes, and some of their other IDEs. But also it talks about this semihosting, and printf via UART, which is one we usually use.
(00:27:57):
Printf via SWO/SWV, which is kind of the same but not quite. It seemed to use fewer interrupts but still used the SWD lines. It felt like it was sniffing your debugger, which, I mean, fine. Anyway, I will link that, because it was interesting to have a microcontroller vendor have this application note on TIM debug tools.
(00:28:33):
It was funny that I needed to explain semihosting multiple times this week. It is like there were bootloaders, and I had a whole bunch of people to talk to about that. And then last week it was semihosting. Maybe this week is about linker files, because I like linker files.
CW (00:28:49):
I do not.
EW (00:28:50):
I am finding that I like being able to explain them.
CW (00:28:57):
Yeah. I do not like their format. I think they are not-
EW (00:29:00):
Oh, no. It is a language that is worse than C.
CW (00:29:03):
Human readable or understandable. Yeah.
EW (00:29:06):
Okay. From Sahil, there is this comment, "Maybe this is just relevant to my current situation, but I am curious at your thoughts on master's degrees for embedded software, and how you recommend people build up their skills tree."
(00:29:23):
There is some discussion of the skills tree I put together with MakerQueen, Steph, which is just a list of embedded skills with cute little icons. I think there are only a couple of spelling errors still.
(00:29:40):
Do you have any thoughts on how do you build up your skills in embedded? This is a hard question, because it is like, "Well, first you go back to 1990 something."
CW (00:29:55):
Yeah, applying to myself, that is not going to be useful for anybody. The original question was master's degree?
EW (00:30:02):
Sure. Let us go with that.
CW (00:30:03):
I do not know.
EW (00:30:05):
It is like you being in a PhD department in physics, and then us finding out that all of the PhD people in physics we knew were software engineers.
CW (00:30:17):
This is going to sound weird, but I do not know what a master's in embedded development would look like. A master's degree, it is not quite a PhD of course, but you are still doing research. I guess you can do a project. I do not know, I just do not think a master's degree is like industry. I guess they are. I do not know.
EW (00:30:38):
You have a master's degree.
CW (00:30:39):
I have master, but not in computers.
EW (00:30:41):
In physics.
CW (00:30:43):
Yeah. So it is not really applicable or comparable. It is certainly valuable from an employer standpoint to see that. I think you can pick up skills doing that. Certainly. Especially if you pick a research project that you do your thesis on something that is going to force you to learn some things and do some development on your own. I just do not know what a curriculum, a master's curriculum in embedded development, would be. A lot of EE stuff, I could see that.
EW (00:31:15):
It is only going to be a one or two year program.
CW (00:31:17):
It is not going to be CS stuff, because that is wildly not applicable most of the time.
EW (00:31:23):
Depends on what your original degree was in. Original degree was EE, then CS stuff makes sense. If your original degree was CS, then...
CW (00:31:30):
Sort of, but I still think most of the CS stuff is not applicable.
EW (00:31:34):
You do not think so? So many times I look at code and I am like, "This could be solved with data abstraction."
CW (00:31:43):
Yeah. I do not know. I am probably a bad person to answer this. I am not thinking about the CS courses that the CS department taught when we were there. They are not applicable partly because they are out of date.
EW (00:31:58):
My AI class spent so long on Boolean logic.
CW (00:32:02):
Well, I mean...
EW (00:32:03):
I got so good at just De Morgan's law in everything, that I could use it in the cafeteria when people were making bad arguments.
CW (00:32:10):
But a lot of the teaching was in Scheme or Lisp or C++, which was new-ish at that time. There was no C. All the C I learned, I learned working for the CS department writing utilities.
EW (00:32:23):
<laugh> Yeah.
CW (00:32:29):
To answer this question, I would have to look and see what a modern- I guess it is a master's in computer science with a focus on embedded. I do not know what that looks like. So I would have to see what they were teaching them.
EW (00:32:42):
A master's in robotics would be fine. <laugh>
CW (00:32:44):
Yeah. Yeah. That would be fine. But that going to be a lot of EE and motion control. I guess I would say...
EW (00:32:51):
Motion control or navigation systems. Any of the algorithms.
CW (00:32:55):
Yeah. It is going to be an EE heavy thing, where you are kind of on the edge and learning how to interface with physical systems or other things. I would not go in and get a master's degree in CS and focus on AI or any of the more theoretical stuff.
EW (00:33:11):
I do not know, because there is so much inference we do now, that having an AI-
CW (00:33:14):
Yeah, but embedded developers are not going to do anything, except take a model from somebody down the street and stick it on an NPU.
EW (00:33:21):
And yet employers would be thrilled to get an AI trained embedded systems engineer. I do not know why. I just know that for a limited time-
CW (00:33:31):
Well, yeah, if you are trying to game your resume, that is a separate issue from trying to learn skills. I am very muddled about this, because I do not actually have a good sense for what that degree would entail. A master's degree is supposed to be two years. So figure what you can do in two years of intensive study, and is that the right use of your time?
EW (00:33:56):
I do not think it is a bad use.
CW (00:33:57):
It is not a bad use, but it is expensive. There is opportunity cost, because you are not getting paid to work.
EW (00:34:02):
Right.
CW (00:34:04):
That is two years of prime employment ladder climbing, slash...
EW (00:34:10):
But it may work out later, because a master's degree is slightly more valuable, but not hugely more valuable.
CW (00:34:16):
I have not found that to be particularly true.
EW (00:34:18):
Yeah, because- Yeah. Okay. I do not know.
CW (00:34:25):
Yeah, it is not a satisfying answer.
EW (00:34:27):
I have not found it to be anti true. I have not found it to be that a master's degree is less useful than being in industry for those two years.
CW (00:34:34):
Really? According to- Okay.
EW (00:34:37):
It is just a lot of times I have been asked- I have seen job things that want master's degrees.
CW (00:34:43):
That they want master's degrees? Okay. For development jobs?
EW (00:34:46):
Yeah.
CW (00:34:47):
Okay. I have not.
EW (00:34:47):
But also for higher level IC in architecture jobs.
CW (00:34:52):
Okay. But that is someplace, yes. If you are doing digital design, you should have probably an advanced degree.
EW (00:35:00):
No, IC in architecture embedded software jobs.
CW (00:35:05):
Designing ICs?
EW (00:35:06):
No, no. "Individual contributor." I am so sorry. I have been looking at hiring stuff, and got into the jargon.
CW (00:35:16):
Ah. Okay. Yeah. Sorry. Yeah. So yes, of course, right. I do not know, I have not been looking that much.
EW (00:35:22):
Individual contributor, as tech lead or engineer beyond developer.
CW (00:35:28):
But are they looking for- Yeah. Okay.
EW (00:35:31):
Just masters. It was not even masters in our area of application.
CW (00:35:36):
Yeah. I do not know.
EW (00:35:38):
Okay.
CW (00:35:39):
Just do not do what I did and take seven years.
EW (00:35:42):
You were working for at least half that time.
CW (00:35:45):
Working for probably 75 or 80% of that time <laugh>.
EW (00:35:49):
And you chose a field that you hated. So that was unclear.
CW (00:35:52):
Did not hate it while I was doing it. I hated it before. I enjoyed it.
EW (00:35:58):
<laugh> Peter asked about how to move up the skills tree if you get stuck. My answer to that is, "Jump to another branch."
CW (00:36:06):
Yeah.
EW (00:36:09):
It is not a ladder, it is a tree. So if you are stuck here, just go around.
CW (00:36:14):
Especially since- As an analogy for other things, like when you get stuck practicing an instrument or something, the common advice is to stop on what you are doing, and do something else and then come back to that. Because sometimes you just need a break. Sometimes you need other-
EW (00:36:35):
Different perspective?
CW (00:36:36):
A different perspective or insight.
EW (00:36:37):
You find a new tool.
CW (00:36:39):
And sometimes the thing you were working on is not that useful, that you are stuck on.
EW (00:36:46):
Yeah. Peter also asked if we had thoughts and speculations about the new RP2350.
CW (00:36:52):
No, not really. I have seen it and read about it. I think there are some interesting parts on it.
EW (00:36:57):
My thoughts and speculation are all about how their documentation is going to go from the best documentation ever, to same as everyone else's.
CW (00:37:07):
Why? Just because they have too many variants?
EW (00:37:09):
The variants are very far apart. They are named closely.
CW (00:37:16):
Well, it depends on- Hmm.
EW (00:37:17):
They are going to want to put the same API, but it is not going to fit exactly the same.
CW (00:37:22):
Maybe not.
EW (00:37:23):
So you are going to get good documentation, and then you are going to end up having something that does not quite work for your version of it. This is product proliferation. You have one product, you can do a really, really good job explaining it. Once you have two, you are not going to put twice as much effort into it.
CW (00:37:44):
Yeah. But I do not know that they are going to continue maintaining all of them all the time.
EW (00:37:50):
I think the first one is worth maintaining, at least for a good long time. I do not feel like this was a replacement. It is much bigger.
CW (00:38:00):
I do not know. It has some interesting things. I think it comes down to, "Do you need more power? Do you want to play with RISC-V? And do you need that fancy serial communications thing, that can do-"
EW (00:38:14):
The whole FPGA thing?
CW (00:38:16):
No, no, no. They have a new interface that is high-speed serial, that you can make do things like high-speed video, and other stuff. Is there an FPGA block too?
EW (00:38:28):
That high-speed serial thing was-
CW (00:38:29):
Has some programmable I/O. Because they have their whole PIO thing and that is the same.
EW (00:38:34):
Well that is one of the things that I am talking about, is going to want to have the same API. Because people are going to think those are similar, but they are going to behave so drastically different. The same API is going to make it worse instead of better.
CW (00:38:47):
I do not know. That is a peripheral, that is not part of the- So they could probably have just taken that same peripheral.
EW (00:38:56):
No, no. The 2350s is huge and massive, instead of-
CW (00:39:02):
The what? Oh, the PIO stuff?
EW (00:39:04):
Well, the high speed-
CW (00:39:06):
That is different. That is a new peripheral.
EW (00:39:08):
Okay.
CW (00:39:09):
The PIO stuff is the programmable I/Os. So you could have a little assembly language thing, you could make it write UARTs and stuff in that. I think that is the same, but I do not know, maybe they changed it.
EW (00:39:22):
Okay. Oh, I have those two in the wrong order. I was not sure if we had covered this one last time you and I talked. Simon said they hear complaints like, "The microcontroller was chosen without any consultation with the firmware team." The question is, "How should EEs or component selectors go about choosing parts, if they do not have immediate access to or the ability to question the firmware team?"
(00:39:54):
"Are there obvious things to avoid, like cursed IDEs, zombie docs after a corporate merger that no longer work, or weird peripherals that are easy to spot from a distance, given you probably do not have time to test drive every part under consideration?"
(00:40:07):
It is funny, I was going to read this as, "Are there obvious things to avoid, like cursed IDEs <whisper> MPLAB, zombie docs with corporate merger that no longer work <whisper> Atmel?"
CW (00:40:18):
<laugh> I do not think we have answered this.
EW (00:40:21):
Yeah. <sigh> MPLAB. It just came up this week, and- Oh! Man! MPLAB. Augh!
CW (00:40:30):
I am going to give a high level answer that I think works for most cases. Assuming you have a choice of parts that fit your application, find the one with the largest community online talking about it.
EW (00:40:49):
I had that "microchip Madness," and one of the criteria was, "Search for the part number, and whoever had the most hits wins," as one of the-
CW (00:40:59):
And make sure that they are not, "This part lights on fire," and there are a million complaints about the part lighting on fire.
(00:41:05):
But if you are choosing a part and there is not a lot of communication about it, there is not an active forum, or it is very new and not a lot of people have experience with it, you are going to have a bad time. Your firmware engineers are going to have a bad time. And that is a red flag.
EW (00:41:26):
If they want you to sign an NDA to even look at the data sheet, your firmware engineers are likely to hate you. Because that means they cannot talk about it.
CW (00:41:34):
That is not to say you cannot choose that part, but there has to be a really good reason. Like, this is the only part that fits our application and fits our power profile and does the thing we want to do.
EW (00:41:44):
Evaluate the risk.
CW (00:41:45):
So you are going to have to think about it. But I have seen EEs just pick stuff because that works and that is it, with no consideration for development afterward. And then it is a real problem where people come in and it is, "Well, I have used this PIC for 34 years, and it has worked for me for 34 years."
(00:42:07):
But it is also completely out of date, and people want modern features or something like that. "Oh, I want it to drive a high resolution display and do Bluetooth." "Well, not with a PIC16, you are not." But that is a wild example.
(00:42:23):
So I think having an active community is good, because your firmware engineers are going to want to find information about it. All these parts are different. They all have quirks. They all work poorly in some corners.
(00:42:36):
Being able to talk with other engineers, without revealing the company's secrets or whatever, about, "Oh, this is happening," or communicate with the company's engineers, is really important. If they have a stellar support department and you can somehow find that out, that can replace that. But-
EW (00:42:53):
It is hard though. It is so much easier to go read what other people's problems are, than to try to formulate your whole thing, and then try to get a minimal code set.
CW (00:43:03):
The other thing I will say along those lines is- Well, it is two things. They are kind of related. How much example code is out there?
EW (00:43:13):
Right. Search for this part on Git. On GitHub, sorry.
CW (00:43:15):
How much example code is out there? And, is it well implemented in Zephyr?
EW (00:43:19):
And if it is, run! No, wait, no. No, Zephyr is good. Zephyr is good.
CW (00:43:24):
That gives you an indication how popular it is, and what support is, whether people are going to be able to find how to use it, and issues like that.
(00:43:30):
I think the hardest times I have had is where it is a weird chip. That came up at Fitbit, where we were using-
EW (00:43:39):
Yeah.
CW (00:43:39):
We were using an ARM, a Cortex, but it was from a strange manufacturer. Okay, not a- It was from a manufacturer you have all heard of-
EW (00:43:48):
But not from making chips.
CW (00:43:49):
But not from making chips. It had some funky peripherals. One of the funky peripherals was a 2D graphics engine, which was pretty cool and unusual, at least at the time for a Cortex-M4 SoC. It could do a lot of stuff. But everything to figure out how that worked was in communication under NDA with those folks.
EW (00:44:14):
With a 24 hour turnaround time.
CW (00:44:16):
Longer. Sometimes weeks. Or in their documentation, which was challenging <laugh>. Took probably ten times as long to do things, than if we had chosen something else that was maybe a little less capable, and rolled our own solution for a few things.
EW (00:44:38):
Okay, so let me summarize some of these. We do not want NDAs for the documentation, because that means that we cannot talk to other people about it. And there is a good chance we want to talk to other people about it.
(00:44:51):
We want a community, where possible, and that can be somewhat determined by looking at just how large the community is. You do not need to dig into how good it is.
(00:45:03):
And we want not strange things.
CW (00:45:07):
And that not strange things- You can find a good community, like ST for example, just to beat on ST. They have a huge forum.
EW (00:45:15):
Huge!
CW (00:45:16):
And they have a lot of people talking all the time about all their parts. But you could still pick a weird ST part that nobody is talking about on their forum, and they get into trouble. You have to use a little care there and not just say, "Oh, nobody ever got fired for choosing ST," but then choose the STM32XL37 8-core-
EW (00:45:34):
It is not even a real part.
CW (00:45:39):
_MP-M44, that is not even a Cortex chip. It has got an A-core in there for some reason, and only runs Linux.
EW (00:45:47):
<laugh> A-cores. Yeah. I really am not happy with MPLAB. Even though you can export everything, use GCC and VSCode- Sometimes you want to go back and make sure that you are working within the constraints of the vendor. And wow, they have made that very challenging.
(00:46:14):
Their documentation, it is like you can tell if this was written before Atmel was purchased or after. Before, it makes a lot of sense, it just does not apply to your chip. And after it just, "What? Why are you telling me this?"
CW (00:46:34):
What were some of the other questions? Posts? You had some other...
EW (00:46:39):
"Weird peripherals that are easy to spot from a distance."
CW (00:46:41):
Yes. Well, I do not know about weird peripherals. Certainly if there is a peripheral that excites you, that always makes me nervous.
EW (00:46:51):
<laugh>
CW (00:46:53):
Like a GPU, or a neural processing unit, or image processing block for cameras. All those things are generally- Oh, and USB. Careful with USB.
EW (00:47:07):
USB is the worst. <laugh>
CW (00:47:09):
All those things are generally IP, that the company either bought and integrated-
EW (00:47:15):
And does not really support.
CW (00:47:16):
They do not really understand it very well. Or there is going to be less example code. Or- It is going to be just stuff people are not talking about. But of course, if you need that stuff, you need that stuff, and you have to deal with it.
EW (00:47:26):
But do you need it on the chip? Or can you buy a specialized chip that will go between you and whatever? And not need to have-
CW (00:47:33):
Yeah. Is there another solution? That is the thing is sometimes you have got these chips that solve all your problems, and that often pushes all of your problems to the firmware.
EW (00:47:45):
<laugh> Solved all my problems!
CW (00:47:46):
Because the Homer car of chips might have all of these things, but they might be just not as good as if you had pieced together a system from a multi-chip solution. Is, I guess, the way to say that.
EW (00:48:02):
Yeah.
CW (00:48:03):
It is hard. It is hard. I know EEs get mad every time the firmware says, "Why did you choose this?" And some high percent of the time, it is the situation I am talking about, where firmware did not exist, they come in and stuff is already done and it is just a silly choice of chip.
(00:48:23):
It was not even a chip like, "We need this." It is like, "Well, this is what I used in the past. It is fine." So that is a different situation than, "I cannot work with this." It is that this was not the best choice.
EW (00:48:34):
Oh, right. No.
CW (00:48:35):
This is not the more modern choice that gives you lower power, or a better Bluetooth stack, or access to Zephyr or whatever. It is just you chose this weird old chip, because you used it before. That is a completely different situation. So yeah, definitely look around and see what is current, I would say too.
EW (00:48:54):
Maybe consider listing your needed features in your system. Because again, some of your features can go on to your microcontroller, but might also be better in a separate chip.
(00:49:08):
Like, if you need high speed ADCs, is that going to be part of your microcontroller or not? In the beginning, it does not matter. It can go either way. Now if you are pressed for cost or power or space-
CW (00:49:21):
Well, then there is low noise-
EW (00:49:24):
Or low noise or location where the sensors are, there are all kinds of reasons you would get pushed one way or another. But choose the hardest parts of your system to design around first. In the end, how good the UART driver is in the hardware abstraction layer does not matter.
CW (00:49:45):
Right. Yeah.
EW (00:49:46):
It matters if you truly need high-speed USB, do not get a chip that has never done high-speed USB.
CW (00:49:54):
Yeah.
EW (00:49:56):
Okay. Anyway, I hope that answers your question, Simon. Or gets us fired from all of our contracts. One of the two.
CW (00:50:05):
All the contracts I referred to are in the past, so they can fire me in the past.
EW (00:50:11):
Retroactively?
CW (00:50:12):
Yeah, retroactively fired.
EW (00:50:13):
Okay. We have Guyrandy's typical question, "I recall an episode where you talked about installing an antenna to talk to your dad. How is that going?"
CW (00:50:25):
No progress at this time.
EW (00:50:28):
I recall that about four months ago, three months ago, you started a six month intensive drum course.
CW (00:50:34):
Yes.
EW (00:50:34):
How is that going?
CW (00:50:35):
Which is exactly a year long. It is going well. It is very challenging. It is breaking down my understanding of my fundamentals of playing. It is embarrassing me. Some of the things I cannot do.
EW (00:50:49):
But you have noticed actual progress in some areas?
CW (00:50:53):
I think so. It is hard, because I am focused on exercises and not playing stuff. But yeah, I am trying to practice an hour, to an hour and a half a day, five or six times a week, which is a lot for a person of my age and joints. <laugh> Yeah, no, it is good.
(00:51:14):
Yeah, it is a good course. It is about some fundamentals of stuff, that when you try to learn on your own it is hard. There are a lot of books and things, but it helps to have an instructor. And this course has an instructor who I can meet with a couple times a week, and ask questions of, who reviews video of my practicing and gives me feedback-
EW (00:51:34):
And who cares about instruction, as opposed to is a drummer trying to make enough money to survive.
CW (00:51:38):
How to teach is a big piece of his thing, and the pedagogy. Trying to find the things that have the greatest impact, and focus on those. And not maybe the more common exercises and things that happen in drumming, that people sit at with a pad for 15 hours and do not feel much progress.
(00:51:56):
So he is focused on how can we break down to exercises that have the most impact in the shortest amount of time. A lot of those are focused on things like proper timekeeping, dynamics, and just clean playing and stuff like that. They are very hard.
(00:52:17):
Some of them I have spent weeks on to get anywhere. But that is stuff I would not have done without at least an instructor online occasionally checking in and saying, "That looks fine," or "This does not." Because I would give up because these are so painful and challenging. So it is helpful.
EW (00:52:37):
And you have people around you who are going through some of the same-
CW (00:52:40):
Yeah, there are other students who I can talk to.
EW (00:52:42):
And you are not all in the same chapter all the time, but you do get to see what their exercises look like.
CW (00:52:48):
They are all variants of things advanced, so we all know what we are talking about.
(00:52:51):
The ham radio stuff, no. My dad got Covid back in April, and that slowed things down a little bit.
EW (00:53:03):
He is doing okay now.
CW (00:53:04):
Yeah, he is trying to figure out an antenna outside. He needs a better antenna than he had, which means putting one up outside. But he lives in a place where lightning is actually a thing a lot. So that makes the whole antenna thing a lot more complicated, from a safety perspective. So it is a pain.
(00:53:21):
I am trying to keep up with Morse code, by putting myself to sleep by doing the alphabet in my head. But I probably need to get back onto learning Morse code better. <laugh>
EW (00:53:32):
Do you just go through the alphabet, or do you choose things to spell?
CW (00:53:36):
I choose both.
EW (00:53:38):
That is how I got better at the international phonetic alphabet. I would spell things from my day.
CW (00:53:44):
Yeah.
EW (00:53:46):
Okay. Talking about these things, do you ever feel like you are collecting badges?
CW (00:53:53):
No.
EW (00:53:55):
Okay.
CW (00:53:56):
Sorry? For like?
EW (00:53:58):
Advanced drum play.
CW (00:54:01):
I was an actual Boy Scout. I collected badges.
EW (00:54:03):
Were you?
CW (00:54:03):
Yeah.
EW (00:54:03):
I never knew.
CW (00:54:05):
Yeah.
EW (00:54:06):
How far did you get?
CW (00:54:07):
Not far. I was what they called "in the Webelos."
EW (00:54:12):
Webelos.
CW (00:54:12):
Yeah. They were like the one below Boy Scout.
EW (00:54:16):
Webelos. You push them and they do not fall down?
CW (00:54:18):
Exactly. I think it goes, Webelos and then Boy Scouts and then Eagle Scouts or whatever. Yeah, it was a small troop and we did not do very much.
EW (00:54:29):
Did you get any badges?
CW (00:54:30):
I think so. I do not remember what they were for.
EW (00:54:33):
So this had a huge impact on your life?
CW (00:54:36):
No. It did not inspire me to continue collecting or making up fake badges for myself. No. I did not know what badge I would- You cannot just say "advanced drumming." First of all, I am not advanced. Second of all-
EW (00:54:50):
Okay. Drumming 220 words per minute. Wait, no, that is two different badges.
CW (00:54:57):
That is Morse code. <laugh>
EW (00:54:58):
<laugh>
CW (00:54:58):
220 words per minute for Morse code would be-
EW (00:55:00):
Wow!
CW (00:55:00):
Really impressive, actually. You could be-
EW (00:55:04):
220 beats per minute is pretty fast.
CW (00:55:07):
220 beats per minute is also extremely fast.
EW (00:55:10):
Oh. All right.
CW (00:55:11):
A normal pop song is like a hundred to 120 beats per minute, maybe up to 150 if there is a real bang, fast song. 220 is double time.
EW (00:55:24):
Well, you have the double-
CW (00:55:26):
Double time, not-
EW (00:55:30):
<laugh> I am not being helpful. I am collecting my "Unhelpful" badge. I already have a whole collection, but I am hoping for the advanced version.
CW (00:55:37):
Will you kill that moth?
EW (00:55:39):
No.
CW (00:55:40):
Oh.
EW (00:55:40):
Oh, there it is. Sure.
CW (00:55:44):
Scintillating podcasting. Anyway, do you have anything else?
EW (00:55:50):
Well, you were going to ask me about my collection of badges.
CW (00:55:53):
I am sorry. I am sorry. I got distracted by the moth. Are you collecting any badges?
EW (00:55:59):
It is funny. I saw a book many years ago about collecting badges and women. I do not know, it was maybe "The Well-Rounded Woman's Guide to Badges in Life." I do not know.
CW (00:56:16):
<laugh> What? The first part was okay, but that did not sound-
EW (00:56:26):
It was kind of a try everything sort of book. It was kind of ridiculous, in that it gave you little badges. And honestly, I did not buy the book. This is the book that has most influence in me that I have never purchased.
CW (00:56:41):
Okay.
EW (00:56:46):
<laugh> I think NaNoWriMo was part of that, that I did- Not now NaNoWriMo, which is in so much trouble. But when I did NaNoWriMo many years ago, it was part of that. It was a, "Okay. Yeah. I am just going to try this, to see how it works."
(00:57:09):
So now I am working to collect my "Have an Art Show" badge. So if anybody is in the Aptos area in November, on a day when the Aptos Library is open, you can see my art show.
CW (00:57:31):
Cool.
EW (00:57:31):
Which is entirely self-sponsored, self-produced, self-installed. But!
CW (00:57:39):
It is in somebody else's building. So I think it counts.
EW (00:57:41):
It is in somebody else's building.
CW (00:57:41):
Definitely counts.
EW (00:57:42):
And if they hate it, they can ask me to remove it. And afterwards I will collect a badge of my own making, probably out of origami.
(00:57:52):
Okay. Should I say thank you and everything?
CW (00:57:57):
I think you should say thank you and everything. Yeah.
EW (00:57:58):
Okay. Thank you to Christopher for co-hosting and producing.
CW (00:58:04):
Okay.
EW (00:58:05):
Thank you to JoJo the dog for being quiet and adorable, and only trying to steal my soul with her eyes once. Thank Memfault for their support of the show. We really appreciate it.
(00:58:15):
Thank you to our Patreon listener Slack group for their questions. You can join the Slack group or just email questions. You can join the Slack group by supporting us on Patreon, under the support tab on the website embedded.fm, which is where the show notes will be. And thank you for listening.
CW (00:58:38):
And they can also go to Ko-fi <various pronunciations>.
EW (00:58:42):
Ko-fi. It is also on support thing online.
CW (00:58:44):
Yeah, yeah.
EW (00:58:44):
Last time I read a lot because I wanted to finish the chapter. So now we have a new chapter of "Winnie the Pooh." Sorry, I have to switch glasses because... Old. Okay.
(00:58:57):
[Winnie the Pooh excerpt]