340: The Left Bunny Slipper

Some pies are good, but there's a lot of pie that's not good. You have to get the right crust on pie, and if you don't get the right crust on pie, and this varies from person to person, if you don't get the right crust, then it's a hot jam.

EW (00:01:22):

Hot jam. Okay. So that was the easy question.

CW (00:01:27):

And then you got like your key lime pie and that's sort of a cheesecake, so that's sort of straddles the line. So it's hard to answer that question.

EW (00:01:37):

That's a custard, not a cheesecake. Don't you know your cakes?

CW (00:01:41):

No, I don't know my cakes. And then there's puddings, which in America are the gloppy chocolatey stuff, but in Britain is apparently cake. So this is not a well defined question. I don't know how to answer it. I mean, there's hand pies, which are obviously better than cake because you can eat them with your hands, like the McDonald's thing, except those are too hot to eat with your hands, so they come in a little, pool of metal pouches. But then there's hotcakes, are those cakes? Those are pancakes, but are those cakes also? Because you wouldn't want to have a pancake versus pie because they're not in the same category.

EW (00:02:18):

Do you ever walk into the break room and hear what people are saying and then just walk out again?

CW (00:02:23):

Who would have a break room?

EW (00:02:25):

This is it. This is the break room.

CW (00:02:25):

The break room's in my head.

EW (00:02:30):

Okay. So actually before we get to the Patreon-

CW (00:02:32):

I don't feel like we answered that question. You didn't weigh in at all.

EW (00:02:38):

I said cake of course, except for breakfast.

CW (00:02:43):

Now is pumpkin pie a pie?

EW (00:02:47):

Yes.

CW (00:02:48):

Why are things like pumpkin pie a pie, but things like apple pie are also a pie? They seem completely different. I mean, one's a bunch of fruit and one's like nine eggs and a pound of brown sugar.

EW (00:03:04):

You're supposed to put pumpkin in too and also you forgot the cream.

CW (00:03:13):

Okay. Well, we can-

EW (00:03:14):

What is this show called?

CW (00:03:15):

This show is called pie, pie, pie. Pie, pie, pie.

EW (00:03:25):

Can we actually talk about the things we're supposed to talk about?

CW (00:03:29):

It's your show.

EW (00:03:31):

It's your show too. So before we talk about the Patreon questions that we got in Slack, I want to talk about Patreon, not just because it's very nice that people support us with their one and $5 donations every month, which is very, very nice. And we usually use it for microphones, although we have enough now that we have a little bit of extra, which we've been donating to different causes, but we're going to do something new.

CW (00:04:05):

What is it? What is the new thing?

EW (00:04:06):

Transcriptions.

CW (00:04:08):

Transcriptions.

EW (00:04:10):

We're going to get transcriptions done professionally of the show.

CW (00:04:14):

So people are going to be able to see what we say?

EW (00:04:17):

Yes.

CW (00:04:19):

Instead of having to listen to what we say, they can just search for when I say something egregious and then email me with a direct quote of what I said, forcing me to respond to it with things other than, "Well, I can't remember what I said."

EW (00:04:37):

Yes.

CW (00:04:37):

And we're going to pay somebody to do this for us.

EW (00:04:40):

It's actually quite expensive.

CW (00:04:41):

Okay. I'm on board.

EW (00:04:43):

Yeah, that is one of the reasons we haven't done transcriptions in the past, just because I don't really want to know what I said six years ago. I like to pretend we're not the same people at all.

CW (00:04:56):

10 years ago, we haven't been doing it that long.

EW (00:04:59):

I said six, didn't?

CW (00:05:01):

My brain is still stuck on the pie cake thing. I've got at least 30% of my CPU gone for that.

EW (00:05:07):

Well, hopefully this show will get transcribed.

CW (00:05:11):

Oh God.

EW (00:05:12):

So we'll be able to tell whether it was 6 or 10.

CW (00:05:14):

Wait, wait, I'm sorry to the transcribers, but I have to do this. Deoxyribonucleic acid. Antidisestablishmentarianism, superheterodyne-

EW (00:05:27):

Are you done?

CW (00:05:28):

... rutabaga.

EW (00:05:29):

We're very sorry transcribers. We appreciate what you're doing. Are you going to ask, why?

CW (00:05:36):

Why are we doing this?

EW (00:05:38):

I read a book, I know many of my stories start out with, I read a book. I should just skip that part. I read a book by Haben Girma who is the first deaf, blind student to graduate from Harvard Law School. And I mean, that's kind of impressive. Just Harvard Law School, just go with, that's impressive at least in some circles. And I didn't realize how many biases I had about what they call ableism, which I hate all the isms, but let's go with ableism, ableist ideas, ableist biases. Basically, the idea that people who have disabilities life is worse for them and it can't be made better. And the thing with Haven's book, she talks about going to high school and trying to gain her independence. She went to Mali as a high school student to build schools. I never did that. That is so brave. And I have no disabilities, but she wanted to do it. She was excited about doing it. She was proud of having done it. I would have been proud of having done it, except I was to check in to do something like that.

EW (00:07:01):

And through the book, part of it was that she needed the assistance, she needed the technology and not making the technology available really was rude. Rude, isn't the right word. She worked on a case, they went in front of the Supreme court against Scribd because they refused to have screen readers-

CW (00:07:33):

I see.

EW (00:07:34):

... because of course you can use screen readers to copy things.

CW (00:07:36):

Well, Scribd is weird anyway, but yes.

EW (00:07:39):

But they have a huge library and they absolutely refused screen readers. And-

CW (00:07:45):

That would be Scribd? Isn't it. Sorry. Not important.

EW (00:07:49):

Yeah, maybe it's Scribd. Anyway, whatever it is. She talked about how that meant everybody who was blind couldn't get access to that. It was like building a desert, an information desert for people and it just wasn't right. When much of the accessibility features are useful to all of us, curb cuts. We're in the sidewalk, there's a curb and then there's a ramp and they had to fight so hard to get those for wheelchair users. And now, the robots use them and get in the way.

CW (00:08:29):

The robots use them.

EW (00:08:31):

But strollers and deliveries. They were always the right choice, so why did people have to fight so hard? And so having read the book, which I would totally recommend because it was many charming memoirs and the occasional realization that I was on the wrong side of that story. So yeah, I-

CW (00:08:53):

No, it makes sense. And a lot of many, many podcasts do transcriptions, especially the big ones which we are not, but the ones produced by large, publishing-

EW (00:09:06):

Empire. Yeah, all the big ones.

CW (00:09:07):

... the Empires and all. And then they do it for precisely that reason than others, it's this is information that if you're not able to listen is gone, there's no access. So that makes a ton of sense and yeah, I think that's great. And we should do it. I had another thought.

EW (00:09:29):

But not everyone is going to get to read this transcript immediately. So I won't talk about the process a lot, unless that thought comes back.

CW (00:09:36):

Sure, sure. It might come back while you're talking.

EW (00:09:40):

So initially it's only going to be available to Patreon subscribers. That's only initially, it will come out. I just need a little bit of time to iron it all out. The first one I got of Dan Zimmerman's episode about voting, I really, really missed having timestamps. So the next one that we get, this one will have timestamps. Just things I'm learning. So it's going to be an RSS feed that is separate from the RSS feed that is the show feed that all of the overcast and the apps pick up.

CW (00:10:11):

Podcast, iTunes, whatever.

EW (00:10:13):

This would be more like a news RSS feed. It'll be all text with a link or two back to the audio portion of the show. And it's going to go to Patreon and if you think you would truly, truly use the transcription, really want it, email me and I'll send you the link. Initially, I just want fewer people to tell me I'm doing it wrong. So all of the Patreon folks do that. And if it encourages you to join Patreon for us, great, super. Super duper. I think that's it.

CW (00:10:54):

That's it. No, that's something I've been thinking about independently of your book to not transcriptions, but accessibility and a lot of it's through osmosis doing the iOS, Apple development and the Apple frameworks and SDK put a huge emphasis on making accessibility easy. And the excuse for not doing it in an application is very slim because mostly it just takes naming your UI elements with things that could be spoken because the OS actually handles all the, if you're in accessibility mode, it handles all this stuff for you. You can be in a state where you tap on something and it says, "That's the button, just click okay." And then you click it again to click okay or whatever. So the whole interface can be built that way and it's really easy to do in applications. And I expect Android does similar things.

CW (00:11:41):

So I just encourage people, if you're making something, whether it's an embedded device or an application, look into the accessibility options your OS and SDKs give you because they might be not that hard to do. It opens up a lot of possibilities for people who might not otherwise be able to use what you make.

EW (00:12:02):

And then naming your UI element's a good thing anyway.

CW (00:12:06):

Yes. Well, yeah, but-

EW (00:12:09):

I mean, it's an extra step, but it maybe an extra step that's useful.

CW (00:12:11):

... you have to name them something and there might be localization issues that come along with that. So now you're opening up a little bit of a can of worms that will cost you some time and money, but even if you don't localize it, it still helps a lot of people.

EW (00:12:25):

Now, onto other things, questions, we got an email and things we talked about or questions we got in Slack to talk about. Let's see, Mikel noticed that our licenses are wrong on our website. Thank you. I will change that eventually. I promise. This is creative commons, no derivatives, no commercial. I don't know. It's all there. It's just not in the right order-

CW (00:12:58):

It's true. We released the whole thing in Apache.

EW (00:13:01):

... it's just not quite linked properly. So thank you for noticing Mikel. We will fix it.

CW (00:13:06):

I'm so self conscious.

EW (00:13:09):

I know. The reason not to get transcripts is exactly what you said and it's hard. I mean, because we say a lot of sarcastic things and that doesn't come through, sorry, in transcription. It'll be hard, but we shouldn't be self conscious about it. An email, Patrick said he is searching for full-time jobs in the near future and has a question about what to focus on. There are two paradigms of embedded engineering, bare-metal and embedded Linux. Patrick has had a class experiencing both, building real projects in both, but is more inclined towards the latter, the embedded Linux. Should someone hoping to break in the industry be proficient in both or should Patrick focus on embedded Linux if that's what appeals?

CW (00:14:05):

Your silence indicates that I should go first.

EW (00:14:07):

Yes. And I needed a breath.

CW (00:14:11):

So I take issue with the premise. First of all, that there's two paradigms of embedded, bare-metal and embedded Linux. There's a big continuum between those. Bare-metal implies to me that you get a board and you start writing software and you're not using an OS. You're not using anything except maybe the C standard library and CMSIS or something like that. But in between that, as we've talked about in other shows, there's smaller OSs, RTOSs. There's the how, which I don't consider bare-metal if you're using STE stuff. Cube Amex, which is not... I mean is that bare-metal or is that bare-metal with a huge assist? Anyway, there's a continuum there and the expertise you have to have on the below embedded Linux side is wide, depending on what you want to do.

CW (00:15:03):

And we had a discussion about this on the Slack I think yesterday and this morning too, where somebody was asking about is it a good exercise to bring up a board from total scratch and with no assistive software really to do the startup code in C or in assembly and all that stuff, set up the interrupts and all the peripherals and just go from there. Is that a useful skill that everyone who's doing embedded should learn? And I kind of was on the side of, no. You shouldn't have to do that. And yeah, you might learn a few things. You might take you longer to experience elsewhere, but I haven't done that hardly at all on arm, maybe never on arm. In the distant past on x86 and stuff. So the whole bare-metal thing is a bit of a mishmash.

CW (00:15:53):

I don't really feel like you have to make a choice between embedded Linux and not embedded Linux. That's what I'm going to call everything else. A lot of the concepts are the same, especially these days where you have the chips coming from ST and NXP where you've got a cortex and an acer is glued together where you're running Linux on one side and whatever your favorite thing on the cortex is on the other side. And they communicate and they use similar APIs and they even have peripherals that are shared. So I don't feel like there's a big distinction. It's a lot to learn, but I don't feel like you need to be proficient at both. I don't feel like you have to be pressured into learning embedded Linux because you think that's where the industry's going, but I don't think there should be an artificial distinction. Am I making any sense?

EW (00:16:52):

Yeah. Although Patrick likes embedded Linux.

CW (00:16:55):

Well, then do embedded Linux.

EW (00:16:57):

That was what I thought too. Yeah. As somebody who came at it from the bare-metal side and I did... I have written the Buddha vector assembly for processors.

CW (00:17:10):

For a cortex though? For a recent cortex?

EW (00:17:14):

Cortex-M0+ when that chip wasn't quite out yet. So yeah, that was forever ago. There's a lot about embedded Linux I don't know. And if somebody was an embedded Linux, pretty good at it, that's a skillset that there've been times I would have hired that into companies I worked.

CW (00:17:40):

Yeah. Definitely.

EW (00:17:42):

So if that's where you're inclined, having seen both that's enough. You don't have to proficient at both. You just kind of have to be able to tell the difference. And if you like embedded Linux, go there, it won't be hard to go to Zephyr-

CW (00:17:56):

Yeah, Zephyr.

EW (00:17:56):

... or free RTOS or some of the other larger RTEMS style operating systems, which is still a huge class of embedded systems. It may be a little harder to go to the eight bit, but I don't know. I don't think that's going to be around forever.

CW (00:18:19):

Not much longer.

EW (00:18:22):

Even thought they're power efficient sometimes.

CW (00:18:24):

I think maybe the unspoken question here is where's the best place to go to get a job?

EW (00:18:32):

Patrick said expressed interest in the aerospace industry.

CW (00:18:38):

Well, a lot of the satellites and small satellites are all running Linux these days, not even necessarily consider embedded. That's the thing about embedded Linux is it's a lot of times it's just the Linux and the embedded part is just getting your distribution together and getting yanked out of work and-

EW (00:18:57):

Stripping it down, so it doesn't have all the other.

CW (00:19:00):

... putting together the kernel you want. And if you on a weird chip, then you have to do some weird stuff and write drivers and BSP parts, but most of the time you probably don't. So after that, it's kind of Linux.

EW (00:19:13):

It's all good. Just be prepared to learn everything else because eventually you'll have to.

CW (00:19:18):

Yeah, but if you want to pick a focus, nobody should try to focus on everything at once. So if you like embedded Linux, start there and there's definitely plenty of jobs and getting more so because the embedded world is moving up, up, up, up in capabilities.

EW (00:19:37):

Another question from someone named Chris that I don't know. So not my Chris, [inaudible 00:19:42] not Gammell.

CW (00:19:43):

Someone else's Chris.

EW (00:19:44):

Somebody else's Chris. I wanted to ask a question about the embedded industry, as long as we're on that topic, it seems like embedded companies cluster in a few States, is that because the upfront cost of starting a hardware business and finding embedded developers? Are there embedded companies that are not startups or government contractors?

CW (00:20:05):

I think they cluster in a few States just because of historical reasons like Silicon Valley is Silicon Valley because that's where the tech industry was primarily born or where chip manufacturers were and companies clustered around there. And that over time, it just kind of accreted, that is my favorite word on this podcast.

EW (00:20:26):

It really is.

CW (00:20:27):

Same thing with Boston and there was a lot of networking stuff there and bio and there's a few places like that.

EW (00:20:32):

Austin, Texas for TI and North Carolina, I keep hearing things from North Carolina. People are asking for development.

CW (00:20:42):

I don't think it's any more-

EW (00:20:44):

Utah and Nevada. I don't know. I'm about to list all the States.

CW (00:20:47):

Utah and Nevada?

EW (00:20:48):

I'm actually working for a company in Hawaii right now.

CW (00:20:50):

Yeah, but I mean, they're an outlier.

EW (00:20:53):

They are. They're a startup.

CW (00:20:55):

Is that because the upfront cost of starting a hardware business and finding embedded developers? Well, I mean, yes, definitely finding embedded developers. There's more of them wherever there are more of them.

EW (00:21:05):

Tautology.

CW (00:21:07):

The cost, no, I don't think makes it as much of difference now as it used to, especially now that so much is remote, because you're not usually working with a CM that's necessarily nearby, so it really doesn't matter that much.

EW (00:21:20):

That's true. I mean, I work with some CMs in Fremont, which is an hour and a half from here, but I haven't been to any of them in a while. And not just because I haven't been anywhere in a while.

CW (00:21:34):

Second part of the question is a little more interesting. Are there embedded companies that are not startups or government contractors?

EW (00:21:40):

Sure. I mean, Google's got embedded, Microsoft's got embedded.

CW (00:21:43):

All the wearable companies.

EW (00:21:44):

Facebook has embedded.

CW (00:21:46):

Facebook, yeah.

EW (00:21:48):

Fitbit.

CW (00:21:48):

Fitbit. Netflix does not, as far as I know.

EW (00:21:53):

No, I would agree. Netflix, maybe not.

CW (00:21:55):

Apple has a ton of embedded. It's pretty much any company that makes hardware of any kind has embedded development because subsystems of any product are embedded.

EW (00:22:08):

I mean, if you think about cameras.

CW (00:22:11):

Cameras, laptops have tons of embedded. They're just computers, but yeah, they've got tons of little computers in them doing purpose-built little tasks. So yeah, the answer to your question is yes.

EW (00:22:24):

Yes, definitely.

CW (00:22:25):

And if you're interested in embedded and you don't know where to apply, think of those big companies too because they have a lot of stuff.

EW (00:22:37):

I guess we should mention iRobot.

CW (00:22:39):

iRobot? I would consider them all embedded, right?

EW (00:22:42):

Yeah.

CW (00:22:42):

Except for their iPhone app stuff.

EW (00:22:47):

Yeah. And even-

CW (00:22:47):

Which supports their embedded, so it's not... If they didn't have the embedded stuff, they would need the app.

EW (00:22:56):

Sworn enemy asked about-

CW (00:22:59):

Chris Gammell asks.

EW (00:23:02):

... asked about RTOSs and has decided to go with free RTOS to start with, wanted to hear about Gochas for underdeveloped firmware people such as himself, especially trouble shooting steps to get things started. How do I isolate processes as necessary and properly tell where I am in execution? Are there tools? Are there resources?

CW (00:23:27):

No. Nothing's available.

EW (00:23:28):

I don't think so. He's just going to have-

CW (00:23:33):

Next question. No. We actually talked about this on the Slack a bit before I realized it was in the podcast questions section. So I kind of answered them there and some other people did as well. So I think where I would start is getting a fundamental understanding of what an operating system is, what they provide, what some of the terms are. For example, the difference between processes and threads?

EW (00:23:55):

What is the difference between process and threads? Are those still in memory sharing?

CW (00:24:00):

Sort of. I mean, that's a piece of it. A process is kind of a term that came out of like Unix. So it's a complete program, it's running in its own address space under virtual memory. It can't access anything in another process except through inner process communication means which tries to protect each process from the other. They usually have their own set of stacks and heap, everything's in their own memory space. And they run concurrently with other processes on the system although that's just the schedule of flipping things around, they're not necessarily running at the same time, unless you have multiple cores. A thread is a piece of a program that's executing independently and perhaps concurrently, but it's in the same memory space. It's part of your single application or program and it's not protected from any other thread.

CW (00:24:58):

So it can read memory from thread A, thread B can write memory to threat A and they can go back and forth. And the only way to protect between them is with crude things like locks, like, okay, there's this piece of memory over here and I don't want thread A to write it while thread B is trying to write it. So we lock that memory while the operation is happening and then unlock it when it's finished so that nobody steps on each other. Threads have their own stacks, but that's about it for separating memory and the stacks are not separate. I can stomp on somebody else's tech. So knowing the difference between those two things is important because in embedded, 99% of the time, you're on a chip that does not have what's called an MMU, a memory management unit. The memory management unit is a thing that allows you to have processes that have separate memory spaces. You don't have that, you don't have processes. So most RTOSs and things like that are-

EW (00:25:56):

Most smaller RTOSs. The R is not the important, it's the embedded OSs.

CW (00:26:00):

Most embedded operating systems have things called tasks, which are just fancy names for threads. And everything's operating in the same memory space on your chip, and anything can stop anything else. So you don't have protection. So when Chris asked, "How do I isolate processes?" You don't. There's a little caveat to that because the cortex, the M3s and M4s have a thing called an MPU, which allows you to kind of set permissions on blocks of memory, very crudely. So you can kind of statically set up protection between different areas of your program, but it's pretty cumbersome. It does work, but it's nowhere near the dynamic sort of thing that a process with virtual memory would give you. So that's that bit.

CW (00:26:46):

And so that just goes to knowing what certain words mean. Knowing what a scheduler means knowing what the difference between preemptive and cooperative or under completion scheduling around robin, all of those things are important to know because they're going to come up. And if you don't know what they mean, and you set up something as preemptive, it's going to behave way differently from something that's not preemptive. And you might be very confused about how your system is executing. And these are not hard concepts. You could learn most of the terms I think in an afternoon, if you have the right kind of material to learn from.

EW (00:27:25):

Yeah. And it's sometimes hard to find the right material. I often suggests Labrosse's MicroC/OS book, but I'd be careful because the later additions are a lot more complicated. It was the first and second, the early additions, definitely the second edition because that's one I have, that really didn't talk as much about the code as it did about the concepts and that's what you need. Jackie Poo suggested Miro Semex video. It's a whole series and Chris Gammell went away and watched it and said it was really great. So that might be a good way to learn some OS concepts. And there are some things in Chris Campbell's question that are interesting, like how do I isolate not processes, we call them threads now, how do I isolate threads as necessary and tell where I am in execution? So that can be a problem, even if you're running to completion, even if you don't, you have cooperative threading because that's much easier to deal with. It's still a little hard, which thread is running right now can be confusing.

CW (00:28:45):

In a debugger or in what context?

EW (00:28:48):

In whatever context you have. In debugger, you should know the debugger has a stack.

CW (00:28:53):

Debugger tell you which thread you're in and it stays with that thread until it...

EW (00:28:58):

But sometimes the debugger won't tell you why something else isn't running.

CW (00:29:00):

Why something else isn't running. Sure.

EW (00:29:04):

Why am I here and not over there? Is always a question that I wish I could figure out. There are some tools, some compilers have some nice tools that show how operating systems work. I'm mostly thinking about some TI tools which doesn't help most people.

CW (00:29:26):

There's usually a lot of extensions for like GDB, for your particular OS too. I remember we're using ThreadX. We had all kinds of info in GDB. You could do info threads just normally and it would tell you which are running. The problem is switching between threads and debuggers for embedded stuff doesn't tend to-

EW (00:29:46):

No.

CW (00:29:47):

... work like on a Unix system. When I was debugging a multithreaded program, you could switch to a different thread, if you knew it was executing. Maybe it wasn't, maybe it would just be idle, but you just bounce around and it would switch the stack and everything, but embedded tends to stay where it's executing and you can't really look at another threads context, which is always very upsetting because sometimes I needed to look at this thread's local variable which you can get to, but you kind of have to stand on your head or remember what it's called and jump to get the context.

EW (00:30:19):

And that's why sometimes we make all of our variables global. It's so embarrassing.

CW (00:30:25):

They're still accessible, it just depends on if GDB happens to decide that you can look at it because a lot of stuff gets optimized away in weird ways. But other tools, I think there's some, I wouldn't say there's anything you need to buy.

EW (00:30:44):

No, I think the tools are out there and I just haven't used big enough RTOSs recently to do that.

CW (00:30:53):

Printf is fine.

EW (00:30:56):

Printf is usually fine. Although-

CW (00:30:59):

Multithreaded Printf can be a little-

EW (00:31:02):

Multithreaded Printf can be a little exciting.

CW (00:31:02):

So that's another thing I'll add. If you are going to be using-

EW (00:31:06):

Printf, you should flush-

CW (00:31:08):

... threads.

EW (00:31:08):

... threads which you will be in with an RTOS. Usually the RTOS takes care of a lot of the kind of scary things like making sure to turn the FPU state back on and off when it transitions from one thread to another. But there's a lot of stuff like that where the C library, some calls will be, what's known as reentrant and some won't be.

CW (00:31:34):

Dirt hook.

EW (00:31:35):

A lot of the string calls are just a mess. You should stay away from them if you can. But they cause a lot of trouble because you might get interrupted in the middle of some string operation and things go horribly wrong. So you should, if you're making a lot of library calls, especially with strings, you should make sure you look into the reentrancy of various of the calls is the term you want.

CW (00:31:58):

And so there's magic bullet. It is hard. There are GDB tools, FreeRTOS, I believe has some pretty good ones. So we aren't the right people to ask, but definitely go look for them.

EW (00:32:16):

I've done tons of multithreaded in RTOS development, but usually I just stick with GDB and home-brewed statistics and Printf.

CW (00:32:24):

I have in my resume, debugging methodologies, a little section. I have oscilloscope and logic analyzer. I have Printf up there because come on.

EW (00:32:38):

The truth is if you architect your software cleanly, it shouldn't get into, I know it's a huge if.

CW (00:32:49):

Well, if you Printf into a serial port and maybe not Printf whatever your logging function is and you change threads, you do need to flush the serial board because you can get weird things happening.

EW (00:33:06):

You get the quick, brown fox jumped airline 32 over the lazy dog and stuff like that.

CW (00:33:14):

Okay, next question.

EW (00:33:16):

I guess so. Oh wait, no, no, no sworn enemy. He has a new podcast.

CW (00:33:23):

Chris Gammell.

EW (00:33:25):

Right. Chris Gammell has a new podcast. It is called contextual electronics. I wonder how he came up with that name.

CW (00:33:32):

I don't know.

EW (00:33:33):

The name of his class. So people should check it out. Might be cool.

CW (00:33:39):

He's doing video too because it looks like they're going to be talking about, and he's already got three or four episodes up, he'll be talking about specific problems and stuff in detail. They're going to share screens and look at designs. So there's a video component as well.

EW (00:33:54):

That'll be me actually watching people program and debug and-

CW (00:33:58):

Instead of holding up our hands to the microphone and saying it's about this big, and if you look over here, you'll be actually able to point things.

EW (00:34:08):

That'll be me. So in a different area of questioning, circular buffers. What's a circular buffer?

CW (00:34:22):

When you have a car, often the finish gets kind of pitted after you drive a lot and you want to have a good layer of wax or another polymer on top of it. So what a circling buffer does is it kind of rotates around and usually you want to get the random ones because then they don't go in a perfect circle and then you don't get the swirl marks. So you put a little thing over that, and then you put the wax down and you just do that over your car. It polishes and it spreads the wax evenly.

EW (00:34:52):

I had no idea where you're going when you started with a car. In an embedded context, what's a circular buffer?

CW (00:35:01):

A circular buffer is-

EW (00:35:04):

This is a quiz. You wanted to know if I was going to ask you embarrassing questions.

CW (00:35:08):

This is not embarrassing. I just implemented one a couple of months ago. A circular buffer is a thing where you never run out of space.

EW (00:35:20):

That answer is incorrect. Don't use it in your interview.

CW (00:35:23):

A circular buffer is one for where you can put things into it and then another entity takes them out. But as you add things in, if you overrun, it goes around to the beginning again. So you can run out of space, of course, but it overwrites the older data.

EW (00:35:40):

Ideally-

CW (00:35:42):

Ideally, you're keeping up on the other end, but if you're not, then the overrun is something that is not a disaster.

EW (00:35:48):

Overrun sounds bad as opposed to circling around, which is ideally you're running around in circles with somebody else.

CW (00:36:02):

The idea is mainly because you have two entities, threads or an ISR and your main program, and they're both trying to deal with the same pool of data at the same time that's coming in serially.

EW (00:36:14):

So an interrupt is reading data in?

CW (00:36:16):

Usually, you're right.

EW (00:36:18):

Well, ADC interrupt is reading data in.

CW (00:36:21):

Oh, I see what you're saying. Not reading into the buffer, reading it from a sensor.

EW (00:36:24):

Reading it from a sensor and putting it into the circular buffer and then your thread is reading it out and doing interesting things with it. Maybe reading it out one at a time, maybe waiting until there's a block of 10 whatever.

CW (00:36:39):

But there's a latency. The reading out thing is way behind or somewhat behind the thing that's writing it.

EW (00:36:49):

It's at least one point.

CW (00:36:50):

It's at least one behind.

EW (00:36:55):

And so that's the basics of a circular buffer.

CW (00:36:58):

Why would I want to use one?

EW (00:37:01):

Well, let's say you do have an ADC system where you want to read in data and then perform an action on 256 of it, maybe you want to FFT it or something. So you read the interrupts in and you get, okay, here's one, here's one, here's one and you put it in a buffer, but your FFT takes longer than one ADC cycle. Maybe it takes longer than a hundred ADC cycles. So where do you put the new data while you're dealing with this buffer? You could just have another buffer and then that's what we call a ping pong buffer and you go back and forth. Now, what if that somehow fills up? What if you need another one?

CW (00:37:47):

Then you need a third buffer.

EW (00:37:50):

And it's like you can't do a ping, pong, pang buffer, you end up with a circular buffer. That was really a good expression. Anyway, it's better to just go ahead and use a circular buffer if you're going to have data that needs to flow.

CW (00:38:13):

In different rates.

EW (00:38:13):

To flow in and flow out.

CW (00:38:14):

Different rates. I mean, if you can guarantee that what you read is read immediately and dealt with, then you don't need it, but that's pretty rare.

EW (00:38:23):

Yes.

CW (00:38:24):

And you can get a lot of trouble if you assume that that's true and then it subsequently becomes untrue for some reason. Like your main program, you add something to it and now it no longer can keep up 1% of the time and stuff like that.

EW (00:38:38):

So for circular buffer gives you some margin and it's usually, there's some element of it circling around or overrunning or being a circle where the first element is next to the last element. And so when you subtract how many elements you have available to read, then you have to take into account that you may be going over that discontinuity of wrapped, subtract is usually what I call it.

CW (00:39:12):

I have a question. Why are we talking about this?

EW (00:39:17):

Because Jordan asked a question about, well, he asked a question about volatiles and then we ended up at circular buffers.

CW (00:39:22):

Okay.

EW (00:39:25):

It was a good question, but I kept asking, why do you want to know? And then we ended up in circular buffers.

CW (00:39:30):

I just want the listeners to understand why we suddenly started talking about circular buffers. I think again-

EW (00:39:37):

Again-

CW (00:39:38):

I'm sure we've talked about it.

EW (00:39:39):

... I'm sure we've talked about it before, but it's one of those things that's-

CW (00:39:41):

It's really useful.

EW (00:39:42):

... So one of the things I have come across with circular buffers that is fairly mind-bending is if you have multiple pointers into it, maybe you have a right buffer that is writing the data-

CW (00:39:59):

Right pointer.

EW (00:40:00):

... right pointer, sorry. Right pointer that is writing the data from the interrupt and then you have a conditioning pointer that is taking the data from there and slowly a median filtering it or doing some small averaging. And then later you have a different thing and each one is working through the dataset-

CW (00:40:25):

At different rates.

EW (00:40:26):

... at different rates, usually because of different windows or because of the length of time it takes to actually do things or maybe you average it all and then you start to look for spikes or whale songs or whatever you're doing. Okay. I'm sorry, circular buffers. This is not the first time-

CW (00:40:46):

There's a lot of Gochas with them and if you just kind of naively implement one based on what we've said, you'll do it wrong. So definitely read about it. I think your book has a decent section. Very decent, very good, excellent section on it, calling out a lot of the coaches that I'd forgotten when I went to look at implementing one a couple months ago. And some of them are things that are just like, like you said, where you run over, trying to subtract to figure out where you are or how much is left across the discontinuity. Lots of stuff like that. And plus there's some atomicity issues because even though the circular buffer makes it safer for the ISR and your code to use the same-

EW (00:41:32):

Because the ISR is only using the right pointer.

CW (00:41:35):

Right, but there's still other things associated with the circular buffer that are not super safe for them both to be talking about like computing the length or some of the things. I can't remember the specifics of the Gocha, which is why I always have to reread that section, but-

EW (00:41:50):

It's about computing the length because one or the other could be changing that variable.

CW (00:41:54):

While you're-

EW (00:41:55):

While you're...

CW (00:41:56):

... computing it.

EW (00:41:57):

... mid computation. The other thing is if the beginning and the ending point, the same value, do you have zero things in it or do you have one thing?

CW (00:42:10):

Yes, which is always better to just waste-

EW (00:42:13):

It's better to waste.

CW (00:42:15):

... unless you've got a circular buffer of large buffers or something. What a waster we've done.

EW (00:42:22):

Okay. So enough of that. Let's see. We got some really nice emails regarding the show where I interviewed you. Some really nice emails. People liked it, but one person had a question about LISP.

CW (00:42:42):

Yeah. And I didn't really talk in great detail or explain LISP very well.

EW (00:42:49):

Happily, Chris had the same set of questions that I usually have when you talk about it... even though I've heard you talk about it before. Everything I have already has a MAC address, isn't that a unique enough ID? And what about IPv6? How-

CW (00:43:05):

So I'm going to have trouble explaining any of this without an hour to two hours of a show, just talking about it because it's not a simple protocol and it's built on more not simple protocols like TCP/IP and routing and things. So I can try to address some of these, but if you don't understand TCP/IP and how routing works and the internet kind of works, then a lot of this isn't going to make very much sense. So the main question is why was a new protocol LISP necessary? Not strictly necessary, but it solves a lot of problems that are solved with kind of a hodgepodge of other solutions, VPNs, VLANs, things. You don't even know what they are, but LISP solves a whole bunch of problems that a lot of other solutions had been kind of cobbled together to do.

CW (00:44:03):

So to go back. LISP is locator/identifier separation protocol. When you're on the internet, when you're using IP to communicate with devices, you have an IP address and just stick-

EW (00:44:18):

182.168.0-

CW (00:44:18):

... with IPv4, it's a four byte address, and that's assigned usually to a particular network interface. IP is what's called a layer three protocol and TCP/IP is a layered system that goes from the wire all the way up to the application. And there's a different layer for everything's built on top of another layer. And so the IP layer is sort of at the bottom middle, and it's where things are put into packets and then shipped across the network and can be routed. So IP is a routable protocol. That means it can get from your computer, to your router, out of your house, to Comcast network, to the internet backbone, to somebody else's ISP network, to their house router and then to their computer. And there's a whole series of things that have to happen for that to work.

CW (00:45:18):

Usually your IP address is just assigned dynamically. Your cell phone talks to AT&T and AT&T says, Oh, well, right now your 10.1.3.5. And then maybe five minutes later, they decide, "Hey, I need that back. You're 1.2.6." It can change. It can change when you disconnect. It can change while you're using it, usually it doesn't. And within your house, there's this whole other complication called NAT. Usually there's only one IP address for your house and then everything inside it is kind of collapsed to that IP address and there's magic to make that happen. The IP address doesn't identify a thing. It kind of defines loosely a thing for a short period of time in a particular place in the network. So the identity and the location of the network are kind of collapsed into this, not very meaningful identifier.

EW (00:46:14):

Kind of like if you have an address on your RV.

CW (00:46:18):

Yeah, I guess so. Except-

EW (00:46:22):

And if you park your RV in your driveway forever, which I mean, isn't that what most RVs do, then that's fine. Everybody knows how to get to your RV if you have that address. It's probably the same address as your house or maybe-

CW (00:46:36):

But if I moved my RV, nobody knows how to get to my RV based on my license plate without communication.

EW (00:46:41):

The RVs address doesn't make any sense anymore.

CW (00:46:45):

So the idea behind the LISP is to take the location and network part and the identity part and split them and kind of the genius behind it and a mentor of mine is the main inventor of this.

EW (00:47:01):

Dino.

CW (00:47:01):

Dino Farinacci. The genius behind it is keeping all that still within an IP. So you get two new addresses now, one is called the EID, the endpoint ID, and that's the thing that you want to be sticky to your device. And the other thing is called the R lock, the route locator and that's the thing that says this guy is at this position within this network and they're both IP addresses. So how it works is it's almost like a VPN or a tunneling protocol and so you have one IP header inside another IP header, and the outer one has one of the things and the inner one has another. And the routers in between know the outer one says, "Oh, this is for this EID, strip that off. How do I really get to it?" Oh, there's the locator. And so I can look at the locator IP address and I'll route to that until it gets to the other thing. Was not explained well.

EW (00:48:04):

Is this like putting a sticky note in your driveway saying I went to the park?

CW (00:48:07):

Sure. Yes, exactly. Well, yeah.

EW (00:48:10):

And so the EID is the RV address and the sticky note is the-

CW (00:48:14):

The EID is your license plate number and it stays with you, but also you have a GPS, it's constantly reporting your location so that people can find you. And the GPS address, it's not a really GPS address, it's an IP address.

EW (00:48:28):

It's a sticky note in the driveway and it's updating

CW (00:48:30):

Your location is IP address. And so to make this work, there's a whole mapping system. It's kind of like DNS, it's DNS-like. DNS is a similar thing. You have two IP addresses, but you have a name that gives you an IP address. In this case, you have an IP address that gives you an IP address. So if a router sees this person wants to send an IP address or a packet... this person wants to send this packet to EID 1.1.1.1, the router that's running LISP knows that's an EID because it's got this protocol header that identifies it as such. I need to get to route locator block because that's where he is. And so it knows to route that packet and decapsulate it on the other end and then deliver it to 1.1.1.1. A lot more complicated than that. That's the best I can do right now. Especially, since I haven't used it in years and I would need to refresh my memory on some of how the details work of the encapsulation and the mapping system.

CW (00:49:34):

So doesn't the MAC address already uniquely identified a device? No, and that's not the right question. Anyway, because as I said before, TCP/IP is layered, MAC address is not involved in TCP/IP, it's an ethernet address or it's a LTE address or whatever your physical network is, it's an address to operate on that physical network. It doesn't care about IP. It doesn't know about IP. It's assigned usually to a particular piece of hardware, like your ethernet card, dating myself, like your internet port or your wifi or whatever. And they're typically static for some of those things, but there's a lot of overlap. They're designed by vendors. They're kind of is a mishmash because they don't really uniquely identify anything. And even if they did, they don't really apply because you can't route a MAC address.

EW (00:50:37):

We're sitting in the same room, we have MAC addresses that are completely different. And so how would anybody be able to say, well, you should at least send your traffic towards California if you want to get to our laptops?

CW (00:50:51):

It doesn't-

EW (00:50:51):

It doesn't make any sense.

CW (00:50:52):

... it doesn't make any sense. It's like saying, how do I make this analogy? It's like sending the envelope tells you where to go instead of the address you wrote on. They're two different things. One that's not good either. The internet address, the MAC address, I keep saying internet, but the MAC addresses for all kinds of access media. It's how do you talk to something that's local to you, connected on your local wifi network, connected on your ethernet, connected on your cell phone network directly. And so it's the kind of the lowest level of those layers and IP is inside that.

EW (00:51:35):

On top. Inside?

CW (00:51:38):

Logically, if you laid out an entire packet of ethernet packet with all the stuff in it-

EW (00:51:45):

If you looked at all the bits.

CW (00:51:46):

... the first header is the ethernet header and inside that is an IP header. So that ethernet header gets consumed by things on your network. So if I send a packet to Yahoo, dating myself again, it first goes from the wifi on my computer to the wifi on my router. That's done with MAC addresses. My computer knows how to get to my router because it knows its MAC address, but that's the end of the MAC address fun. At that point, the router gets it, take that ethernet header off, throws it in the trash, reencapsulates it with the ethernet for the next router, along the line, the MAC address or whatever layer to address and then sends it. The process of sending an IP packet is encapsulating and decapsulating these layer two things, which are different protocols. Anyway, this is not a good idea.

EW (00:52:49):

I know what gets you to talk about networking enough.

CW (00:52:51):

Am I making a mess of this?

EW (00:52:52):

No, I think it's pretty cool. But I'm really sad they can't see the hand gestures.

CW (00:52:56):

The MAC address doesn't really help and it doesn't help for a whole variety of reasons. Not the least of which is the MAC address is just not involved in wide area networking.

EW (00:53:09):

What about IPv6? I mean, can't we just have everybody have their own unique IP? Isn't there enough addresses in IPv6 to like give an IP address to every star-

CW (00:53:23):

How does this become you asking me more questions again?

EW (00:53:26):

Every grain of sand and some nonsense

CW (00:53:28):

Yes. There's a lot of IP addresses, a lot of IP addresses now. It's great. No. So the problem is-

EW (00:53:35):

Is routing again.

CW (00:53:36):

You got it. It's routing. If you just willy-nilly assign an IP address to everybody randomly-

EW (00:53:43):

You get one.

CW (00:53:44):

... without structure-

EW (00:53:44):

There's one under your chair.

CW (00:53:46):

... your left bunny slipper gets one, your coffee maker gets one, your fork gets one. If they're all just kind of handed out, we got plenty of them. It doesn't matter.

EW (00:53:56):

They're candy.

CW (00:53:58):

The problem is the internet runs on tables, databases, and the internet needs to run fast. So when it gets a packet from who knows where, and it says, "Hey, I'm trying to get to-

EW (00:54:09):

Left bunny slipper.

CW (00:54:10):

... 128 bit address for left bunny slipper, that router has to figure out what the next person to send it to is to get it on its way, which means it has to have a database, a database that can be fast enough, that it can look up that 128 bit address in microseconds to find the next place to go. If you can't aggregate, if there's no structure to IP addresses, you can't aggregate them like saying all of the stuff that starts with 32 is out this interface to this guy. If you can't do that, it just does not scale. You can't make a database that's both, I mean, you could. You could put all the Ram in the universe in one box and have an entry for everything that 128 bits codes for. I leave that as an exercise to the reader, to how large a black hole that Ram would create. You can't do it. So things have to have structures. That's why just ignoring IP and saying, "Let's use MAC addresses everywhere." Doesn't really work. There's no structured MAC addresses either.

EW (00:55:20):

I don't think a lot of people realize there's structured IP addresses. We get so used to seeing-

CW (00:55:25):

Well, it was a big deal. It started with structure, when it was first created, there was each of those bytes and the structure very simple. It was bite by bite. So everyone got a network that either had 256 entries or 65,000 entries or whatever the next one up is, three bytes worth 16 million. I don't know. 21, doesn't matter. It was a later innovation to divide it up bit by bit, so the structure could be even finer grained. And that allowed routing tables to collapse in the '90s to be usable.

EW (00:56:04):

But we forget because we often use the 192 or the 10 networks that those are really internal networks and places like Xerox park has their own top level IP address. That means that they have one of the 256 numbers in the very beginning.

CW (00:56:29):

Have an IP4 address, yes.

EW (00:56:31):

Yes. They have a whole-

CW (00:56:34):

But even they, at this point, are using probably-

EW (00:56:36):

... No, it's probably sold off.

CW (00:56:37):

... private networks and normal structures. There's a lot of IP addresses sitting around unused, but yeah.

EW (00:56:44):

But what that means is that if some router sees that bite in the very first-

CW (00:56:51):

Indoors.

EW (00:56:51):

... spot, it says, "Oh, this is going to a park, so I'm sending it there."

CW (00:56:56):

And there's a lot of interesting things with how the, God, I'm so off topic now, a lot of interesting things with how this works because if you want a fast database to do this, you want to be able to short circuit, like if you start looking at three bits of an IP address and you know after three bits where to go, you stop looking. And so the tree structures to make those are really kind of interesting in how they work with the hardware so that you can just, Oh, okay. These three bits, that's a leaf on this tree that points to this database entry. Go. So you don't have to look at the whole thing.

EW (00:57:29):

That's the goal is to look at as few pieces of information as possible.

CW (00:57:32):

If you have to look at all 128 bits, you're in deep trouble for an IPv6 address, but KBX 81's questions and CIP address, IPv6 address is so huge, theoretically, each device could have several unique IP addresses. You could have a different identifier and stuff. Is it still a thing in that world too? The answer is yes.

EW (00:57:55):

Because of the routing?

CW (00:57:55):

Because of the routing, because there's still no distinction in IPv6 between identifier and locator, it's the same problem. And if you want to solve that problem in the IPv6 world, you got to do it that way too. On top of that, you could do really interesting things with LISP. You can encapsulate IPv4 inside IPv6, you can have an IPv6 EID and an IPv4 locator or vice versa. And we played with that. It was a way to do IPv6 through networks that you couldn't necessarily do it yet. So you could have an endpoint that was using an IPv6 address, traverse an IPv4 network, and then pop it out on the other side and still be IPv6. So there's a lot of games you could play that way too. So no, there's no reason that it doesn't work in IPv6 and lots of reasons to use it.

CW (00:58:47):

Nobody asked really what it was for. There's a lot of different things that can be solved with it. It's kind of a dynamic tunneling protocol. So you can run an overlay private network, so you can have your company's network use your company's IP addresses across the internet using this system. There's security things with that, of course. There was in data centers, oftentimes virtual machines move from server to server, blade to blade within a data center, or maybe from data center to data center, but you want to still be able to access that. And so there's a lot of games they play with IP addresses to try to, okay, this thing moved physically in my network, but I want to keep the same IP address. Solve some of those problems too.

EW (00:59:39):

The one thing that made sense to me, although it wasn't really-

CW (00:59:42):

I should have Dino on to explain this because I probably did a terrible job.

EW (00:59:42):

You should ask Dino to come. One of the things that made sense to me, although it wasn't a target application, was the idea that my phone could be an end point and it could be a server of data. Right now, if I want to-

CW (01:00:03):

Stream video distribute video privately.

EW (01:00:05):

... stream video, I have to somehow hook up to a server, my phone hooks to the server, somebody else hooks to the server and now my phone streams data to the person, but it doesn't, it streams data to the server. If I want a direct connection, they have to be able to reach me. And this LISP would give me my phone, okay, here is the address. You don't have to go through another server.

CW (01:00:28):

And you'd get a permanent address or a semipermanent address. Yeah, what you said was the demo that I helped put together, which we had a phone and another cool thing which I didn't mention is because the ID stays the same and the locator changes, your connections stay up even if your locator changes. So the cool thing we did was a simple little demo streaming video from an Android phone from its camera and then turning off wifi. And then the phone would go to cellular because it lost wifi. The locator changes because you've moved from your local wifi network to Verizon or whatever and they give a new IP address. So the interface changed from wifi and LTE, so the IP address changes, but your EID didn't. And so that TCP connection doing the streaming or RTSP, whatever we were doing, stayed up because the network didn't know, it just sees traffic between this address and this address, and this address didn't change even though the underlying thing did. And so that's kind of one of the things we could do with it too, is it just doesn't matter if you are roaming, your connections will stay up and there's applications for that is that. The network mostly handles those things anyway, but there's definitely applications where you would like to keep a continuous connection without having to reestablish.

EW (01:01:53):

Okay. Well, I think that's it for the show.

CW (01:01:56):

Oh God, thank God.

EW (01:02:00):

You said that last time. Right. Don't just start reading Winnie the Pooh, you have to actually finish the show. I don't have to ask if there are any final thoughts. But I should say thank you to Christopher for producing, co-hosting, being guests and all the other things. Thank you for listening. Thank you to our Patreon supporters for giving us a couple of bucks and letting us do the transcripts and sending mics to guests. It makes it just that much easier to know that you support us enough that you want to keep things like that going. If you would like to be Patreon supporter, it's patreon.com/embedded, or you can go to the embedded.FM website and look for support. And that is not going to get you any support, it's going to give us your support.

EW (01:03:02):

If you would instead like to receive support, you'll have to contact or email us, show at embedded.FM and now some Winnie the Pooh to leave you with. [Buy the book!]

EW (01:04:09):

Embedded is an independently produced radio show that focuses on the many aspects of engineering. It is a production of logical elegance and embedded software consulting company in California. If there are advertisements in the show, we did not put them there and do not receive money from them. At this time, our sponsors are logical, elegance and listeners like you.