Transcript from 387: Bucket of Spiders with Elecia White, and Christopher White.

EW (00:06):

Welcome to Embedded. I am Elecia White, here with Christopher White. This week it's going to be just us talking about, well, talking mostly about my homework.

CW (00:20):

Okay. It's been a while. I feel like we haven't done this in a while.

EW (00:24):

Well, we haven't had a new show in a while either.

CW (00:27):

I know.

EW (00:27):

We got a complaint recently.

CW (00:28):

A complaint? Oh, good. I like complaints. I'm good at complaints.

EW (00:32):

Apparently there's not enough origami on the show.

CW (00:35):

Not enough origami on the show. Okay. I'm not sure if a podcast is the right medium for origami, but -

EW (00:42):

And then you fold it 45 degrees.

CW (00:45):

Yeah.

EW (00:46):

Yeah.

CW (00:47):

And then look! A swan. Imagine a swan.

EW (00:52):

I mean, part of the problem is that I haven't been doing as much origami. I mean, I've been doing origami that's not my own pattern, so I don't talk as much about that.

CW (01:03):

Well, I mean, that's how a lot of people's projects work though. You could still talk about it. Not everyone's doing something from total scratch.

EW (01:11):

Okay. Well, I made a bug last week, on Sunday. And Chris was walking by, and I said, "Did you see the bug?" And then he screamed.

CW (01:25):

I didn't scream. It was more of a "Aah!" A scream is longer. It was more of a... -

EW (01:33):

Yelp?

CW (01:33):

A yelp. Yes.

EW (01:36):

Okay.

CW (01:36):

Yes, very good. But the moment you pointed at it and said, "Did you see the bug," and I knew it was origami, it fell over and it was ten feet away from you. So it was a miracle of timing.

EW (01:49):

It was a beetle with super long horns. I think he was just afraid.

CW (01:52):

I don't think I yelped. I think I said, "It moved. Did you see it move?"

EW (01:55):

No. There was some definite -

CW (01:59):

There was a yelp?

EW (01:59):

- exclamation-type actions.

CW (02:02):

Alright. Well, my reputation as a non-bug-afraid person has been ruined.

EW (02:10):

You still have to take out the spiders though.

CW (02:13):

That makes it sound like there's a bucket of spiders I need to take out somewhere. Is there something I don't know?

EW (02:21):

I did rearrange my office. There were some things in there.

CW (02:25):

It's Wednesday. It's time to take out the spiders. They pick them up at Thursday morning.

EW (02:30):

I also haven't been doing origami because I had jury duty.

CW (02:33):

Yeah. It was a thrilling experience for everyone.

EW (02:38):

Jury duty here is, normally, you get sent a letter, and then you have to call in every day, or maybe you have to go in for a couple hours, maybe at the worst you have to go in for two days.

CW (02:50):

And they dismiss you.

EW (02:51):

And then they dismiss you.

CW (02:52):

Yeah.

EW (02:52):

And even you were on a trial?

CW (02:54):

I was -

EW (02:55):

How long were you on the -

CW (02:56):

Well, so -

EW (02:57):

You had two.

CW (02:58):

Well, it depends on what we're calling here. So Santa Cruz is here, but -

EW (03:02):

In San Jose, you had to.

CW (03:03):

In San Jose, which is Santa Clara county, I served on one full trial as a juror, which was probably a week or so. And then I served on another as an alternate, which meant I had to sit as if I were a juror, but then I didn't get to deliberate. That was also a week or so.

EW (03:20):

Really?

CW (03:20):

...Maybe longer. I can't remember. They were a while ago.

EW (03:23):

See, I don't remember them being that long.

CW (03:25):

They were definitely at least a week.

EW (03:27):

Oh, okay.

CW (03:27):

Because they do the voir dire thing, which takes at least a day, a day and a half. And then there's openings...Yeah, they were at least a week, those. And then there was a third one, which I was in danger of being impaneled on, but...there was a plea deal moments before they started jury selection.

CW (03:47):

So in that one, I think I sat around for most of a day, and then they hauled us into the court. And then the judge lectured us for...an hour, even though we were dismissed.

CW (04:01):

Because she wanted us to know all about this interesting case, which I didn't care to hear any of the details about, because it was actually quite horrible. Anyway, so yeah, I've been on two and a tenth.

EW (04:13):

So I've been on the jury selection process before, the voir dire, where they ask you questions, and they try to kick you out of the jury if you know too much about it, or you've been influenced by news, or something.

CW (04:25):

Or you're just not the right kind of person...they envision making the decision. Yes.

EW (04:30):

I always don't want to be the right kind of person, but I ended up on the jury, juror number five, for a two-week case, not including jury selection. So it was more like two-and-a-half.

CW (04:43):

Plus there was a week break?

EW (04:45):

There was a week break too.

CW (04:46):

In between jury selection. And so, yeah, it was weird.

EW (04:49):

And it was in-person. And I mean, everybody wore masks when we were in the building, but it was eh. And it's funny, when you're going through jury selection process, they say, "Have you ever been on a jury?" And you say, "Yes." And they say, "Well, how was it?" And everybody answers, "It was interesting."

EW (05:15):

Sometimes there's a pause. "It was, hmm, interesting." It was not interesting. It was awful. It was exhausting. It was emotionally difficult. It was time-consuming. It was time-wasting. Oh, my God, the amount of time spent doing AV, just figure out how to switch screens...God, it was awful.

CW (05:40):

So you're not hyped up on your civic duty.

EW (05:44):

I mean, I admit parts of it were interesting.

CW (05:46):

Yeah.

EW (05:47):

But parts of were just catastrophically irritating.

CW (05:52):

Yeah, that was kind of my impression, was...a lot of TV, wow, we're really off topic. A lot of TV is NCIS, and SVU, and all these cop procedural shows and stuff, and courtroom dramas. And after serving on a couple juries, well, that's as realistic as Star Wars. It's all fantasy.

CW (06:16):

And the lawyers aren't as smart as they seem on TV, and they make all kinds of mistakes, and they're not good at public speaking. And there's a lot of wasted time, and cops lie, and yeah, it's eye-opening. It's good to have gone through, I think. But yeah.

EW (06:34):

The thing that was most eye-opening to me is something you've always said, which is, if you are a pulled over, well, not necessarily pulled over by a traffic cop, but if something happens, don't say anything.

CW (06:51):

Yeah.

EW (06:51):

Don't talk to the police. And I admit, I mean, the defendant, if she had said nothing, she probably would've gotten off. She was convicted mainly by her words.

EW (07:01):

But we have this thing called Miranda rights, which we see in the procedurals. "You have the right to remain silent. Anything you say can and will be used against you in a court of law."

CW (07:11):

"You have the right to an attorney. If you cannot afford an attorney,...one will be appointed", something like that. Yeah.

EW (07:16):

But they tell you that when they arrest you, because that's when they're supposed to read the Miranda rights.

EW (07:23):

What they don't tell you is that now, with body-worn cameras and cameras inside the police cars with microphones they wear, everything you say, from the moment you talk to them is being recorded, can and will be used against you in a court of law.

CW (07:40):

Yeah. Yeah.

EW (07:40):

And so that was really eye-opening. I mean, it was good, because we saw the raw footage. We didn't have to deal with the reports.

CW (07:56):

So-and-so said this, and, yeah.

EW (07:56):

And I mean, we saw a lot of the reports too, but we had all the raw footage of as soon as the officers got there...I was really surprised at how that whole "you have the right to remain silent" thing really doesn't apply anymore. I mean, you still have the right to remain silent.

CW (08:12):

You have the rights. It's that they don't have to tell you about your rights -

EW (08:15):

Yeah.

CW (08:16):

- in a timely manner. Yeah.

EW (08:18):

So I have been very cranky because of jury duty. Because I was completely stressed out by it. And...I have so much work. I should not have taken on the third client, which -

CW (08:31):

And the fourth client, boy, that was a real mistake.

EW (08:36):

No. Yeah, the third client I'll get to talk about next month, but -

CW (08:40):

That's not so much a client.

EW (08:42):

Well, that's true.

CW (08:43):

It's a project of your own.

EW (08:44):

It's a project I'm doing for a company though.

CW (08:48):

Yeah, but -

EW (08:48):

But it's not my usual type of thing.

CW (08:52):

It's your thing.

EW (08:53):

Yeah.

CW (08:53):

...It's not a product that somebody has asked you to work on. It's somebody's -

EW (09:00):

Something else.

CW (09:00):

Yeah. So, that's cool.

EW (09:04):

So yeah, I shouldn't have accepted that, because I had too much work to do this summer. And then I took a forced and stressful two-and-a-half week vacation that...led me to much demotivation just because I felt -

CW (09:19):

Tired.

EW (09:20):

- too much.

CW (09:21):

Yeah.

EW (09:22):

Yeah. We have gotten some good guest suggestions in the mail recently, and I'll be following up -

CW (09:27):

In the mail.

EW (09:27):

In the emails. Sorry.

CW (09:30):

That sounded funny.

EW (09:30):

I'll be following up soon on those, because I am a little behind on my scheduling. Okay. So now there are three topics.

CW (09:40):

Three topics.

EW (09:42):

Four.

CW (09:42):

Four topics.

EW (09:44):

Five.

CW (09:44):

I'm going to say there's seven topics?

EW (09:47):

Probably.

CW (09:47):

Okay.

EW (09:48):

How is your Kickstarter going?

CW (09:51):

The Kickstarter is over. And I'm shipping records, and I'm doing it myself. So I'm going to hopefully finish up shipping everybody's stuff this weekend, or at the latest, early next weekend. Early next week, not next weekend.

EW (10:10):

You did this thing called BackerKit.

CW (10:12):

Yeah. Yeah...So with Kickstarter the fulfillment thing is always sort of nebulous. And it wasn't clear to me what Kickstarter did for you, and going through it, kind of figured that out over time. There's a lot that needs to be done when you're finished, because you have some number of people who've ordered things.

CW (10:34):

And they've all ordered different amounts of things, maybe one record and some stickers, or...a different variant of a record and something else, maybe one record and nothing, so that everybody's ship list is different.

CW (10:49):

There's everybody's addresses. And so at the end of it, you don't have any of that information, like addresses and stuff. You just know the country, and the person's name, and their email.

EW (10:59):

So even though I've told Kickstarter at my address -

CW (11:00):

They have not shared that with anybody. Yeah.

EW (11:03):

Okay.

CW (11:03):

So in order to get that you have to do surveys, which entails sending an email to everyone and saying, "Please tell me your stuff."...So that's one thing that would be difficult to manage on your own. Kickstarter will help with that...

CW (11:16):

And then there's mundane stuff, like sending everybody a digital download code, or getting everybody's address and buying postage for everything, dealing with tracking numbers. So what BackerKit does is, for a percentage of your take, they will manage a lot of that stuff for you automatically.

CW (11:42):

So they import everything from Kickstarter. They know who all your backers are. They send the surveys out. They automatically remind people if they haven't responded to the surveys. Yes, I know it's annoying. I'm sorry. And they manage everybody's stuff. And then at the end you can buy postage for every grouping of different items.

CW (12:02):

And then...you just print that label out and slap it on there. And then they will send the tracking stuff. So it handles all this stuff that would be a nightmare of spreadsheets and mistake making for somebody to do manually.

EW (12:14):

Mail merging.

CW (12:14):

And the other kind of cool feature is, they kind of had a last minute changes feature where you could say, "Add stickers," or something if you decided at the end, "Oh, I really did want stickers," or, "I really did want a different record," or "I ordered an additional one."

CW (12:29):

Which was kind of nice, because we got enough out of that to actually pay the BackerKit fee.

EW (12:34):

Nice.

CW (12:35):

So it made it totally worth it to do it, because it was paid for by last minute additions from people, and it saved us a huge amount of time...It was kind of tricky...I don't know if it could have been made easier. There was some weird stuff.

CW (12:49):

I really wanted to send the download codes before people paid us, because I promised, and there was no good way to do that with BackerKit. So I ended up doing that manually. But then BackerKit sent them out again, once people did pay, -

EW (13:05):

And everybody clicked on them again.

CW (13:05):

- and people were like, "My code doesn't work." I was like, "Well, you already used it." And it was like, "Oh, that's the code for the album, not the -" Yeah. Anyway, so there's stuff like that.

CW (13:16):

And it was tricky, because the add-ons were tricky. And...there were things that could've been made easier that took hours of time for me to figure out, but far fewer hours than sitting in front of Google Sheets...We didn't have that many people. It was 100 people or so. So...it didn't seem unmanageable, but -

EW (13:37):

You could have done it all yourself.

CW (13:37):

I could have done it all myself, but it would have been painful. And just using the USPS website to buy postage, that would have taken forever. So, yeah.

EW (13:49):

So it's going. And you've been complaining a lot about CAN in Python. Do you want to share your pain or just whimper?

CW (14:03):

I can share my pain. I don't know that anybody will be able to fix it for me. So here's what I'm doing. I have a device. I can say what it is. It's a radar, and it's an automotive radar. And the way it reports things is it has a different CAN message. I'm assuming some knowledge of CAN for this discussion.

CW (14:24):

So I'm not going to explain it all again. It sends a different CAN message ID for every track. So if it's tracking 64 objects, there'll be 64 different CAN messages. You're looking at me funny.

EW (14:38):

No.

CW (14:39):

Oh, okay. There's 64 different CAN messages, one for each track, and those come in sequence. But then after that sequence, there's ten additional messages that are packed with additional information about each track. And inside each one is seven or eight, seven or eight, wait, it's ten messages -

EW (15:02):

It's okay. You can totally lie. It's not -

CW (15:03):

There's some amount that divides evenly, of additional information about each track, packed into each of those ten messages. And the last one just has one for the remainder, because it doesn't divide evenly. And those come right after.

CW (15:18):

So I...have some other information coming in in a timely manner and I want them to be synchronized. And so I'm using that other information with non-CAN to drive the read of the CAN information. So I get the other information, and then I say, "Okay, time to go look for my tracks."

CW (15:35):

And so I sit and wait for the first track to come around, because it's continuously sending, and I don't want to start in the middle. So once I get that first track, I do a continuous read using the socket interface to get all of my messages.

CW (15:51):

What's happening is...that periodically, and often enough to be completely annoying, I'll get to, say, track number 40, and then I'll get track number zero again.

CW (16:03):

And then it'll start over from there. And at that point, the entire sequence is botched. And I can't do anything with the data that was corresponding to the other thing that I downloaded.

EW (16:14):

Do you get socket errors?

CW (16:16):

No, there's no errors. If I look at the CAN interface statistics in Linux, there's zero errors of any kind. It's all perfect. I had some inefficiencies in my code that I removed.

CW (16:31):

And so now it's just doing a tight loop with receivefrom, gathering all the messages contiguously, and then I process them all later, which is the right thing to do if you're trying to do something in a timely manner. But even then it's still happening pretty frequently.

CW (16:45):

The weird thing is there's a candump command on Linux that just spits out hex of everything it's getting on the CLI. If I use that this never happens. So it's not that the radar is actually deciding, "Hey, I'm going to drop a bunch of stuff and start over."

CW (17:04):

It's not that the CAN interface is dropping a bunch of stuff and I'm just missing it. It's got to be something like my code just goes out to lunch for long enough that it's missing a bunch of transactions.

EW (17:17):

Your other asynchronous input, if you just say that's always there and don't pay attention to whether or not it really is, can you keep up with the sockets?

CW (17:29):

That was the next thing I was going to try, was to just take that out and read continuously. But I took a lot of that out, so I'm...triggering, triggering?

EW (17:42):

Triggering? Triggering? That word now sounds wrong.

CW (17:43):

...Triggering. I'm triggering off of a signal I get for that thing, and then I execute a callback function that I provide it. And I took out everything related to that in the callback function. So it was saving a file before doing the CAN read. I took all that out. So that doesn't work.

CW (18:03):

So the next thing is to just take that thing out entirely, the additional information, and just do a tight loop, and see -

EW (18:08):

Yeah.

CW (18:08):

- if it ever happens. But then, okay, say it goes away. Well, that's informative. It doesn't make a lot of sense...and doesn't really lead me to a way that I can solve the problem because I -

EW (18:22):

Sure it does.

CW (18:22):

Well, I don't want two things running simultaneously, because synchronizing the timestamps will be very difficult.

EW (18:32):

But it will be more precise.

CW (18:34):

I don't care about precision. I just don't want it to stop. The thing that's frustrating is I'm not doing anything hard. It's ten lines of code. It's nothing. And it can't keep up. And this is not a slow processor. It's a Xavier. I've got it cranked up to max power. I've turned everything on that turns the CPU up.

CW (18:57):

So it's very confusing. I don't really understand it. And I actually did some things today where I was like, "Okay, let's see if it's going out to lunch." I took a timestamp every time I did receivefrom, and then got an array at the end with the deltas. And then I looked at the deltas versus where it started over.

CW (19:15):

And there's maybe it taking a little bit longer, but we're talking a fraction of a millisecond. So it doesn't make any sense. It's not like it's saying, "For a quarter of a second, I'm taking a break."

CW (19:31):

So it's...one of those embedded systems problems that's just stupid. And I should probably go rewrite this in C and never think about it again, but there's reasons to do it in Python that I'm rapidly thinking about ignoring.

EW (19:53):

Everybody's going to ask you questions, like, "Have you tried the CAN_RAW_FD parameter in the socket library?"

CW (20:03):

I don't understand the question.

EW (20:05):

Well, the socket library, since I have Googled it for you -

CW (20:08):

Yeah. Thanks. I've spent many, many, many hours reading all of that documentation, but go ahead.

EW (20:16):

No. Yeah, there's just -

CW (20:17):

I've tried the waitall notification...What I really wanted to do was read a batch...without doing receivefrom. So the thing with CAN in Linux is, apparently, and people can correct me if I'm wrong, but I'm not wrong. It's a datagram-oriented protocol.

CW (20:36):

And so the way the socket implementation works is, you can ask for as much as you want, but you're still getting the 16 bytes of a CAN message only once a read. So it would be nice if I could say, "Hey, read a whole block of this stuff and give it to me as a chunk," but it won't do that.

CW (20:53):

It only returns one CAN message at a time. So there's time in between each read for something bad to happen, which is what I suspect is happening...There was a socket option that said, waitall that sort of indicates..., "Give me everything I asked for, the number of bites I asked for," and just sit around until you get it.

CW (21:11):

That does nothing. Yeah, I'm using the CAN raw interface. There's another CAN module for Python that's not built in called socketcan. I have considered trying that, but that's built on all the same stuff. So I don't understand how it would be any better, but maybe it is for reasons I don't understand.

CW (21:34):

So I might try that. I don't know. If somebody's had good experience with that, they can let me know, but I doubt people are doing a lot of python-can.

EW (21:42):

We should probably sit down together and look at it. I mean, I did some in CAN. I don't remember having problems keeping up in Python.

CW (21:51):

Yeah, well -

EW (21:51):

But I did it a totally different way. Okay. Now for my homework?

CW (21:58):

Okay.

EW (21:59):

Are you ready? So I have been asked to answer a bunch of questions. Let's see. There are 17 of them. We don't have to do them all.

CW (22:09):

Okay.

EW (22:11):

And I figure I'll ask you the questions, and then I can answer them on my own later, having...cribbed your answers.

CW (22:21):

I'm supposed to answer as you?

EW (22:23):

No, for yourself.

CW (22:25):

Oh. And you're going to use my answers in your - ?

EW (22:29):

Only the parts that are relevant.

CW (22:31):

Okay. Alright.

EW (22:33):

I mean, if I was copying your English essay, I wouldn't do it word for word.

CW (22:38):

You wouldn't be able to read my writing.

EW (22:40):

That's true. What do you wish you had known about embedded systems when you were first starting out?

CW (22:47):

This is hard to just answer on the fly. What do I wish, I'm going to repeat it back, because that's the technique for stalling.

EW (22:56):

And then say that's a good question. Don't forget that.

CW (22:57):

That's a really -

EW (22:58):

Yeah.

CW (22:58):

That's a really deep and interesting question. I think I wish I had known more about electronics to start with. And...this isn't a technical thing necessarily, but I think I wish I had known how messy everything is, mostly because there were a lot of frustrations...

CW (23:26):

I was coming from a development environment that was technically embedded, but it was writing stuff for routers. And so I wasn't dealing with hardware very often. I was at the protocol level. So I was talking to networking device drivers and things and writing networking device drivers.

CW (23:47):

And so...turning the question around, what things did I learn quickly that were important, that became apparent quickly, were things like, "Wow, cables are really evil, and if you don't have the right electrical signal integrity, your communications don't work," or timing.

CW (24:18):

Timing, if you have an electrical timing issue, even if that's controlled by software, you can have intermittent, strange problems that take a lot of time to figure out. So...yeah. I mean, I think it's back to what I said originally.

CW (24:33):

The electrical part of it was stuff I didn't know that would have helped me go a lot faster if I'd had a background and somebody saying, "Here's how these communication protocols work at the wire level, and here's why you can get into trouble." But I mean, I knew some of that stuff, so I don't know.

EW (24:59):

Sorry. There's the sound of me typing.

CW (25:00):

The sound of you typing?

EW (25:02):

Okay. So what I got from your answer is embedded systems exist, which I didn't know starting out, -

CW (25:10):

I knew that.

EW (25:11):

- and are everywhere.

CW (25:14):

What? Wait a minute.

EW (25:14):

I mean, when I first started out after school, I didn't know there was, I mean -

CW (25:20):

But you're now answering the question.

EW (25:23):

Yes. I'm paraphrasing what you said.

CW (25:25):

Okay. I didn't know how to work in an interdisciplinary team.

CW (25:30):

That's fair.

EW (25:31):

I didn't know how many different applications there were and how my specific knowledge could be used.

EW (25:38):

And that was one of the great things about embedded systems for me, was that I went from having a software education and a bunch of Fourier to actually having a career in signal processing. I didn't know that I could do both.

CW (25:56):

Yeah. Yeah.

EW (25:57):

And then from your answer, I think, I wish I had known not more about electronics, but more about debugging hardware, about the simple stuff of checking cables, and looking up information, and datasheets, and understanding the logic analyzer output.

EW (26:12):

Well, actually at that time, lifting the logic analyzer would have been a process.

CW (26:17):

Well, just using it period, since I'd never come across one before.

EW (26:18):

Yeah.

CW (26:18):

Yeah.

EW (26:20):

Okay. What factor, or set of factors, is key for someone to start a successful career in embedded systems programming?

CW (26:33):

Patience.

EW (26:35):

Patience or persistence?

CW (26:37):

What's the difference? I guess persistence is when you're patient, but also mad about it?

EW (26:48):

I don't know that I would have termed it that way. No. I mean, patience is waiting for something to happen. Persistence is keep trying until you figure it out.

CW (26:56):

It's both.

EW (26:58):

Okay.

CW (26:58):

There's a lot of waiting around for something to happen in embedded. Have you ever tried to download an image and do it to an old slow device? Key factors.

CW (27:08):

I think having an understanding of how computers work is really key, and not necessarily like my previous history, not necessarily electrically, but really having an understanding of what a microarchitecture for a CPU.

CW (27:28):

What that consists of, what the parts of the CPU are, what things you have access to and what they control, how program flow works,...what the different kinds of memories on a device are and how they are used.

CW (27:42):

Because with embedded, at least until very recently, if you're debugging problems, or you're trying to optimize your code, knowing those things is hugely useful.

CW (27:57):

Because you'll find yourself in various situations that you just can't solve if you don't know what a stack is, or a program counter, or the return register, or all of these intricate little parts of a device that you have access to.

CW (28:15):

It doesn't mean you need to know assembly language necessarily, but I think having an understanding of computer architecture is pretty key.

EW (28:28):

Okay. So a general understanding of microprocessor architecture, patience and persistence, minor willingness to learn, puzzle abilities, and creative thinking,

CW (28:40):

Puzzle abilities. I'm not sure about that.

EW (28:42):

That's not really the right word, but being able to deal with resource constraints, and trading off RAM versus speed, and that sort of thing.

CW (28:51):

Yeah. Yeah. I don't want people to feel like Google interview questions are necessary to be answerable as an embedded engineer.

EW (29:04):

Yeah.

CW (29:04):

The goat and the lettuce or whatever is not going to help you. But you're talking about optimization, and yeah, fitting things, rearranging.

EW (29:16):

Alright.

CW (29:16):

Tetris.

EW (29:16):

I need a different word. Alright. I'll think about that. What is the biggest frustration that people go through when trying to learn embedded systems, and what are our recommendations to overcome them?

CW (29:30):

Well, we'd have to lean on what people have told us.

EW (29:34):

I don't know. We know a lot.

CW (29:36):

But my frustrations are not going to be somebody else's frustrations. I guarantee it. My frustrations are that I have to sit in a desk, in a chair, typing -

EW (29:45):

Your life is so hard.

CW (29:45):

- staring at a screen, not moving.

EW (29:49):

I mean, you could -

CW (29:50):

Frustrations -

EW (29:52):

This desk works as a standup desk.

CW (29:54):

I use it. I also stand in frustration. Frustrating things. I think one of the frustrating things for me was coming to terms with realizing that there are going to be problems, two kinds of things. A, there were going to be problems that took weeks to figure out.

CW (30:18):

And at the end of the two weeks was going to be one line of code or a change in a configuration register. Because of that microarchitecture thing, even if you have a good understanding of it, every chip is different, every system is different.

CW (30:33):

And the kinds of issues you're going to have, there's always going to be something that's surprising about any system you develop. No matter how genius your software development methodology is, you're still gonna have bugs.

CW (30:48):

Your programming language is not going to prevent you from writing code that has bugs. Your hardware will have bugs. And so there's always, in the development of any device, at least a couple of times where you're like, "This doesn't make any sense, and I can't figure it out," and it takes weeks to solve.

CW (31:06):

Especially if you're in a interdisciplinary team or a large software team and there's different parts of the system, and somebody off in a corner just happens to write something that occasionally writes a byte that is over in your software, in your part of the code's memory.

CW (31:22):

And...you have no responsibility for that, so you wouldn't go think to check the other person's code. So there's a lot of stuff like that where...there's just these mysteries and they take a while to solve. The other frustration, which is worse, is there are occasionally problems like that that you will not solve.

CW (31:41):

And there's plenty of times that I've shipped stuff, and it's got a bug, and we can't figure it out. And that's just the way it goes. And the best you can do is have a workaround, because the trade-off is sometimes, "Well, we could spend six months trying to figure this out, or we can accept it and move on -"

EW (32:01):

Patch around it.

CW (32:01):

" - if it's not a critical issue." Yeah. Then that's really frustrating, because you feel like you've completely failed, or you're an idiot, but sometimes the systems are just so complex that it's like, "Well, I don't know where this is coming from."

CW (32:14):

And that's usually true when something's really intermittent too, because it's really hard to solve problems that you can't reproduce. And that's one of the differences from embedded. And I am talking, and talking, and boring everybody.

CW (32:25):

But one of the big differences for embedded is your visibility into things...varies wildly compared to when you're writing application code. When you're writing application code, oftentimes you have these big debug suites that will graph things for you, and show you memory leaks, and put up things that say, "Oh, I detected this and that."

CW (32:45):

And we don't have that a lot in embedded. So a lot of the times problems just don't get noticed, or the system is so complex that it's very difficult to have a problem be reproducible. Because it might happen in a customer's premises, but not in your lab for some subtle, tiny difference, and you can't recreate the scenario.

CW (33:10):

So, yeah. I mean, one of the reasons for that is embedded systems are supposed to be cheap, and lean, and lightweight, and not have a lot of extraneous structure and things. And that limits what you can do in terms of logging and putting in extra stuff for debugging.

EW (33:30):

I would add that our compilers still suck, which is kind of part of your visibility, and IDE, and debugging suites.

CW (33:39):

Okay. I mean, some compilers suck.

EW (33:42):

Compared to a software suite of tools, our tools -

CW (33:47):

Our toolchains.

EW (33:47):

Our toolchains are not great.

CW (33:49):

Yeah, are limited.

EW (33:49):

The compilers are fine. Sorry.

CW (33:51):

Yeah.

EW (33:51):

The whole toolchain is -

CW (33:52):

Yeah, I agree.

EW (33:52):

- kind of weak and fragile. Anybody who's had to use GDB server, and had it crash, and wondered why you were debugging your system and it just wasn't working.

CW (34:07):

Yeah.

EW (34:07):

I also think that one of the difficulties is having to navigate hardware and software when you've probably only really learned one in the past. And then the thing that was so frustrating for me were things like STM32F03, blah, blah, blah, blah, blah, blah, blah. I mean, it's just impenetrable jargon.

CW (34:34):

Well, it's not jargon...That's just junk.

EW (34:38):

But no, I mean, you need to know it's, "Okay. So I said F0 instead of F3 or 4 so that indicates something important. SPI and I2C, we just throw these terms around, because of course we all understand. And UART and USART, does it matter if I say the 'S' or not?"

CW (34:57):

Yeah, I don't think any of that's unique to embedded though.

EW (34:59):

...There is a lot of impenetrable jargon. I mean, even looking up I2C, does I-two-C and I-squared-C mean the same thing? Yes, yes they do. But -

CW (35:12):

[Ah], but I3C.

EW (35:13):

But Googling it isn't as easy as it should be.

CW (35:17):

I think one of the differences too, is that you mentioned the compilers and stuff, but it's not just that, it's the languages. And I know people are going to shout Rust at their earbuds or whatever.

EW (35:30):

Rust!

CW (35:30):

But a lot of stuff's still done in C, and if you're coming from learning software in college, you probably haven't done a lot of C. And so it's a bit of a brain shift to something that's ancient and brittle, but also powerful in what it allows you to do, but also gives you enough cliff to jump off of. I don't know. That's not a good metaphor.

CW (36:02):

So that's a hard thing. I think, getting more so. Because modern languages have moved so far beyond C that if you've cut your teeth on software development and that stuff, coming to C is like, "What is happening," or more likely, "What isn't happening?" It's not helping you about anything.

EW (36:24):

Let's see. These two I think we'll combine into one. What's the biggest opportunity for people getting into embedded systems, and why should anyone choose to pursue a career in embedded systems?

CW (36:34):

Biggest opportunity? I don't know. I don't know.

EW (36:42):

I wasn't quite sure how to take that.

CW (36:44):

I think they're asking what's the best sub-area to focus on?

EW (36:50):

Oh.

CW (36:50):

Like IoT light bulbs, or -

EW (36:54):

Oh.

CW (36:54):

- automotive stuff. I don't know what's hot. I mean, everything's hot. I think there's enough available to do in embedded that you can kind of seek out things that interest you. Yeah, I don't know where to go with that precisely. What was the second part?

EW (37:11):

Why should someone choose to pursue a career in embedded systems?

CW (37:16):

Well, you get to see a lot of different things. You get to learn a lot of different things. You're not limited in the kinds of products that you might work on.

CW (37:27):

You do get to work in a more interdisciplinary environment where you might regularly talk to electrical engineers, and mechanical engineers, and even optical engineers, and system engineers, and all kinds of people, and learn some things that aren't necessarily software development.

CW (37:46):

I think that's the answer I'd stick with. I wouldn't go much deeper than that.

EW (37:51):

What should people do to ensure they don't get stuck in their career?

CW (37:55):

People do to ensure they don't get stuck in their career? Quit their job?

EW (38:01):

Okay. I'll write that down as a suggested answer. I had learn, whether it's reading, or experimenting on their own, or teaching and sharing their knowledge with others, it's just important to keep learning.

CW (38:18):

Yeah. Okay.

EW (38:22):

Okay. What's your favorite embedded system development board, and why?

CW (38:28):

Pass.

EW (38:29):

I wrote, "That's unfair. I can't choose. I guess it's the one that is on my desk, because it's on my desk."

EW (38:37):

What programming languages are used most by industry? Go on. Shout Rust again. We don't mind.

CW (38:44):

Wait, what was the question?

EW (38:45):

What programming languages are most used by industry?

CW (38:49):

C. Followed by C++, followed by assembly language, possibly followed by obscure weird stuff like Ada.

EW (39:03):

I think I'm going to go with C, C++, and that knowing Python is helpful for making additional tools.

CW (39:09):

Yes. Rust is up-and-coming, but I don't think it's -

EW (39:16):

Yeah.

CW (39:16):

- a high volume of stuff right now.

EW (39:19):

Just let me know when that balance checker works for you.

CW (39:20):

...I'm not doing it.

EW (39:24):

I'm not saying you're doing it. Okay.

CW (39:29):

It's the borrow checker, see.

EW (39:30):

Is it the borrow checker?

CW (39:30):

It's the borrow checker.

EW (39:30):

Oh, man. If I'm going to get my burns right, I have to get them right.

CW (39:33):

Yeah, see now people are going to really yell at you.

EW (39:36):

Yeah. I deserve it too. let's see. What is the state of art in terms of embedded software development today? How can you answer that question? Why would you ask that question?

CW (39:48):

I don't know...State of the art is GCC plus Clang and VSCode? I don't know. What are they asking, the tools?

EW (39:58):

I don't know. But I'm totally stealing your answer on that one. What is the...best way to prepare for an embedded engineer interview?

CW (40:12):

Get a good night's sleep. Make sure you breakfast. Shower. Go for a nice walk first, and review the basics of some things like real-time operating systems, communication interfaces, like UARTs, and I2C, and SPI, and do some bit twiddling problems. I don't know. It depends on what people ask. I don't really like interviews.

CW (40:47):

I think they're useless for the most part. So I think people do hang their hats a lot on technical questions. So the kinds of things you're going to be asked by a competent embedded engineer asking you questions is the things I mentioned...

CW (41:04):

What is a thread?...How do you synchronize between two threads? How do you read data off of SPI bus, [blah, blah, blah]. What's DMA? That kind of stuff. Just all those three letter acronyms, three and four-letter acronyms.

EW (41:27):

It's tough to know all of those.

CW (41:28):

It's tough to know all of those? It's half a day to know all them to a level where you can say what they are.

EW (41:36):

That's true.

CW (41:38):

And...I would say, if you don't know what an RTOS is, or I2C, or SPI, and you're interviewing for an embedded interview, -

EW (41:46):

You may not be in the right place.

CW (41:46):

- you probably shouldn't be interviewing for an embedded interview.

EW (41:50):

...My recommendation as the best way to prepare is to consider doing a portfolio project, something you can hold in your hands, something you enjoy talking about and understand.

CW (42:02):

Yeah.

EW (42:02):

Because you can derail the interview by talking about something you know instead of answering questions about things you don't know.

CW (42:08):

That's not derail. It's -

EW (42:11):

Manipulate?

CW (42:12):

Not manipulate. It's guiding the interview toward things you know.

EW (42:17):

Okay. Sure. Sure, sure.

CW (42:18):

It's being in control of the flow of the interview to stay away from things like bit twiddling.

EW (42:25):

I love bit twiddling.

CW (42:27):

Yeah. But you can look it up.

EW (42:29):

You can look up what RTOSs are too, but it takes longer to understand the concepts behind the RTOSs.

CW (42:35):

That's like saying you can look up physics.

EW (42:37):

Well, yeah.

CW (42:37):

Sure you can, but if I'm looking at how to, I don't know, do one of those weird XOR to count the bits things, knowing that offhand saves two minutes. Knowing what an RTOS is saves you a month.

EW (42:57):

Yeah, exactly. What do you wish your students knew about you?

CW (43:06):

I don't have students.

EW (43:08):

Fair enough. My answer to that was kind of backwards. How about what I wish they never found out. I don't know what I'm doing. I have to look up a lot of stuff. Sure, I can translate between binary and hex.

EW (43:22):

I can explain how direct memory access works in theory, but when I need to implement it on a project, it's a matter of looking at the user manual for the particular chip, sorting out what bits need to be set on which registers.

EW (43:35):

I'd really rather students thought I knew everything, but I don't. And that's okay. And let's see, a funny story/memory of your career. See, I can't steal these from you.

CW (43:47):

Well, there was the time that the guy was arrested, no.

EW (43:54):

Okay. What funny stories from my career that don't involve me blowing up boards, which would ruin my credibility?

CW (44:01):

Well, there was the time you melted your shoes in the desert. There was the time -

EW (44:07):

That was on a bomb range.

CW (44:08):

There was the time you were on an active bomb range with signs saying things are going to explode.

EW (44:13):

They didn't.

CW (44:13):

...It turns out the same time, all two of those things, there was the rattlesnakes everywhere.

EW (44:21):

I had a snake bite kit.

CW (44:23):

Well, that's great. That's fine then. You have to let them bite you. Yeah. Yeah, that was -

EW (44:25):

And boots. Well, until I had to -

CW (44:27):

That was fun.

EW (44:29):

- put them together back with duct tape.

CW (44:30):

There was the time...you got to ride in a stock car at 200 miles an hour. There was -

EW (44:41):

And I left you that voicemail where the car was coming into range, and it just got louder, and louder, and louder, and louder until it was just screaming as it drove by at 200 miles an hour.

CW (44:50):

Yeah, I don't remember that exactly.

EW (44:52):

...It was a good voicemail.

CW (44:54):

Yeah, those were a couple good ones.

EW (44:58):

What work are you most proud of and why?

CW (45:03):

I think the work I'm most proud of, although the company was kind of a mess, was the stuff we did at a company called Avinger...We made imaging and interventional catheters to address peripheral artery disease in people.

CW (45:24):

Long story short, it saved people from having amputations, because that's the usual treatment once it's gone far enough. And using our device, they could actually repair people's arteries so that their legs healed. And...peripheral artery disease is a big problem with people with diabetes and older people.

CW (45:45):

And...the company, it still exists, and they're still doing stuff, but financially the company didn't do well, and it was hard to work for for various reasons.

EW (45:57):

7:00 AM meetings.

CW (45:59):

7:00 AM meetings, and actually having to go to hospitals, and observe stuff, and animal studies, and other unpleasant things. Knowing that something I had a big hand in probably saved a few people's lives and quite a few people's ability to have a good quality of life, that's pretty good.

EW (46:23):

Yeah. Will you answer that question for me? Because I don't have as good an answer.

CW (46:28):

Toys. You made thousands, tens of thousands, hundreds of thousands of children's playtime fun and educational.

EW (46:39):

Well, I mean,...yes, I am actually really pleased with that, because...I know there's at least one boy who probably would have had to go through kindergarten again if not for a toy that I made. And I don't know who else were affected by them, but I'd like to believe there were people out there who had an easier time learning to read.

EW (47:11):

I was thinking about saying, I might say I'm most proud of the podcast, for all that that's weird.

CW (47:18):

Well, that's a good answer.

EW (47:19):

Because I know a lot of people -

CW (47:21):

I don't consider that part of our embedded career, which it totally is. So, yeah.

EW (47:26):

I mean, getting to talk to all the people we've gotten to talk to, and having people say that it helps them stay engaged in their career, I'm pretty pleased by that. Because it was hard.

CW (47:40):

I'm glad they didn't ask least proud of, because I got two -

EW (47:44):

The list is long.

CW (47:45):

- competing answers for that. One is working on the internet, which I think may have been a gigantic error. And I apologize for any hand I have had in making the internet work better, or faster, or any of that. The second one is kind of the dermatology medical thing.

CW (48:10):

I think it did help quite a few people, so I'm not so down on it, but...it was hard, because the product did help people who needed help. But it also had a large application space that was purely vanity. So it was kind of a -

EW (48:28):

You got into it for scar removal.

CW (48:31):

Scar removal, and people with large birthmarks and things that caused them emotional trouble and stuff. It could address those, but the main use case it turned was -

EW (48:47):

Wrinkles

CW (48:47):

- people with a lot of money who wanted to smooth out their face and stuff. So, I mean, it was a toss up. So...the other medical device was more clear cut, too. It was -

EW (48:58):

It was a little easier, good.

CW (48:58):

- pretty obvious, the application there. And the thing I'm most sad about that is the market is weird and things don't take off the way you think you're going to, especially in healthcare.

EW (49:14):

Okay. I think that covers my homework. There were a couple more, but I can answer them.

CW (49:18):

The other least proud thing is the apocalypse machine I made. And...know how I say never work on time? I think I may have screwed up the clock on that. So, sorry.

CW (49:34):

Well, it was set to go off in a thousand years, but one of the important things about embedded systems is knowing the difference between milliseconds, seconds, and those things.

EW (49:48):

And uint8_t versus uint32_t.

CW (49:51):

Yes. Yes. There may have been a rollover bug there, but since it's already shipped,...we didn't do OTA for that, because...we did that in the early 2000s.

EW (50:01):

Well, something to look forward to.

CW (50:02):

What's the next topic?

EW (50:04):

I found out about this thing called TRIZ. T-R-I-Z. And...it's old. It's a way of solving problems. It's from, let's see, Genrich Altshuller, who was a science fiction writer as well as a Soviet engineer and inventor. And the idea is that there are methods to doing invention.

EW (50:41):

And..they did magazines and lots of instructions in the Soviet Union around this. But I just discovered it recently. And I wonder if other people out there have heard about it. It's not forced creativity, but it's thinking about problems in a more structured manner.

EW (51:07):

And when I say thinking about problems, it's things that require inventions. And if we want to think about how we're going to solve big problems in the future, this has a methodology to go through. And they start talking about different tiers of problem-solving and how most of us are in trial and error.

EW (51:33):

If I have a problem, I think about what I've done in the past, and then I try something. And if that doesn't work, I try something else that's similar. But if you have a problem that's really different, that's going to get to a non-optimal solution.

EW (51:51):

Because you're going to get to a solution that is close to what you know, but if it's a new problem, being close to what you know isn't the right choice. And so -

CW (52:03):

I'm thinking of examples of that right now from companies I was at where...they knew how to do lab stuff. And so they built this deeply embedded system with National Instruments boards, doing everything, like GPIOs. They'd spend $2,000 on a GPIO board.

CW (52:20):

So they knew how to build lab equipment and buy these parts from National Instruments. But they didn't realize you could do stuff with a microcontroller, because that was just not in their brain.

EW (52:29):

Yeah. Yeah, exactly. And then...tier two is searching for a solution kind of outside yourself. Tier three is using multiple disciplines..., searching through multiple disciplines. But there's this whole process...I've never really heard of before. And I found it pretty interesting.

EW (52:55):

I'll put a link in the show notes. The original one is in Russian, but there are a couple of different books, and there are a couple of translations.

CW (53:08):

Yeah, never heard of that.

EW (53:08):

It's just been kind of interesting to think about. Apparently there are 40 steps, different problems you can solve using different ways. One of the examples given in Altshuller's book was a simple, "Okay, so how do they put the raspberry goo inside the little bottles of chocolate?"

CW (53:37):

What?

EW (53:39):

So you have a candy that is a -

CW (53:43):

Oh, the bottle-shaped candy.

EW (53:44):

The bottle-shaped candy, and then inside they have liquid.

CW (53:46):

Got you. Sorry, that didn't make any sense.

EW (53:47):

I'm sorry...I was in my own world.

CW (53:52):

Sure.

EW (53:52):

...How does that work?

CW (53:53):

Or the ones that are filled with alcohol?

EW (53:55):

Alcohol. Okay. Yeah. So do you have any idea how they build those?

CW (54:00):

How they make the candy?

EW (54:02):

How they make a liquid inside the solid chocolate?

CW (54:06):

I had assumed that they make hollow chocolate and inject it.

EW (54:10):

But if it's the kind of gooey raspberry jam consistency -

CW (54:17):

Yeah.

EW (54:17):

- then you'd have to heat the jam to be able to inject it in. And that would heat the chocolate.

CW (54:27):

What if I, okay, I'm not going to solve this problem.

EW (54:30):

No, no, no.

CW (54:30):

Yeah.

EW (54:30):

This...by the way, is not an interview question. Do not ask this as an interview question. But one of the first tenets of this whole process is to think about how to change the state, the physical state, of the problem you're trying to solve.

EW (54:50):

And so the solution with the candy is you freeze the jammy stuff -

CW (54:54):

Sure.

EW (54:54):

- in the middle, and then you dip it in.

CW (54:55):

Sure, sure.

EW (54:55):

Which as soon as I said it, you were like, "Of course, that's what you did."

CW (54:59):

Sure. And that was often the alcohol, which you can't do that with alcohol.

EW (55:02):

Right.

CW (55:03):

Right.

EW (55:05):

...Okay. So if you have a problem, one of the first things to do is, can you change the state of it? And another one is, can you do the opposite of what you expect?

CW (55:19):

Okay.

EW (55:19):

Which was, I mean -

CW (55:20):

Yeah.

EW (55:20):

- in the candy one -

CW (55:21):

Yeah.

EW (55:21):

- that is, instead of -

CW (55:22):

Instead of filling.

EW (55:22):

- heating the jam, you're going to cool it.

CW (55:25):

Yeah.

EW (55:25):

And so you end up with a better solution.

CW (55:29):

Well, when a problem comes along, you must whip it.

EW (55:38):

That reminds me, it's about time for me to have tea and whipped cream, which is what I put in my tea.

CW (55:43):

Tea and whipped cream? That sounds decadent.

EW (55:46):

It's delicious. Do you have anything else you'd like to talk about?

CW (55:51):

No. I should probably get back to CAN.

EW (55:56):

I had one more topic, but I think I'm going to save it.

CW (55:58):

What is it? Just to tease people.

EW (56:01):

I had this idea about how to introduce people to hardware, how to introduce them to schematics and reading datasheets. And I think the way to do it is maybe to use the Arduino board, to actually take off the hood, and look at the board, and what you could do with it if you weren't in Arduino land.

CW (56:29):

That's certainly something that's available.

EW (56:31):

And I'm hoping it's a way that people who are a little concerned about learning hardware would be willing to follow.

CW (56:40):

Yeah.

EW (56:40):

Because it's something they already have familiarity with.

CW (56:44):

Okay.

EW (56:45):

Oh, and Remoticon. Hackaday is having their Remoticon in November, 19, 20th.

CW (56:54):

Okay.

EW (56:55):

And their call for proposals is open for some period into October. I assume everybody can look up Hackaday Remoticon.

CW (57:05):

Okay.

EW (57:06):

Okay.

CW (57:07):

Alright.

EW (57:07):

Alright. Thank you for listening. Thank you to Christopher for co-hosting and doing at least part of my homework for me. Now we have some Winnie the Pooh.

CW (57:20):

Thank you to the Patreon, Patro, Pay -

EW (57:25):

Patrons.

CW (57:25):

Thank you to the calzones, patrones -

EW (57:28):

Patrons.

CW (57:29):

Patrons.

EW (57:31):

Our supporters on Patreon.

CW (57:33):

Thank you to you who have supported us for supporting us.

EW (57:40):

Yes, we really do appreciate you. And should anyone want a sponsorship, they should email us.

CW (57:48):

Email sponsorship@embedded.fm.

EW (57:52):

Okay. Now for Pooh?

CW (57:55):

Sure.

EW (57:57):

[Winnie the Pooh excerpt].