347: Be Careful About the Bits
Transcript for Embedded 347: Be Careful About the Bits with Christopher White, and Elecia White.
EW (00:06):
Hello, and welcome to Embedded. I am Elecia White here with Christopher White. This week it's just going to be the two of us chatting with each other.
CW (00:16):
I'm so sleepy.
EW (00:17):
Well, I hope you wake up because I can't chat with myself.
CW (00:20):
I hope that doesn't come across at all.
EW (00:22):
Let's see. Let's do announcements first. Big thank you to our Patreon supporters, which of course you can join if you want to. We're going to talk about a question from there today that's got a lot of really good advice, but we appreciate it and it is help paying for the transcripts.
CW (00:44):
And mics.
EW (00:46):
And mics. If you have any comments on the transcripts, please let me know. I'm still pretty new at it. I am finding it pretty amusing to read through them as I post them.
CW (00:58):
A little strange. It's a little strange.
EW (00:59):
It is. It's weird to be searchable.
CW (01:02):
It's also strange to see you when you talk for what you feel like is five minutes and it turns out to be a paragraph.
EW (01:06):
Yeah. Okay. So what'd you want to talk about today?
CW (01:11):
I don't want to talk about anything. I want to go back to bed.
EW (01:16):
You could have left me on the couch then.
CW (01:19):
I thought you had an agenda.
EW (01:21):
I have sort of an agenda, but first I wanted to talk about what we've been doing besides work.
CW (01:27):
I plead the fifth amendment.
EW (01:35):
Not true. I mean, you've started a couple of projects. You're still working on the album.
CW (01:39):
I'm becoming a professional doom scroller.
EW (01:41):
You can't do that.
CW (01:44):
Yes, I can.
EW (01:44):
This is not good.
CW (01:44):
I can do it. I assure you I can do it. I'm really good at it.
EW (01:49):
Tell me where the professional part comes in then.
CW (01:52):
Well, it's just the level of skill involved.
EW (01:53):
Oh, okay.
CW (01:56):
No, I've been working on the album. We have one more song to do. It's a difficult song and I'm having trouble writing a part for it and getting up the gumption to fight with it is a little tricky, but I think I'm close.
EW (02:11):
And you took over part of the garage, put away my soldering station in order to do so.
CW (02:17):
Your soldering station. But I think I use it more than you.
EW (02:22):
Yeah, you probably do.
CW (02:23):
Yeah, I'm building a ukulele from a kit. It is a baritone ukulele, which is the largest form factor, I think, of a ukulele. It still has four strings, but they're tuned like the highest pitch four strings of a guitar. So I don't have to learn a new ukulele tuning, which wouldn't be a big deal, but I'm lazy.
EW (02:48):
Which is why you're building one from scratch because you're lazy.
CW (02:51):
Well, it's not from scratch exactly. The kit's pretty nice. It's harder than I expected given when you look at the pictures on the website and they show you what parts they send you and what's preassembled. It's like, "Oh, this won't be that big a deal." But it's going to take me a few weeks. It's from a company called Stewart MacDonald. Their website is stewmac.com and they have all sorts of guitar building lutherie tools, and parts, and kits, and blank wood. If you're pulling totally from scratch, then you can spend $5,000 on very expensive wood, if you're insane. But this kit was $120 or something like that. It was pretty reasonable.
EW (03:36):
You've already talked someone into sending it to their parent for a gift. So I thought that was pretty funny.
CW (03:42):
But it's a nice kit. The hardest parts are already done. So like the neck is finished. It's already shaped. That's a super hard part of building a guitar or any sort of string instrument. They've slotted the frets. That's also a hard part because those are precisely measured in distance and the body sides are already shaped, so they're curvy body. You don't have to shape them yourself, but you have to a glue everything up and put in the bracing and install the frets, which I'm not quite looking forward to, even though I've already done it on a guitar at one point. I remember that being sort of fun, but getting them filed to shape and stuff is kind of tricky. But yeah, a lot of glue.
EW (04:20):
Clamps.
CW (04:21):
Clamps and glue. Yeah, the hard part is having enough woodworking tools that you need, but you can get by with most things. The most frustrating now is when I need something, I can't just pop down to the hardware store, wander around and find a part I need. So I have to look like on Amazon for L brackets or plywood or something. But anyway, yeah, that's sort of fun. Very low tech.
EW (04:50):
Yeah. I've been trying to find things that keep me away from the internet and computers in general, but even origami, I've now got a Python script to help me design seashells. It's not going the direction it was supposed to go.
CW (05:10):
The highest tech part of this project is the YouTube video I'm using as the instruction manual.
EW (05:15):
Okay. Let's see what else have I been doing. Reading. I've been doing a lot of reading, science fiction, fantasy, mystery.
CW (05:23):
What are you reading?
EW (05:23):
Romance.
CW (05:24):
What do you recommend?
EW (05:26):
Well, let's see. I guess the most interesting book I've been reading that I'm willing to admit is one by, I think his name is Vermeij. It's a natural history of seashells and it is really, really interesting, but it is a textbook. I'm doing okay, but I had to get it in paper because it didn't come electronic. So I keep pushing the words to figure out what apical means. And it means towards the apex. I got that from context clues, but yeah, I missed the definitions.
CW (06:18):
Was that the book you said was authored by a blind person?
EW (06:22):
Yeah. He's a professor at Davis who was, I believe, blind as a child and still blind. He is the foremost expert on shells all by touch.
CW (06:38):
That's cool.
EW (06:39):
It's really cool. A book of seashells like this would normally be 90% pictures, but this one actually talks about it and gets a little bit into the math and evolutionary biology in terms of economics. And how much energy things cost and why you would design it, why shells end up this way instead of that way, because of cost and how different formations of the shell are more expensive in terms of energy or upkeep. It's weird. It's cool. It's one of those things that when you tell people you're reading it, they're like, "Why?"
CW (07:26):
I don't.
EW (07:28):
No, you just made me show you the pictures. So we do have a couple of emails. I have one from James. This one was hard. James is an electrical engineer and part of the job involves writing firmware in C. James wants to know if we have a device for writing bare metal code, in particular to MCU specific registers. And he does say that it's possible that after we read the following, the advice would be stay away from firmware. But I doubt it.
EW (08:08):
Where is the best place to put MCU specific code for configuring IO pins, clocks, peripherals, and interrupts? James, the usual method is to have one or more board specific hardware dot C files. That sounds really familiar, like something that would happen.
CW (08:23):
C files.
EW (08:25):
Dot H and dot C.
CW (08:26):
But he says for configuring. So I assume that's the actual action of configuring. Okay.
EW (08:32):
And those file map the pins to the functions and hardware dot H to have macros to map pin controls without exposing the MCU internal registers. Peripherals are more different to place UARTs, end up with a UART driver and a init function, send, receive. I mean, basically James, you're doing the same thing we're all doing.
CW (08:55):
Yeah. I mean, the idea is to layer things and have abstraction layers for their appropriate ...
EW (09:02):
It's called a hardware abstraction layer for a reason. But now let's get to the actual questions. Is it okay to define the MCU specific registers or is it better to use an inline function in case you want to port your code to a controller without say UART peripheral and you need to write a bit banging style routine?
CW (09:25):
I'm trying to parse that.
EW (09:29):
Do you do pound defs or do you write a proper abstraction layer that makes it so you can do it on a different processor?
CW (09:37):
It depends on if you anticipate having to port your code to a different processor.
EW (09:42):
That's actually a really good point. I mean, there's this YAGI principal, like the DRY principal. And where DRY is don't repeat yourself, YAGI is you aren't going to need it. Apparently I didn't have the acronym right. YAGNI?
CW (10:00):
There you go.
EW (10:01):
YAGNI. But you aren't going to need it. A lot of times we try to design for portability when the truth is, it's not likely to be portable and if it is need to be portable, we can make all the modifications then.
CW (10:18):
Sometimes. I mean, I do agree that portability is sometimes ... it's one of those premature optimizations that people get excited about. I do think it's overvalued sometimes. But again, it depends on what you're doing.
EW (10:36):
It totally depends on what you're doing.
CW (10:39):
I would say if you're building like a platform of products, like your company wants to make a new product every year, and reuse the code, and upgrade the hardware with new features, but you want kind of a fixed code base that you build on over many years, then yeah, thinking about this very carefully is warranted. Because you are going to find new chips, cheaper chips, more capable chips that you want to port to over time. So yeah, portability is important. But if not, and if it seems like the next time you make a product, you're going to start over from scratch, which happens a lot more than people, I think, want to believe, then optimizing this upfront may not be worth the time in complexity. I don't know.
EW (11:24):
Let me go on with James's email a little further. Where do you put peripheral interrupts that have MCU specific labels? Do you put them in the driver or the callback to a higher level function or put them in the application code and accept that if you compile for a different processor, you need to remember to change that?
CW (11:45):
Putting interrupts in application code makes me nervous.
EW (11:48):
And with the UART driver, there might be a command handler, which he would put in a different file, but are there any rules of thumbs for the maximum number of functions or lines of code in a particular file? I mean, we've all seen the EE who puts all of their code in a single file and I think we've all agreed that that may not be the optimal solution, but is it possible to have too many files? I've seen that.
CW (12:19):
My gut feeling is that if you've designed your code well, this sort of falls out. Like if you have areas of functionality that are modules, that are common, that fit in a file, it'll be capped to inappropriate size. If that starts to grow too big, then that's probably not one module or one area of functionality. And you actually have two that have snuck in there. You should split it. Having too many files, I mean, there there's big projects, but the only way to get to too many files is if you're artificially limiting what you're putting in them. And then you have the other problem where you have a module that spans too many ... you've divided up something that should be common into a bunch of stuff that isn't common, but should be.
EW (13:07):
Exactly.
CW (13:08):
What do you think?
EW (13:09):
Yes. The problem with software engineering as a whole is that there are no hard and fast rules.
CW (13:20):
Yeah. The compiler has some.
EW (13:23):
But this architecture part, it's blurry. And with the hardware, it's kind of easy to say, "Okay. So if it touches the hardware, it needs to go over here. And if it doesn't, then I'm going to put a layer in between hard hardware abstraction layer and keep them separate." But that doesn't really work when you start thinking about command parsers, because that depends on UART. You can't pretend that it doesn't. I mean, you can-
CW (13:58):
It depends on input-
EW (14:00):
A circular buffer.
CW (14:02):
... from a stream.
EW (14:03):
Yeah.
CW (14:05):
So you could draw the line there and then you could just as easily hook your command parser up to an internet socket.
EW (14:13):
Which has a different speed, so therefore it has to be checked at a different speed. It's software and nothing's impossible. It's just, how do you design it in a way that makes it easy and streamlined, and you don't write the code you don't need, and you write the code you need in a way that when you have to look at it in a year, it still makes sense. James, yeah. Okay. We all have this problem. You're definitely not alone in that one. You did mention in the email, which I didn't say it was that you have my book. So I'm actually going to go back to my book on this one. I don't know what the word for that is, self-aggrandizing.
CW (15:02):
Don't think so.
EW (15:04):
Anyway, in the book, there's the section about creating system diagrams and while I am not really a visual learner, I do tend to think more if my hands are involved. So in the book, I suggest that you write a block diagram. Simple. Okay. So here's how the hardware looks. Here's what's connected. This is over in SPI. This is over in UART. And you end up with this picture of the system. And maybe you also add a few more things like your command handler is going to go near your UART. This sort of thing makes sense. And if you have a display, you have a rendering engine, that means you have fonts and text and images. And you start to collect all of the pieces of the system. If you aren't responsible for all of the architecture, you can just put up your piece. And in James's case, that would be, there's a hardware piece that talks to the hardware, and that would be on the outside of everything.
EW (16:15):
And then there's maybe the interface layer that makes it into a genericish UART, and then there's the circular buffer, and then there's the command handler. Okay. So you write all this together and you have this block diagram of how the hardware works and a little bit of how the software works. Now you're going to put that block diagram to the side. You're not going to worry about it anymore other than now you have sort of a list of parts. Do an entirely different organizational diagram that looks over with the shared resources is. This is more of a tree thing. We have main on top and main calls like a display or call sensors or calls logging. These are separated out like a family tree, although, of course, they may end up talking to each other. And then you end up with this idea of where the organization goes, where the control flows through, and how it flows, and who ends up talking to each other.
EW (17:19):
Okay. So you finished that drawing. You put that to the side and now you do one more drawing. This is a layered view where you are trying to figure out what touches and the other two drawings will inform this. But the idea is that if you have the UART commands, then it's going to touch the UART interface and the UART interface is going to touch the hardware. These are going to be lined up together. The boxes now are about the same size proportional to their effort. And if you think about the UART interface layer, that's going to be pretty small, probably, especially if you pull out the circular buffer. The thing about drawing on a screen, rendering is going to be a big thing, but it's going to touch how it gets images, how it talks to the LCD, and then how the images are going to talk to maybe a SPI flash, which, I mean, there's a lot of layers here. The idea is you end up with this diagram that shows where the connections are and what pieces are big and what pieces are small.
EW (18:35):
Okay. So you have these three drawings. Now you go back and you answer this question. The drawings are not that important. You're not going to keep them forever. You don't need them to be beautiful. It's a way of thinking about the system that starts out with how is the hardware organized? How does my information flow? How is control flow? And then what needs to talk to each other? And it's that what needs to talk to each other that starts informing what your modules are and how big they are, because you can think about, if my rendering engine gets so big, then I can break it into text and images. As long as they don't share too much, that's fine. You start seeing the natural break points and that's what you're looking for. It's really bad when you try to break along edges that you shouldn't have, then you end up with spaghetti code and globals all over the place. The goal is to find a nice crystalline structure and crack it that way. Have I gone to far into metaphor land?
CW (19:45):
Oh, no. That's all really good. I have opinions and specifically I have opinions related to this, which is good because you don't want to hear me just rant about-
EW (19:59):
I've heard some of your opinions.
CW (20:06):
How do I start this? Firmware and electrical engineers, we tend to get excited and focused on peripherals, hardware, MCUs, registers, very low level things. We tend to worry about them because they're difficult and it is difficult to organize all this stuff and get it to play nicely together. So I think there's a bias towards designing from the bottom up. And that's okay because most of these systems end up looking very similar. So from the bottom up, the layering, you're always going to end up with something that's pretty much what 90% of the other engineers in the world are coming up with.
EW (20:50):
Sensor, display, interface to the world ... There are only a few things.
CW (20:57):
I think these four diagrams, you had. Three or four.
EW (21:00):
Three diagrams.
CW (21:00):
I think these three diagrams are great and should absolutely do that. I think you should draw a fourth diagram that's from the top down. If this is part of your mandate at your company is to do the whole application, the whole MCU thing, it's to start from the application requirements and the top, the user interface, and do a diagram that's driven from the top down.
EW (21:24):
The second two are driven from the top down.
CW (21:26):
They sort of, but you were still talking about interconnections and flow rather than features.
EW (21:34):
Ah, yes, yes.
CW (21:36):
The trouble that I've gotten into with the bottom up stuff and just looking at the interrelationships between modules is if you ignore the top level features, what the thing is supposed to do, how it's supposed to interface with the user, sometimes you can get into trouble with your interface diagram, your APIs and other things to where you design yourself into a corner. That's all I'll say about that. It's just make sure you think about your application either alongside or as a separate exercise while you're doing those.
EW (22:11):
Yeah. To your point about how we tend to focus on the hard parts or the parts we find fun for those of us who really like the hardware-
CW (22:20):
It's the part we interact with the most. So that's where the bias comes from, right?
EW (22:25):
It's the parts we worry about the most too, because you have to read the data sheet or the manual, and you have to be careful about the bits. It's hard to maintain, so you worry about that. But the truth is every time you go in there, you're going to have to read the manual and figure out the bits again. I try to make nice comments and I don't always have to, but it's better to just assume that. So if that piece of code is always going to be hard, then maybe you should spend your attention on the pieces that you can make easier. So back to James's question, you have it right. You need to build the layers and how many modules you have, how big your modules should be, it should be along the lines of things. If you have, I don't want to say object oriented because that's not quite what I mean, but it is-
CW (23:22):
functionality oriented.
EW (23:23):
... functionality orientated. If you have a SPI flash-
CW (23:26):
That's a thing.
EW (23:27):
... you should have a file that is called SPI, and you should have a file called flash. And then if you have a storage system, you should have a file called storage system. Don't put them all in one system in one file because as you talk about them, they're different things. And as you talk about a command handler, you can talk about a UART and a command handler. Now, because I kind of agree with you that if you're thinking about someday you may have to port it, then you need a little hardware abstraction layer and then the command handler. That makes a lot of sense.
EW (24:06):
You don't have to do that, but if you're going to do that, you should think about making drivers similar to the POSIX model, which is open, close, init, read, write, and ioctl, which is the catch all for IO control, "I don't know what I'm doing. Please, don't look over here." Sort of function.
CW (24:32):
The setup function.
EW (24:32):
But if you think about the, "Okay, I'm going to have an init function, I'm going to have a read and a write. If I have more than one thing using this, I need an open and close." That works for ADCs. It works for UARTs.
CW (24:45):
It works for everything.
EW (24:47):
It really works for everything. Sometimes you need to keep pipes open, a connection to the internet that you're waiting for something to respond to. But it's a good model to start with.
CW (25:00):
I mean, DMA sometimes gets in the way of that, but you can hide that under read-write too, if you want.
EW (25:07):
Because you're either blocking or you're checking to see if it's there, and yeah.
CW (25:13):
Yeah. And that's what I've done in the past. We've even gone as far as to do the virtual function table for drivers. Like we didn't have an OS. Oh, we had Thread X, but-
EW (25:23):
It doesn't count.
CW (25:24):
For drivers, they all had to have those functions and we just filled in the structure and it did its thing. Yeah, that alone is 95% of an abstraction layer that's going to solve a huge number of problems. And if you're doing it in C++, you can do this with objects and you can replace the IO thing with the virtual functions that you're talking to. Where am I headed with that? Oh, the great thing about that, the virtual functions either in C or C++, is it's really easy to make mocks for doing your-
EW (26:00):
For unit testing.
CW (26:00):
... tests. You can just rip that out and have read and write go to a file or CLI or something like that. So, yeah, that's a huge thing to think about. It's just to have a common interface.
EW (26:16):
Yeah. James did follow-up with what about naming? Do you worry about prefixing?
CW (26:23):
Huh?
EW (26:25):
Like when you have uart_ead, ()is it in your uart.c? Or do you do something else? If you have a command handler, if it's called commandhandler.c, is it commandhandler_check?
CW (26:44):
Yeah. Yeah. Yeah. Because you want to be able to search for these functions.
EW (26:49):
I tend to agree.
CW (26:50):
If they're all called read, you're going to have a really fun time.
EW (26:53):
Really fun time. So I do prefix, if not with the file name, then at least with an ID. So commandhandler.c might be cmd.
CW (27:05):
Yeah, yeah. You can shorten. The only other thing I would suggest is there's a lot of open source stuff out there to crib from. So if you want to see an example of how this is done-
EW (27:16):
Oh my God, there's so much Arduino code.
CW (27:17):
... you can look at Arduino code. You can look at some of the RTOSs that are out there. They have to do this and they have to be fairly general because they work with any application in any processor. So they've got all this kind of problem to solve as well in a much more general way than somebody writing a purpose-built application does.
EW (27:36):
But we go back to, if you aren't going to need it, don't make it too general. Make it a little general. I have two things I'm working on. One with the GPIOs, are basically, you tell it sensors SPI chip select on. So I can send GPIO and then I give it my sensors SPI select and then I get it on. And then I might have a different project that is pin port C, pin three, active high.
CW (28:25):
I would argue that's never appropriate. You should at least abstract that away.
EW (28:30):
I actually have. Yeah. Of course. But that whole having to have three variables instead of two causes a lot of problems. So yeah, there's a time to abstruct those away. But when you think about some of these abstractions, they take like 15 minutes to write once you've thought about what you need them to do.
CW (28:52):
And they save you days.
EW (28:54):
And they save you days later. So as much as I say, "Don't write things you don't need." All right, maybe it is time to have a function that just is GPIO, sensor A, SPI on, instead of trying to do it all. All right. Well, James, I don't know if I answered your question, but that's this month's effort for it. APIs are really, really hard to design. And the thing is, you won't know until long after you've left the company.
CW (29:30):
Yeah, the thing is you're going to get it wrong. It'll work, it'll be fine. You get it wrong. And then you'll realize the next time, what you got wrong. So then you do it the second time. You'll do it a little better and you get that one a little bit wrong. I think we're all on our eighth tries on some of these things.
EW (29:48):
Yeah. I think most of us end up with the POSIX model because it makes sense.
CW (29:55):
There're problems with POSIX model.
EW (29:57):
There's totally problems. There's problems with everything.
CW (30:00):
No, no, no.
EW (30:01):
Go with what is fairly easy to understand.
CW (30:05):
And the truth is this applies to everything. This is not a C specific problem. This happens in every language and it's just a general software development problem of how do you organize code and do meaningful modules without doing too many?
EW (30:25):
Yeah, I suspect just about every profession has something similar. That it's a balance of too much and too little.
CW (30:36):
Yeah, surgery.
EW (30:38):
Not surgery. Okay. So that was James's question. This other question I have is actually been answered. I'm going to change the name and hopefully Chris will change it in all the places I forget to change it, but it happened on the Patreon Slack. Emmet had a career question. He said, "So I have a career question." Okay. We got that part. Sorry. I should have started ahead. "I joined a startup where I was the only hardware engineer. We were acquired by Big Tech. Last week the product I worked on launched. It was a Herculean effort. I used to report to the CTO. I loved my job and had a lot of fun. Now my boss hired a gray-beard with 30 plus years of experience who wants me to report to him."
CW (31:36):
Not those guys.
EW (31:37):
"Fair enough, I suppose, but he's been an individual contributor and my boss wants him to work on a clean board design, a design I had proposed and was to work on last year. I have so many concerns that I thought I should ask you. A, how to deal with the disappointment in what feels like a demotion. The reporting chain is one step removed from decision-making. Two, how to advocate for myself and keep learning. I told my current boss that the gray-beard has to share the work and I won't settle for the scraps off the table, but I don't know if that was too confrontational." Yes, yes, it was. "And three, how do you share your baby? It was supposed to be my design. I had evaluated the parts, lined up suppliers, got the dev kits. I feel guilty about asking for more because the startup has been so good to me and my boss has gone above and beyond.
EW (32:32):
The gray-beard is also much better than me, has been working for longer than I've been alive, and the imposter syndrome is so real now. Oh, and we're also remote. So not like we have eyes on the other person is doing. Do you have any advice?" Before we start with the advice, the thing with the Patreon Slack that I am so pleased by was that he got advice from many people. He got advice from the gray-beards. He got advice from people who had been in similar situations and it was pretty much all good advice. I didn't see anything that I wanted to say, "No, no, don't do that." But it was all very, "Yeah, we understand, of course. This has happened to just about everybody and it's normal." The feelings are normal and you don't want to be a cog, but to some extent you are.
CW (33:28):
I had completely forgotten. I didn't really jump into this when it happened on Slack. I was trying to think of a parallel situation in my career when this had happened. I wasn't coming up with anything. I realized just now there was a huge thing that happened to me like this. I think I wasn't mapping it because of the gray-beard part. And stop me if this is not helpful, but when I was at Procket, my first big job there was to do OS architect, how parts of the OS were going to work, some of the memory management interprocess communication stuff. But my real job once that was done was to work with the routing team to implement routing protocol. I was doing OSPF.
CW (34:16):
After about 18 months, I had implemented most of it and it was working fairly well. But they hired a new guy. The new guy came in and they gave OSPF to him. He was originally supposed to help me, I think, but he basically took it over and spent a lot of time telling everyone what was wrong with it, going over the design with a fine tooth comb and tearing it to shreds, saying he's going to have to rewrite it from scratch. So it was a very similar kind of feeling, maybe not the same situation. But the feeling of I worked on this for years and now this person's coming in and they're taking it over. Whether or not they're saying it's bad or not, taking it over still feels like a demotion or like you've done something wrong. I don't remember how I dealt with it. I think I just got very angry and-
EW (35:12):
So that's not maybe the healthiest.
CW (35:13):
... went off to work on something else and let him have it.
EW (35:15):
Didn't you get an ulcer around that time?
CW (35:19):
I can't remember if that was before or after. But yeah, this is super common, especially in startups because new people come in and there's a big, "You did it wrong." And I have to prove myself kind of tendency of new people coming into startups.
EW (35:39):
Well, when you see a design for the first time, you mostly see the flaws. Part of that is because you don't understand it, so everything looks like a flaw. It's a hard not to be the person who fusses over everything. You have to make an effort-
CW (36:04):
I think I've been that person too.
EW (36:06):
... to keep your mouth shut until you understand enough and to figure out what the important changes are. This is why when I do code reviews, I have a level called nit, which says my OCD compels me to point this out, but it need not compel you to do anything about it. Because sometimes I have to say, but I know it doesn't make any difference, but I still have to say it.
CW (36:33):
Yeah, we had nits at Fitbit, the iOS group, except they were required to fix.
EW (36:36):
Well, that's not what a nit is.
CW (36:41):
Those were the days. So back to his question. I didn't really have any good advice. It was confusing to me to read, because I couldn't formulate what I thought was good advice, but thankfully other people did.
EW (36:58):
Well, I'm going to go with my advice and then I want to look at some of the others. That is for people who are in this situation, and maybe not for what you've experienced, but somethings to remember is the gray-beard is good at general stuff, but you're the domain expert. You aren't coming into this working relationship with nothing. You've already succeeded. Emmett succeeded in shipping a product in this domain. And that is something. It means you've already seen the mistakes possible, possibly made a few. And if your new gray-beard manager has been an individual contributor-
CW (37:45):
He's a manager?
EW (37:45):
Well, he was going to be a manager now, but he's previously been an individual contributor. So that's always tough knowing that your new manager isn't even really a manager. That's how I read it.
CW (38:00):
Oh, he says he was doing design works. That doesn't sound like manager to me.
EW (38:03):
Emmett said he was getting a demotion.
CW (38:07):
Well, okay. I took it as a technical demotion. But that's fine. Doesn't matter.
EW (38:16):
But my point goes back to remember you have your own experience and for all that this new person deserves respect, so do you.
CW (38:25):
Yeah. It's specific experience in this particular product at this company.
EW (38:29):
And given remote work these days, screen sharing and being on the phone is really handy. You don't have to do a Zoom with 95% of it being you on a camera, trying to look interested and not like you just want to doodle. Oh wait, I'm sorry. I'm sure that doesn't apply to you. You can do this thing where you just screen-share and you don't have to fill the silence. You're just working in an office together, looking over each other's shoulder. Okay. So for question three. Let's go back. What was question three? How do you share your baby? You show what you got. You don't hold on too tightly. And hopefully, the gray-beard will want to explore what you've done in order to get his information. Even if he's starting over, understanding someone else's work is useful, offer up your knowledge as an overture of possible work friendship.
EW (39:29):
It will feel terrible, but it's a way to get further and it's a way to maybe get further as a team. So to some extent you're advocating for him as well. You want him to succeed and you will succeed if he succeeds. That's what teams are about, is not trying to make it all about you. Okay. So back to how do you advocate for yourself and keep learning? Well, you have to do that whether he's there or not. Didn't you do that before? How does the gray-beard fit in? It isn't a zero sum game. If the gray-beard is as lazy as I am, he'll be thrilled to share the work with you because it means less work for him. Offer to take over parts, be proactive about asking for advice when you ready for it, don't just ask for it in general. Use the gray-beard for his knowledge, learn everything he can teach you, even if he does it gruffly.
EW (40:39):
It isn't a one way street. You're bringing stuff to the party and you're taking home goodies too. You can say things like, "I think maybe we should go to this virtual conference and it might be good for us." The gray-beard may say, "Oh God, no. Please, no. Please don't make me go to another conference. We don't have time. I've been to so many of these." And then you can say, "Was it useful? Should I go?" so you're actually advocating for both of you to go to a conference or to learn something. And if the gray-beard says, "No." You're actually in a better position than you would have been if you were just advocating by yourself. Does that make sense?
CW (41:21):
Yeah, that makes sense. I think it's hard because it's going to be the specifics of the person on the team.
EW (41:26):
Yeah, the gray-beard.
CW (41:29):
The description is detailed, but somewhat vague at the same time. So it's not clear to me what's really happening. What the gray-beard is, what their working relationship is supposed to be. So it's very hard. I think you have it right, that taking the time and taking the deep breaths to not get your dander off. Dander up, dander up? It's not dandruff, not get your hackles up. That's what I'm thinking.
EW (42:04):
Sure.
CW (42:07):
At least too soon before there's a real reason to be upset. And trying to make it into a team because that's what's happening, right? You're in these startups, you start out and it's sort of a team, but really, really it's just you off in a corner and the rest of the company as a whole team, but there aren't enough people to actually have a whole technical team. And then people start coming in and you have the situation. This is just what happens. It's a little hard because, and this is what threw me a little bit with this story, is it's pretty unusual to be at a startup, be the main person, and then have somebody extremely senior brought in just wildly different, added to your team and basically say, "Okay, now this is the guy." I'm thinking how I would feel about this. You're the person who's done all of this. You're at a startup. You think, "Oh, I'm in a startup, so this is my opportunity to really shine and-
EW (43:09):
And grow.
CW (43:10):
... and own this, grow, become a really senior person. This is a springboard. And then now here comes somebody who's been doing this longer than you have alive and it's there now." That's really hard to deal with.
EW (43:27):
Well, Emmett said that had been working really hard, Herculean task, lots and lots of effort to ship the product. I expect that his CTO manager felt the same Herculean project, and then they just got bought by some big tech and maybe a gray-beard was hired from the outside, maybe gray-beard was transferred. So there are multiple ways he could have gotten there. But the manager, CTO, might have said, "I don't have to do all the work anymore. This is so exciting. And neither does Emmett. He doesn't have to do all the work anymore. We'll give him some help."
CW (44:05):
If they say that openly, that's great.
EW (44:09):
Well, it may not be said openly because you don't want to discourage people from working.
CW (44:16):
Then you run the risk of pissing people off.
EW (44:18):
Exactly as happened. It's really hard to give up what you've worked so hard on. It's just one of those situations where even in these trying times where a two week vacation may end up-
CW (44:34):
Being the kitchen.
EW (44:34):
... being in the kitchen. The distance, the perspective, is what's needed so that you come back to it with less resentment and more, "Okay, let's do something different now. Let's learn something different."
CW (44:49):
And that might be the option is like, "Okay, I'm in Big Tech now. This guy has got this project." If things aren't enjoyable, you have places you can go, other projects potentially. That's what happened me in Procket. I was like, "Okay, you got it. I'm over here now. I'm going to work on ... " Whatever I was doing, nulticast forwarding engine or something. I don't remember what I moved to, but I keep going back to it, it's hard to not feel like he says, this is reflecting on his performance.
EW (45:28):
And it may have nothing to do with his performance.
CW (45:30):
Yeah. Maybe it's something the boss was told to do.
EW (45:35):
And it may be something that does reflect on his performance, but it may not be entirely negatively. It may just be finally, I get to give this kid someone to learn from. You don't know that. I always made the mistake of assuming maliciousness when usually it was either just neglect or someone actually trying to help in a way that I didn't understand. That brings about a lot of frustration that isn't necessary at all.
EW (46:14):
Let's see. I want to talk a little bit about some of these other suggestions. JakeyPoo wanted to point out to try to look forward to working on a team. The teams come with more processes, communication, documentation. This might not be a demotion. It may just be a formalization. Being the person who won't let go of a design, it isn't healthy. It's better to look forward to working with others to make it better. You can make sure it's a strong design by making sure there are strong processes. So if that's interesting to you, you should do that. Jakey Poo also wanted to point out that there are a lot of people who have low pay because it's a scrappy startup. Now that you've been bought by Big Tech, you should be getting paid well.
CW (47:12):
Yes, yes, yes.
EW (47:13):
Dial back the effort, don't overwork yourself. Take this as an opportunity to rest for a little while.
CW (47:20):
Yeah, once you're required by Big Tech, you're no longer a startup and you are no longer required to act like you were at a startup.
EW (47:27):
And now the work that you've been put in have been replaced by you and someone else. So it's important to look at those. Let's see. Patrick chimed in that this wasn't a demotion and the CTO's job is not to interface with every individual engineer on every project. They don't have time for CTOing if that's what they're doing. Consider it more like a role that wasn't being filled beforehand. And then he talked some about the individual contributor track versus the management track and how those are separate, and thinking about that is tough. A lot of people on the Slack channel were kind of irritated that he didn't get to interview somebody who is supposed to be his new boss or his new major coworker.
CW (48:15):
Tech lead. Yeah.
EW (48:18):
I agree with that. I would be very irritated if somebody hired my boss without me interviewing them.
CW (48:27):
I didn't get to interview the guy who took over OSPF, but he did crash my car.
EW (48:35):
Oh, no wonder you just said, "Fine, goodbye." I remember that.
CW (48:46):
It wasn't my car at the time. It was his car. It was your car.
EW (48:50):
It was my car, but we had sold it.
CW (48:51):
We had sold him the car.
EW (48:52):
This is not an important part of the story.
CW (48:54):
Sorry.
EW (48:56):
But he hadn't actually registered his car.
CW (48:58):
We sold him the car, the next day he crashed it. He walks into my office and he says, "Do you have insurance?" I said, "That's not how it works, buddy." Sorry, sorry. Off track.
EW (49:13):
Let's see. SMG had some good advice. And it's mostly about being nudged in the right direction and learning as much as possible. Ben had some good advice that ended with, "If it all turns sour, get out."
CW (49:33):
That's advice all the time.
EW (49:36):
Well, it is, but in this particular instance, it-
CW (49:39):
That's what I was saying about being in Big Tech too. You have options beyond leaving the company. You should have options beyond leaving the company.
EW (49:48):
Tom pointed out that at least you got to ship. Imagine if this had happened right before you shipped and your design had been trashed. That would be horrible. I mean, that would be so much worse. At least you got to say, "Yes, that's mine."
CW (50:05):
Well, that's the thing with the imposter syndrome too because you can look back and say, "I did this. This is a thing. It is out there. It's done. Was it perfect? Maybe not. It doesn't matter. It's done. I did it." Not many people can say that. So you can feel whatever imposterness you want, but it doesn't matter. You did it. They can come in and say, "Well, we didn't like that." But who cares? You accomplished it. The goal was done.
EW (50:32):
Tom had a point about gray-beards that I thought sounded pretty relevant to me. "Here's a tip about gray-beards. They don't appear to do much actual work. The good ones enjoy hiding the little real work they do and are good at showing credit or even dispensing undue credit. They know this is how you get people to want to work with you." I hope you got one of those because it's true a lot of times when you're pretty senior in your career, you don't need the credit anymore. It's better if you don't get it because if you keep getting more credit, they keep giving you more work. But if you give the credit to other people, those people get the work.
CW (51:16):
That's true. And if you don't get any credit, you rarely get blamed.
EW (51:26):
Tom does point out that gray-beards do things. It just doesn't look like they do things, which I thought was a good point about how some people work versus others. There's a lot about slow down. Slow down now. Don't keep working these 60 hour weeks.
CW (51:48):
60 hour weeks? Are there that many hours in a week?
EW (51:50):
I don't know. I think there are. Yes.
CW (51:53):
Wow.
EW (51:53):
Yeah. Take a step back.
CW (51:57):
Yeah. You can't do that for very long. You end up with an ulcer.
EW (52:02):
Take this as an opportunity, not a demotion, not somebody stealing your baby, which it never was your baby. It's a company.
CW (52:11):
That's true.
EW (52:14):
Take it as a well-deserved vacation.
CW (52:17):
You know what's funny is I think about all these things and I may have talked about this on the show where you asked me questions, but you think about all these companies, how invested you get in the thing you work on there and how attached to it and how upset when things don't go your way or when somebody impugns what you've done or get credit. And then when you finally leave a place, at least for me, within like two days, I no longer care at all. Zero. It's like, "What was that? Why was I so angry? I'm no longer there." It's very strange psychology.
EW (52:56):
I mean, that's part of why contracting works for you, is that you lose some of that emotional investment and gain the distance.
CW (53:06):
But it's just a disconnect. Once you've quit, all of that emotion is like, "Oh, that guy was such a jerk to me. I don't work there anymore." What is that?
EW (53:14):
And yet now if you went back, having cleaned the slate, it might be easier to work there.
CW (53:20):
Maybe.
EW (53:22):
I remember I was at LeapFrog. I had a project I was working on that was not part of the normal toy lines. It was outside. I was told I had to get one of the toy managers to like it.
CW (53:39):
Toy manager.
EW (53:41):
So I got one on board and he liked it. We agreed that we were going to show it off together. I was excited because this was a possibility for my toy, just mine. He took it and he showed it to a bunch of people and it got okay reception, didn't turn out to be a toy. But I was really upset. I went to the vice president of engineering and said, "It was like he took my kids to kindergarten and I didn't even get to say goodbye." The sad thing was that Mike said to me, "Well, I have to tell you, I dropped my kids off at kindergarten this morning." I was like, "Okay, maybe that's not what I should have said. Maybe I should have chosen something else." So yeah, it's easy to get invested in these things. We put so much time and effort in them that if we didn't like them-
CW (54:36):
That's how they get you.
EW (54:37):
I know, but if we didn't like them, it would feel dumb and pointless.
CW (54:44):
It would feel like normal work.
EW (54:44):
Well, that's the thing, is that we forget that this is actually work. It's not us. It's hard to remember that.
CW (54:53):
Anything else?
EW (54:55):
That's actually all I captured for ... I thought I had more, but that's all I have right now. So do you have anything else?
CW (55:02):
I don't have anything else.
EW (55:05):
I forgot to bring down Winnie the Pooh.
CW (55:08):
Do you want to go get it?
EW (55:09):
No, no. This time let them make their own story up. We'll let Pooh and Piglet continue to wander around looking for woozles.
CW (55:19):
Is this airing?
EW (55:19):
Sure.
CW (55:19):
All right.
EW (55:25):
Thank you to Christopher for producing and co-hosting.
CW (55:29):
You got it.
EW (55:33):
Thank you to our Patreon supporters for being great and our Patreon Slack group for also being great. If you have any questions, concerns, jokes, whatever, show at embedded.fm, or hit the contact link on embedded.fm. I think that's all.
CW (55:53):
All right. Bye.
EW (55:55):
Bye.
EW (55:58):
Embedded is an independently produced radio show that focuses on the many aspects of engineering. It is a production of logical elegance and embedded software consulting company in California. If there are advertisements in the show, we did not put them there and do not receive money from them. At this time, our sponsors are logical, elegance and listeners like you.