438: There Is Nothing That Is True
Transcript from 438: There Is Nothing That Is True with John Taylor, Chris White, and Elecia White.
EW (00:06):
Welcome to Embedded. I am Elecia White, alongside Christopher White. Our guest this week is an author, John Taylor, author of "Patterns in the Machine."
CW (00:18):
Hi, John. Welcome.
JT (00:20):
Hello. Thank you for having me.
EW (00:22):
Could you tell us about yourself, as if we met at an Embedded Systems conference?
JT (00:29):
My name is John Taylor. My day job is, I am principal software engineer for medical company, Avanos Medical. I have been in the embedded space ever since college, about 30 years plus. Obviously, I have written a book, "Patterns in the Machine," we are here to talk about. That is pretty much me, other than I like doing this for- I write software even when I am not working, so that makes me a serious software nerd.
EW (01:02):
All right. We do this thing called Lightning Round, where we ask you short questions and we want short answers. And if we are behaving ourselves, we will not ask why and how, and are you sure, and what about. Are you ready?
JT (01:13):
Yep.
CW (01:14):
Should you trust the software folks?
JT (01:17):
No, never.
EW (01:18):
What is the best open-ended question?
JT (01:21):
What is the best open-ended question? I think that one right there <laugh>.
CW (01:26):
What is your least favorite programming language?
JT (01:32):
That is a good question. I have been doing C/C++ for so long, I do not do much else.
CW (01:37):
Well, I guess C/C++ then <laugh>.
JT (01:40):
That is one way to look at it. Yes.
EW (01:43):
How often is the hardware late?
JT (01:46):
It is always late. In fact, I just got the display for our board that I was supposed to have in June, and I got it Friday.
CW (01:57):
What was your favorite fictional robot?
JT (02:00):
Robby the Robot from Forbidden Planet.
EW (02:03):
Which direction should you move the cursor?
JT (02:06):
Move it to the right.
CW (02:08):
If software is not soft? Is hardware not hard?
JT (02:11):
Hardware is always hard.
CW (02:12):
And where does firmware come into that? <laugh> Is it firm or not <laugh>?
EW (02:17):
This is so weird!
JT (02:21):
It is mutable, but it is hard.
CW (02:22):
All right.
EW (02:23):
Do you have a tip everyone should know?
JT (02:29):
In my- You have made reference to it. There is a chapter 17 in my book, which is a bunch of rules for development that we have sort of talked over. My favorite one, and this is paraphrased from Reddit, from just a few weeks ago, "Two weeks of coding can replace two hours of planning."
CW (02:50):
<laugh> That reminds me of something a co-worker used to say, which was, "Oh, we do not have time to do it right, but we have time to do it twice."
JT (02:57):
Yep.
EW (02:58):
I have had people say that to me seriously, and I feel like maybe they did not understand the inherent sarcasm there.
JT (03:05):
<laugh>
CW (03:05):
<laugh>
EW (03:07):
So, Patterns in the Machine. This sounds like a William Gibson novel, is it?
JT (03:15):
No, I am not that talented. That was just the title that I came up with. I am not exactly sure how I got there. I just remember driving to work one day, and by the time I got there, that was the title I had decided on. But, to be honest, I really liked the title, and I think that is why I got publishers interested in the book. Not because of its content, but simply by its title.
EW (03:43):
It is a good title. But what do you mean by "patterns"?
JT (03:47):
"Patterns" is basically just a glorified buzzword for best practices, and obviously "best practices" is yet another buzzword. So it is really just- This is not going to be a short answer.
EW (04:06):
Oh, I am sorry. We are done with lightning round. Take as long as you want.
JT (04:08):
Okay. So best practices are really just ways of doing things, that your company, or you evolved over your professional career, or you have read books, or gone to conferences that, "These are very similar things that everybody has to do. Here is what we have decided that works best from our experience."
(04:30):
That is really what the book is about, is a bunch of patterns. They are not new or anything. What is different is the emphasis more on strategic thinking and strategic patterns, versus tactical patterns. So it is really just, "Here is a recommended way to do something that has produced results in the past for other people."
EW (04:54):
You have something called the "model pattern"? Is that what it-
JT (04:58):
Data model?
EW (04:58):
The data model.
JT (05:00):
Yes.
EW (05:00):
What is that?
JT (05:07):
That is just the name I have got. It actually started when I worked with Carrier back in early around 2012, 2011, somewhere in that timeframe. And they called it the "data dictionary." They had actually borrowed it from some concepts from AUTOSAR. We had a bunch of automobile refugees, when we did some hiring in HVAC.
(05:30):
Basically it is, you have data that exists, and your system is data driven, and nobody owns the data. The data basically is ownerless and it is unto itself, and you can read and write and get change notifications off the data. So it is not necessarily anything new. It may be new on how it gets applied in an embedded system. Originally when first experimenting with it, and doing it, it was like, "Oh, this is model-view-controller."
CW (06:04):
Hmm. Yeah. I was going to say, sounds similar.
JT (06:06):
Model-view-controller has all sorts of different connotations to it.
EW (06:11):
And that has changed over the years.
JT (06:13):
Yes, it absolutely has. So that is where the "model" part from "data model" comes from, is the model-view-controller concept, sort of the original MVC concept, before it was bound up into all this GUI stuff.
EW (06:28):
What about design patterns? It sounds a little like you are describing publish-subscribe, or is that not related?
JT (06:39):
It has publish-subscribe in it. So the core things of the data model are, one, it is type-safe. People think, "Well, that is not a big deal." But yeah, it really is <laugh>. So that is huge.
(06:58):
The other is, it is thread-safe. So I can safely read and write model points, regardless of what thread I am in, and I do not have to worry about thread safety.
(07:07):
And then they have publish-subscribe semantics. So a chunk of code, a module, can subscribe to change notifications to a model point, which solves a lot of- It is a very good golden hammer. Sometimes, I will admit, I abuse it when I really should not use it that way, but it works well for a lot of things, the publish-subscribe semantics.
(07:33):
There are a lot of other things that you can do with the data model, that is really handy. There are a lot of things, especially when it comes to debugging and troubleshooting, that the data model is very helpful with.
EW (07:45):
So is this a design pattern? You said "best practices," are these the same as what we are calling "design patterns," like model-view-controller?
JT (07:55):
So the data model is more of a design pattern, as you could think of out of the Gang of Four book or whatever. And then obviously, the book provides a concrete implementation of that pattern.
EW (08:10):
Right. You have a GitHub repository.
JT (08:13):
Yes.
EW (08:14):
But it has got like a whole framework thing.
JT (08:19):
Yeah. So the core of- So part of the book was- If you are going to have a book, you got to have example code, right?
EW (08:29):
Well, I did not, but I understand that some people might want to. O'Reilly certainly wants me to <laugh>.
JT (08:37):
Over the years, obviously I have worked on this framework. I call it loosely the "CPL C++ class library." It is just a realization of a lot of things that are in the book. It is by no means the only way to do what I talk about in the book. But the core things of that framework is it provides a platform independent OSAL, and then some very basic middleware type functionalities. One of them is an implementation of the data model pattern.
(09:14):
So yeah, there is a framework there. Then there is an example thermostat application that is built on top of it. Part of that was to validate some of the original concepts of the data model, and actually putting into practice. It is one thing to say, "Oh, hey, this is a great idea for a framework or a pattern." But until you actually use it in a real application, you usually find some rough edges that have to get knocked off and reworked, and things like that.
EW (09:49):
So you said "OSAL", operating system abstraction layer?
JT (09:52):
Correct.
EW (09:52):
It is like a hardware abstraction layer, except with any operating system, whether you are using FreeRTOS or Zephyr, or...
JT (09:59):
Correct. So it abstracts away the core OS primitives. Things like mutexes, threads, semaphores, elapse time, which unfortunately gets complicated on an embedded system if you want to be portable. So it is basic core functionalities like that.
(10:18):
One thing I talk about in the book is being platform independent. And once you can do that, you can write automated unit tests. So if you have an OSAL, whether you run on Windows, you run on FreeRTOS- There is even a bare metal implementation of it in the library, so you can run if you are bare metal on the target. Once you have that decoupling, it opens up a lot of flexibility on how you actually go about developing.
(10:49):
Going to go off on a little tangent here. On my day job, I do probably 85 plus percent of my work on the simulator, which is a functional simulator, not a dedicated hardware simulator of instruction set or anything like that. Not having to burn the flash every time, is a huge time saver.
EW (11:15):
I totally agree. Having an offline simulator is so important to me, especially functional. I do not necessarily want to test the SPI flash driver on a simulator, but I do want to test my KV store, if I am using a little database or EPROM like thing.
JT (11:36):
Right. So the functional simulator is not about hardware performance or anything like that. It is just if I can exercise my app, and yet I will worry about the performance, once we get on a target, and once, "Oh, I finally have my LCD, so I can see my UI on the target." We have developed a third of our UI, and have never had an LCD yet for the target. Anyway, I can talk a long time about the benefits of having a functional simulator.
CW (12:07):
I tend to agree. And there are many, many projects I wish I had one. But if you do not do them early, it is often too late in the project. I have come into many projects where it is like, "Where is the simulator?" "Oh, we do not do that."
JT (12:20):
Right. That is the point. That is absolutely, if you- It is really easy to do from the very beginning and almost no extra overhead. But if you try to retrofit it in later, it is almost impossible, depending on how later, later is. So it is one of my claims of, "You can do things the right way and not take longer." It is not hard to build a simulator. It does not add any extra work to build a simulator. It just requires more extra planning upfront.
CW (12:51):
While we are on that subject, how do you like to build simulators? Is there a platform or framework you go to? Or do you build from scratch?
JT (12:59):
It is just built. So my approach to building simulators, once you have the OSAL, you can run on any RTOS, on any operating system, right? And since it is functional, you do not really care about the realtime part of the OS. So you can run on Windows or Linux for your simulator.
(13:14):
When it comes to the hardware, you just do things on a case by case basis. It is like, "Oh, I have got extra off-board EPROM. Do I want to try to simulate the SPI driver, the I2C to it? Probably not. Do I want to simulate the command set to the E squared? Probably not." But there is another layer that says, "Oh, I am going to read and write to the EPROM. I am going to give it an offset. I am going to read so many bites, or I am going to write so many bites." That layer is very easy to write a simulator. You just replace the EPROM, the reads and write, with file operations. So you just do it on a case by case.
(13:58):
Like I talked about our project with the LCD. Our graphics package provides a windows simulator, so did not have to do anything there other than utilize the feature that came with the library. And some features you do not simulate, because it is not worth doing. So there is not a framework per se, other than the only minimum framework is that you have an OSAL that really is platform independent, and you just go from there. It is like the console UART, you use standard I/O on the simulator for your console UART.
EW (14:34):
Phillip Johnston over at Embedded Artistry wrote a blog post about the open-closed principle.
JT (14:41):
Mm-hmm <affirmative>.
EW (14:42):
He said that your book was important for that. And that he actually modified one of his courses, the "Design for Change" course, after thinking about how you put together your open-closed principle. What is that?
JT (14:58):
So, the open-closed principle, I do not have the text in front of me. It basically says something like, "Your code should be open to extension, but closed to modification." At a high level, what that means is, instead of editing existing code to add new functionality, you just add new code. If you go in Wikipedia or go to other people's more definitions of what open-closed is, it is basically talking about putting in interfaces, and then decoupling clients from the implementation.
(15:43):
For me, I somewhat take the open-closed principle pretty literally. My claim is if you write decoupled code, your code is change tolerant. And if you are change tolerant and you are decoupled, then if you want to add something new or go in a different direction, you are not editing existing code, you are just adding new code. You may be throwing some old code away, but you do not go in and hack up the existing code.
(16:17):
If you build things that way, there is sort of an art to it. I will go into that a little bit. What that does is, basically now you are building your application by off en la carte menu. It is like, "Oh, I need an egg roll <laugh>, let me go grab an egg roll. Oh, I want a taco, let me go grab a taco." I apologize, I am winging my menu metaphor off the top of my head.
CW (16:45):
<laugh>
JT (16:45):
But what you want to do is, when you build a widget, you want to break it down to- First thing you want to do is, you want to separate on how you create the widget, from how you use the widget. Because usually when you create a widget, there are some very specific things, some very dependencies that you cannot get rid of.
(17:06):
Take for example, file operations. A read-write operation on a file is pretty generic, has very little dependencies if you look at the actual function signatures and the data types that you need, right? So it is very easy to have a very abstract read-write interface to a file. But when you go to create a file, that gets pretty complicated, right? "Oh, it is a file name." "Well, is there a path in the file name? What is my file system? Do I have forward slashes? Do I have back slashes, for directory separators? Do I even have directory separators?" So if you can separate creating a file, from how you use a file, then that lets you basically by doing that- I know I am sort of meandering. You decoupled the usage from how you create it.
(18:02):
What that means is- I talk about this in the book, it is called the "main pattern," pretty unimaginative name. You push all the high dependencies to one spot, basically one place where you create everything, and then you wire all these subcomponents together. So you pass people reference to a file, instead of passing them a file name. This is just a rough example.
(18:33):
So your main code is not reusable and yeah, it is platform dependent, or at least parts of it are to a certain extent. But that is a very small portion of the code. That is like 3% of your code, and the rest of it is independent, and does not have these dependencies.
(18:52):
What that means is, if you want to add something new, you are not going to go in and- Now I have a new file that now runs on LittleFS on my board, I do not have to go in and hack up anything. I just add a new instance that knows how to use the generic file, read-write. Yeah, it has a specific create, but I can pass now an instance of my interface implementation that runs on LittleFS. Works just as well as if I was passing it a POSIX file or a Windows file handle.
(19:32):
What I was trying to get at is, if you want to add new things, set your code up so they are adds, they are not rework into the core code. Now, that main code that I talked about, that has all those dependencies, you do not waste your time trying to make that generic. You just say, "Okay, this is specific to this platform, building this variant of the product, and you just leave it at that."
EW (20:03):
That actually was going to be my next question about, how much effort do you put into this? There is another concept, the "you are not going to use it" idea of, do not overbuild your code. How do you balance that with the open-closed principle?
JT (20:22):
So from my perspective, first off, 80% of your code is never going to change. 20% of it is. You know a good idea of what some of that 20% is going to be. But the reality is you do not know. For my perspective, the easiest way to deal with it is, just assume everything can change.
(20:41):
But at the same time, when you go to, "Okay, I want to have a generic read-write file interface," coming back to that, do not spend hours or days trying to figure out, "Well, what is the best general purpose, best abstraction ever for that?" You say, "No, what maps? How can I define an abstract interface for my persistent storage, that I have on my board, that I will not need any glue logic to convert that interface into my implementation?"
(21:20):
So basically you build it for what you have, you put those interfaces in, and those semantics of the interface may, are going to, map one to one with your platform, but you still have that interface. And once you have that interface, that is your decouple point. So do not spend time trying to come up with the one rule, one interface, that is perfect. Do one that gets you the decoupling, that is very lean and easy to implement for your platform.
(21:49):
Worry about the second implementation or the third implementation later when you get there. Because a lot of times you never get a second or third implementation. Sometimes you do. Processors get changed in today's world, right, with the supply chain issues.
EW (22:05):
That was an interesting exercise in what we usually think does not change. I have seen companies that are always Microchip, always ST, you do not change the chip. Then suddenly they are changing the chip, and it is really traumatic and different compilers...
CW (22:25):
Do you have anything that can compute?
EW (22:27):
Right!
JT (22:28):
<laugh> Yes. There are advantages to- Oh, well today you do not really have a choice, but there are advantages to staying with a particular tool set. Because your EEs are familiar with it, your software guys, you learn the nuances of the chip. It is one less tool set you have to learn. But if you assume you will always use the same micro, that is- You cannot do that anymore, at least not on new designs, that would be very foolish.
EW (22:56):
So this CMSIS ARM interface? Does this fall under the patterns that you like to see? It does abstract a lot of the operating system away, as well as the hardware.
JT (23:11):
Yeah, so you are talking about the C M I S from ARM?
EW (23:16):
C M SIS.
JT (23:18):
Yes.
EW (23:19):
But yes, we are talking about the same thing, whatever, whatever the letters are.
JT (23:22):
Yeah. So that is ARM's attempt to provide some basic OS abstractions. And there is nothing wrong with that. Though in my experience, none of the available, whether they are open source or commercial, RTOSs really take advantage of it. There is usually some reference, if you look at FreeRTOS, there is a little bit in there that they utilize that for. I am not saying you have to use PIM's OSAL, you just need an OSAL. If the ARM one works for you, great.
(24:01):
What I am against is, and I cannot remember which, whose it was, but they defined their OSAL by their particular implementation. There was no definition of, "Here is the interface." They just say, "Okay, on Win32, here is what it is. On this hardware, here is what it is. And on Linux, here is what it is." Yes, they all had the same methods at the end of the day, but there was no one definition of what the OSAL was. There was just implementations of it.
(24:40):
So if you are trying to figure out, "Well, what is the source of truth for how mutex is supposed to work?" You did not have that. All you had was, "Oh, the mutex API is the same function signature across all three RTOSs." But there was no definition of the semantics for it. So if you are going to do something, if you are going to have an abstract interface of whatever kind, whether it is an OSAL or something else, the key thing is you need to define the semantics for it. There needs to be a single source of truth for that.
(25:11):
And that really should be in, as I talk about in the book, that needs to be part of your header file. So for you hardware guys out there, I am a firm believer in when you write documentation for your software and the header files, that, ideally, they read like data sheets. They are not just, "Oh, here is the syntax." But, "Here is what is expected of them. Are they thread-safe? Are they not thread-safe? Who owns the memory? Who does not own the memory? Does the caller have to supply it?" Things like that.
EW (25:42):
A lot of people have asked me, how do I define an API, or an interface, if that is what we want to call it. If you have a pile of code, and you have some ideas of what you want it to do, how do you get from here to there?
JT (26:01):
Um...
EW (26:03):
While following the open-closed principle, and making good APIs.
JT (26:11):
There are a lot of questions in that question.
EW (26:13):
Oh, but this is a whole career. Yes.
JT (26:16):
<laugh> Right. There are two ways you can do it. You can go top-down or bottom-up, or you can do a combination of top-down and bottom-up. So you either start at the top and you start your big bucket, and you start breaking it into smaller buckets, and you just repeat ad nauseum until you get down to something that ends up being at the code level, right?
(26:40):
The other is you can start, "Well, I know I have this CPU, and I know I have this, and I am going to have these many I2C peripherals, and these SPI peripherals, and I am going to have these A-to-Ds. Well, I am going to need drivers for all those. And this I2C is a humidity sensor, or a moisture sensor, and so I am going to need something for that." Right? So part of this is how do you decompose a big picture into something manageable?
EW (27:16):
Yes.
JT (27:21):
To be honest, I have been doing this so long, it is hard for me to think- I can get to A to Z quick, because it is like, "Well, I can go from A to D, and then from D, I can go to J <laugh>, because I have been there, done that. Then I can get to Z pretty quick." I do not think about it anymore, so that is hard.
(27:43):
Now I will back up a little bit. Let us say we talk about the driver up. We are doing a bottom-up design. Once again, I have done this enough, so I am going to say this is a way to do it. I am going to come back to the data model. One thing, and I do not talk about this in the book, but how you write drivers and how you write application code, is two different ways of doing things.
(28:14):
This was very apparent when I was at Carrier, and we were looking at the AUTOSAR stuff. They have very specific rules and layers when they come doing drivers, and all that kind of stuff. There is a reason for it. Your concerns are different when you are writing a driver or a COM stack or something like that, versus an application.
(28:38):
One thing the data model is very good about, is you can decouple your application from your driver, by putting in model point instances. So if I have a driver that reads 12 A-to-Ds, it can read those A-to-Ds, do whatever filtering however fast it needs to do it, all that, and then dumps the final result into a model point. And then the application just reads the model point. So an application can get an A-to-D value without having to know anything about the driver.
(29:10):
So one thing, if you are starting from the bottom up and it is like- Well first off, you should have done a system architecture, we will come back to that. And if you are doing a bottom-up, it is like, "Oh, I have got to get this data, whether it is inbound or outbound. If it is inbound, I am going to read this data, stick it in some model points based on my requirements and I will be done. If it is outbound data, I am going to get the source values from the application, it is going to write into these model points. I am either going to pull it it or get change notifications. Then I am going to go drive them out on whatever the physical hardware interfaces are."
(29:46):
Anyway, blah, blah, blah, I do not think I answered your question. It takes experience to learn how to take whatever requirements, because you never get all the requirements at once, then to try to figure out how to decompose that into something that you can actually implement.
CW (30:13):
I want to go back to the model for just a minute, because I really like the idea. Like you said, it is similar to how modern applications on desktops and mobile are often architected, where there is a model that is the business of the application, but it is not the thing that is necessarily displaying or moving the data.
(30:33):
But embedded, sometimes we have a lot of performance requirements, that require us to do DMA, or to have things be as low-level autonomously shipped from one end of the program to the other without a lot of layers in between. How can that be squared with the model?
(30:52):
Say I am doing something with networking where I have high performance, and I have got an interface on one side. I get packets in, I have got to do something with them, and route them out to another interface. And I have to do that as quickly as possible.
(31:04):
It strikes me that shoving that in a model, and then having somebody grab it out and do stuff, would add a lot of latency. Is there a way to have model and non-model things side by side? Or is there another way to square that?
JT (31:18):
Oh yeah. So to your point, the data model only works for certain things. If you are pushing a lot of bandwidth, or you are pushing something very fast, the data model is not really appropriate. It does not really work, for some of the reasons you just said. And that is fine. It is not an all or nothing thing.
CW (31:47):
Yeah.
JT (31:47):
Right? So some of your data, by the time it gets to the application, there is no super realtime performance that your application cannot meet. Or you are not streaming out a video stream, where you have got this constant data going out.
CW (32:11):
Sure.
JT (32:11):
So on one project I worked on, that we used the data model, is we had a cell modem, and the data model was used to provide the data that would go into the outbound. We would end up eventually doing HTTP requests. But that was all handled inside the modem. So we had this stack. We had a separate interface to that whole modem, which was a ITC, inter-thread communication, that was separate from the data model. So if we wanted to issue an HTTP request, or get notifications of the response, that was through a different kind of mechanism. But basic config data, data that was not necessarily static but did not change very often, when we were building or parsing those HTTP request-response, they would get either pull from model points or get stuff. But trying to actually do the HTTP requests, or do the responses, you use whatever is appropriate.
(33:31):
One example we had, we had to stream data between the off-board realtime processor and the Linux GUI board. That interface had realtime requirements and all that. That was not a model point based interface. That was a data packet driven interface. And yeah, some of the data that went across there, ultimately either came or was sourced from or stuffed into a model point.
(34:05):
The thing about PIM is it is very much, it is not, you can only do it this way. You do whatever way makes most sense, for whatever that is. If you end up having multiple sub-architectures, that is okay, with the one caveat. You actually thought about it ahead of time, and decided that was okay. And not necessarily just, "Oh, I ran into a new problem. Let me figure out how to- I will solve it now."
(34:32):
That brings me back to, you really need to do your software architecture first, and that is a true waterfall activity, before you get into detailed design. Those kinds of things would have been addressed in architecture. Because one of the things you want to do in the architecture is, what are my real-time constraints? What performances do I have? What issues do I have about RAM? Do I have enough to do it this way, or do I need to really save every bit and byte of RAM?
EW (35:01):
Does your book talk about how to do software architecture?
JT (35:06):
Yes. I am pretty sure there is a chapter in there. My definition of software architecture was pulled from somebody, I cannot remember the reference, but it is the important bits <laugh>. It is not that it always looks like X, Y, and Z. It is like, after your project was over, what were those- If you do a postmortem on it, what were the really important aspects? Or what you had wished you had not had a change by the time you were done? Those are the kinds of things that you need to talk about in your architecture. So if you have some real-time performance requirements, those need to be discussed in your architecture document, and how you are going to address them.
(35:58):
The easiest way for me is, let me start with my hardware. Let me go down every hardware interface and say, "What am I going to do with this? And what is its realtime performance?" "Oh, it is relatively slow, so I do not need to do a DMA driver for it." And it is like, "Well, no, I have got to sample my A-to-Ds at two kilohertz. And I have got 12 channels I have got a sample. I am going to want to DMA that, and set that all up. And I need to make sure that nothing else in the system is going to prevent that. And if there are any limitations on how much jitter I can have, and those kinds of things."
(36:40):
Those are the kinds of questions that you need to think about. I am just talking about drivers. That is easy when you talk about embedded systems, to focus on the hardware specific things. But there are other aspects to your application that you need to talk about. You need to ask the question and then say, "Yeah, this is a concern." Or, "No, it is not." Or, "It is somewhere in the middle, and this is how in general we are going to address it."
EW (37:06):
So when you say architecture and postmortem, and you mentioned not necessarily being able to describe how to do the layering of an API, how does your book help people who have not been through it? Or is it for engineers who have already been through a project or two, and are looking for more information?
JT (37:37):
It is not a beginner's book. It is more of a intermediate. It is definitely for people that have at least done a embedded project. It is not a recipe book. It is not, "Here is a template for this kind of document, or whatever." It is really talking about, "Here are things you need to think about. You need to put some focus on strategic thinking, and those problem solvings, before you jump into the tactical.
EW (38:20):
What made you decide to write a book?
JT (38:23):
I honestly do not remember. I do remember- So this all happened around the COVID timeframe, slightly before. I just decided that I was going to write a book, sort of knew what it was about, not exactly. And I was going to do it whether I had a publisher or not, even if I had to self-publish, I was going to go ahead and finish it. It went from there.
(38:56):
It took probably about a year and a half from start to finish. Now, all that said, I had been working- I pulled from my code bases. I had been doing the example thermostat app in my spare time, for years <laugh>. It may have took less than a year and a half to go from, "Okay, I am writing the book," to having it published. It took decades <laugh> of source material to get there.
EW (39:23):
What books, blogs, or conferences have been influential for how you think about embedded system development?
JT (39:30):
I am a computer science graduate. I am not an EE. But I have only done embedded. That was my first job out of school. If you remember "Embedded Systems Programming" when it was the magazine circa 1990s? And when Embedded System Conference was a conference in person? COVID pretty much killed that. I think it had started to go away before COVID though. So things like that.
(40:01):
My second job out of college, I was working for Allen-Bradley, up in Wisconsin, working on their industrial automation. Their group had complete- If you remember back in the methodology war days, there was a methodology called Shlaer–Mellor. They were really into C++ and Shlaer–Mellor methodology, and they sent everybody off to training and we had all these case tools. So that was my big exposure to C++ and object oriented and all that.
(40:40):
If you are not familiar with Shlaer–Mellor, which most people are not, one of its key tenant was, their approach was, it was a translation step from design to code, and not an elaborative step from design to code. So basically they were advocating, figure out all the problems in your design, solve the solution in design. Once you have done that, if you have the end-all be-all case tools, you could push a button and your design documentation, they could auto-generate the code. They never got to that point, but that was the idealized concept that they had.
(41:22):
That was a big influence on me. Another was my first boss out of college. He was a really <laugh>. He had a lot of patience with a new grad <laugh>, let me put it that way.
EW (41:40):
I have some listener questions. From William, "What is your take on Rust?" Although I feel like that may be kind of a...
CW (41:51):
Bait?
EW (41:52):
Bait. Given that you have a blog post about that.
JT (41:57):
Yeah. My problem with Rust- Well, Rust itself, I do not have a big problem with. Anything that finds errors at compile time is a good thing. My problem with Rust is the problems it solves. I am trying to figure out the best way to say this is, I do not see those problems day to day. I cannot remember the last time I had a data corruption issue due to threading. 99% of my designs do not do dynamic memory allocation. Yes, there is an occasional buffer overflow error, but memory errors are not- They are very infrequent.
(42:48):
So all the things that Rust pitches that it solves, those are not the big ticket items, when it comes to the problems we have in the projects I work on. So part of it is Rust solve some problems, but it does not really- It is not solving problems that I have on a day-to-day basis. Those problems just are not there. Yes, there are bugs that fall into those camps, but they are the exception. They are not the rule.
(43:21):
The other complaint I have about Rust is, at a big picture, I see this as companies and big tech basically wanting to commoditize developers. They want the lowest cost engineer. Instead of realizing that, "Oh, software is hard and there is a certain skillset, certain ability that you need when you hire engineers." They want to get it dumbed down enough, that anybody and everybody can write software. And now we are the equivalent of McDonald's.
CW (44:03):
I am going to push back on that a little bit. Have you tried to use Rust?
JT (44:06):
No.
CW (44:06):
It is not for people who do not know what they are- I have tried to learn some Rust, and it is extremely difficult. <laugh>.
EW (44:15):
One of the key Rust phrases that I have heard from people who really like Rust is, that it is hard to write bugs in Rust because it is hard to write Rust.
JT (44:24):
<laugh>
CW (44:28):
I agree with a lot of what you are saying, especially on the solving problems that maybe are not applicable to all situations. I think they are situating themselves to safety critical, or security focused things, where you really do not want people having any buffer overflows, because that would be a disaster and would lead to privilege escalations, and things like that. Which maybe not what happens on a small control system somewhere. So yeah, like you said, it is not applicable everywhere.
EW (45:03):
But it is not easy.
CW (45:03):
<laugh> No, it is not.
EW (45:03):
It is not an easy language.
JT (45:07):
Actually, that makes a lot of sense. Because when I look at the projects I have been on, the issues do not go away. How do you own- You have to pay attention to thread safety. You have to pay attention to who owns the memory, right? How do you make sure you do not have buffer overflows? Those issues do not go away in a language like C/C++, it is just those burdens are shift to your design. So if you have a crappy design, you are going to have those problems, right? So it is still hard. It is just that-
CW (45:42):
Moving the hard around. <laugh>
JT (45:43):
Right. You are moving the hard around. You are moving it from, "Okay, here is the language I have been using for 20 years. How do I solve the problems?" Or, "Here is a brand new language, that is different, that prevents these problems, or poses a model on you that then prevents those problems." But I claim that even if it prevents those problems, it still leaves me with the ones I face today.
(46:06):
So we had a race condition- In fact, there is a race condition in the model points in their publish-subscribe. It is not a data integrity issue. It is an issue of how the observers get synchronized with the data point value. And if the data value changes very fast, or changes at exactly the right time, your application appears to get a double notification. Right? So there is what I consider a race condition, but it is not something that Rust is going to solve. It is a problem that the definition of the interface, and the semantics of it, was not correct. I do have a fix for that now.
(47:00):
So it is like, "Okay, if I spend all my energy to learn Rust, it gets rid of a class of bugs that I do not already have. But I still got to deal with the bugs I already had." I do not see- Anyway, I am not saying I will never do Rust. I am just not in a hurry right now.
CW (47:18):
Fair enough.
EW (47:19):
Fair enough. We will still get emails, but that is fair.
CW (47:21):
We will not send them on to you. We will absorb them.
JT (47:24):
<laugh>
EW (47:26):
<laugh> We will put them in our Rust bucket.
CW (47:28):
<laugh>
EW (47:31):
William also asked, "When you see good patterns used technically correctly, but with no thought put into the usability of the product, creating a poor user experience, what do you do?"
JT (47:48):
Uh...
EW (47:49):
He goes on to say that, "We have all used software and hardware that is a technical marvel, but it is awful to use for anyone who is not an engineer." Just to give you more.
JT (47:58):
Oh, I see. Yes. One of my pet peeves on that one is, it is a little better now, but if you have ever owned a Ford and the Ford Sync, that user interface for the Ford, especially the first two or three generations, is horrible. Unless you have the super top-end screen size. I do not know if you are familiar with that, with the Ford Sync interface.
CW (48:25):
I think I accidentally used it in a rental car once.
JT (48:28):
It is just horrible. And I can see where it makes perfect sense to an engineer. But it makes no sense to your typical driver. I am an engineer and I do not understand it. The moral of that story is, you do not have your engineers design your user interface, because you are going to get a user interface that makes sense to an engineer, and they are so not the demographic of your end product.
(49:03):
How do you do that? That is really your project, your company, how are they going to resource your project, right? If they are not willing to pay for somebody for industrial design, or design factors, you are going to get what you get.
EW (49:24):
One last question about your book. You wrote your book with your brother. Christopher writes music with his brother. What was it like writing a book with your brother?
JT (49:37):
It went actually pretty smoothly. I do not have anything to compare it against. I will say when we were growing up, we did not get along as young kids, or teenagers or even early college. We just did not get along. But we had both gone and gotten married, and been physically separated. Then when we came back together, I guess we forgot why we did not like each other. I do not know.
CW (50:10):
<laugh>
JT (50:10):
<laugh> But it worked out really well. Shortly before we wrote the book, Wayne and I actually worked at the same startup together for a while. I left shortly before we started writing the book. It went really well. I know my limitations as an author. You should have seen it in the emails. So if Wayne had a suggestions, it was like, "Okay, fine, I will take it. I am not going to push back." One of the things he would always push back when I was generating content was, do not write a API reference book. You are writing for- You need to keep the readers engaged. So it is important to get ideas and concepts across, and not worry about if that sentence is 100% factual in all cases. Does that make any sense?
EW (51:12):
So much sense. That is the problem with embedded systems. There is nothing you can say that is 100% true all the time.
JT (51:19):
Right.
EW (51:21):
"Use DMA." No, there are plenty of places where you should absolutely not use DMA. "Use a watchdog to keep your system more stable." Well, yes, but do not let it kill you when you are doing over the air transfers, because then you will die, and it will be bad and you will get a brick. There is just nothing- I teach a class about embedded systems, and they ask questions and I answer them, but I always have to say, "Well, in most cases. In general. For most of the things I have worked on. Except for some cases <laugh>." There is nothing that is true.
JT (51:58):
Right. And I would want to put all these qualifiers into one sentence, and he is like, "No, do not do that <laugh>." Really Wayne is actually a lot more interesting than I am. His sense of humor is a little dry, but he is hilarious. If he had been on this call, he would have kept us entertained.
EW (52:25):
Actually I do have one more question. What is your favorite part of the book?
JT (52:30):
Obviously the chapter 17 was a fun chapter to write. "The Tao of Development," or I forget what the title is. After proofreading the book so many times, I have a hard time reading it over again. It has been a couple years now. I really enjoyed writing it all. Wayne knew when to make a comment, because he has done co-author stuff with other people before.
(53:03):
I sort of knew what I wanted to say. So, once I was out on paper it was like, "Okay, we are good." It was a really interesting experience. I plagiarize myself at work. I will sit there and copy paste stuff out of my book into documents. Or if I am having to write up a how-to or something, I will grab a paragraph here or sentence here, and paste it in. So part of this is, all this stuff that was in my head, now that I have it on paper, I do not have to keep remembering it all.
EW (53:44):
Oh yeah, I do that too. Also, if you go to an in-person interview, and you take your book in your bag, there is plenty of fun with pulling out the book to answer a question. "See here in my book I wrote, how did you solve this problem?" <laugh> It is just brilliant.
JT (54:03):
Yeah.
EW (54:05):
Well John, do you have any thoughts you would like to leave us with?
JT (54:08):
So I was thinking about that, and obviously there are a lot of different topics in the book. There are rat holes you can go down. But I would say, and it frustrates me at work, because I just see history repeating itself over and over again. So my parting thought would be, "You do not have to settle for the status quo. You can write quality, robust software in the same time or less, than just sitting down and hacking it out at the keyboard."
(54:47):
I wish I was able to be able to convey that, in such that people can understand that and like go with it. But my practice, until somebody actually is exposed to it and has that light bulb moment, they do not get it. But you do not have to settle for the status quo. We can write software that is change tolerant, in just the same amount of time or less, than writing crappy software.
EW (55:17):
Our guest has been John Taylor, author of "Patterns in the Machine: A Software Engineering Guide to Embedded Development." He also has a blog on PatternsInTheMachine.net. And he is a principal engineer at Avanos Medical.
CW (55:33):
Thanks John.
JT (55:34):
Thank you.
EW (55:36):
Thank you to Christopher for producing and co-hosting. Thank you to our Patreon listeners Slack group for questions. And thank you for listening. You can always contact us at show@embedded.fm or hit the contact link on embedded.fm.
(55:49):
Now a quote to leave you with, from Jeff Atwood of "Effective Programming: More Than Writing Code. "As a software developer, you are your own worst enemy. The sooner you realize that, the better off you will be."