401: Oil and Water
Transcript from 401: Oil and Water with Miro Samke, Elecia White, and Christopher White.
EW (00:00:06):
Welcome to Embedded. I am Elecia White, here with Christopher White. We are happy to welcome Miro Samek back to the show.
CW (00:00:16):
Hi, Miro.
MS (00:00:18):
Hi, thank you for inviting me back.
EW (00:00:20):
Could you tell us about yourself as if we met at an embedded systems conference?
MS (00:00:27):
Yeah. Thank you for inviting me again to your show. I have been developing embedded real-time software for almost three decades now. And I started with programming nuclear physics experiments. And later on I worked for GE Medical Systems where I worked on diagnostic x-ray machines.
MS (00:00:51):
Then I moved on to Silicon Valley, where I worked for a couple of companies working in GPS technologies. But around 2005, I founded my own company, which represents my main interest, which is modern embedded programming.
MS (00:01:08):
I also wrote a couple of books about it, about practical aspects of using UML state machines, UML statecharts, and event-driven active objects in the context of deeply embedded systems, such as microcontrollers.
EW (00:01:25):
So are you ready for lightning round, where we ask you short questions, and we want short answers, and if we're behaving ourselves, we won't ask "How," and "Why," and all of that?
MS (00:01:35):
Sure. Please go ahead.
CW (00:01:36):
Do you like to complete one project or start a dozen?
MS (00:01:40):
Complete one project.
EW (00:01:42):
True or false? Most embedded systems are either primarily state machines, or continuous monitoring devices, or some combination of the two, but usually more one than the others?
CW (00:01:52):
That's a long question.
EW (00:01:53):
It really was a long -
CW (00:01:54):
A lot of hedging.
EW (00:01:54):
It was a lot of hedging there, wasn't there?
MS (00:01:56):
Yeah. Yes. I would agree by and large.
CW (00:02:00):
What else is there? What's a favorite course that you took in your academic career?
MS (00:02:08):
Well, I studied physics, so they grilled us on math a lot, analysis, and geometry. I think maybe quantum mechanics was the most interesting.
CW (00:02:20):
Yeah...I took that too. Pretty cool.
EW (00:02:24):
Do you have a preferred way to learn new technical things? For example, reading the manual, watching videos, or trying it out without reading the docs?
MS (00:02:31):
Oh, that's difficult. Well, I certainly read first, but then I experiment on my own. I'm a hands-on person, so I need to experiment.
CW (00:02:42):
We might have asked you this one last time, in which case I'm interested to know if the answer changed, but I'd have to go check. Favorite fictional robot?
MS (00:02:50):
...I watched recently Terminator 2 again, and so T-1000, Arnold, I think is the top of my list now.
EW (00:03:02):
Do you have a tip everyone should know?
MS (00:03:05):
Yeah. Well, I would say that in embedded systems, nothing works until everything works.
CW (00:03:12):
Oh, I thought that was going to be a much shorter sentence.
EW (00:03:15):
You were going to stop with nothing works?
CW (00:03:17):
After the last couple of weeks I would stop there. Yes.
EW (00:03:22):
Actually, could you expand on that? Nothing works until everything works.
MS (00:03:27):
Well, we need to start, always start, with a working system, and only make...small incremental steps to keep it working at all times.
MS (00:03:41):
Very often, for instance, users of my software take the software, and immediately want to go to their own board, and their own example that interests them, instead of starting with a set of tested examples that are provided as well. So I would recommend just buy a very inexpensive board...even though it's not exactly your target.
MS (00:04:10):
And try those examples first as is without changing anything. You already would be testing many things. The software, the compiler, everything around it, the connectivity. And then you can only make very small, incremental steps from there.
CW (00:04:30):
I didn't do quite exactly that thing, but that's what I did kind of starting with a new client this month is, they had a particular target. And I got some dev boards that were similar and got familiar with the IDE, which I'd never used.
CW (00:04:43):
And I just got the examples from the vendor, and tried a bunch of them to make sure everything worked, and got familiar with things. And it made it a lot easier when I finally did import the client software and start building it.
MS (00:04:55):
Yeah, absolutely. It builds the confidence that the stuff works, first of all...And then people take too large steps. And in order to move fast, you have to take baby steps, really small steps.
MS (00:05:10):
If you take one bigger step, you find you will waste a lot of time trying to find out why things don't work anymore. And taking smaller steps actually ends up that you will be moving faster.
EW (00:05:28):
I feel like this relates to when I read recipes online where you'll read a cookie recipe, and they'll say, "I replaced the sugar for honey. And it didn't work at all." And it's just, yes, I see that a lot with embedded systems as well, that you have to run what they said to run first and then make the changes.
MS (00:05:50):
Right. There's a million ways it can go wrong. And only a few ways for this to go right. Probability points to the way of doing this.
EW (00:06:01):
You have a YouTube series, teaching embedded systems programming, teaching Modern Embedded Systems Programming. What's the modern there for?
MS (00:06:12):
...Well, first of all, what I define as modern, everything invented past mid-1980s I call modern...And the other stuff I call traditional, that the traditional approach is still the dominating, absolutely dominating, approach in our field.
MS (00:06:30):
So we probably didn't really...learn many lessons past mid-1980s, I would say, which is a kind of sad state of affairs here. But that's why I call it modern. I'm trying to point out interesting and important developments that happened in the '90s and maybe early 2000's.
CW (00:06:58):
But even if you apply more modern techniques, we're still stuck with a lot of old stuff. I2C and SPI aren't going anywhere, and they're from pre-Pac-Man era.
MS (00:07:07):
Right.
EW (00:07:09):
Pre-Pac-Man era. Thanks. That's a nice line in the sand. Okay. So, I mean, we've gotten past assembly.
CW (00:07:19):
Yay.
MS (00:07:22):
Yes. So the whole premise of this course is, I see it as a missing curriculum, because our discipline falls, as you know, somewhere between EE, electrical engineering and CS, computer science. And yet our field is distinct and different from both of those.
MS (00:07:38):
...And not many universities actually teach this middle ground. So there is of course a need, but also an opportunity to fill the gap and to teach this part.
EW (00:07:56):
Do you think that we will be seeing more embedded systems curriculum going forward? Or do you think it will always be, "You learned half of your education in embedded systems in school and the other half you just have to pick up on the job?"
MS (00:08:12):
I don't really know what's on the mind of universities and schools that teach this. Some schools do. Some schools do, and they are proudly specializing in producing embedded engineers, but not too many right now.
CW (00:08:31):
I mean, it's certainly better than when we went to school, because we basically got C++, and/or Java a little bit after us. And then if you wanted any embedded, there was a microprocessors course in the engineering department, right?
MS (00:08:43):
Right.
EW (00:08:44):
Yes. Yes, of course. And now there are far more robotics teams.
CW (00:08:47):
Yeah.
EW (00:08:48):
And other hands-on interdisciplinary clubs.
MS (00:08:53):
Right. But they often start with Arduino and maker platforms like this. And, for instance, in my course, I deliberately didn't choose Arduino for the reason that we probably will discuss later.
MS (00:09:06):
But...it is geared towards newcomers to the field, and makers, and hobbyists, but not so much to professional embedded developers that will then later go into the industry and develop production-level code.
EW (00:09:27):
It's not for professional - ?
MS (00:09:30):
I would say that Arduino isn't -
EW (00:09:32):
Oh, okay.
MS (00:09:32):
- meant for that.
EW (00:09:33):
Sorry. I thought you were talking about your course and I was like, "I think, maybe?"
MS (00:09:37):
Yeah...I'm talking to what Christopher just...said, that it's now easier. Yes. And we have maker platforms like Arduino which didn't exist a few years back, but they are not necessarily geared towards professionals.
EW (00:09:56):
Fair enough. I mean, I wrote a blog post about not using Arduino.
CW (00:10:01):
I think it was called "Don't Use Arduino (For Professional Work)," so you're right on there.
EW (00:10:05):
Yeah. So I think we're in strong agreement on that. And I'm teaching a course too, which is totally different from yours. And it's been a little hard to convince people that you can't take a project that is based in the Arduino IDE into an interview, and talk about it.
EW (00:10:24):
And...even if you put a lot of work into it, you can't expect them to take it as seriously as if you just used a compiler.
CW (00:10:37):
Yeah.
EW (00:10:37):
GCC, IAR, Keil, anything other than Arduino IDE.
MS (00:10:43):
Right.
EW (00:10:45):
...You end up showing a lot of code in your videos.
MS (00:10:52):
Right.
EW (00:10:52):
And you're compiling for the Tiva platform. Is that right? Actually, maybe I should let you tell. What platform are you compiling with? And what tools are you using?
MS (00:11:03):
Yeah. So please remember that the video course started back in 2013, right? So it's a while back. And so back then I chose Tiva C Launchpad board...and the choice was quite deliberate. I had the whole list of features that the board should support.
MS (00:11:26):
First of all, I wanted it to be based on the modern ARM Cortex-M processor, as opposed for instance to AVR or something older. Second, I wanted to have a debugger, real debugger, interface built into the board. I wanted to have a few LEDs and a few switches on the board.
MS (00:11:46):
I wanted to have accessible pins, because in later lessons I used a logic analyzer, so I wanted to easily connect to the board. I wanted to have a virtual COM port. Right now I don't make much use of it, but in the future, I will talk about concepts like software tracing and testing on the target. So this will be very useful.
MS (00:12:10):
And I wanted this to be available for a long time. And so far, Tiva is still available. It can be bought almost everywhere, from Digi-Key and other vendors. And last but not least, it had to be quite inexpensive. So it retails for...12.99 or something like this.
MS (00:12:32):
And this is important, because many people who participate in the course come...not necessarily from the U.S. or the European Union, but from countries like the Middle East, South America, India, China. And for them it would be a problem if the board would be too expensive.
MS (00:12:51):
So that's why I chose Tiva. If I...had the choice right now I would probably go with STM32 Nucleo, although Nucleos don't have enough LEDs for me.
EW (00:13:09):
There are some Discovery boards that have rings of LEDs now.
MS (00:13:13):
Right. These are nice. But some Discoveries didn't have virtual COM ports...This was an important feature for me. What did you choose?
EW (00:13:27):
Yeah...A Cortex-M was all I limited it to.
MS (00:13:33):
Right. So you don't specify beyond that.
EW (00:13:36):
I recommend strongly that they use an STM32 and a Discovery board. And I gave them a list of things that they could use. And that has worked out well for me, because I have cohorts, and they go through together. And then they talk about their boards, and they end up with more information than I could possibly give them.
MS (00:13:54):
Yeah. STM is a good choice right now. So -
EW (00:13:57):
Yeah.
MS (00:13:57):
Yeah, absolutely agree.
EW (00:13:59):
So your class has been going on since 2013, you said? And you have, was it 60, 80? It was a lot. You have a lot of courses, a lot of videos.
MS (00:14:11):
Over 40 now.
EW (00:14:12):
Over 40.
MS (00:14:13):
43 or something.
EW (00:14:14):
Have you planned out what you're going to teach when?
MS (00:14:18):
Yeah, well I drew a chart, a dependency chart if you want, so which subject depends on which other subject. And so I wanted to start from the beginning to kind of establish the terminology and common understanding of things. But then it took me eight years to get to my favorite subject, which is state machines.
MS (00:14:42):
So that's why I had to talk about things like RTOS much earlier than that...So you see that RTOS is a little bit lower level than hierarchical state machines and active objects. Yeah. So, yeah. So the dependency, what depends on what, and what I have to talk about first, and can then build upon it, determines my order of presentation.
EW (00:15:13):
How did you choose your theory of order? I mean, some things you do have to explain earlier, but did you base it on your book or just how you wish you had been presented the material?
MS (00:15:27):
I wanted to end up with something like active object framework. And so I worked backwards, so to speak, working which concepts need to be presented to get to that point. And so for instance, at some point I had to talk about object-oriented programming.
MS (00:15:46):
Because it's a framework based on classes, and instances, and inheritance, and polymorphism. So I needed to cover those concepts. I needed to talk about real time, and scheduling, and real-time issues, and some things like rate monotonic scheduling analysis...
MS (00:16:04):
Mutual exclusion blocking had to be covered before that. Of course the interrupts and race conditions have to be covered even earlier than that. So that's why I worked backwards. So only now I'm slowly getting where I wanted to end up.
EW (00:16:24):
It looks like you're finally to state machines, which is one of the things that I think you really enjoy talking about.
MS (00:16:31):
Yes. I find this subject a little bit hard to explain, because...I see a lot of misconceptions. And the subject is complex, because there are so many kinds of state machines even. And there is no single terminology to even name them.
MS (00:16:55):
For instance, I talked at some point about input-driven state machines and contrasted them with event-driven state machines. Now input-driven state machines is something that I just found in the literature, but some people call it periodic state machines or even some other names.
MS (00:17:15):
But the point is that they are not exactly the same, and they cannot be translated into UML state charts later on. They are not quite compatible. So that's why state machines, the subject of state machines, takes eight lessons. And probably it's not the end of story yet.
EW (00:17:34):
It was odd for me to see that you had eight lessons on state machines and four lessons on RTOSs. And to me, those are opposite levels of complexity. Why, I guess?
MS (00:17:47):
Yeah, well, exactly, because RTOS is maybe more crisply understood concept. Of course you have to start with the definition, what you consider RTOS. Some people consider RTOS the whole software. It's the kernel. It's the file system. It's the communications stacks, and so on and so forth.
MS (00:18:12):
So I like and I prefer to use an as narrow as possible meaning of the word. So for me, RTOS is the RTOS kernel only. This allows me to be more precise when I talk about things, not to put too much meaning into one word. So for instance when I narrow it down that way RTOSs could be explained in about six or seven 30-minute lessons.
MS (00:18:48):
Of course I didn't cover in detail the communication mechanisms within the RTOS. But it was precise enough for my purposes.
MS (00:18:58):
Interestingly, things that many people will consider tricky, like context switch, could be explained within maybe the first two lessons, actually only the second lesson showed how to automate context switch in assembly.
MS (00:19:16):
But things like semaphores, mutexes, and other communication mechanisms could not be covered, could not be explained that quickly at all. This is where the complexity of the RTOS is.
MS (00:19:27):
And many people don't appreciate it, that in order to use those mechanisms correctly, you have to be quite versed. And these mechanisms are also very tricky to use correctly.
EW (00:19:42):
Because of concurrency or other reasons?
MS (00:19:46):
Yeah, well, I think that the biggest problem within RTOSs is the issue of blocking, and this is the root of many problems in an RTOS. So of course you can have race conditions, and there are mechanisms to prevent them, such as mutual exclusion mechanisms, such as mutex, for instance.
MS (00:20:06):
But then you introduce additional blocking potentially. And this, of course interferes with your real-time response. For instance, I saw on your show, it was an interesting episode, 335, I have here. Patching on the Surface of Mars with Joel Sherrill, I guess. He talked about RTEMS, open source RTOS.
MS (00:20:29):
And...among others, he talked about a hiccup in the real-time response, where a highest priority task that normally took one millisecond to execute all of the sudden started to execute in 17 milliseconds or something like this, order of magnitude worse.
MS (00:20:45):
And in the same episode, he explained rate monotonic scheduling and this method of handling real-time response.
MS (00:20:56):
And then the problem is immediately apparent for me that RTOS, even though it has been designed specifically to enable real-time monitoring scheduling, for instance, at the same time, because of those interactions and mutual exclusion mechanisms defeat the purpose right there.
MS (00:21:18):
Because sometimes you have a completely unresponsive system that apparently doesn't meet any real-time deadlines.
CW (00:21:26):
Yeah...That's an interesting point because I think of mutexes, and semaphores, and all those things, it's fairly simple concepts, but most of the places I've used them...haven't been in places where preserving a real-time response was super critical.
CW (00:21:43):
They were using mutexes to keep things from clobbering other things, make sure things were happening in the right order.
CW (00:21:50):
But adding the additional requirements of not having those things get in the way of your real-time response just makes things a lot more complex for figuring out how to architect your system. And they're not great tools for that, like you said, because they're blocking.
MS (00:22:04):
Right. Rate monotonic scheduling, for instance, does not take into account mutual exclusion. And if you want to, really, then you would have to take the worst possible case and count this towards the CPU utilization, the central concept in rate monotonic scheduling.
MS (00:22:24):
...And then what you will end up doing is that most systems wouldn't be schedulable, meaning that they will have no chance of meeting the deadlines always. This would work for soft real-time systems, but for hard real-time systems, it is unacceptable.
EW (00:22:40):
So should we ditch all the real-time operating systems and just use state machines?
MS (00:22:45):
No, no, no. That is exactly the point, that I introduced a real-time operating system and had to do this. Because it's a stepping stone in understanding active objects and hierarchical state machines, event-driven state machines that work with active objects.
MS (00:23:05):
So what active objects are, there's actually a design pattern that prescribes how to use an RTOS, if you will, safely.
MS (00:23:17):
So there are just simply restrictions on the naked threads, on top of the RTOS threads, that when those best practices are implemented, you have a chance to build a correct system much, much better than without them. So first of all, you should avoid any sharing of resources among threads.
CW (00:23:40):
Any sharing of resources among threads but you can communicate between threads. I could pass something.
MS (00:23:46):
Right. Now, the second thing is, in order to communicate, you have to designate special objects for that.
CW (00:23:52):
Got you.
MS (00:23:53):
And these are called events or messages, and they are specifically designed for communication only. And they are exchanged between those threads in a thread-safe manner. And it is kind of a component that is on top of the RTOS...Many RTOSs, for instance, provide message queues.
MS (00:24:13):
So message queues could be used to exchange those events safely between those tasks. And so this means that you should prefer asynchronous communications by means of those events. By asynchronous I mean that you just post an event with a queue, but you don't wait for the handling of this event for the processing.
MS (00:24:37):
And then the third thing is that you should structure all your threads around the event loop, meaning that there will be just one single point of blocking within the endless while (1) {} loop in each thread or each task at the top of the loop.
MS (00:24:53):
And then when an event arrives the event, this blocking call unblocks and the event is processed in the application level code. So some application level code is called to handle this event. The event has to be handled without any further blocking. And then you quickly return back to the event loop.
EW (00:25:17):
It's funny that you're using the TI board. Because this is a pattern I see a lot in the TI code, especially their BLE subsystems, that every thread really is its own little kingdom, and it spends most of its time waiting. And then it doesn't do any waiting until it gets back to waiting for whatever happens.
EW (00:25:41):
So there's no, "Oh, I received some data. I'm going to print it out the serial port and wait for that to be done." No, no, no, no. If you receive the data, you can print it out the serial board, but you can't wait. You have to go back to your single point of waiting.
MS (00:25:56):
Right.
CW (00:25:57):
It's funny, because...when I started my career, I started in networking protocols, and this was how all code was written. Because networking protocols work this way. You're passing messages between separate devices, and you are receiving messages.
CW (00:26:13):
And then you see what kind of messages, and that's an event. And then you have this loop that handles it, and then you handle it, and you get out. And that's how everything was architected. And so when I did other things that seemed like a natural way to do it, but I don't see a lot of people using this pattern that often, -
MS (00:26:29):
Right.
CW (00:26:29):
- day to day anymore.
MS (00:26:31):
Yeah. So there are of course many, many dangers now with programming, I mean many pitfalls that you potentially can fall into when you start doing this. So first of all, RTOS provides a whole assortment of mechanisms, all of them based on blocking.
MS (00:26:51):
And so it is very tempting, for instance, in your event handler, in the code that is supposed to handle an event very quickly and return, to call something like a small delay, blocking delay, or you have a situation where you sent some information, for instance, a packet through I2C port, but then you expect some reply.
MS (00:27:15):
So what you do is you wait in line for the reply. So this obviously holds up the whole event loop processing, and you are becoming unresponsive to any new events that start accumulating. A new event queue can actually overflow it. So this would actually derail the whole mechanism.
MS (00:27:36):
The point is, though, that RTOS provides so many of those mechanisms. And when you buy an RTOS, you pay mostly for those mechanisms, and each one of them is actually something that should be avoided.
EW (00:27:54):
So you're paying for the things you shouldn't use.
MS (00:27:56):
You shouldn't use them.
EW (00:27:59):
...Are you familiar with any of the FreeRTOSs? FreeRTOS, or Zephyr, or I think there was a -
MS (00:28:06):
Yes.
EW (00:28:06):
- MicroC/OS one that was out?
MS (00:28:09):
Yeah. For instance, the QP frameworks are ported to all of those that you mentioned, to FreeRTOS, to MicroC/OS, to ThreadX, to embOS from SEGGER, and of course, to POSIX. The QP can run on top of Linux and other POSIX compliant RTOSs. So yes, I am familiar.
EW (00:28:30):
The QP framework. That is what your company actually makes, and that's what you sell.
MS (00:28:34):
Right. That's right.
EW (00:28:36):
And how is that related to active objects?
MS (00:28:41):
Well, this is an active object framework, so it provides active objects, first-class items, in the framework. So there is an active object class from which you can derive your own class and specialize. There is strong support for hierarchical state machines. So the behavior of your active object is specified as a state machine.
MS (00:29:05):
So this is supported. There is of course support for events and thread-safe event passing. You can pass the events directly from one active object to another active or from ISR to an active object.
MS (00:29:19):
But there is also the publish-subscribe event delivery where you can publish an event and...the event will be multicast to all those subscribers and delivered in a thread-safe way again.
MS (00:29:34):
There is support for dynamic events with parameters,...the mutable events, that are then allocated from thread-safe and deterministic event pools.
MS (00:29:45):
And then the whole event recycling is handled automatically in the framework. And there are many other things that the framework supports also.
EW (00:29:57):
But to some extent, it's about keeping things, I don't want to say clean, because that's not really the right word. Keeping the objects -
CW (00:30:10):
Separated.
EW (00:30:11):
Separated.
CW (00:30:11):
Yeah.
EW (00:30:12):
As an architectural thing, not as a last-minute, "they shouldn't be spaghetti" thing.
MS (00:30:20):
Right, because you see, when you take an RTOS, just naked RTOS, just for instance FreeRTOS, then you have all those mechanisms, event flags, and message queues, and so on and so forth. Time delay. But they are not very useful, as I said. Maybe they are even harmful.
MS (00:30:42):
On the other hand, you don't have events. You don't have publish-subscribe and other thread-safe mechanisms for delivering those events. Because the events don't exist in the first place. You don't have any support for state machines or let alone hierarchical state machines.
MS (00:31:00):
So what you end up doing is you have to build all those mechanisms on top of the RTOS. And instead of everyone doing this on their own, I'm just thinking all of those things are completely reusable. And so you can think of this as just the extension of an RTOS.
MS (00:31:20):
The most important thing that differs,...there's a difference between a framework and an RTOS, is that a framework works by inversion of control. So when you use a framework, you write the code that the framework will call.
MS (00:31:35):
So you will write for instance a state machine code that...will be called by the framework when the event is available. This is in contrast to writing threads in the real-time operating system, where you write the whole body.
MS (00:31:51):
So every single time you start with while (1) {} loop and you choose which blocking mechanism you will use in this particular thread or task. So the inversion of control is just the defining characteristic of the framework versus a library like an RTOS.
EW (00:32:10):
But you did say that it works on top of an RTOS.
MS (00:32:14):
Right.
EW (00:32:15):
Okay.
MS (00:32:16):
Because those state machines, they need to be scheduled.
MS (00:32:19):
And of course you could use very simple mechanisms that would, for instance, execute...every such state machine to completion and then look at what is the next active object that has some events in their event queue, and maybe choose the highest priority one, and then execute this one in such a cooperative non-preemptive way.
MS (00:32:45):
This is one way of doing this. But it is also possible to combine the whole approach with a preemptive kernel. And in that case,...for instance, one active object can be running a very long run-to-completion step.
MS (00:33:01):
For instance, in a GPS receiver, a position fix is a floating point calculation iterative that takes a few hundred milliseconds to complete on a small ARM processor. In the meantime, you have to track the GPS signals every few hundred microseconds really. That fast.
MS (00:33:20):
So...in this case, you could use a preemptive kernel to do the signal processing at much higher priority, every single time preempting a very long run-to-completion step. And this run-to-completion step then will eventually complete. And everything is fine.
MS (00:33:38):
Because there would be no processing of, for instance, the next event. It's still in the middle of processing the first one. So this will be all fine from the perspective of state machine, but you will be able to meet a hard real-time deadline in a high priority thread.
MS (00:33:59):
And this system is, by the way, much more compatible with rate monotonic scheduling, because...you don't have situations where you share something through mutexes or something like that. And...you don't extend artificially your CPU utilization.
MS (00:34:22):
So the whole method works much easier and cleaner. It's much easier to analyze such a system.
EW (00:34:27):
You don't spend a lot of time talking about your framework on the videos I saw from your YouTube channel on Modern Embedded Systems Programming.
MS (00:34:36):
Well that's right, but it is coming more and more, and it's already kind of being used. I smuggle it piece by piece, so to speak. For instance, in the lesson of RTOS, on RTOS, I've used, not Free RTOS, or at the very beginning, maybe I used Micro C/OS or something like that.
MS (00:34:55):
But at some point I switched to the kernel built into the framework, because the framework comes with a selection of a few real-time kernels. And one of them is called QXK. It has been used in the last few lessons where I exactly demonstrate hard real-time behavior of an RTOS using illustrations with logic analyzer traces.
MS (00:35:18):
I show, for instance, priority inversions and how to remedy them with priority ceiling protocol and so on.
MS (00:35:28):
So yes, the framework is being increasingly used. In the lessons for state machines, I used it, for instance, to demonstrate semantics of hierarchical state machines,...what actions exactly are triggered by events in a hierarchical state machine, in which order the exit actions and entry actions are handled and so on.
MS (00:35:53):
And this was already using the framework.
EW (00:35:56):
Okay. Clearly I skipped around. But you have more than 40 lessons at 30 to 40 minutes each that start with interrupts and getting started...But I didn't get any commercials on YouTube,...and I wasn't even using Christopher's account.
MS (00:36:19):
Yeah, well -
EW (00:36:19):
This is a lot of work. I mean, this is a lot, a lot of work.
MS (00:36:25):
Yes. And because you do something similar, I know that you have an idea how much. Well, that is because the revenue of the company is not coming through this. The company survives on the revenue from selling commercial licenses to QP frameworks.
MS (00:36:46):
So QP frameworks are available under open source licenses, but this is GPL. So it has the strings attached, meaning that if you use the GPL framework, you are obligated to publish your application-level code as well.
MS (00:37:03):
And it would be very easy for companies to publish the code, create a free GitHub account, and put the code out there. But actually most companies don't want to do this. They prefer to pay money and buy a commercial license, which frees them from the obligation of GPL.
MS (00:37:22):
And so this is how Quantum Leaps makes money. And the YouTube course is just one of many channels of promoting this to the developer community.
EW (00:37:42):
I feel it's as good a channel of promoting QP as the Embedded podcast is a channel for promoting Logical Elegance, which is to say, you don't do a lot of promotion, do you?
MS (00:37:54):
That's right. Well, you are doing the exact similar thing, I suppose. You just educate, and you teach in a sense. And that is what small companies can do. We can out-teach the competition, so to speak. For some reason, big companies like Wind River will not teach for free, but we do.
EW (00:38:22):
It's nice. I mean, I totally agree. And clearly, because we're doing similar things, but it is weird that it's the smaller companies who tend to want to reach out more. And the larger companies are going to instead make it all about their company or make you pay for lessons.
MS (00:38:46):
Right.
EW (00:38:46):
It's odd. Talk about an inversion of how things should be.
MS (00:38:52):
Right. I never understood, for instance, the policy of Microchip, that they were still selling professional or higher optimized versions of compilers for their PIC processors.
MS (00:39:03):
They make money by selling tons and tons of chipsets, chips out there. And they should be giving the best possible compiler in the hands of the developers and not charge -
EW (00:39:17):
Yes.
MS (00:39:18):
For them, these are really pennies....I never understood their policy. But I think that in bigger companies, everything is about return on investment, and they have to have some demonstrated value immediately available.
MS (00:39:32):
If they don't see it, they don't do it. As a small company, I don't really know what my return on investment will be. I have only sensed that it is worth doing.
EW (00:39:44):
And because I think with the small companies, the return on investment may not be cash.
MS (00:39:50):
Right.
EW (00:39:50):
We don't necessarily do everything by revenue. We can have our return on investment satisfaction, and people telling us thank you, and that sort of thing.
MS (00:40:01):
Right. Yeah. And...for instance, when I mentioned that there is a need to fill the missing curriculum, if you will, for embedded developers, that is also an opportunity to exactly provide this for them.
MS (00:40:18):
And if you do, you will earn their trust that you know something about that, that you helped them. And then they are more likely to use your product.
EW (00:40:28):
Yes, you too were influenced by, "If you build it, they will come." I understand that. Okay. But back to the videos, how much of what you do is pre-written versus how much is done on the fly?
MS (00:40:44):
Everything is pre-written for me.
EW (00:40:45):
Me too.
MS (00:40:46):
I actually write down my script. First of all, I start with a code. If I don't have a good idea, what kind of code I will present, I don't have a lesson yet. So I think a lot about the code. I experiment.
MS (00:41:02):
You'll be surprised how difficult it is, for instance, to show failures, for instance, stack overflow, or race conditions, or some other failures. So I experiment with what is possible. Then I write down the script, and then I edit it down.
MS (00:41:22):
Well,...I remember one thing, a story somebody told me that Blaise Pascal, that was this French mathematician from the day, he started one of his letters like this, "I apologize that the letter is so long, but I didn't have the time to make it shorter." So I do take the time to shorten my original script.
MS (00:41:47):
And so in the end, just so people maybe understand, I figure that it is about an hour of my time for every minute of the video. So a 30-minute video will cost me about 30 minutes of time.
CW (00:42:03):
30 hours.
EW (00:42:04):
30 hours.
MS (00:42:04):
30 hours, sorry. Sorry, sorry. Yes. This includes writing the code, preparing the diagrams that I show, the editing, and so on.
EW (00:42:17):
Does that make it hard for you to go to conferences that won't pay you, and be okay with putting together a conference talk, knowing how long it takes you to present things?
MS (00:42:30):
Well, again, I see it as if it was easy, then anybody could do it. So if it is hard, and I chose to do this the harder way people appreciate. People notice, I had some comments that some people wrote to me that...they watch my videos at 1X speed. And this is probably one of the only few that they actually...don't speed up the videos.
EW (00:43:00):
That is a real compliment.
MS (00:43:03):
I take it as a compliment, right.
EW (00:43:07):
You show a lot of code in your lessons. How much time do you think people spend writing code versus reading other people's code?
MS (00:43:20):
Well, I think that code...is read much more than it is written. So it has to be legible. It has to be nice and elegant. But speaking of writing the code,...I still, for instance, try to write as much code as I only can. And I take pride in this.
MS (00:43:51):
This is an important part of my day as opposed to many companies now outsource it to the least bidder or somewhere. And the code is not the highest possible quality.
MS (00:44:05):
So I would just say, pay attention to how you write your code. And then this will pay off in every possible way. This is the most valuable part that you can contribute to your company.
EW (00:44:23):
I agree with you. And I must have gotten my question wrong, because I wanted to know for people who are learning, they often jump in to wanting to write code, which is fun. Writing code is fun.
MS (00:44:40):
Right.
EW (00:44:41):
But they often don't want to read other people's code. And so you put a lot of code on the screen, and I appreciate that, because that encourages them to read it. But do you think outside that screen time people are becoming accustomed to reading code instead of just starting to type?
MS (00:45:01):
Yeah. Well, just like...even literature, you have to read a lot before you start writing. So...I would say people new to the field should read a lot of good code, and choose wisely which one they consider good, elegant code. And then emulate that.
EW (00:45:23):
Any recommendations for good and elegant code? I somehow think it's not going to be the Linux kernel?
MS (00:45:29):
Probably not.
CW (00:45:30):
Are you trying to get emails?
MS (00:45:34):
Yeah. That's why, for instance, books like Jean Labrosse's "MicroC/OS," they were very good source of elegant code. I might be biased. I would recommend QP, which is also open source and available in GitHub, to read that. So there is good code, well-explained code, out there.
EW (00:45:57):
I agree. I usually suggest Adafruit's code as a very nicely put together system where...you're implementing a lot of different features. It's still code that maybe isn't quite as optimized as it could be.
EW (00:46:22):
But if you are thinking about a system where you need to implement 10 different sensors, and you want one interface, that's the code to look at. Because they've thought about that.
MS (00:46:33):
Yeah. Well, someone said, I think that Donald Knuth was it, that, "Premature optimization is the source of all evil." So yes, I would agree that overly-optimized code is obscure and not as clear as it could be. And we should probably put clarity of the code above the absolute last byte of memory and absolute last CPU cycle.
EW (00:46:58):
Especially with our processors now. They're big enough. They're fast enough. We don't need to optimize, and when you do need to optimize, start with what the compiler wants to do. Unless you're on a PIC in which case you don't want to pay for that.
EW (00:47:14):
I have some listener questions. One is from Simon, and he asks, "What mistakes/problems can I avoid building into my system by properly using state machines? What are the easiest ways to use state machines improperly, and how can I avoid falling into those traps?"
MS (00:47:34):
So if you start using state machines, and by that I mean event-driven state machines, in which you'll generate events externally to the state machine, and then you send those events to the state machine for processing.
MS (00:47:52):
So if you mean those state machines, then if you start using them in the context of active objects, and by active object, we talked about it. It is the set of those three best practices that they encapsulate. So if you start using those, you can avoid a lot of problems.
MS (00:48:13):
You will not have to deal with race conditions, because if you don't share, things that can lead to race conditions, you are safe with them. You will be sharing only events that are guaranteed to be delivered thread-safe from the source to the destination.
MS (00:48:28):
So no race conditions. No mutual exclusions mechanism...You won't need to even know what the mutex is. You will also be able to meet your hard real-time deadlines much easier and apply RMS, rate monotonic scheduling methods, for instance, to mathematically prove that you will meet your hard real-time deadlines always.
MS (00:48:54):
...It will be also easier to reason about your system, because every event will be processed to completion as opposed to having some convoluted paths through your code. And also if you start using state machines you will avoid a lot of spaghetti code.
MS (00:49:17):
We talked about spaghetti in my last time at this podcast. So now what are the easiest ways to use state machines improperly?
MS (00:49:29):
The improper use of state machines would be if you would start mixing event-driven paradigms with sequential paradigms, by which, I mean, if you start putting blocking calls inside your state machines.
MS (00:49:45):
So for instance,...as an action of your state machine, you will call a delay, time-delay function, or you would call a blocking semaphore, or something like that. Don't do that. So you will be better off not to use state machines at all than to put a blocking call inside a state machine.
MS (00:50:05):
So my point here is that...always realize what kind of paradigm you are using in which thread...You should not never mix them within one thread. You can make a hybrid system, so one thread could be completely event-driven, and the other thread could be sequential.
MS (00:50:24):
That would be alright, but don't mix them in one thread of execution. And so always realize which one paradigm you are using and stick to it. Don't mix them. They don't mix. This is like oil and water.
EW (00:50:38):
This is why we need names for them. And...we need shared names for them. So that when you look at somebody else's code, you're like, "Oh. You're mixing the interrupt or event-driven state machines with the sequential state machines. And you shouldn't do that. So let's separate those things out."
MS (00:51:01):
And not only this. When you are using state machines, you need to realize that, "Oh. I am using an event-driven state machine here," or, "That is a case that I'm using an input-driven state machine." Input-driven state machines would be quite easy.
MS (00:51:14):
They are quite easy to recognize, because each state starts with the if statement, in which you test some global variables. In UML nomenclature this would be called a guard condition. It's a fancy name for if and else. So if every case in your state machine starts with an if, this is an input-driven state machine.
MS (00:51:39):
It is not an event-driven state machine. So it is all fine, but I'm just suggesting that you recognize which type of a state machine you are dealing with and then stick with that. Don't mix those either.
EW (00:51:55):
Okay. Keep separate event-driven, interrupt and sequential. Okay. DougG asks about using QP on multicore/multiprocessor systems. Have you done it? How'd it go? And any interesting notes?
MS (00:52:13):
Well, as I mentioned, QP has been ported to RTOSs and among others, to POSOX. So if, for instance, you use it on top of embedded Linux and, and this embedded Linux runs on a multicore then, then QP is running on multicore.
MS (00:52:33):
But that is mostly transparent to the programmer, the symmetric multiprocessing, typically, and you don't really do anything special. I have not used the QP yet on a small multicore. Now we have chips that have, for instance, a small M0, Cortex-M0, and the M7, or something like this. I have not used them in such systems yet.
MS (00:53:00):
I have, however, used QP on distributed systems, meaning that there were two microcontrollers talking to each other over a serial port. And it turns out that the active object model is very suitable for this.
MS (00:53:16):
Because it is kind of like a thin wire communication through events only, and nothing is being shared, which is suitable for just serializing those events, and sending them over the network, or over the serial connection in this case. And then deserializing them on the other end.
MS (00:53:38):
And actually, if you use so-called communication proxy design pattern, then you can have a proxy active object that actually does not do the processing, but just only sends events back and forth between those two processors.
MS (00:53:52):
However, this active object looks to other active objects in the same address space as a real thing. They don't know that they are posting events to an active object that actually lives in some other address space in the processor.
MS (00:54:11):
So what I'm trying to say is that the active object model is very suitable for distributed systems and will be probably also quite suitable for multicore, but I have not made that experience quite yet.
EW (00:54:26):
You've mentioned hierarchical state machines. Can you give an example of what that is?
MS (00:54:35):
It happens often in the state machine design that you have several states that need to handle the given event in the same way.
MS (00:54:46):
However, in normal, classical flat state machines without any hierarchy, because they react to some other events in a different way, you will have to put them in a separate state and then replicate the same responses, the same transitions, in all those states. So, this of course inflicts repetition.
MS (00:55:08):
And that's why normal finite state machines, traditional ones, tend to explode. In the 1980s, David Harel came up with a more advanced version, which he called statecharts. Later on they were adopted into UML and sometimes also called hierarchical state machines.
MS (00:55:30):
And this is the mechanism, that you can now put states within states. And those states that are nested within a bigger state, they inherit so to speak the behavior from the superstate. So now you can have this common transition or common reaction to events coded only at the level of the higher level state.
MS (00:55:54):
And all states that nest within, if they don't prescribe how to handle this event, this event will not be forgotten or dropped on the floor. It will be propagated to the higher level state and handled there. And so, in that case, you will have a mechanism of overriding this event, if you like.
MS (00:56:15):
So the substate can prescribe how to handle this event on its own. And it will be the case of overriding a virtual function, or if it ignores it, it will be handled in a common way by the superstate.
MS (00:56:28):
In that way you can reuse the behavior. And so you avoid the repetition. And that's why those hierarchical machines, they no longer explode as your problem grows in complexity.
EW (00:56:43):
I tend to like really hands-on examples. So I'm going to try to give one, and if I'm wrong, you can tell me. So I have a toy and it talks, "Blah, blah, blah. Play with me." And if I hit the volume button, it has a click, high click or low click, depending on which way I put the volume.
EW (00:57:04):
But if I hit a different button, the A button, it goes into a different thing. It starts talking about the letter A, and in the letter A, if I hit the volume button, it still gives me the same clicks. But if I was in an old-timey, traditional, finite state machine, the A state would have to point to a volume that then would have to point back to an A state.
EW (00:57:30):
So it'd be a volume that was in the A state and only belonged to that. But with hierarchical, I can say, "Look, if anybody presses the volume, you're going to play the click sound. And if you don't want that to happen, you have to replace it with something else."
MS (00:57:45):
Yeah. That is a good model here. Yes. That would be a good application of state hierarchy. Right.
EW (00:57:53):
I tend to use toys for all of my state hierarchy, because children's toys have so much state in them.
MS (00:58:01):
Right.
EW (00:58:01):
And it's easier for me to think about, "Okay, well, how does that apply here?" Let's see. Oh, I saw a new book from the Raspberry Pi Foundation about teaching computer science called "Hello World." Have you seen that?
MS (00:58:19):
No. No, I haven't.
EW (00:58:21):
Okay. Well then I won't ask you questions about it.
CW (00:58:24):
You can.
EW (00:58:25):
But someday someone is going to say yes, and I'll ask them questions, although it may be somebody I invite from the author list. Because teaching embedded systems and computer science, it's different than teaching just about everything else. Do you agree with that?
MS (00:58:43):
Well, I have not taught anything else.
EW (00:58:46):
Okay.
MS (00:58:46):
So it's difficult to make comparisons...In my course, I focus on fundamental concepts, such as object-oriented programming, and event-driven programming, and RTOS, and real-time, and state machines, and active objects. So that's why I believe that fundamental concepts, it is always good to understand those deeply.
MS (00:59:12):
And if you do go and show people how they work at the lower level, they tend to understand it better and deeper, and then use this more efficiently. So this is my approach that I take. In my education as a physicist, we had actually a course for teaching physics.
MS (00:59:35):
And I taught physics at the level of middle school. And I always try to show them experiments, and I believe that showing and demonstrating things to people is very effective to teach them.
EW (00:59:53):
I totally agree. I have one more listener question from, I don't remember your first name. How would you pronounce that? Christopher? Which one? M-C-N, M-C-E-N -
MS (01:00:14):
It's some kind of a handle.
EW (01:00:16):
I'm going to go with Kevin.
MS (01:00:17):
Right.
EW (01:00:17):
Because I didn't remember. I should, but I don't, who made a really wonderful joke about, "Are city machines less complex than state machines?" But the -
CW (01:00:34):
It is Kevin.
EW (01:00:36):
It is Kevin. Okay. "How far can we, or should we push concurrency on bare metal through state machines before it really makes sense to move to an RTOS?"
MS (01:00:51):
Well, Kevin probably at this point meant the input-driven state machines that combine event discovery with event processing.
MS (01:01:03):
So this is, again, the situation that you start every case in your state machine, for every state, with an if statement, which is supposed to test your inputs and find out if an interesting change in those inputs has occurred that you'll need to handle. So you discover your events, and then you try to handle them.
MS (01:01:26):
The point is that, because it's a non-blocking process, if there is nothing interesting to do, you don't block. You very quickly return, or you just pass through that code. You can string such state machines, and put them one after the other in the forever superloop, right, in the foreground/background system.
MS (01:01:48):
So in this sense, people think of FSMs, of finite state machines, as a mechanism that handles some kind of concurrency before they switch to an RTOS. With event-driven state machines, the situation is different, and they are actually at the higher level of abstractions in RTOS itself. So they build on top of the RTOS.
MS (01:02:13):
So how far should you push concurrency before it makes sense to move to an RTOS? I would say don't use a naked RTOS at all. Use RTOS only with those three best practices of concurrency. That means use the RTOS only as an active object, as a vehicle to partition your problems into encapsulated threads called active objects.
MS (01:02:42):
And then you can use event-driven state machines inside each of those threads. So this is kind of like a inversion of the question, because Kevin apparently puts an RTOS at the higher level than state machines. And I'm saying this is only because he probably thinks of input-driven state machines.
MS (01:03:05):
Otherwise I recommend to use active object design patterns with event-driven state machines on top of the RTOS. So go beyond the RTOS with that, not somewhere below the RTOS.
EW (01:03:20):
I would add that we've already talked about mixing the types of state machines is bad. And so an RTOS gives you a way to mix the types of state machines without it being bad, because it limits their scope between them.
EW (01:03:36):
So if you have a sequential state machine and an input-driven state machine, you need an RTOS so that they don't fight.
MS (01:03:45):
Well, actually RTOS limits your choice,...unless...you sleep for a tick, and then you start checking for your inputs. Then you could possibly use an input-driven state machine.
MS (01:04:03):
Otherwise RTOS prefers probably event-driven state machines, because those the tasks can block on the empty event queues. And RTOS needs blocking, because during the blocking time in one thread progress can be made in other threads. So RTOSs necessarily needs blocking to manage the concurrency that way.
MS (01:04:28):
So I would say that RTOS strongly prefers event-driven state machines. But actually RTOSs can be bad for state machines for some other reasons. And that is mixing the sequential blocking programming paradigm with state machines.
MS (01:04:47):
As I said, it is very bad to put any blocking calls inside the state machine, in the action of the state machine. So...this is the danger of using RTOS and state machines together. Because there are so many temptations, so many mechanisms within an RTOS that would block, all of them based on blocking.
EW (01:05:12):
Okay. I think I understand. Christopher? Nods. One more question for you. I think, your book is "Practical UML Statecharts in C/C++: Event-Driven Programming for Embedded Systems." And you have another one, "Quantum Programming for Embedded Systems," which is practical statecharts in C++.
EW (01:05:36):
You really like UML statecharts. I've never gotten into them. Convince me, please?
MS (01:05:45):
Well, the statecharts title was suggested at some point by my editor. And we stuck with that. Maybe I should have called this book maybe somewhat differently, and maybe the subtitle would be better. "Event-Driven Programming for Embedded Systems."
MS (01:06:07):
Yeah. Yes, I was trying to, in the first part of these two editions of my book,...explain the state machines and their full potential.
MS (01:06:20):
To actually have them working at their full potential, it turns out that you need a framework around them that works on the principle of inversion of control and provides the run-to-completion execution context for each state machine. Because the state machines need to run to completion and be undisturbed during that time.
MS (01:06:46):
So they are vulnerable if somebody would call, for instance, a state machine from within another state machine and so on. So...I came to this realization after reading, of course, many books that event-driven state machines actually do need a framework.
MS (01:07:05):
And you see this in all UML tools capable of code generation. For instance, the Rhapsody tool from IBM is based around the framework called OXF, Object Execution Framework. There are some other frameworks also supported by Rhapsody, for instance, IDF, Interrupt-Driven Framework, and many others.
MS (01:07:30):
But all of them are frameworks...And there are other tools also based around frameworks. So it turns out that UML state machines do require framework. And the contribution of QP is actually to provide something very, very lightweight, capable of working with small microcontrollers.
EW (01:07:53):
Miro, thank you so much for speaking with us today. Do you have any thoughts you'd like to leave us with?
MS (01:07:58):
Well, not really big thoughts. I just maybe want to reiterate that you can use different paradigms, and it is alright. But it is always good to realize which one you are currently using.
CW (01:08:16):
Yes.
MS (01:08:17):
And then, you need to know which ones can and cannot be, or like to be, mixed together. So it's alright to use different paradigms, but sometimes they just don't work well together.
EW (01:08:35):
Our guest has been Miro Samek, Founder and CEO of Quantum Leaps. You can easily find his YouTube series by searching for his name, but there'll be links in the show notes to that, and many other things.
CW (01:08:47):
Thanks, Miro. It's good stuff.
MS (01:08:49):
Thank you for having me.
EW (01:08:50):
Thank you to Christopher for producing and co-hosting. Thank you to our Patreon listener Slack group for questions. And thank you for listening. You can always contact us at show@embedded.fm, or hit the contact link on embedded.fm.
EW (01:09:05):
And now a quote to leave you with, from Immanuel Kant. "Space and time are the framework within which the mind is constrained to construct its experience of reality."