479: Make Your Voice Heard
Transcript from 479: Make Your Voice Heard with Carles Cufí, Christopher White, and Elecia White.
EW (00:00:06):
Welcome to Embedded. I am Elecia White, alongside Christopher White. We promised a show about Zephyr, and here it is. Zephyr and Nordic at the same time. With our guest, Carles Cufí, here to answer our questions about Zephyr, Nordic, and I do not know, maybe some other stuff.
CW (00:00:26):
Hi Carles. Thanks for joining us.
CC (00:00:27):
Hi there. I am glad to be here.
EW (00:00:30):
Could you tell us about yourself, as if we met at lunch at Electronica?
CC (00:00:36):
Sure. I am an embedded software engineer. Although for the last few years I have been working on open source, and specifically on Zephyr, my career has really revolved around Bluetooth.
(00:00:47):
I started many years back now, back in 2000, working on what was then to become the first ever hands-free car kit compatible with Bluetooth. It was the first device you could actually install on a car, and have your phone connect wirelessly to it. So that was pretty nice.
(00:01:10):
For me, it was an introduction to a technology that essentially has been with me for my whole career. Not always as my first area of focus, but usually related to, or in very close contact to. After doing a little bit of Bluetooth in France back then, and after shipping that first hands-free car kit- Which was made by a company called Parrot, which went on, by the way, to make drones.
(00:01:44):
Then I started working on an operating system, actually, in the UK. This is a company that disappeared, but it was a very, very interesting company. I will always remember it very fondly. We were basically doing Android before Android existed.
(00:01:56):
Instead of using Linux, we had our own kernel. And then all of the applications were written in Java. This was 2003, so well before Android. We had some really top-notch engineers in there. Unfortunately the startup itself was not successful. Although we had funding from the mobile operators, and we had a lot of support from a lot of people, but ultimately we did not succeed.
(00:02:21):
But the codebase and in general what I learned there, that is where I got my first contact with Linux, Unix, and in general with the whole philosophy behind all those operating systems. So this was for me a huge learning experience. I learned to write Unix based code, as well as learning Java a little bit.
(00:02:43):
This was my second big experience, and one that took me to the more powerful chips. So away from MCUs, from microcontrollers, and then going to the big chips that power the mobile phones today. Then I went back to microcontrollers for a little while, in a small startup, before landing on Symbian. I do not know if you remember that operating system.
CW (00:03:08):
Yeah.
CC (00:03:08):
<laugh> I guess you have to be a bit old.
CW (00:03:09):
Hey!
CC (00:03:09):
It used to power- <laugh> Well, speaking for myself. I used to work at Symbian. It was very interesting actually. They made the operating system that powered the smartphones of the day. Mostly Nokia, as well as a few others. It was an operating system based around C++, and very interesting and intricate concepts from the software development point of view.
(00:03:34):
It was hugely successful at the time. It had a massive market share, especially because, or maybe related to the fact that, Nokia was basically owning that market at the time. Then what happened is that the iPhone appeared. Nokia scrambled to- At the time- I guess things must have changed so much, but at the time they scrambled to try to counter that.
(00:03:57):
I remember they even set up a high speed camera, one of those that can record thousands of frames a second, just to figure out whether the original iPhone's UI was indeed running at 60 frames per second. They wanted to know whether that was actually technically possible with the technology at the time, and it was.
(00:04:13):
It was just that Symbian never did, right? We never managed to get to that level of fluidness in the user interface. I remember that also as a learning experience, that engineers sometimes from another side of the world can surprise you with something that you never expected.
(00:04:30):
After a while, I am working on different odds and ends here and there, mobile applications, things like that.
(00:04:35):
I landed at Nordic. That was 2010, and Bluetooth Low Energy was about to become a thing. The first version of the specification I think was released either late 2010 or early 2011. Nordic wanted a chip- Or Nordic had been designing a chip already for Bluetooth Low Energy. In fact, Nordic was one of the promoters of the Bluetooth Low Energy technology, which was partially based on some earlier Nordic chips.
(00:05:03):
So I joined the team. The team that was making the first ever Nordic Bluetooth Low Energy chip. A very simple chip, with an 8051 microcontroller. Was not even ARM at the time. It was very limited, but we actually shipped in many products.
(00:05:17):
I guess it is safe to say now, Casio watches, not smartwatches. I am talking about the ones with the LCD screens. They actually sold some of those with our chip in them at the time. I think it was called "nRF8001," the first one we released. It was a very simple chip, but an interesting one. It gave us a first foot in the market of Bluetooth Low Energy. So it was interesting.
(00:05:44):
After that I became very involved- I started with several people, the SoftDevice project, and that is what we did for a long time, actually. The SoftDevice, for those that do not know, that are not familiar with the Nordic software architecture of the last ten years or so- Which would not be that surprising. It was, or is, if you count that it is still supported but no new features come to it, is essentially a binary blob that you flash in Nordic devices, and then you can interact with it via supervisor calls.
(00:06:23):
So it is kind of always there, always available, no matter where you are, what you are executing in the chip. So it has some very specific and particular properties, that we made sure were designed so that you could have Bluetooth no matter where you are doing in the chip. So if you are updating an image, if you are in the middle of a bootloader operation, you always have Bluetooth available. So it is a bit special, but I think it worked well.
(00:06:51):
This was very well received, and for a long, long time we were supporting it. Well, we still do, like I said. But after a while we saw that we needed something else. So I was indirectly accidentally put in charge of designing a next generation software development kit for Nordic chips.
(00:07:16):
Well, that is what I have been doing for the last few years. It has been mostly working on Zephyr, because we chose Zephyr. I am sure we will talk about Zephyr more, so I am not going to get ahead of myself. But we chose Zephyr for SDK as most people know now, especially engineers that work with Bluetooth chips.
(00:07:34):
We have been using it, we have been modifying it, we have been contributing to it, we have been expanding it. We have been doing a lot of things with it for the last I would say seven years, I want to say. Perhaps even eight. That is my job today, still now.
(00:07:49):
I contribute to open source, mostly Zephyr, but other projects too. I interact with the projects. My team acts as a bridge between the internal Nordic engineers and the open source projects. We do many, many more things.
(00:08:05):
So that is a summary of my professional career. I am not sure if that is where you were asking, but I got on a tangent and went this way. <laugh>
EW (00:08:14):
That is fine. But we are going to do lightning round next, where we ask you short questions and we want short answers.
CC (00:08:18):
Sure.
EW (00:08:18):
Are you ready?
CC (00:08:21):
Yes.
CW (00:08:23):
Okay. Barcelona or Real Madrid?
CC (00:08:24):
Barcelona.
EW (00:08:26):
Worst Zephyr macro?
CC (00:08:32):
<laugh> I am just going to use the word "macrobatics". All the macrobatics in devicetree probably. Although <laugh> I cannot choose, there are so many. <laugh>
CW (00:08:43):
GDB- Well, GDB is command line debugging. I do not understand the question.
EW (00:08:48):
GDB or printf debugging?
CW (00:08:50):
Ohh <laugh>.
CC (00:08:54):
Definitely printf.
EW (00:08:55):
Best O'Reilly cover animal?
CC (00:08:58):
Oh. Real or fictitious? <laugh> I do not know, honestly. Off the top of my head, I really do not know. <laugh>
EW (00:09:07):
You always choose your own cover animal.
CC (00:09:09):
Oh! That is true. Of course. Yes. I am really bad at this. What was it? <laugh>
EW (00:09:15):
You do not even remember?
CW (00:09:15):
<laugh>
CC (00:09:15):
It has been so long, and I have had the son in- Anyway. Yes. I do not remember <laugh>.
EW (00:09:23):
Okay.
CC (00:09:23):
What was it? Now I will have to search it, huh? Because you have put me on the spot now. <laugh> It has been a long time since then.
EW (00:09:32):
It has been a long time.
CC (00:09:33):
Yeah. What is it then? Oh yes, it is one of- But I do not know the name of the bird in English. Do you know it? It is not a robin. It is a jay, maybe. Could it be a jay?
EW (00:09:45):
It makes sense if it was a blue jay.
CC (00:09:47):
Yeah, it is probably a blue jay. Can I say "blue jay" then?
EW (00:09:51):
Sure.
CC (00:09:51):
Yeah. Okay. Blue jay it is.
CW (00:09:55):
Would you rather work on applications or operating systems?
CC (00:09:58):
Operating systems.
EW (00:09:59):
Complete one project or start a dozen?
CC (00:10:02):
Complete one project.
CW (00:10:04):
Favorite fictional robot?
CC (00:10:07):
<laugh> I have to say I am not a big sci-fi fan. But I am going to say the robot, the female robot, I think she is called "Maria," from a film that I really enjoy, "Metropolis."
CW (00:10:19):
Oh, that is like the original sci-fi film, right?
CC (00:10:22):
<laugh> Sort of. Yeah. It is.
EW (00:10:25):
Do you have a tip everyone should know?
CC (00:10:29):
Use git bisect. No, really, no. I know. Yes, everybody should know about connecting vessels. I remember when I learned this blew my mind, like how to pour water using tubes and difference in height. Connecting vessels. Or git bisect. I do not know. Both are important.
EW (00:10:47):
They are both good, but only use one with code.
CC (00:10:51):
<laugh> Right. <laugh>
EW (00:10:53):
So Zephyr. Chris and I have been getting a crash course in Zephyr, after seeing it for a while. But as we are working in Zephyr on multiple platforms, multiple processors, one of which is a Nordic, we are coming across a lot of things that are good and bad. Do you have favorite parts of the system?
CC (00:11:22):
I do actually. My favorite parts of Zephyr are actually the kernel APIs, I would say. I think they are very clean. I have always liked it in general, and I have always liked them. I think they were designed. Some people do not think the same, by the way. This is not a widespread opinion, or at least I have not-
(00:11:40):
But in my particular opinion and compared to others, and I have had a chance to compare to other kernels, I think they are great. Maybe it is because I am biased, and I have worked with them for too long now. But I really, really, really think they are in general well-designed and they are very functional, easy to understand.
EW (00:12:01):
And these are the ones that let you sleep the processor or-
CC (00:12:05):
For example.
EW (00:12:05):
Let you send signals.
CC (00:12:09):
Correct. So among others, they will allow you for sleep for a certain amount of time, yield the thread, take a semaphore, spawn a thread, pull for example on multiple synchronization objects, and so on. But some of the ones that I think nail the use case, and have been widely used over and over and over, are some of the ones that perhaps are lesser known by those starting in Zephyr.
(00:12:39):
The message queue, for example, k_msgq, that is a great API. It does require you to incur an extra copy, so to speak. But other than that, it is super practical. It is one of the easiest ways you have to pass arbitrary data between two threads.
(00:12:55):
So I definitely recommend people coming to Zephyr to look not only at the ones that you come across always, like k_sem_take, k_mutex, k_thread_create, and so on. But actually go beyond those and look at the documentation, which I have to say in my opinion, the kernel one is very high quality. And look at those extra ones. Look at those additional ones, that you do not find on every single sample. You actually have to go looking for them a little bit.
EW (00:13:22):
Newcomers to Zephyr is a good thing to talk about. Do you have suggestions? When I started with the nRF53 Audio DK. I know codecs pretty well, so I was comfortable with the audio. I know BLE enough to know GATTs and peripherals and all of that.
(00:13:50):
But I did not know the audio BLE part, which was big. I did not know Zephyr, which was large, very large. And I did not know Nordic's Zephyr, which was different. Do you have a good way of getting started, without getting lost in any of these rabbit holes?
CC (00:14:13):
I want to say "yes," but that would be optimistic. The point is, Zephyr has a steep, steep learning curve. That is a fact. This is something we have repeated time over time, and we have tried to mitigate that. When I say "we," I am talking here- I am actually putting my Zephyr hat on, and then changing it for my Nordic one, and so on, then switching them. Because we have tried on both sides, very hard. I think we have accomplished some of it, to help out new users.
(00:14:43):
The first thing to know is that you need to take it slowly with Zephyr. You need to start with a simple sample. Understand perhaps just how a thread is created, how a semaphore is posted, the very basics without trying to understand everything at once.
(00:14:57):
Do not go and looking, "Why is there an overlay file here? What is an overlay file? Why is there a .conf file here?"
EW (00:15:04):
<laugh>
CC (00:15:04):
<laugh> Start with the code. Run it. Compile it for your development kit. It is almost certain that your development kit, or a very close variant, will be supported. There are so many boards supported in Zephyr. Not in NCS, but in Zephyr, yes. In NCS, we reduced the number of boards to those, obviously sold by Nordic, but-
(00:15:23):
By the way, sorry, "NCS" means nRF Connect SDK, which is Nordic's SDK, that comes with- It is not really a flavor of Zephyr, it includes Zephyr. So it is a bunch of things, including Zephyr. That whole package, which is really a large number of git repos, that whole thing it is called "nRF Connect SDK." That is what we maintain at Nordic, and that is our software solution for all of our developers.
(00:15:49):
If we talk about Zephyr, upstream Zephyr, vanilla Zephyr, as people tend to refer it. Then there are so many boards, in that you can pick your own board, or buy a cheap one. You can even use QEMU. You do not even need to buy hardware.
(00:16:03):
The point is you try a little bit, you add the printk statement. Perhaps you add one semaphore, and you post the semaphore between two threads, and you start playing with that.
(00:16:14):
Then slowly you jump into what probably is the hardest part when you start with Zephyr, which is the building configuration systems. They are really tricky, but they are tricky for a reason.
(00:16:26):
When you have an operating system like Zephyr, that supports not only so many different boards, but so many different SoCs, so many different applications. So we go from a 16K of RAM into gigabytes of RAM, right? Because Zephyr actually runs on big iron as well, on some applications. Some of our member companies, some of our users, actually use Zephyr in very, very big chips. But at the same time, you can still build and run Zephyr, including a Bluetooth stack, for the BBC micro:bit, which is the original version, which is a Cortex-M0 with 16K of RAM.
(00:17:04):
Why am I saying that? I am just trying to explain why the complexity is there. You could skip some of that complexity by essentially compromising on the scalability, compromising on the extensibility. But if you do not, then you pay a price and the price is complexity. That complexity is there to avoid adding complexity to your application later.
(00:17:28):
You do need to understand that basic frameworks that allow you then to take your application, and be close to the dream of a rebuild a way to switch chips. I can switch then to another vendor.
(00:17:44):
That is why we have things like a config and devicetree, which are very complicated, we all agree. Which have thousands of entries only in the main Zephyr tree, let alone if you then use an extension of Zephyr like the Nordic's SDK or anything else. There are many extensions to Zephyr that can be downloaded.
(00:18:02):
The point is take those slowly. and try to understand them one after the other. So try Kconfig, create your own Kconfig option, change the values of existing ones. See what happens then in the code.
(00:18:16):
Do the same for devicetree. Devicetree is particularly difficult. Because it involves you understanding not only the actual devicetree source, which is a language to describe hardware, which is complicated in of in itself. But on top of that you need to understand the schema for those files, which essentially is what we call "bindings." Those are YAML files, that describe what you can write in devicetrees.
(00:18:45):
So the whole thing is overwhelming. I understand. I have not even mentioned the fact that we use CMake, but we have our own set of CMake APIs that begin with Zephyr underscore. There is a very valid reason to that that I will not go into today. But if you are curious, you can ask around in the community, they will explain [to] you why. So there are a bunch of things that you need to start taking on very, very slowly.
(00:19:07):
Now, there is one series that I really, really enjoyed. I read it even though I have been working on Zephyr for years. It is from a company called Memfault. They do very, very good blog posts.
CW (00:19:19):
Yeah.
EW (00:19:19):
Yes.
CC (00:19:22):
Yeah. They have this website or this blog called "Interrupt." In it they have this series called "Practical Zephyr." I would very, very much recommend those. They are very good. I think they did do a great job at explaining the basics, Kconfig and then devicetree. It stops there, but they do a great job.
(00:19:42):
But of course also Nordic- We have put a lot of effort into also training. Training solutions, training materials. We have the DevAcademy, to which I am happy to say I have contributed myself as well. As a technical consultant, so to speak, although I have not written the content myself. I think that those courses are also absolutely great.
(00:20:05):
The only downside, if you call that the downside, is that they are obviously oriented towards Nordic's SDK, and less so to Zephyr. But there is a lot of content in there that is applicable to both Nordic and Zephyr. So DevAcademy, if you are using Nordic, that is unmissable. You have to read it. You have to go through the courses.
(00:20:28):
And no matter what you are using, if you are using Zephyr, be it with Nordic or not, I would very much recommend the Interrupt series from Memfault, "Practical Zephyr." That would be what I would do if I studied now.
(00:20:38):
The other thing that I would also absolutely recommend, is going to the YouTube channel that Zephyr has. There are some really great videos there. For driver development. For devicetree, as well. Kconfig. The build system. There are all sorts.
(00:20:51):
If you sort by views, I am sure you will get a pretty good feeling of the ones that have been successful, and there is a reason for that. The presenters there, they did a great job two, three years ago. It has aged pretty well, by the way. I was rewatching one of them earlier this month. Those also, if you prefer listening or watching to reading, those do a great job at introducing those as well.
(00:21:18):
Finally, Discord. Please join Discord. Please ask questions on Discord. Avoid creating- If you fail to understand something, avoid creating a GitHub ticket, because that is really not their purpose. Their purpose is to contain actual bugs. We have had so many users use GitHub tickets for questions.
(00:21:40):
Either use the GitHub discussions. I do not use them myself that much, but I know they are popular among the community. But Discord. Discord is perhaps the king of real time communication in, well, I guess in the world almost right now. Perhaps not. But definitely for Zephyr.
(00:21:54):
Zephyr, we have a Discord server. It is extremely active. People help each other a lot every day. Every day I see dozens of conversations. We have channels for all possible topics. I very, very much recommending joining this Discord server, as soon as you start your path of Zephyr discovery.
EW (00:22:15):
I did the Memfault Interrupt blog, "Practical Zephyr." That was really good. DevAcademy, I did a little bit of, I am more of a reader than a watcher. And Discord. I have been on that Discord for a long time, but it is so noisy that I just kind of forgot about it. <laugh> Just let the messages pile up and did not really think about- And I was worried that I was not sure if my questions were Nordic or Zephyr, for a long time.
CC (00:22:48):
Oh, but that is fine. We have a Nordic channel, right, so you can ask them in the Nordic channel. And if they are not Nordic related, perhaps- I do not want to blow my own horn, but I think we are relatively good at redirecting people towards the right channels. And it is perfectly okay.
(00:23:02):
Sometimes we get questions about a devicetree in the random channel. Or in the Discord channel, which is supposed to be about the Discord server itself. And that is perfectly fine. Please do not hesitate to ask. We in general, those on Discord trying to help other users, I am sure we will redirect you properly. And if not, it does not matter. Many questions get answered in the wrong channel anyway every day, and that is perfectly fine.
EW (00:23:30):
Well, that is good. I have a UART to DTS question, that-
CC (00:23:37):
Okay.
EW (00:23:38):
I will be posting there very shortly. <laugh>
CC (00:23:42):
<laugh> All right. Sounds good.
CW (00:23:43):
One of the difficulties I had just on top of- I did not find devicetree all that intimidating, maybe because I have seen things like that before. But the language of it and how it describes hardware, I kind of like that.
CC (00:23:55):
Mm-hmm.
EW (00:23:56):
It is a giant macro system. It is what I would have designed.
CW (00:24:00):
I do not- Under the hood it is a macro system, but that is not how it appears in the sourcing.
EW (00:24:04):
Okay.
CC (00:24:04):
No.
CW (00:24:07):
But the trouble I had most was okay, yeah, I understand Kconfig. Yeah, I understand DTS. It is the hierarchy and the inheritance, that happens from board to peripheral to-
CC (00:24:19):
Oh.
CW (00:24:20):
Maybe SoC. There are all these DTS files that-
EW (00:24:23):
And application ones too.
CW (00:24:24):
Build on each other in the application. Walking and seeing how those flow from one to another throughout the entire source tree- Especially when you have a complete Zephyr repo with everything in it. It got very confusing to see, "Oh, this is not working, because 14 DTSs upstream, this pin was set to be something else."
CC (00:24:48):
Right. Not only, it is a very good point. It is one that we have been struggling with for years now. The problem is you have to cater for those end users that use Zephyr exclusively to build their own application. They are mostly worried about their final result. So they have their application, and they have their devicetree files, their board potentially if they are not using a DK.
(00:25:10):
But then you also need to support this operating system in general. Meaning we have to build it against a thousand different targets, combinations, and so on. So then it becomes essentially a compromise or a balancing game. Between, "Let us make it clear where these files come from for users," and that is no easy task, as you have discovered yourself.
(00:25:30):
But at the same time, we have to be so flexible that if you want to change a signal devicetree node, you can do it at the SoC level, at the board level, at the application level, or even at the command line level. We do that, well in part, because we think it is the right thing to do from an architectural point of view. So having multiple entry points.
(00:25:50):
But most of all we do that because we need to. If we did not do that, then we would have to duplicate stuff and we would end up with tons of duplicate. That is the reason really. But I understand.
(00:26:00):
What we have done to mitigate the issue that you have just described, is to try in the different parts of the Zephyr documentation- Which I agree could be improved in that regard. But what we have tried to do, is have bullet points for the sequence of inclusion for devicetree and Kconfig files.
(00:26:18):
If you go to that, we have many sections in the documentation that deal with that. But there are two main ones. One is called "Application Development," inside "Developing with Zephyr," in the documentation in the main vanilla upstream documentation.
(00:26:36):
The other one is inside "Build and Configuration Systems," where especially if you go into the "Build System" and "Sysbuild"- Which is something else we can talk about. They try to make it clear how this inheritance works. Not only for Kconfig and devicetree, but also for other files that are also inherited. So that is the reason and that is how we try to mitigate it. But I completely agree with you, it is very complicated.
(00:27:05):
The downside also is that although for those systems, you do get a consolidated view, unique view in your build folder, of the whole devicetree once processed, once massaged, once everything has taken place. And also the same for Kconfig. You cannot blame that. You cannot "git blame" that, so you cannot know who introduced what, because that is generated, built in.
(00:27:31):
So like you say, I find myself now not seeing my UART output anymore, and I have to go- Either I go to that file, the consolidated final file in my build folder, and look at the nodes there. Or I have to make an exercise of jumping back and forth between board, SoC and application configuration files, configuration overlays or overrides, in order to find out what on earth happened. It is difficult. I agree with that.
(00:28:00):
It is difficult. We will try to improve it. We always will. But I think we have a renewed energy now towards improving this, after the introduction of sysbuild as well, which complicate things even more. Because it is another layer on top of everything else. So, yeah.
EW (00:28:16):
I have a question from a listener, Tim, who says, "Nordic seems to have gone all in on Zephyr over the past few years. What has that process been like?" How has it been to go from the SDK and the soft devices, to switching over to this big thing that you do not really quite control?
CC (00:28:39):
Well, it has been difficult and challenging and interesting and fun. For me personally, this has been almost my- Well, not almost, it has been my main project at Nordic for the last few years. So for me personally, it has been a big part of my life actually. Not only Zephyr itself, but actually using Zephyr at Nordic.
(00:29:06):
I think the hardest part honestly, that we got through, that we essentially overcame, was not too hard, but it was a matter of convincing internally those that were in charge at the time, that using open source was not only a good idea, but was also the future. There was a lot of hesitation at the time. There were internal voices that decried the effort, decried the proposal, obviously, and it is normal.
(00:29:39):
I do not think it would have made sense if it had been anyway else. Because you have to understand that 2016, things looked very different than what they do now. Zephyr was a newcomer, had just been unveiled. There were a couple more RTOSs that were mildly popular, but by no means were taking the world by storm.
(00:30:00):
Bare metal was still the standard, at least for many. Or bare metal combined with FreeRTOS. But always using silicon vendor HALs or similar drivers. Something as wide encompassing as Zephyr, people were afraid of it for many reasons. And honestly, rightfully so, as in this was a huge bet.
(00:30:27):
The thing is, it surprised me. It was not as hard as I thought. Probably because we did a good pre-study. So we sat down and said, "Okay, look, let us take Zephyr as it was back then, 2016, 2017, and let us try things around. Let us build a small sample with Bluetooth. Let us see what the state of it is, in all its areas. File systems and networking stack and Bluetooth and kernel and drivers and so on and so on."
(00:30:53):
So we did all of that and we documented it thoroughly. But most importantly, I think what we said is, "There is a lot to do, but there is a good foundation. I think we can start from here."
(00:31:03):
We had vendor neutral backing on the side of the Linux Foundation. So we knew we were not going to be tied to a particular architecture or vendor. The decisions would be taken by committee. So that was great.
(00:31:20):
We knew that the codebase we started with, and that we would start contributing to, was already of good quality. We knew that there was a focus on test and security, which is something that we really wanted. In many of these open source projects, security and test, they are an afterthought. Not in Zephyr. They were there from the very beginning.
(00:31:39):
There was a point- I gave a talk last year and I mentioned a quote from a meeting. At some point someone said, "Okay, look, we can either wait to see if Zephyr and in general open source microcontrollers end up happening, or we can make them happen. Nordic is not a huge company, but we are popular. We make popular MCUs. I think we could make it happen, or at least help make it happen."
(00:32:05):
Obviously it was not us who invented Zephyr or started Zephyr. But I thought we have enough power that I think with our support, Zephyr could become at least more relevant. I do not think we ever dreamt of how widely used Zephyr has become now. But at the very least, we knew that it could be a strong player in the world of open source RTOS. So that was actually not too bad.
(00:32:29):
The hard part- So coming back to the original question, the hard part was actually moving to a new development model. Making people inside the company work together in a way that was compatible with upstream Zephyr.
(00:32:41):
At the same time that helped us provide value for our customers. Because we obviously do not want to be just a company that takes some software that is made elsewhere, and puts it on some chips. We actually want to add, like every other company, some special sauce here and additional algorithms, features, et cetera.
(00:33:01):
Maintaining that balance between working downstream in the SDK, working upstream so that we ensured that the Zephyr was a success. That our chips and our boards were usable. Not only usable, but were actually optimized for Zephyr upstream as well.
(00:33:15):
Maintaining that balance, and at the same time reorganizing internally. So that we would commit to using GitHub, to changing the review process, to dismantling the silos that we used to have in software development, and trying to come all together and contribute to a single codebase. All of that was hard, very hard. That really took a while.
CW (00:33:40):
Was there anything that you found surprisingly easy? <laugh>
CC (00:33:43):
<laugh> Aside from convincing everyone? I think the transition to GitHub. I was expecting it to be worse, because everybody was used to another code review system, an internal one and so on. People actually liked it. Unlike other things that we changed, this was actually in general very well received.
(00:34:02):
It surprised me because I thought, "Oh, my. We are going to get a thousand proposals to use something else for code review." And we did. But when we said, "No, look, we are sticking to GitHub just to be consistent with Zephyr. Also because it is simple, it works. Everybody uses it, so why not?" We did not really get any pushback.
(00:34:23):
In the same way that also surprised me, another one that was easy actually, is the change to the coding style. We changed from an internal coding style we had with the nRF5 SDK, all our software really had that internal coding style, to Zephyr's. Why? Because, well, you do not want your customer to have to switch between two coding styles, when looking at the codebase. So we made this decision. That was surprisingly easy as well. People seemed to, if not like, at least adapt very quickly to it.
EW (00:34:49):
Let me ask another question that turned out to be very popular, although I am not quite sure how to ask this. Let me start with Bluetooth something something, Wi-Fi something something release date.
CW (00:35:04):
What?
CC (00:35:04):
Sorry? <laugh>
EW (00:35:07):
I am trying to lead him into telling me when we are going to have a BLE chip and a Wi-Fi chip.
CC (00:35:12):
Ahh.
CW (00:35:12):
Oh.
EW (00:35:12):
When it will be released to the market.
CW (00:35:14):
I see, I see.
(00:35:14):
They are not going to tell you that. <laugh>
CC (00:35:16):
I cannot say that. Yeah, I checked before the interview. I checked and unfortunately I cannot share anything that is not on our website already. I did see also <laugh> a question in the Google Doc, but unfortunately I cannot say.
(00:35:29):
Well, we do- I do not think it is a mystery that we are a company that makes Wi-Fi chips and Bluetooth chips. That is obviously well known to everyone. So I would say it is highly likely that we release one that combines both. When? I really do not know. I do not know myself, to be completely fair and honest. I do not know. <laugh>
EW (00:35:51):
Jakeypoo suggested the question, "How hard is it to integrate Wi-Fi and BLE into one IC?"
CC (00:35:57):
What?
CW (00:35:57):
<laugh> I feel like he is trying to- <laugh>
EW (00:36:02):
Again, leading questions.
CW (00:36:03):
Very, very clever workarounds to asking the question.
CC (00:36:06):
Yeah, that is a very clever workaround. Well, look, I will tell you from- Because I am not a hardware engineer, I will tell you from the software perspective, given the architecture we have now.
(00:36:13):
So from the software perspective, it is actually not too hard. Because Zephyr, and by extension NCS, allows you to enable and disable everything you want. And in general it is very well tested against running things in parallel and concurrently.
(00:36:28):
So running the TCP/IP stack, which is by the way the stack we use in our Wi-Fi products today, and the Bluetooth stack, concurrently, it turns out to be a fairly well-known and popular use case already now. In different chips and combinations, with and without Nordic chips.
(00:36:46):
Actually the software architecture changes, if any, to ensure that both the Wi-Fi stack- Which in our case means essentially the higher layers, right? The lower layers actually run typically in a small core within the Wi-Fi subsystem of the chip or whatever, and that is the case for many other vendors as well.
(00:37:06):
But the higher layers, combined with the Bluetooth protocol stack- The entire Bluetooth protocol stack, often including both the upper and lower layers. They are very easy to combine. That should not be a big problem.
(00:37:20):
So from that perspective, speaking from the software side, it is easy. <laugh> Or rather nothing is ever easy, but it is certainly designed for. So then it comes to hardware. But unfortunately I do not know anything about hardware, or very little.
EW (00:37:37):
Well then let us go back to software, and ask the question from Timon about why NCS exists. I mean it is hard because Zephyr does so much, but they are also modules and you can pull in and out things.
CW (00:37:53):
Not modules. That was another thing I was going to ask about. Never mind. <laugh>
EW (00:37:57):
There are lots of subsystems? Why is it Nordic's Zephyr, instead of Zephyr's Nordic?
CW (00:38:08):
Well, he said. It is Nordic SDK, that includes Zephyr.
EW (00:38:12):
Exactly.
CC (00:38:12):
Correct.
EW (00:38:13):
But why is it not the other way?
CW (00:38:15):
Right. Okay.
CC (00:38:16):
Okay. Yeah, that is a very fair question. The straight answer is, because when we started, nothing of what you see existed. Almost nothing of what you see today. The extensibility of Zephyr, using Zephyr modules, for example. The ability- The west tool did not exist even.
(00:38:34):
So although we started gearing up towards developing all of these tooling, changing Zephyr so you could do almost everything out of tree- We did that for two reasons. We did it for us, and for our customers. When I say our customers, I mean Nordic customers, but I could also say for our users, referring to Zephyr users.
(00:38:54):
If you are writing an application, the last thing you want is to modify someone else's C file, and then have to commit that diff. What you want is to have your files outside in your own repo, and use Zephyr and everything else. See if Zephyr has anything else that is provided for you, essentially as a big huge library that you use, and then update when you need. But you do not need to touch, unless strictly necessary.
(00:39:15):
We did not do- Or Zephyr did not contain NCS, because Zephyr at the time did not have the ability to do things like that. Once it did, we still wanted full control. So it is really very much- There are multiple factors. One is control. Why? Well, problem with Zephyr is that it is a- It is not a problem. But Zephyr is an open source project, and that means that its development is led by agreement of the different parties contributing to it.
(00:39:47):
Now we are a hardware company. We sell chips. There are times when we have to release on time for our customers. So those two are not really compatible. That is why we have a very lightweight Zephyr fork. Meaning we take the Zephyr tree, the main tree, and we have our own variant of it, if you want. But it is very lightweight. We try to keep the changes in there to a minimum, and instead we put all the functionality around it.
(00:40:13):
Now that functionality for the most part, does not overlap with what Zephyr offers. So that means that for the most part, what we are offering there is additional value. Additional algorithms, support for hardware, applications, extensions, all sorts of things that are useful to users, but to Nordic users. This is part of the added value that Nordic sees when delivering the SDK.
(00:40:42):
Now, could we do it the other way around? Not really. Because even though Zephyr does include now the functionality to extend it, in a manner that you do not need to modify its code, it does not mean that we would be able to do what we do, if we relied on the open source project.
(00:41:00):
The fact that we need the control. The fact that we sometimes also do things that are simply not allowed in Zephyr. Until very recently, you could not distribute binary blobs, as in pre-compiled libraries, as part of Zephyr. We, as in Nordic, championed the introduction of it, with other companies, in order to enable vendors- Not Nordic, because we already had our solution. Other vendors, to be able to provide their own binary blobs.
(00:41:30):
But in fact what we do, what we have been doing since the beginning, is to have a special repository where we put our binary blobs. Those are important, because on some of those we can share the code. Because perhaps we do not know- That is very common in silicon vendors, you do not all know the IP you have sometimes. So perhaps you can share the code, so you have to ship it as a pre-compiled library.
(00:41:49):
Or in other cases it is just easier for the user, because the pre-compiled library, it is pre-qualified, for example, for thread. You do not want them to have to go over the thread qualification. So you provide the pre-compiled thread stack that they can use, and they can skip that. So there are multiple reasons, and until very recently, that was not possible with upstream Zephyr. So shipping binary blobs was an important thing as well.
(00:42:11):
Then there is all the branding, our own documentation, our own extensions. Everything that turns or that goes or that is part of the ecosystem, but not part of the software trees themselves. Those are very important as well. Having our own distribution, our own SDK, controlled by us, managed by us, released by us. That was also one key requirement for us going jumping headfirst into open source. So there is all of that.
(00:42:43):
Will that change in the future? I do not have a crystal ball. I do not know. But how we do it today was a logical sequence of decisions, based on what was there at the time, what we wanted to give our users, and the approach we took. So, yes.
CW (00:43:03):
Is that something that you could foresee reversing in the future, when Zephyr becomes capable enough, that it makes more sense to go the other direction?
CC (00:43:15):
It is a good question. It is one that I really do not have an answer for. Not in the short term. There is too much in NCS. There is too much in our SDK. There is one additional factor did not mention, that I will add now for completeness and full understanding of this. That some of our source code, it is actually not open source.
(00:43:36):
The one we ship in our SDK, although good parts of it are shipped as source code. The license we use is essentially a modified BSD license, where we add a clause saying you can only use this software with Nordic chips.
CW (00:43:52):
Right.
CC (00:43:52):
This is quite common in other SDKs from other vendors, but you cannot call this open source, because it is not part- That license is not in the list of OSI approved licenses. So that means that we could not contribute that code to Zephyr directly, if we wanted to. We would have to change the license, which potentially we could.
(00:44:09):
But does Nordic as a company want to do that? That would fall on people above my pay grade, that decide this sort of things. Until now they decided "No," they want to keep the value added and use that license. Which I think makes a lot of sense when you look at the amount of things we have contributed to Zephyr so far, and the ones we plan to contribute as well, to keep a little bit for ourselves.
EW (00:44:37):
That is always a hard balance. Because yes, the company needs to make money. On the other hand, we want to say "open source" and we want it to mean open open source, not mostly open source.
CC (00:44:52):
Correct. <laugh> Yes, and it is very hard. This is actually going back to the earlier question. It was one of the hardest part, is to just decide for every new module, framework, sample, does this go up or downstream, right? We have to make a call for that. We have our own internal processes for that, so that we all agree within Nordic to an approach to going up and down, and then we act accordingly.
(00:45:17):
But it is hard sometimes, because there are risks with doing things downstream. For example, a competitor of the same functionality may appear upstream, and suddenly we have two implementations of the same thing. That is a problem of if you keep it downstream.
(00:45:31):
But at the same time, certain things, certain functionalities specifically, that we know gives us value when compared to other silicon vendors. We want that to be part of NCS, and not usable with other chips, just because it makes sense, right? In that regard from a company perspective.
(00:45:55):
There is this fine line you have to walk, where you want the project to be successful, the Zephyr project. You want as many contributions as possible, so that you ensure that. You want also to remain optimized and compatible upstream. And at the same time you have to save something for the SDK.
(00:46:19):
My opinion at this point is that we have refined the process so much, that it is pretty clear by the moment we conceptually come up with an idea and say, "We need to do this," where it is going to go. We know each other pretty well now.
(00:46:31):
Everything that is a big system ends up upstream. Because we cannot start modifying all of our files in a direction that is incompatible with upstream. That makes sense because it is the only sensible approach. And then individual features, individual samples, things that are more self-contained, that encapsulate a particular feature, or extra functionality, those typically stay downstream.
EW (00:47:00):
You work almost entirely on open source?
CC (00:47:05):
Yes. Well, go on one. Sorry.
EW (00:47:09):
I mean to the extent that we have been discussing that it is open source.
CC (00:47:14):
Exactly.
EW (00:47:15):
That is really cool, that you get to work on open source and get paid for it, which is good. Has that become a core part of your next job? I am not saying you are leaving Nordic. I have no information, no reason to think that. Stop panicking Nordic folks.
CC (00:47:33):
<laugh>
EW (00:47:33):
But having worked on open source code, is that something you think is important for your career? Or do you think it is just one step, it is just programming?
CC (00:47:48):
No. It is absolutely- Let me put it this way. I do not think I would go back to only proprietary source code. It is fine to have some proprietary source code. It is fine. I absolutely think that combining proprietary and open source software is the right thing to do, in some occasions. I think that companies do it for a purpose that makes sense. I am all in in that.
(00:48:10):
But only proprietary, not using an open source project as a foundation. I do not think I could go back to it now. Perhaps there are many reasons, but the main one I think is the fact that working with open source allows you to meet so many talented- This may sound a little bit like a cliche, but it really is true.
(00:48:31):
The sheer number of people that contributes to Zephyr. The different companies, the different coding styles, the different approaches, the different backgrounds. All of that basically enriches you, in a way that I think working in an office with a few developers would never give you.
(00:48:49):
So there is that, and then there is the fact that I enjoy open source as a philosophy. I actually think it is a very good philosophy. And a very good way of developing software beyond all of the political, or- I just think it is a very smart way of developing software.
(00:49:06):
So much so, that I think it will be a mistake not to use that way. This mechanism, this approach nowadays. Especially for complex software. And software or microcontrollers has become so complex, that you either do it with open source in a collaborative manner among all the vendors, or you end up with a clutch.
(00:49:26):
What would have happened, we would end up with a clutch. A mismatch of different open source projects, in part proprietary modules, all together tied with string probably, if we did not have Zephyr. Because there would not be a unifying factor.
(00:49:44):
Zephyr gives you that central point where Mbed TLS integrates with Trusted Firmware-M. All of these additional- MCUboot. All of these additional satellite, I am going to call them, projects- Perhaps not the greatest word, but they are satellites to Zephyr. All of those work together, because Zephyr tests them together, and we ensure that those work together.
(00:50:05):
If we did not have that, I do not even want to think what the codebase would look like, to be honest. I do not think it would be good. Might be wrong. So going back to the question, absolutely, I do not think I would go back to working on proprietary software exclusively. If I ever change jobs- Again, not in my plans. I would definitely go for open source. Definitely.
EW (00:50:27):
How does Zephyr test all of the different boards? Is there some Zephyr room that has a thousand tiny microcontrollers?
CC (00:50:38):
No, not really. This has changed over the years. The way it is done now essentially, is that all of the CI of the continuous integration in Zephyr happens only by building samples and tests. And running them on servers, meaning on QEMU or native_sim.
(00:50:55):
This additional mechanism that allows you to compile Zephyr into a Linux application, that then can be executed natively on any computer running a Linux distribution. All of that happens on server services that are maintained by the project. We actually have our own servers. We have had them for a year or so now. But that is it.
(00:51:19):
We do not have test farms. We do have them in the individual vendors labs, of course. That is essentially what we do. So the individual vendors run the same test suites that are used in the emulated or simulated platforms. They execute those on their own, in their test labs.
(00:51:41):
So Nordic for example, has I think two daily builds that take the latest Zephyr, the latest and greatest, and they run it on our boards. Only on our boards, of course. Every time we find a bug, we report it. That happens with other vendors as well. So that is the way it works. It is essentially people pull their results together using the GitHub issues, then we fix those issues as they come.
(00:52:06):
But the actual execution on hardware is done by the silicon vendors, not by the project itself. That means that it is completely optional as well, of course. Some silicon vendors do it, some do not. Of course it is in your interest as a silicon vendor to do it, because then you ensure that there are no regressions on your particular hardware. But of course that depends on your involvement on the project.
EW (00:52:27):
Okay, so Audio DK. No, sorry. I have been working on the Nordic nRF Audio DK. It has a lot of examples and applications. I do not want to ask you about those, because I understand that is not your area of expertise. But Zephyr as a whole has so many examples!
CW (00:52:50):
Which is great.
EW (00:52:52):
Except for the part where none of them do what I want, and they are all so different, that I cannot figure out what it was I was supposed to do.
CW (00:53:01):
<laugh>
CC (00:53:02):
Yes.
EW (00:53:03):
How do I untangle those? I mean, I am an experienced embedded software engineer. I feel like I should not have to read the code ten times, just to figure out whether or not it is using a ring buffer. I say that as though I pulled that out of nowhere, but it was <laugh> a discussion recently. Or what was the button? We had a discussion recently about Zephyr and button handlers, and-
CW (00:53:29):
The person we were talking to, was complaining it did not have a button driver, but it does.
EW (00:53:34):
But it does. But then I read that button driver and it did not do what he wanted.
CW (00:53:39):
Oh, I see.
EW (00:53:43):
Examples, good or bad?
CW (00:53:43):
<laugh>
CC (00:53:48):
<laugh> <sigh> We had the same problem in our previous SDK. I think many SDKs have them. Because when you set out to develop something for your users, but you have an operating system that has, I do not know how many thousands of Kconfig options and how many modules, framers, et cetera. How do you do it?
(00:54:04):
I can tell you what we did in Nordic, in order to make it easier for our customers. To try not hit that wall, where there are just too many samples. Each doing an individual small task or accomplishing a goal, but then it makes it really hard to put together. What we did is we divided the samples into samples and applications. Then there are basically two categories.
(00:54:30):
The samples are samples, so they are testing or they are showing you how to do one particular thing. So this will be, for example, a BLE throughput sample, where you connect two boards and then it sends data as fast as possible. That is a sample, because typically you would never ship that in a product, unless you just want to show off how slow Bluetooth is in general with low energy, when compared to Wi-Fi especially. You would not do that. So that is a sample.
(00:54:52):
But then we have applications. Those are actually close to what you would call "reference designs" in hardware. It is essentially they are more tested. They include fully featured applications in the sense they have firmware update, they combine multiple subsystems.
(00:55:09):
We actually execute them in some cases in specially designed hardware. Like the Audio DK, for example. The Audio DK is designed for that application, and that application is essentially the only one that runs on that hardware. We have a similar one called "nRF Desktop," that showcases how to do a mouse and a keyboard using Bluetooth Low Energy and NCS Zephyr.
(00:55:36):
That is how we approach this. I think it is not a bad approach. I like it personally. I think it gives users a starting point. But what happens if your future application does not fall into the category of the ones- Does not match any of the ones we offer? Then you are back to square one. Like in Zephyr, where you have to pick a sample, and start banging away your code, and trying to figure out how to put together all these things.
(00:56:03):
I think that at least with Zephyr, you get the easy enablement side of things. With a Kconfig option, you can enable a subsystem and relatively easily add functionality to an existing sample. But that said, that still falls short. I agree.
(00:56:22):
The thing is, I do not have a magical solution, because we still need samples to test things. Beyond tests, we still need samples even for ourselves. When we are developing, we need samples. That is what we use. And not only tests, we use samples very often.
(00:56:34):
They are also a good starting point for some applications. But inevitably you are going to hit that wall of too many samples, and then them not being useful for anyone. I am afraid I do not have a great solution to the problem. But like I said, we can mitigate it. You can mitigate it.
(00:56:54):
Perhaps this could be something we discussed in- We have many meetings in Zephyr where we- Especially during in-person conferences, where we meet and sit down. This is actually a great topic that we could discuss there. How to improve the sample situation in Zephyr, where there are too many and perhaps not complete enough.
EW (00:57:17):
There are sometimes multiple ways to do things, which is understandable given the organic nature of Zephyr. But it is hard for new people to see the trade-offs. If there was a canonical solution, sometimes it would be better.
CC (00:57:41):
It would. I agree with that. I have experienced that myself, where I have seen basically two different samples or tests solve the exact same problem, using a different kernel primitive, for example. The problem is that you have developers who are very often very creative, very smart, and they want to try things out.
(00:57:59):
At the same time you have users, which probably at that point in time, they just want to get their stuff running. They want to boot up their board, bring it up, and do whatever they need to do with that firmware. It is difficult because you need flexibility, and at the same time, you do not want to be able to do things in a thousand different ways.
(00:58:21):
But if you have ever programmed in Python for example, it suffers from much of the same, right? Yeah?
CW (00:58:26):
<laugh>
CC (00:58:26):
<laugh> I was actually writing some Python code this afternoon. I said, "I can do it in a thousand ways. I can either iterate, or I can use a set, or I can-" I ended up going for one of them, but then thought, "I do not really know if there is an inherent advantage to one of these."
(00:58:39):
I think the moment you give people flexibility and freedom, you hit that. You hit that, and it is very hard to avoid. I think the best thing to do in this cases, is to look at a code that you consider of good quality, be it Zephyr or not. Not all of the Zephyr samples are going to be at the same quality.
(00:58:59):
But if you see them, some of them referenced again and again and again. Or if you like the way a particular contributor does things. That is actually quite often the case. We have people that other people's contributions. So if you do that, then I think that is also a good starting point, to try and select how to achieve one particular thing.
EW (00:59:19):
It is funny, I did not think about that. I remember in college having favorite teachers, but I do not ever really think about looking at who writes the code. Why have I never thought about that?
CC (00:59:34):
<laugh> Well, I do not have a few actually in upstream Zephyr. Obviously, I am not going to name names. But you end up- Perhaps you have a mind that is more alike, so you tend to select the same resolution to a particular problem. Maybe it is just because the way they write their code, is also from your perspective, the ideal or the optimal way.
(00:59:59):
Because very often you are mistaken about that. We are all mistaken. Because you try something out, and you actually look at the assembly code in the list file, or anywhere in the listings. And it is just nothing at all what you expected. And the compiler has done something completely different. And then your assumptions about what was better optimized, completely fall through.
(01:00:20):
But having some people that you trust how they code, and that you agree when they take technical decisions, I think it is something that is definitely worthwhile.
EW (01:00:31):
Well, the temptation to ask you to name names is very high. But I do not think that would be prudent.
CC (01:00:39):
<laugh> I really do not want to name preferences here. <laugh>
EW (01:00:44):
Well, Carles, it has been really great to talk to you. Do you have any thoughts you would like to leave us with?
CC (01:00:49):
Yes. In line with one of the questions that was asking, was it hard or not to transition from a proprietary development model entirely, to then switching to one that is based on open source, that contributes to open source, that radically changes the way software is developed in a particular company.
(01:01:13):
One thought that I would like to share with listeners is, the fact that you see how things are done, and that there is an established approach to solving problems in a company, I do not think that is always a reason to abide by it.
(01:01:28):
I think that sometimes things around you- When I say "around you," I mean in the industry, among your peer programmers that might not work at the same company, change. Things change, things evolve, and not everybody in your company and not everybody working with you, may be on board with that.
(01:01:46):
But I think it is very important to make your voice heard, if you think that the company could do better. I am not even talking about mistakes. Everybody makes mistakes, and that is always great to point them out. But those are usually very often pointed out very quickly in a company.
(01:02:02):
What is harder to point out is the general direction. If you think that the general direction a company is going- I am not even talking about Nordic now. I am looking completely in general. If you see that the way your company is doing things technically of course, is not in line with the direction of the world- When I say "the world," obviously I mean the technical community, is going towards, then I think it is very important to speak out.
(01:02:30):
Not because you want to be the person that has changed the way things are done in a company. But because if you do not say so, you are actually harming the company, from my perspective. Because if you know something or you think you know something, that is going to be a deal breaker, or a major change, or something that is going to completely overhaul how things work in your industry, and you do not mention that, then it is sort of like holding out on the rest.
(01:03:02):
You may end up not only harming the company, harming yourself, in the sense that you will not be able to develop the software you always wanted to. You will not be able to see your products succeed, et cetera, et cetera.
(01:03:14):
Even if you have to go to the CTO, the worst you can happen if you suggest going in a particular direction, is perhaps you will have wasted a few moments of someone's time. But at best, you can actually change the way a company approaches technical challenges. It might well be that one day you will find out that that was the right thing to do, and then you will probably be happy for it. That is the parting thought I wanted to share with you.
EW (01:03:50):
Thank you. That was good. Our guest has been Carles Cufí, Open Source Software Engineer at Nordic Semiconductor.
CW (01:04:00):
Thanks Carles.
CC (01:04:01):
Thank you.
EW (01:04:03):
Thank you to Christopher for producing and co-hosting. Thank you to our Patreon listener Slack group for their questions. And of course, thank you for listening. You can always contact us at show@embedded.fm. Or hit the contact link on embedded.fm, which is where the show notes will be. Which is what will contain links to things like the Memfault "Practical Zephyr" introduction, and Nordic's DevAcademy, and all of that.
(01:04:29):
And now I have a quote to leave you with. The problem is, once you start on "Don Quixote" quotes, you really just tune out for the podcast, so that you can read "Don Quixote" all over again. So let us go with this one. "Finally, from so little sleeping and so much reading, his brain dried up and he went completely out of his mind.” Alternatively, "The proof of the pudding is in the eating," which I did not know where that came from, but apparently "Don Quixote."