An Introduction to the Tricky Parts of Embedded Systems

Embedded software is not like other software; working close to the hardware has a special set of challenges.

In my career, working on consumer, medical, aviation, municipal, and industrial applications, I’ve seen a lot of systems, some good and a few great. It makes me sad when people give up on the joy of working with hardware just because there are a few oddities in the niche of embedded system software.

Ok, there are many oddities. These are not discussed in traditional software engineering or computer science curriculums. When I asked another embedded software engineer what he thought the difficult areas were, he came up with the list below.

I think he was kidding but there is a lot of truth here. Take number five: “Eclectical Engineers”. I thought he meant to type Electrical Engineers (who are tricky, but not that tricky). On the other hand, as embedded systems software engineers, we do have to be pretty eclectic. It is an interdisciplinary field; working with software, hardware, algorithms, mechanical, and manufacturing. So, yeah, once you think about it, most of these points are reasonable— in some way. However, this is not really my outline, though it is not the last you’ll see of these points.

Before we dig into the oddities, and to make sure we are on the same page, I want to define what embedded software is. To someone without a technology background, I usually say that  embedded is software for things that aren’t computers.

A more formal definition would be that embedded systems are those that are purpose built for their application. That usually means they are resource constrained, because the devices have the minimum features needed to do the job. Despite that, the systems still need to be safe, reliable, easy to use, cheap, and/or low power.

A classic question in this field is: “Is a phone an embedded system?” I tend to answer no, because it has a complex operating system and is a multi-purpose, even general purpose, device. However, somewhere out there is a person programming a phone’s motion sensor driver so the processor goes into a reasonable sleep state while still waking up the phone when the user moves it. That person likely has a different opinion of whether a phone is an embedded system.

Instead of tracing through exactly what qualifies, I propose a specific example, a simple consumer project: an internet enabled light.

Many projects boil down to this in some way. Perhaps our IoT Light is a set of holiday lights you can leave up all year long and they use the day of the year to change colors (Perennial Lighting). You would never forget Valentine's Day was coming if your house started to get pink on February 1st. Or maybe our IoT Light is like the one in the image where Aaron, Chris, and I ran around a park in the dark, painting with light itself. Or possibly, it is a door light that comes on when it is dark and your phone comes into range.

I would very much like to make fun of the IoT Light concept. “Internet of Things” anything is such an easy target because there are so many ridiculous examples and worse implementations. However, our lives really are becoming more connected and there are many real, useful applications. Thus, the IoT Light is the kernel of our example. And since the last idea, the door light, can best be done with BLE and BLE tends to be popular right now, I’ll use that as the concrete example.

To get started, and to get funding for this idea, we’ll need a proof of concept. There are many possible platforms and processors that can work. The Qualcomm lines are cheap and low power but often difficult if you want to write the code yourself. TI’s CC2640 processor is wonderful in many ways though the dev kit is $300 on TI’s website. The dev kit is also physically large so it won’t look good in a demo. The Nordic nRF line is incredibly popular and offers many examples. Plus it is easy to get a lovely, small development board, as there are many vendors like OSHChip and BLENano.

We could also look at Adafruit and Sparkfun. Some people dismiss these as hobbyist or maker sites (though they are more than that). In this case, their target audience is a big advantage as their boards are tested, documented, and debugged. This benefit should not be overlooked when working on prototypes. Choosing a platform is difficult. If you can find many that meet your basic requirements, seeing what is available for easy use is a better selection method than a dartboard.

At this point, so early in the process, it seems like it doesn’t matter what platform we choose for the proof of concept. That should be true but once you’ve gotten a lot of code on one processor,  it is excruciating to throw it away for another processor that might not be better overall.

I’m going to go with the OSHChip because I know Philip Frieden, the guy who made the system. Plus, it is nice to look at and open source so we can be confident of reproducing the parts we need later in production. The OSHChip uses Nordic’s nRF51822. I have a good feeling about that because it was the chip that I saw in other, similar products that I happen to take apart when considering the IoT Light idea.

With a BLE processor selected, now we need an LED. It could be a simple one but more colors would be better. If we have more than one LED, we can have more brightness and power consumption options for our customer (or our marketing department). Overall, the WS2812 (NeoPixel) is widely popular, fairly inexpensive, and lets us defer decisions about LED quantity to a later time.

We obtain the parts and start putting together the kits. If we pull down Nordic’s sample code or follow OSHChip’s tutorial, talking to a BLE app on a phone is easy. Next step is to light the lights.

There are some WS2812 drivers for our processor but there are also many people who are complaining that these LEDS don’t work well with the BLE chip. Worse, the LED example code is in hand-tuned assembly! No one wants to deal with that at this point in a project.

The BLE chip is running a little operating system and the WS2812 LEDs have very specific timing requirements. It is possible to make them work but… well, it may not be as smooth and nice as you want. Worse, as marketing talks inevitably about the possibility of adding more LEDs and fancier patterns, perhaps this isn’t scalable.

To be safe, we’ll add another small processor in here, an ATTiny. The ATTiny is a way to reduce complexity of a system by dividing a complex task into multiple simpler tasks. If you look at existing projects in the wild, there are a lot of ATTinys controlling WS2812s. We are just making a proof of concept after all. We’ll get an Adafruit Trinket and it will solve this strange timing problem.

(Did you hear ominous music?)

Let me pause here because I’ve been throwing around all sorts of letters and numbers in this post, as though you should be familiar with every chip out there. In the fake outline I gave you above, we would be skipping ahead to Liturgy Memorization because that’s what it amounts to: random memorization in order to fit into the group. Sadly, I’m going to continue throwing around the company and component names as though you’ve already memorized it all. However, the specifics are not critical to catch, and the links are there if you want to know more.

I recognize that this assumed knowledge is kind of annoying as we haven’t gotten to the tricky parts yet. Sure, we just saw a timing issue but if we were making the IoT Light prototype, we probably discovered those from forum comments and application notes. It wasn’t a painfully learned lesson discovered by working on the problem ourselves.

Jargon is terrible but essential. It is exclusionary but makes for a great shorthand. Remember: no one is born knowing all these letters and numbers. Once you know them, you tend to fling them around with abandon. I am aware it makes me sound unintelligible sometimes.

Through the magic of hand waving, off-topic asides, and other people's code, our IoT Light proof of concept mostly works. It wasn’t trivial to put together but it wasn’t that hard: we used Nordic’s example code to make a BLE interface, debugging it with a smartphone application (like LightBlue). We can send it BLE commands and it changes the LEDs. We had to figure out how to make a serial driver talk to the LED controller and spent a fair bit of time coming up with a clever way to tell the ATTiny what to do with the LEDs. But it works.

Also, we checked with your EE: the two processors can go into low power sleep modes so if we decide to go for a battery powered device, it isn’t that big of a deal.

The proof of concept comes together so easily that we start to think, “Sure, that marketing schedule is reasonable, we can ship it in eight weeks.”

Well, that’s the thing, right? Everything from here gets much harder. To some extent it is worse knowing all of your intended functionality can work. It means that all this other fiddly stuff isn’t even important. Or at least that is what it feels like as you are working on it.

Now is when things get tricky.


This a blog post is part of a series based on the Embedded Systems: The Tricky Parts talk I gave to the Silicon Valley IEEE Computer Society. A video of the talk is available on YouTube.