Transcript from 428: Sprinkling a Little IoT with Jonathan Beri, Chris White, and Elecia White.

EW (00:00:06):

Welcome to Embedded. I am Elecia White, alongside Christopher White. We've talked about Memfault. We've mentioned Renode.io and maybe Ubidots and Golioth and Particle. I have just discovered these are all Pokemon and that we need to collect them all. Jonathan Beri is going to help us unravel which one will win at which Pokemon towers. Is that right?

JB (00:00:38):

Maybe if the Pokemon have antennas in their tails.

EW (00:00:42):

More seriously, we're going to talk about the Internet of Thing and the tools that we can get around the Internet of Thing. They are not in fact Pokemon.

CW (00:00:51):

How are you doing, Jonathan? Welcome back.

JB (00:00:53):

Thanks. Great to be here.

EW (00:00:55):

Could you tell us about yourself as if you weren't on a show 200 episodes ago?

JB (00:01:05):

Yeah. Quickly about me. My name's Jonathan Beri. I'm the founder and CEO of Golioth, an IoT platform company, but I've been in embedded IoT for the last decade or so. At big companies like Google Nest. At tiny companies, like Particle, not so tiny anymore. And started Golioth in this similar space.

EW (00:01:25):

All right. We're going to ask about Golioth, but we are also going to ask about a lot of other things. Before that we want to do lightning round. Are you ready?

JB (00:01:36):

Yes.

EW (00:01:38):

Complete one project or start a dozen?

JB (00:01:40):

Definitely start two dozen.

CW (00:01:42):

What does the S and IoT stand for?

JB (00:01:47):

Well, generally not security.

CW (00:01:50):

<laugh>

JB (00:01:52):

Sometimes I like to say sane defaults, but that doesn't quite have the same ring.

EW (00:01:58):

What is your favorite video game?

JB (00:02:01):

Favorite video game. Oh, Pokemon. <Laugh>

CW (00:02:08):

Should we bring back the dinosaurs?

JB (00:02:10):

Five movies tells us no.

EW (00:02:13):

If you had your own podcast, who would you want to be as a guest? And what would you ask them?

JB (00:02:18):

Oh, I wasn't prepared for any of these lightning round questions. <laugh> I would have a fictional guest, Jean-Luc Picard and grooming tips. <laugh>

CW (00:02:36):

We may have asked this the last time you were on, but we can go check to see if you give a different answer this time. Favorite fictional robot?

JB (00:02:43):

I probably have a different answer every time that question is asked, but probably Lieutenant Commander Data, going on the Star Trek theme.

EW (00:02:53):

And one last one. Do you have a tip everyone should know?

JB (00:02:57):

I remember my tip from last time, which is still very valid, which was, don't cross the red and black wires. <laugh> Things go wrong.

EW (00:03:04):

It's still valid. Yes.

JB (00:03:06):

I think this time it's more, once you get the bootloader working, don't change it. <laugh>

EW (00:03:15):

Yes.

CW (00:03:16):

And might I add, don't put too much in the bootloader.

EW (00:03:20):

<laugh>

JB (00:03:20):

Yes.

CW (00:03:21):

And don't look at my PR that I just filed yesterday, which has too much in the bootloader.

EW (00:03:27):

When we last talked to you, you worked at Particle. What does Particle do?

JB (00:03:33):

Particle is an IoT platform. They are a full stack solution. So you can buy a component, whether dev board or module from them. That includes the operating system, the software update capabilities, connectivity. So if you are building cellular products, they even give you a SIM and then the tools to manage the devices.

EW (00:03:54):

And now you have founded a company called Golioth. What do they do?

JB (00:04:00):

Similar space. We help hardware companies who are designing internet connect devices, by providing those capabilities like software updates, secure communication, but in a different approach. Where we say, "Hey, you've got your chip, you've got your vendor of choice. Your product is working. Now, how can we help connect that device online? But in a way that is working with the components you've chosen, or the operating systems you want to use." A little bit more bring-your-own-device kind of way.

JB (00:04:34):

Our approach to that problem space is, let's look at an IoT device and we'll definitely go into IoT I'm sure. And the cloud parts of that are very complicated, very specific and hard a lot, especially for hardware companies. So can we be that team who's figured that all out, what do you need from the cloud? And let's provide that. So kind of a longer answer then. It's slightly different.

EW (00:04:57):

So I've been doing embedded systems for what seems like forever, but isn't. I definitely started after networks, but I went to several companies who were building various IoT devices as we would call them now, who would start with the idea that they should build their own radios or their own GPS receivers. I would go, "No, maybe we should buy those, even though they seem super expensive, we should maybe do whatever it is you wanted to do." But we would still have to run our own servers, build our own bootloaders, update our own firmware, monitor devices for system health and why they crashed, and whether or not the bug that they have actually has to do with something crawling inside the box, because Boston has salt fogs. All of this was from scratch. Sometimes it was an assembly even, but now there are tools for all that. There are so many tools. Okay, we have Memfaults. Do you remember what Memfault does Christopher?

CW (00:06:09):

Yes.

EW (00:06:10):

Okay. Tell me.

CW (00:06:12):

I'll record it and insert it here, <laugh> after this show. Memfault does a lot of diagnostic stuff and capture a lot of telemetry and things from IoT devices, if I remember correctly. So they provide infrastructure so that you can have your device and have it send in a compact way, statistics and things, as well as from our update things.

EW (00:06:39):

But they didn't care about the data. It was all about the operations aspect.

CW (00:06:46):

Yes. From my memory.

EW (00:06:49):

Jonathan, Golioth provides, you just said. I don't know where that fits in. It's not AWS. AWS provides, like-

CW (00:07:01):

I think it is on top of AWS. If you want to do IoT, and I'm going to stop talking, let John talk in a second. If you want to do IoT, if you do it from scratch, then yes, you put a bunch of stuff in front of AWS, but that's a lot of hard work. You don't just connect AWS to your device. I think maybe Golioth is the thing that makes a lot easier. Do I have that right?

JB (00:07:25):

Yeah. It's going back to what does an IoT device need from the cloud? And you could totally build everything that Memfault and Golioth and Renode does yourself, but-

EW (00:07:36):

<laugh> I feel like I have. <laugh>.

JB (00:07:38):

Yeah. Like many people have, I mean, in Nest, we had everything from scratch. Even the way we use databases, creates a lot of the capabilities that Amazon has today. But when you're talking about a product, for the most part, those things are actually not part of your business. It's not core to the value you create, because if you're building a smoke detector, safety, reliability, user notifications, that's your product. Great hardware, reliable sensors, that's your product.

JB (00:08:07):

The things in the middle, the things that go into developing your product, the things to monitoring your device, your product, the things to making sure that you can scale your product, again, is not part of the thing your business wants to build, and hopefully we will be successful with. To your original point, over time, more and more people have realized that. Whether it is coming from the open source world or commercial offerings, like Golioth and Memfault, that should be a commodity. You really wish that you wouldn't have to start with, okay, let's write our own assembly bootloader, then bring in our own IP stack and then write a custom driver with our modem vendor.

JB (00:08:43):

And then, okay, now we have to figure out security and communication and encoding, and now we have a database. I haven't even built the mobile app yet! So how do we go from have a chip on our desk, to having that end product. That's where a lot of the solutions of space have start to be developed and show up over the last five, ten years.

CW (00:09:03):

And sort of parallel to IT where five or ten, probably more like ten or 15 years ago, if you were at a company, you had a server room, it was packed with servers and you had an IT department that was tasked with managing those servers and everybody's files went on those for the company. Now, nobody does that. It's all elsewhere and managed by some other company, because if I'm running a company that makes X, running a data center, shouldn't be necessarily part of X.

JB (00:09:32):

Yeah, exactly. Why does most companies use Gmail for their email server, right? Doesn't help your business by running your own SMTP server anymore.

EW (00:09:42):

What? This is so true, but it's hard to understand the landscape. And to some extent, it's easier for me to explain to managers and clients what they have to do to make it work, instead of trying to figure out which company I should be recommending and which ones won't exist in a year.

JB (00:10:15):

Yeah. And, I think there's a third question. What do you need from the cloud? Not everything is an IoT device, and not all IoT devices are created equally and don't have the same requirements. As an example, I met with a large consumer device manufacturer last year and they have a lighting solution. They're very much privacy focused and their customers don't really want to use a cloud service. So by design, they made sure that their lights and light switches just work and they get paired in the factory. And there's a mobile app for some minor tweaking, but it doesn't talk to the cloud, mostly. I say mostly, because they have a, let's say, compliance requirement to have the ability to software update those devices in case there's a vulnerability or in case there is a major feature that they want to push out, but they plan never to use it.

JB (00:11:11):

So going from, I have an offline, no wireless link between two devices to, now I have to be able to push a firmware update to millions of devices, sometimes is this huge lift. And so I call it, "Sprinkling a little IoT." It's just as hard as building a full-fledge IoT system. Well, that company has to figure out OTA, and fleet management, and being able to push signed firmware binaries down to a subset of devices, to a hundred percent of devices, et cetera. For them, they just need to go figure out the OTA problem and have a solution. Versus, let's say, a agriculture product, which needs to sense a lot of different sensors all in near real time, combine that with weather data and location data, and other telemetry from third party vendors. They have a whole other set of problems, so it's hard, it's messy, it's complicated. But first it's useful to figure out, or to determine, or even just to ask, to figure out what questions to ask, what do I need from the cloud as part of my it IoT solution, and what don't? Because that will help you figure out which vendors in the space actually serve your needs.

EW (00:12:26):

Okay. So OTA, over-the-air, firmware update, that's one giant feature that many people don't expect the complexity of, even though we talk about security and how important it is not to create bricks, it's still something that gets tacked on at the very end of projects.

JB (00:12:47):

You forget to have enough flash on your device for A/B updates and didn't choose the right component with hardware accelerated cryptography, so you can't actually do the OTA fast. Yeah. Usually it happens too late in the process.

EW (00:13:00):

Another thing that happens often very late in the project is realizing that you need to manage the device. You need a heartbeat to figure out if the devices are still out there, and you need them to check in to see if they need to be updated. You may need debug ability, so that when you do send out firmware updates, you know that their batteries are dying, or there's more crashing of this kind, this whole, what I'm saying, the operations side. And then maybe you need to command something, maybe the agriculture device needs to command something that says, tell the farmer to turn on the water.

EW (00:13:45):

Or maybe the farmer says, okay, now I want you to turn on the water and it commands down to the device. So there's this command and control, and it can go either way. That most people get. Most people say, okay, I understand what that's for. And I understand why my product would need it. But the first thing most people think about are the data streams, the sensors, where are they going, which way do they go? And how do I collect that data? Are there other things I'm missing here?

JB (00:14:22):

You're definitely right. The data is usually the first thing, and sometimes the only thing, companies think about as relates to the IoT device, because that's where the value is, right. Maybe there's an aspect of command and control, but how they get the data, how much data they're going to have, how they're going to store it, these other elements are really part of the early phases of the product development life cycle, as well as the operations of the product development life cycle. Those are kind of the main things. I would say, the different phases, you might actually be using different tools. Like we talked about, observability, monitoring the device, the things that Memfault does really well, that's great for when you're initially developing your solution, but also as you operate the device. And some things are actually even more important at the development phase, versus the operation phase.

JB (00:15:13):

Being able to test at scale, there are companies that help with in the cloud, let's say scaling out your virtualized tests. Like we know as an example of a tool that helps you do development in the cloud, or assist you developing with the cloud capabilities. Those four things you hit on, which you can probably drill in if we want to go a little bit deeper, device management, that sort of command and control, the data, and really just interfacing with other systems. Yeah, that's probably the last one that we didn't touch on. Sometimes you have to talk to other systems. It's another team in your cloud, application team or your mobile team, and we can call that data, but it's actually quite a different problem. Because if you get a raw sensor reading, let's say of an accelerometer, you get its raw values, you might send that to the cloud, denormalize it into actual coordinates. Well, the application team, who's trying to show the tilt of a trashcan and a mobile app has to then process that in something else. And that goes onto the cloud, the general cloud world, so there's some tie in between the world of device and sort of device management device data, the world of application and application data. So there's that bridge.

EW (00:16:29):

Renode, R E N O D E dot I O.

JB (00:16:35):

Mm-hmm <affirmative>

EW (00:16:37):

They do simulations. They don't solve any of these problems actually. Is that right?

JB (00:16:45):

Well, they actually solve a good chunk of these problems. I was talking about the role of the cloud and I've started this mini-series that I call "the clouds of IoT" and it's probably a good framework for us to talk about. You're giving me flack before we started, that I hadn't finished the blog series. It's coming, it's coming.

EW (00:17:06):

Was a great start. And then I was like, okay, what about the other four? But go ahead. Talk about it please.

JB (00:17:12):

It's trying to debunk part of these challenges you're asking about that there's not just a air quotes "cloud". It's not just someone else's machine, right? But the capabilities of a solution that is enabled to the cloud for IoT is actually specialized. And I talked about five different types of clouds and, to be very clear, there might be one company that does more than one of these clouds, but at least I try to point to one specific example or two specific examples.

JB (00:17:42):

So Golioth and Particle and a few others in this space are what I would call a "device cloud". That's the thing that helps you connect the cloud to the internet, security, software updates, the thing that you, as a firmer engineer, might be using on a regular basis while you're in the prototyping phase, when you're in the pilot phase, even operating a device at scale, pushing new firmer, that's the device stuff. That's what a lot of conversations end up talking about, we talk about OTA, et cetera.

JB (00:18:15):

But there are other types of clouds that are part of a solution. And, I mentioned, the "connectivity cloud" is another kind of cloud. We can double click on that. I'll list the rest before we go on in more detail. What I would call the "data and data cloud". I'll rattle these off a little bit faster, so it's not taking too long. "Application cloud" and the "development cloud." As an example, a connectivity cloud, if you're building a cellular based device, you might need a SIM provider, [inaduible] think Verizon or Hologram. If you're building an application, so a mobile app or a dashboard, you might use tools like Node-RED or Blynk or Ubidots.

JB (00:19:02):

A development cloud is a tool that helps you in the initial phases of development and through the development life cycle, as you continue to iterate on your product. I would categorize Memfault in the development cloud. It's a key part of writing your firmware, testing your firmware, doing new releases that is actually made better because the cloud exists. Versus, having a similar tool on your desktop, that you would batch download log files, and then try to process those locally. It's just a complete different kind of user experience, than having a real time tool that can aggregate all that information and make use of it. So those are the five clouds: device, connectivity, data, app, and development, in no particular order.

EW (00:19:41):

Okay. Ubidots and red somethings?

JB (00:19:45):

Uh, Node-RED.

EW (00:19:46):

Node-RED. Where do they fit in?

JB (00:19:49):

What they help you do is, well, they kind of overlap. Data and apps tend to kind of share because first you need to be able to collect all the IoT data, and then you need to be able to create dashboards, mobile apps, etcetera. So Node-RED is a great tool for taking in sensor data or other sources of data with a low code experience, a lot of time, IoT, and then process it, so it can be averaged or summarized or reformatted, so it's more compatible with, let's say, your dashboarding tool, like Ubidots.

EW (00:20:22):

Grafana, where I can just graph things, would fit in there?

JB (00:20:28):

Yeah. It's a actually pretty common, we see a lot these days for the application. When we talk about application, it could be anything from, hey, is the data coming in and can I see it on a chart, to, I'm literally building an native Android and iOS tool. And Grafana is definitely an app builder for a lot of other companies, especially if you're not building a consumer product, where there's a installer or a product manager or a building manager. As an example, they just need that dashboard. Grafana would fall in that category.

EW (00:20:59):

The data cloud is that just always AWS and Azure and whatever Google canceled last week?

JB (00:21:06):

<laugh> Yes and no. This is where we're starting to get into that, well, some companies do multiple things. Like AWS IoT has some device cloud capabilities, device management, software updates, but they of course have a huge swath of stuff around data processing and data storage. But there's a lot of IoT-specific databases. So in general, IoT sensors tend to be time stamped, so we call them "time series databases", and there's companies like InfluxDB or TimescaleDB. They build specialized databases that are extremely efficient, easy to store the sensor based data, but also easy to query it for visualization or later use. A bunch of companies make these optimized databases for things like time series data.

CW (00:21:55):

It seems like Amazon has to a certain extent owned a large chunk of this market with AWS, at least the data storage market. Is there a way to spread the risk if you're an IoT company and say, well, I don't want to put all my eggs in an AWS basket in case the Cascadia fault breaks? Or do you have to choose, okay, this is where my stuff's going, and that's just the way life is if that company screws up and loses my data?

JB (00:22:23):

Yes and no. There's a standards problem in IoT, as we're all familiar, but it's not as the same- When we talk about standards, we talk about data standards and how your Google device for Google Home can't talk to your Alexa. What I mean by that more is, the way that you build a device for AWS IoT, is fundamentally different than you could build it for Azure IoT or for Golioth or for anything you built yourself from homegrown. So the portability across different vendors makes that challenge. Some solutions like Golioth can work on multiple cloud providers, so if you build for that one thing, at least that it can run in multiple places. But all in all, each IoT project, for the most part, is kind of a snowflake, even if it's doing very similar thing to the nearest competitor.

JB (00:23:11):

So if you take on a whole, all-in solution, let's say you built vertically on AWS, you can't really run that against Azure IoT. Either you pick solutions that are designed to have multi environments, which there aren't that many, or you pick some of the pieces. So, a lot of companies that I've come across who, let's say, use AWS for their IoT product, they might just use AWS's database. And then they hired a big team and they wrote a lot of their own IoT device management on top of AWS not depending on their solution, because of that key factor. They want to use a database that's high availability and cost effective. That same database might be running also on Azure, because it's a standardized database, but that's the only real way, except for again, couple solutions like Golioth, that is designed so that it can be portable. Yeah, there's that baked in challenge for most solutions.

EW (00:24:09):

Is it baked into operating systems as well? I mean, I know AWS has a FreeRTOS branch and I think Golioth likes Zephyr. I'm liking Zephyr more every time I see it. And Azure has ThreadX. I'm starting an IoT project. If I choose one thing, does that mean I've chosen everything?

JB (00:24:42):

There's actually a couple good questions in there. I'll start with my recommendation for anyone who's starting an IoT project. Start with your component. What chip do you want to use? What radio technology you want to use, get that vendor down. And each vendor has made their own bet at the OS level. And generally speaking, those vendors have FAEs that will support you on the firmware side and the hardware architectures design side, and they've partnered with the cloud parts. So they have middleware components that you can drag and drop, and pull in. For example, if you're using Infineon with ModusToolbox or MCUXpresso from NXP, and you're starting a project, you're going to have the ideal software SDK starting experience. And they have multiple RTOSs in there too, by the way, for the most part.

JB (00:25:31):

The middleware is the thing you can swap in and out. So before you get too far along into your development, you can build upon well tested and support the key thing, supported middleware, and for the most part, you actually get some choice. How vertically integrated that OS and middleware is, will kind of make it a little bit harder. For example, if you started with the cloud side, if you were to pick AWS IoT and try to design your hardware from there, well, if you're really proficient in mbed, you're kind of stuck. I have a friend who has built an IoT product and they chose AWS IoT and their dev shop chose mbedOS and the two shall never meet. Nine months later, they didn't ship their product and they had to scratch the entire V1 of the solution. They've gone to another platform.

JB (00:26:27):

So if you start with the OS and the hardware, find which one you're going to get the best support, then go from the middleware up, that makes it a little easier for you to design your solution. I mentioned Infineon and ModusToolbox, they support FreeRTOS and Zephyr today. So you can actually go with the software you're most comfortable with, or maybe meet the timing requirements or whatever, and then pick from a growing number of cloud solutions from there. You mentioned Zephyr. We're big fans of Zephyr, but the real driver for us, the reason why we like it so much, it's a good RTOS and has good kernel and and drivers, et cetera, but the key, and maybe sound a little bit repetitive, is that vendor support for Zephyr.

JB (00:27:14):

We know that if customers using an NXP part with Zephyr, we know the FAE who maintains that SDK and integration, and that we can find bugs, whether's our bugs or their bugs, and have that resolution. IoT is such a complicated thing! You want to reduce as many variables of complexity and find the most amount of support you can, picking that hardware OS is usually going to give you the happiest path and then adding the cloud on top of it, will just continue to reduce those areas where things can go awry.

EW (00:27:50):

I have a napkin sketch for internet enabled fishbowl, and I don't write code, and I don't want to know about firmware updates. Is there a completely vertically integrated solution so that my fish can talk to me while I'm at work?

JB (00:28:12):

Yeah. There are projects and there are products. There's something very important to keep in mind. I'm a maker. I build hobby projects on the weekends. I've also worked on products that have supply chain and design for manufacturing and lots customer support and all the other things. And so the category of things you could build for projects, there's actually a huge ecosystem in the maker community. We're seeing a little bit more of that makery low technical effort, making its way into finished products, or tools for creating finished products. But by and large, almost every product I've ever worked on or customers I've ever worked with, they have a C/C++ developer, they have a hardware loop testing lab, they are doing manufacturing with a CM. Where things that are low code and low friction for, let's say, non-engineers to design and develop, really fall in the category of makers, educators, et cetera.

JB (00:29:10):

There are solutions for sure that are great. There's tools like Blockly, which have been integrated in projects like the BBC micro:bit. We're starting to see more of that kind of drag and drop, or don't require you write C code. And things like Arduino, Arduino cloud IoT platform. They have the ability to design a solution and actually generate the code for you under the covers it's Arduino and C++, which is pretty neat stuff. But if you're thinking about making 10,000 of those fishbowls, you probably want to start there, get the solution with a vertically integrated product. And then find someone who can design a PCB or breakout board, write some firmware, think about the OTA, because one, it may not be secure <laugh>, it may not be scalable and it may not be cost effective. So it's a good place to start, but certainly I haven't seen a lot in the productizable realm yet.

CW (00:30:12):

You've made a good point that, this is something that is philosophically where I am as an engineer, is try to make things as easy as possible for yourself. Don't reinvent the wheel, don't go off reimplementing things that other people have implemented, try to find solutions that are quote, "not your job, not part of your product," elsewhere. Are there pitfalls and shiny objects that people need to watch out for, that can drag you away from the <laugh> easy path, because they look like a great idea? Have you seen a lot of that kind of thing, where somebody's like, "Oh, I saw this new technology and I'm going to go for it."? How do you advise people to be cautious about what they're implementing?

JB (00:31:02):

There are shiny objects all the time in software. I think program language is one of those examples. There is amazing stuff happening in MicroPython and Rust. There are people actually starting to ship product with those. But if you're trying to build consumer grade IoT product, are you going to be able to find 30 Rust developers who know the Tokyo real time stack? And be able to interface with the networking, that tends to be a challenge. So I go back to, what is the prescribed path? What is the happy path with the most amount of support? Whether that's from the chip you bought and the vendor who makes it, and the support team they offer you as part of your being a customer, to the modems and the other forms of connectivity. Are there stacks that you can just grab, as opposed to finding other challenges? Because IoT is probably the culmination of most of software, right? It's hardware, it's embedded firmware, it's networking, it's modems, it's connectivity, it's security, it's infrastructure, it's servers, it's databases- <laugh>

EW (00:32:14):

Applications.

JB (00:32:15):

Yes. Mobile-

EW (00:32:16):

Everything.

JB (00:32:16):

Cool. Yeah. So if you can just circle around the path of least resistance, yours to be more successful because there's so many places where you can get lost or confused or fail, really.

EW (00:32:31):

And you choose based on the device first.

JB (00:32:35):

Yeah. I would actually say, choose on the product first. So that fishbowl example, is there going to be a consumer product? Is it going to be in an aquarium? Is it going to be at a Vegas hotel? Who's going to operate it? Does it need to have one controller, multiple people controlling it? Is it part of a larger system? A consumer electronics company who might do what's called a MRD, a market research document, understanding the landscape of how people use products today, existing solutions and how people interact with them. From there, you can start to pan out or build out what kind of connectivity do we need, what kind of on device computing do we need? Do we have to interface with other devices, locally, over the internet, over some other transport? From this pyramid of figuring out what you're trying to build as the product, you can start to develop that roadmap of the hardware design, the firmware design, what kind of connectivity in cloud. Of those five clouds, what do we actually care about, what we actually need in our solution and not just pull on something because it's shiny and new, or someone told you to use it.

EW (00:33:45):

I totally agree with you. I like the way you've broken this up, because the device cloud is an area where I spend a lot of time, and then usually I have a UART or something that goes to the connectivity cloud, which might be BLE, it might be 2.4 GHz radios, and it might be cell modems. For the most part, I don't care. Those are very separate things. What I talk to after that, it's fine. And then as the clouds go up, the application cloud, as long as the data gets to the data cloud, I don't care what you do with the application cloud, as long as I can understand that my device is working correctly. Really that goes back to the development cloud, where somewhere between the device cloud and the development cloud, I know my devices are being updated, I know my devices are alive and that they are performing as their self test indicates they should. But who wants to plan anything? I mean, nobody wants to plan anything! Everybody just seems to want solutions or to write it themselves.

JB (00:34:59):

Yeah. It's interesting. I did a Twitter survey a few months back, about sensor drivers and FreeRTOS as a kernel and their distributions of that kernel. I found out that most of the distributions don't include sensor drivers. Like, Espressif has the ESP-IDF, which is based on the FreeRTOS kernel, and they've brought in a bunch of stuff to make it really easy to use FreeRTOS. But every single person who uses the Espressif ESP-IDF writes their own sensor drivers, it's the little I2C here and a little accelerometer driver there. When I asked why people do, is, "I've always done that." <laugh> Versus grabbing a library, a well tested sensor interface, from maybe the sensor manufacturer. And so I think there's-

CW (00:35:49):

Wait, those exist? Where?

JB (00:35:51):

Some do, or different operating systems include those. Like Zephyr has a whole sensor library with a bunch of sensors provided by some of the sensor makers. For example, Bosch has a bunch of sensor drivers, and some of 'em are actually optimized, and lo and behold, they actually perform, but people still end up writing their own Bosch BME680 sensor driver from scratch. Maybe that's just the way some people are wired, there's not what you can do about it. If you looked around and maybe surveyed the ecosystem, those early decisions, that planning, maybe you picked a solution that had some of that already built in that you didn't have to focus on those things that you were doing over and over again, in every project.

EW (00:36:38):

I think that's one of the things embedded systems is going to be changing in the next five to ten years, is that we're going to start figuring out that we can't keep writing the same-

CW (00:36:48):

Please don't make me write another accelerometer driver.

EW (00:36:51):

I mean, I haven't written a UART driver in years-

CW (00:36:53):

Right. UARTS are the least involved-

EW (00:36:53):

And before that, I used to write one or two a year, and now it's usually I don't have to do that, but I'm tired of writing display systems and...

CW (00:37:08):

Even in display, it's a mixed landscape. I think you're right, Jonathan, when you're planning out your hardware in order to walk that path of making it easy on yourself, picking hardware that you don't have to write a bunch of software, that is not part of your product's job, <laugh> nobody bought your internet connected fishbowl because it had a custom SPI driver for the light.

EW (00:37:37):

Yes. <laugh>

JB (00:37:39):

There's things that do matter, especially in the world of IoT, even a lot of applications that are, say, power sensitive, or memory sensitive. A lot of the companies we work with, they spent a lot of time on power management and actually taking the device driver they got, it was working and they're optimizing the driver. That's one, is more interesting and relevant to your product, but two, sometimes more fun depending on the DNA of the engineer. So it's better to spend time on those kind of things, as opposed to doing the default, pull off the I2C buffer, read it, dump it on the UART and then, oh yeah, cool, that works.

EW (00:38:18):

Okay. So cloud layers, I think I have 'em. Data, that we talked about initially, goes pretty well into these cloud layers. Device management goes into the device cloud, it might go also into the data cloud, but it definitely comes from the device cloud. Is that right?

JB (00:38:41):

Mm-hmm <affirmative>.

EW (00:38:42):

Okay. Then we have the data stream, which comes from the device cloud, through the connectivity cloud, into the data cloud, and then probably the application cloud does something with it.

JB (00:38:54):

Yeah.

EW (00:38:56):

So if we talk about actual, put in some place things, instead of just general names. Let's say, I want to talk about the data stream, and I'm going to choose Golioth, because I'm talking to you and it seems rude to choose Particle at this point. What happens next? What? Where? Which connectivity cloud should I use? What does going to Golioth mean? Does that mean that I also have to use some other piece?

JB (00:39:34):

This is where I have got to say it depends. <laugh>

EW (00:39:37):

Of course it does.

JB (00:39:39):

I get that one card, right? It really does depend on the product and the company, because a lot of products there might be already engineers to that company. They might be already building mobile apps. So you probably want your mobile team to have access to that data coming off the sensor and just let them run with it. But if you are a consultant, and you're working with an entrepreneur who has a great business idea, you're designing the PCB and some of the firmware, you kind of don't have that resource. So you want something that's super well integrated with wherever the data is coming off of, and you can just drag and drop and do what you need on the visualization piece. So we see a lot of mix. If you're not building a cellular device or a lower device or anything with a managed connectivity, you don't even need to worry about connectivity cloud, right.

JB (00:40:29):

You're building a Bluetooth app or you're doing something on ethernet or Wi-Fi, then you can just cross that part of stack altogether. If we go back to your fishbowl monitoring example, it's probably some pH sensor in the fishbowl, maybe some turbidity to measure the activity of the fish, and you want to stream that data and show it on a mobile app. A good example to use as a framework. You have picked your Wi-Fi component and the sensors and you wrote some firmware and you have your UART plugged in the side and you're seeing the data there. Probably the most commonality across all these types of products on whether it depends, is some form of device management.

JB (00:41:18):

So being able to connect that device to the internet, so it has a secure communication, instead of having it on a UART, maybe you have it on our log output in the cloud. And you're like, "Cool, this is connect to the internet." Now I want to test out OTA, so I build a new image, push it through the device management solution and you see it instead of, you know, blinking red, it blinks the green LED, "Great OTA works!" And now I need to show to my investor or my business partner, the values coming off that device. Log output is interesting, but let me just grab Gafana dashboard, wire up the data from the sensor, the streaming information, and then show the pH levels rising and falling, or the turbidity as you are going to twirl water around there. And so that's your prototype.

JB (00:42:15):

That checks off some of the boxes. As you go from that prototype to, let's say a pilot, there's a lot of other things have to happen. But fundamentally you're just building on where you started. Maybe you replace Grafana with a web app that the subcontractor built out, and that's taking the data from your device management like Golioth, and then dumping it into a web app instead of a Grafana dashboard. Then maybe you decide that you want to have a outdoor fishbowl, so you have a swap out of components and you now have a cellular device instead of a Wi-Fi device. Okay, now I have to go find my connectivity provider, bring that in. These two SKUs, one's the Wi-Fi, one is the cellular, one has connectivity and one doesn't. You can continue to refine the product.

JB (00:43:03):

At some point, usually in that pilot phase, you have all the cloud parts figured out. Who's doing what, if you're using a third party vendor, let's say Grafana. Or if that web developer built your own dashboard. Whether you need connectivity or you don't. Your device manager's probably plugged in from early on. The rest is product development, writing new features in firmware, testing, changing from a Adafruit feather board to a custom PCB. One of the key things is, there's a certain point where you don't have to make any of these decisions anymore because you figured it all out. Now it just matched to whatever those initial product requirements were.

EW (00:43:47):

I'm always afraid of that. I mean, what if I chose wrong? What if I chose Sigfox?

CW (00:43:56):

To what now?

EW (00:43:57):

And what if I had a LoRA device, I wanted it to be super low power. And because everybody else was using The Things Network, I was sure it was going to succeed. But now I have a device and I can't talk to it anymore, because I chose the wrong connectivity, or I chose a data cloud that doesn't exist anymore.

CW (00:44:19):

That's why you put eight different radios on every product, so you can adapt as necessary.

JB (00:44:26):

Yeah. I think one of the things that has changed, we talked a little bit about the changes embedded and how reuse is moving towards commonplace. Just the ability to prototype as a professional engineer has also changed significantly.

EW (00:44:42):

Oh, yes.

JB (00:44:43):

Right. You can actually build a working prototype with dev boards and breadboards, and it will be a lookalike. RF might be different in power, but at that phase, unless you have a very specific application, you're not optimizing for those things. You want to know does this work? Does the idea pan out? Can I build it with these basic components?

JB (00:45:04):

Choosing flexibility around the software you're using, we talked about operating systems earlier, having one that will allow you to, let's say, swap out a cellular AT modem for a Wi-Fi AT modem. Those kind of things can help you at least reduce the risk at that phase. And in some ways, even further down the road, as you get to, we deployed our first hundred sensors in the fields. It's like, "Shoot, I need to do OTA. And LoRaWAN doesn't support OTA. What am I going to do?"

JB (00:45:37):

That low cost, almost low cost, prototyping phase, you can go back one step without a significant crazy amount of cost. Versus, you have to manufacture the full thing with enclosures, with IP ratings, with FCC and then find out when you have your first thousand in the field, "Oh, shoot. I can't get through a metal box. I should have thought about that earlier, should've went with plastic."

EW (00:46:03):

<laugh>

JB (00:46:06):

I think those tools, whether it's the cloud-enabled versions, counterparts, or just, we've gone with debuggers and developers and IDEs and just cheap commodity sensors that you can easily wire in, makes it a little bit easier, a little bit less scary, but still risky.

EW (00:46:26):

I was teaching my "Making Embedded Systems" class today and their homework was to make Blinky work. When they heard that last week, I'm pretty sure I could hear their eyes rolling. It's like, "Make Blinky, everybody's made Blinky. That's dumb." Then today, when they had to answer the questions of, "Did you step through the code? What compiler did you use? Did you use one of the hardware abstraction layers? Are you using an RTOS? Did you use the class board or something else?"

EW (00:47:03):

Suddenly the tooling becomes so much harder than the development, than the engineering. I mean, the tooling is engineering. I shouldn't say it that way because that makes it sound like it isn't, it totally is. But it's the part of the engineering problem that we don't always remember. I feel like we're hitting that with the different clouds. Okay, it's all there, and I might be able to put it all together once. But I don't know that I could put it together in a way that I could be confident it would still go together that way in a month. Or put it together in a way that I'm confident somebody else could put it together with my notes. It's a tooling problem right now. Do you see that as well? Or is that just me not knowing the landscape?

JB (00:48:04):

No, I think the tooling problem is one of the things holding embedded back. I really enjoyed Tyler from Memfault, when he was talking about the analogy that, a lot of large software companies have engineers dedicated just to make tooling better, to make those engineers highly productive. There's a humongous group at Google, just building internal tools for code completion, for codesharing, for continuous integration and deployment. I remember, this is now a lifetime ago, there was a public API we wanted to develop, and a single engineer could build that public API for all the code, configured it and deployed it on a global scale. And that's only possible because of the tens of thousands of hours, from hundreds of engineers, who built the tooling to make that possible.

JB (00:48:58):

We don't really have that. I think that we can pontificate of why, in the better world that's the case, but there's at least one component that I feel pretty strongly about, is because even though there were teams of embedded engineers, the embedded work was mostly individual. Like, you were running your driver, you're integrating into some code repository, but the tools that you interact with are the thing at your desk, the code on your screen, the IDE that's configured in your environment. Modern software practices have made things more distributed, but also, it's a complex system of things that you don't even know exist yet, or haven't been built yet. That doesn't work anymore. Things like modern development tools that we are seeing in the classic web server and mobile application are slowly trickling in to embedded.

JB (00:49:54):

I mean tools more of the standard definition, like an IDE, a compiler, debugger, but also the things around those tools. Like, observability for what's happening on your device at your desk or in the field. Or how to get a packet off of a UART into a database and all the things in between. We're still getting those more classic tools to be improved, like Visual Studio Code. When I was working at Particle, we launched a plugin for Visual Studio Code, and no one was doing UART terminal output from a physical device. So figuring out how to plug into /dev/DTV0 on the Linux machine, and then show the log output from a Particle device was actually really novel at the time, and hard to do and hard to repeat and wasn't reusable because it was tied to that specific platform. But now Microsoft has a team building embedded tools for Visual Studio Code. Things like step debugging for real time operating systems. Things like terminal output. We're slowly getting there. I think the tools around, the IoT parts, for that part of the embedded world is also catching up. Like Memfault, like Renode, like Golioth, and others.

EW (00:51:14):

So many people are used to rolling their own, because they control all of the pieces, which means they don't have to worry about a company going out of business, which I think is a common fear. How do you as Golioth CEO say, "No, no, we'll be here next month" and get people to believe that? As an engineer on the other hand, how do you figure out when to trust a cloud service?

JB (00:51:47):

Those are two, three? good questions. I think part of it is just some of the things that have happening in the IoT space. Specifically IoT, not talking about Golioth yet. Like, open source is a thing now. <laugh> Leveraging open source has actually been a big part of our company and our product. I would say ten years ago, we couldn't have built some of the things that we wanted to build, that we could build today because of leveraging open source. That's both on the device side, but also a lot of stuff we do in the cloud. Like our approach to interfacing with device side firmware through the RTOS is we just have a little library. You can look at our library and see all the codes there and all the interaction points.

JB (00:52:32):

If you need to design Golioth out, you can just take that library, figure out what we're doing and swap us out. It doesn't prevent us from going out of business, but it certainly prevents the risk of you being stuck with a service that no longer works. On the other side of, let's say, the UART, is the cloud communication, and the security, and other things that we talk about. Our approach to the device management, device connectivity, security, all those other pieces, is using open standards. Either open source standards or open standards that are defined by big groups of PhDs. So we document all of our APIs on the cloud side, and you can go on our website and see our reference information, and you could literally reproduce what we're doing in the device to cloud communication.

JB (00:53:19):

So the device side is open source. The cloud communications is open and documented. If we went out of business, we could show people today how to design us out, so that you can limp along until you design your own solution. That's like worst case scenario, which people want to hear. From a business perspective, when we have these conversations, especially with larger companies, they have those concerns, not just as an engineer, from a business perspective. They don't want to build a device management platform. Again, because it's not part of their core product and it doesn't create value. But they need to be able to control it, they need to be able to understand it, and they need able to communicate that to their end customers.

JB (00:54:03):

Part of our products has an enterprise version, so we can run a version dedicated just for your company on Azure, on Google, on an oil refinery, if you had your own data server there. That allows them to have that control without having to basically build a Golioth size company. Between the openness of our solution, and the ability to actually run it themselves without having to build it themselves, is how we mitigate some of that. It doesn't make everyone feel good, or answer all those questions, but at least it's different than lots of IoT companies today. Especially consumer ones, they just run out of funding because they built something proprietary and did it from scratch. Literally their gateway devices, their lighting controllers, just stopped working. That's our answer.

EW (00:54:56):

So as an engineering side, how do I avoid vendor lock in for my IoT tooling? Is it even worth bothering to worry about that? Or do I just figure out what my company needs and not worry about?

CW (00:55:12):

Yeah. Three abstraction layers.

EW (00:55:13):

Yeah. How many abstraction layers do I need?

JB (00:55:17):

There's a fine line of course, because the more abstractions you have, the harder it is to maintain. Probably performance gets impacted. What I usually recommend, or the analogy I usually give, when I talk to engineers as opposed to the business person, is when you're designing a PCB, you have a keep out zone, right? To deal with issues like, "Oh, what if there's a component shortage, and I have to redesign my PCB? Do I have to redesign the whole thing or just swap out the modem?"

JB (00:55:44):

Or if you're writing some firmware, you don't have to create your own HAL abstraction layer on top of a HAL abstraction layer. Maybe modularize your code a little bit. That's probably good practice and I would say best practice. Because at the end of the day, the more effort you put into buffering from, let's say, the chance that there's a solution that no longer works, or you have to redesign somewhere in the future- But you can't spend all your time running that abstraction.

JB (00:56:15):

So if you can prevent the inevitable redesign, if that's your fear of that, without having to spend as much time preventing that through abstraction layers, than you are writing firmware code, you're doing it wrong and you really have to strike that balance. The secret, I'll tell this to your listeners, about IoT products is, they're actually pretty simple. All this complexity we talk about, the thing that's happening on a device, where the thing that relates to the IoT parts, is actually usually pretty simple. Other companies take care of it. Like, your module vendor doing the modem, is probably handling security and certificates and network packets.

JB (00:56:56):

The actual thing you're doing is probably just reading a sensor, taking the sensor data and then sending it to a UART. And that piece can probably redesigned if you really need to, without too much pain. The actual application, the firmware, the logic, the algorithms, the power management, all the things that's your product, your innovation, that's always going to be complicated. That's where the time should spent. That IoT part actually tends to be pretty simple, and the least amount of places you have to worry about.

EW (00:57:28):

It's where scaling and large numbers of units come in. That it comes back to being a device problem. Getting a bug report that says unit number 1,000,012 crashed. That's not actionable. <laugh>

JB (00:57:49):

Mm-hmm <affirmative>.

EW (00:57:52):

So that comes back to device management. Does Golioth do any of that?

JB (00:58:00):

We do a bit. We've decided to focus on our swim lane, which is connecting device, securing device, making sure they're online and healthy. Things like understanding the firmware, and what's going on on the particular light device at the stack or heap, we don't do. That's where folks like Memault do really well and kind of where we stop and they start. We see a lot of folks using those solutions together, actually.

EW (00:58:29):

I feel like I need a map. Those cartoon maps that are for tourist cities. Over here is the application cloud, and these are the stores and restaurants over there. And here's the device cloud. And here are the stores and restaurants over here. And look, there's a little road that goes between them.

CW (00:58:50):

I think you need to do a whole book of understanding embedded systems through maps.

EW (00:58:53):

Okay.

JB (00:58:54):

Didn't you do a great design of...

EW (00:58:56):

The memory maps.

JB (00:58:59):

Yes, exactly. I had that visual in my head of IT products and solutions.

EW (00:59:04):

Yes, if somebody could make my Memory Map Land, but instead for IoT solutions, that would be awesome! Then we can have the "Ocean of AWS" and it can have little ports along little areas that it needs to touch, and FreeRTOS can be like the planes...

EW (00:59:23):

Anyway. So you were a product manager with, and did some code for, Particle, but now you are CEO. How's that gig going? How is it different?

JB (00:59:39):

Well, you both run your own companies. I would say I spend a lot more time running a company, than I am product managing or writing code. <laugh> Things like payroll, insurance and weird laws between states, because we're a remote first company. That part aside, it's just been fun to scratch the itch and build a team and work with some great people. It's not for everyone, because it is Saturday and I'm probably going to get off this podcast and do some emails... But I think it's worth it.

EW (01:00:12):

You've built a pretty fun team, it looked like.

JB (01:00:15):

Indeed.

EW (01:00:17):

Let's see, one more question. What are the best and worst uses of IoT that you have seen so far?

CW (01:00:28):

<laugh> Is that a listener question?

EW (01:00:31):

No, but I think maybe they asked what was the worst, and I added what was the best, because you can't just have the internet of fishbowls. It's a fishy example.

JB (01:00:41):

Well, it's worth calling out. There's a Twitter account, the internet, where they literally have an internet connected toilet with health monitoring capabilities. I'll leave it for the listeners to look up what that means. My rule of thumb for connecting a device to the internet is, it should be better than if it wasn't. <laugh>

CW (01:01:03):

Yes! Right!

EW (01:01:08):

Don't take away the features, because you've added the internet and want me to pay for them every month, instead of just the wants.

JB (01:01:17):

Or actually make it perform worse, because now you have to have a 2X size battery to deal with the Wi-Fi radio you put on there. And it literally does the exact same thing that it was doing without internet. You've just created a dashboard. And I think-

EW (01:01:31):

Except now you have to update the firmware every six months, randomly.

JB (01:01:36):

Hopefully, there's a firmware update every six months, you're not stuck with a terrible device. Yeah. My general rule of thumb is, something that senses something in the world that you wouldn't have knowledge otherwise. Something that provides automation, or a meaningful enhancement of convenience. I'll try to describe those. Or, let's say, does something really cool. <laugh> That farming example we used earlier, right? You have no idea how much UV, how much rainfall or moisture's happening to your crop at any form of scale, without having some sort of sensor there. With the cost and deployability of sensors, you now have insight into your crops that you never have before, except for sending someone else to, like, put their thumb into the ground.

JB (01:02:32):

You still wouldn't know about those UV. So that actually improves crop yield and crop health and reduces environmental waste on water and rolling up trucks or tractors to go check on those things. Getting information which you couldn't get otherwise, I think is my favorite use case of IoT. This idea of convenience, not just, "Hey, I'm sitting my couch, I want to turn on my lights." I like consumer products altogether. But, being able to monitor things that you couldn't easily send a truck out to, talk about waste management, elevator monitoring, street lighting poles, windmills, wind generators. It's really expensive to send someone up there, like thousands of dollars. Instead, "Is the generator working?" Thumbs up, great. We don't have to spend, send someone out there, dangerous trip, whatever.

JB (01:03:31):

Those are my two general use cases of when I think about, okay, this is IoT making devices better, products better, our environment better. I just think of cool things like, particle accelerators and being able to beam form crazy stuff, in the convenience of the upstairs lab, where you could have a PhD sit there with a dial and move things around. Being able to have that remote control experience, there's fun examples. There's a example that Zach, the CEO of Particle used to always bring up, which I also agree is a terrible use case. There's a basketball from basketball company? It was a Bluetooth monitor, with an accelerometer, so you can monitor your shots. But they had to put the Bluetooth radio in the accelerometer somewhere. So they put it at the bottom of the ball. Well, if you dribble the ball and it hits on the sensor, it just one, breaks the sensor and two, doesn't bounce back. So you've literally made a basketball, not a basketball anymore. <laugh> It's just figuring out where it can enhance that, versus...

EW (01:04:45):

Yes. Make things better! Jonathan, it's been really good to talk to you. Do you have any thoughts you'd like to leave us with?

JB (01:04:55):

Yeah. I think IoT is such a fascinating space. It's obviously what I spend a lot of my time in my career on, but it's complicated. If your listeners are thinking, wow, there's so much stuff I have got to learn and figure out. Yes! That's what makes it both enjoyable, but also complex. And the advice I give is, find resources. Whether it's, if you're working with a chips vendor, get some help, because they have engineers who are helpful. There's online communities. There's very active Slack, like Embedded and Interrupt, as well as Discords that are popping up all the time. And local enthusiasts who are bridging the divide between embedded and hardware and connectivity. We can figure this out. We see people shipping products all the time. Just know that if you feel overwhelmed, it's not just you. It's a vast, complicated space, but there are people here to help you figure it out.

EW (01:05:51):

If you want to collect all of the Pokemon that are the IoT tools companies, you should search the internet for IoT Pokemon? <laugh>

CW (01:06:02):

I don't think that's-

EW (01:06:03):

I wonder what's going to happen there, if anybody actually tries that.

CW (01:06:07):

New product idea?

EW (01:06:09):

Our guest has been Jonathan Beri, founder and CEO of Golioth.

CW (01:06:16):

Thanks, Jonathan.

JB (01:06:17):

Thank you both.

EW (01:06:18):

Thank you to Christopher for producing and co-hosting. Thank you to our Patreon listener slack group for their support and their questions. Of course, thank you for listening. You can always contact us at show@embedded.fm or hit the contact link on embedded.fm. And now I do not have a quote to leave you with. I think Jonathan has said it all.