Transcript from 415: Rolling Computers with Benny Meisels, Elecia White, and Christopher White.

Welcome to Embedded. I am Elecia White, alongside Christopher White. We're talking to Benny Meisels this week about automotive security, and we don't mean car alarms.

CW (00:18):

Hey, Benny. Thanks for joining us today.

BM (00:20):

Thank you for having me on the show.

EW (00:23):

Could you tell us about yourself as if we met at a large embedded systems conference?

BM (00:30):

So I'm the Lead Solution Architect at CYMOTIVE. We offer various cyber security products and services for securing vehicles in the automotive ecosystem.

BM (00:40):

Previously, I was a security researcher and a manager for the security research team for onboard in-vehicle penetration testing at the company. And before that, I served for five years in the military as a security researcher.

EW (00:54):

Are you more into the security side or the cars side?

BM (01:00):

Security. That's my background.

EW (01:01):

Okay.

BM (01:02):

The car stuff was new when I joined the company.

EW (01:05):

I believe you're familiar with the show, so you know lightning round is where we ask you short questions and want short answers. And if we're behaving ourselves, we won't ask "Why," and "How," and everything like that. Are you ready?

BM (01:15):

I am.

CW (01:16):

Which is better, CAN bus or canned beans?

BM (01:19):

CAN bus. Definitely.

EW (01:21):

What book would you like to write?

BM (01:24):

I would like to write a C++ book in Hebrew. There aren't any good ones, currently.

CW (01:31):

When do you think cars will be really self-driving?

BM (01:35):

I think, within the next 10 years, we'll see commercial cars, as in trucks and taxis, but personal cars I think are much further out for mostly business reasons.

EW (01:46):

Do you have a favorite car?

BM (01:48):

Not really. I actually don't own one, either.

CW (01:52):

Favorite fictional robot?

BM (01:54):

Baymax from Big Hero 6, cute and cuddly.

EW (01:59):

Do you have a tip everyone should know?

BM (02:02):

I think everyone who's doing some protocol research or working with embedded should learn MicroPython and CircuitPython to test protocols and use Compiler Explorer to test assumptions on what their code will actually end up being like.

EW (02:16):

Setting aside lightning round, those are really good tips. I hadn't really thought about using MicroPython as a protocol tester, but that would be perfect. I've been using a lot of Python recently with UARTs, and yeah, that would be really cool.

BM (02:35):

I find that CircuitPython and MicroPython are really good for quick rapid prototyping. And especially with protocol, especially if you have a one-off thing, it's, a lot of times, much easier than getting a board and bringing up whatever you need to work on it.

EW (02:48):

Yeah. And it would be trivial to program a SPI flash or something you have to do in manufacturing later, but you just put it together pretty quick into MicroPython or CircuitPython. Cool.

EW (03:01):

So security, I didn't ask you what your favorite security hack is, but is there something that people do often that just makes you wonder how we get anything done ever?

BM (03:15):

You mean in security specifically or - ?

EW (03:18):

Yeah, or generally, I mean, generally is good too.

BM (03:24):

I think the problem with security is that it's a never-ending process. And I actually had a discussion with some people in the office today about general IT security. And it's a question of threat models, and risk management, and how far do you really want to go?

BM (03:40):

Because you could end up spending years on optimizing some specific security part in any system, but the question is, is it really worth it? The joke, I guess, on vehicles, is if we go back to classic vehicles, they are more secure than today's vehicles, but would you want to own any one of those?

CW (03:57):

Right.

EW (03:58):

I mean, is that really true?

CW (03:59):

Well, it's pretty hard to hack a carburetor over Bluetooth.

BM (04:03):

Although it might get stolen for other reasons.

CW (04:05):

Yeah. Right.

EW (04:05):

Exactly. It might get stolen for other reasons. So what difference does it make?

BM (04:12):

I think there's a big expectation today that the car is more than just a transportation device. I think many people assume that the moment you enter your car, it extends to your phone, and you're able to do anything you would be able to do while walking just with your car while you're driving.

EW (04:32):

It's kind of like how our phones no longer are really just phones.

CW (04:38):

Or at all as far as that-

EW (04:39):

They're not even just communication devices.

CW (04:41):

I don't use mine as a phone very often.

EW (04:43):

Yes, they are so much more, and we do treat our cars -

CW (04:46):

As rolling computers.

EW (04:47):

- as rolling computers and an extension of our homes.

BM (04:51):

And I think that will be the case even more so for the years to come.

EW (04:57):

So it's the worst thing I could do as a hacker on a car that is not secure?

BM (05:08):

So I think many people would think about crashing it into a wall, remotely controlling it. Although that is possible, I don't think most hackers are that interested. I don't think there's that much money involved in crashing people into walls.

EW (05:24):

I mean, unless you're 13 and have been playing far too much of that driving game that's just mean, crashing into walls, I mean, I suppose assassination pays well, `but I wouldn't think that that would be a goal for most people. So yeah. How do you make money by hacking cars?

BM (05:46):

So I think there's multiple ways. I think what's more common today is hacking to open features on the car which you didn't pay for or do some modification, and there's markets for that.

EW (05:59):

Well, wait a minute. I want that. As a consumer, I remember we had a car that we couldn't do something in and Chris got a different module that made it go faster or something.

CW (06:11):

What? I don't remember that.

EW (06:12):

The TT.

CW (06:13):

I didn't do anything.

EW (06:14):

Are you denying it for legal reasons, or -

CW (06:16):

Yes.

EW (06:16):

Oh, okay.

BM (06:19):

I think there's a line between modifications which make sense and the ones which can be dangerous. And I also understand the fact that the manufacturers are interested to sell you a service, for example, and they want you to pay for it.

BM (06:32):

So hacking your way to get that service for free is counterproductive to what they're trying to do. So it makes sense that they'd like to try and protect from that.

CW (06:41):

Well, and it's the trend of everything becoming a subscription that, if you have a rolling computer and you can do general things with it, then you can enable and disable certain features. And suddenly it's not that it's not capable of doing anything.

CW (06:55):

It's that the company would like you to pay them extra to enable that thing. And so yes, a huge incentive to bypass that, right?

BM (07:04):

So that's true. But I also think that once they figure out the way to sell you the services that make sense to sell that way, we won't feel about it the same way. In a sense, people don't feel Spotify is a way to take more money on things. It's useful.

BM (07:19):

So the first, I guess, services will be weird and, "Why are we paying for heated seats when there's a heater in the car?"

CW (07:25):

Right.

BM (07:25):

But eventually they will find services which makes sense and maybe even follow you when you rent another car. So you could just have that service forever. And wherever you show up, if you get a car from a specific brand, you'll have that service.

CW (07:39):

Have a Ford subscription to having power door locks -

EW (07:42):

And it would have your favorite stations and the volume that you prefer -

CW (07:46):

No, that makes perfect sense.

EW (07:46):

- and how you want things.

BM (07:47):

Seat adjustments.

CW (07:48):

Yeah. That makes perfect sense.

BM (07:50):

So I think they're mostly trying to figure that out. I would say, though that what I'm afraid of, and I think will end up happening is as we have a consolidation of the vehicle space, and we're slowly going to cease consolidation of the main components in the vehicle space, you're going to see very similar effects that we saw in the mobile space.

BM (08:13):

When we had our old dumbphones, they were so different, one from another, and they also didn't have so many features. So it wasn't so common to hear about some advanced attack or ransomware.

BM (08:24):

But once we have hundreds of thousands or millions of cars running the same exact code, if you get the right bug in the right place, you can exploit this at scale and probably cause a mass ransomware attack where I think also you're not going to ask for money from the individual user. You're going to ask for money from -

CW (08:44):

Right.

BM (08:44):

- the corporation which built the car, which actually is able to pay.

CW (08:48):

Aren't we close to that already? I mean, I feel like Delphi supplies a huge portion of all the components, and Bosch, and Magna does a lot of the manufacturing. How close are we to just, the brand exists, but most of the components are from ... a small number of suppliers?

BM (09:08):

So there are tier 1s which are very large, but I guess you'll also see more consolidation in the sense that the OEMs, the manufacturers of the vehicles, want to do more stuff in-house.

CW (09:18):

Oh, okay.

BM (09:19):

Most of them want to try and compete with Tesla. And they understand that to be able to deploy features at such a rapid pace, you've got to do more stuff in-house. You can't just be an integrator anymore.

EW (09:30):

I was going say, monoculture is usually bad, but now we have the OEMs, like Ford and Tesla. And we have the people who create the parts, like Bosch and Tesla.

EW (09:45):

And so it isn't entirely a monoculture, except that there are pieces that most of them use the same part underneath. And so attacking that part has a wider benefit than just being able to unlock one person's car.

CW (10:05):

And I think what you're saying, Benny, is that the major car manufacturers are moving to design those parts themselves instead of pay Bosch.

BM (10:16):

So they might do that or they might do a long-term agreement with one of these tier 1s -

CW (10:20):

Okay.

BM (10:21):

- so they secure the supply chain going forward and don't start new projects one at a time. They might say, "We want this component for the next 10 years," and secure the development of that in a more coordinated fashion than they used to.

CW (10:36):

Is that dangerous too? Because now you've lined up a part that might have a vulnerability discovered in two years that you have 10 years worth, or - ?

BM (10:45):

This is an issue with cars in general. And the way this is being solved is by deploying over-the-air updates. Most of the manufacturers are going into this year with new models which, ... most of the components would probably support over-the-air updates at this point.

EW (11:02):

I want to get back to over-the-air updates because, oh, there are so many ways to do that wrong, but you said tier 1, you've said that twice now. And what does that mean?

BM (11:12):

So the automotive ecosystem is a bit different maybe than some consumer electronics. When you do consumer electronics, you might have a company in China producing for you, an ODM or something, but the car industry is much larger and the vehicle is much more complicated.

BM (11:29):

So you have different levels of suppliers, you start with the OEM, the manufacturer. They buy complete components from a tier 1. Tier 1 might develop one or more components.

BM (11:40):

Many of them are known for specific components that they are at top of the market at developing, but they will buy specific modules or software from tier 2s and tier 3s. And you have this whole supply chain, which is very complex.

BM (11:57):

When we find a bug in a system, we don't actually necessarily know outright that it was the supplier who introduced that bug.

BM (12:06):

You might find out that it was actually an SDK, which was bought from a third-party company, which was incorporated into a module, which the module was after incorporated into the larger ECU, the component.

EW (12:20):

We spoke recently to Laura Abbott from Oxide about the TrustZone, and Cortex-M, or Cortex as a whole, and how important it is to be able to be sure that you are running the code you intend to run.

EW (12:37):

As soon as you have over-the-air updates, that root of trust becomes kind of more of a tree of trust and goes all over the place as a developer, how do I deal with that sort of thing?

BM (12:53):

So before you even write one line of code, you really need to have a good concept of how your software is being verified, as in Secure Boot, and also how your updates are being done.

BM (13:04):

I wouldn't necessarily say that it becomes a tree of trust, because there are standard ways of doing this with cryptography which don't require creating very complex structures.

BM (13:16):

It doesn't make it foolproof, but if you follow best practices, you can set up an over-the-air update in a good way, which from my point of view is actually better than saying, "I won't update and therefore, I don't have the attack surface of over-the-air updates."

EW (13:32):

That's interesting because I do usually think of over-the-air updates as being more vulnerable, but that does allow for patches of vulnerabilities as time goes on.

BM (13:41):

So you mentioned before, cars, I think these days in the U.S. are on the road for about 12 years on average.

BM (13:48):

In that kind of timeframe, the chances of not having a major vulnerability when you have such diverse types of code, you have Linux, you have embedded software, you probably even have at this point a web browser, which, bugs are found in those on a regular basis.

BM (14:04):

If you don't have over-the-air updates, you don't have a chance to fix any of that. You have to wait for the person to show up at a dealership. And even then it really depends on the business model.

BM (14:14):

You might not have an update for your system if you haven't paid or depending on where you live. So over-the-air updates are the only way that I see to long-term secure these kinds of devices.

EW (14:26):

I was going to ask about if over-the-air truly meant over-the-air like Tesla does, or if it meant going into the dealer and having your ECM plugged into their computer. So you're saying truly, and is this Wi-Fi networking, or is this cell modem?

BM (14:46):

So in general, I think, most manufacturers are going for cell modem, but I have seen discussions of using the Wi-Fi with the consent of the owner on their personal Wi-Fi as they might have a much faster internet connection, or they might not have good coverage in their area. So I assume you're going to see both models, both cell and Wi-Fi.

EW (15:08):

Okay. So back to the earlier question of, "What can I do if I hack my car? I can get features. Okay. That's great." I can see how the car manufacturers don't want that.

EW (15:20):

I might be able to remote-control drive into a wall which, alright, there will be some people out there who enjoy that, but most people are more sensible that that's still wrong. What are people really doing? Are there more features that are being attacked beyond the monthly subscription models?

BM (15:49):

So I think the most visible attacks which we should care about today are car theft. There's a lot of ways to steal cars using hacks. It used to be people broke the window, took out the ignition wires, and ignited the vehicle. These days -

EW (16:05):

Hot-wiring.

BM (16:06):

Hot-wiring. Yes. But these days, many of the ways to steal cars are by figuring out how the keys work, by relay attacks, if you've heard of those, where you have a keyless entry system.

BM (16:20):

And you just use radio transmitters in order to extend the range of that device, and therefore you could be further away from your vehicle, and it shouldn't open. But because they use these devices in order to extend the range of the key, your car will open, and they can steal it.

CW (16:41):

And that's the normal key fob kind of mechanisms, right? Not necessarily the newer stuff that Apple and Tesla are doing with using your phone as a key, which is a whole different can of - ?

EW (16:55):

Beans?

CW (16:56):

Yeah.

BM (16:56):

So this would be the mid term. You have classic car keys, which basically work when you press the button.

CW (17:02):

Yeah.

BM (17:03):

But these keyless systems, the idea is you show up next to your car, they open.

CW (17:07):

Oh, okay.

BM (17:08):

And the next generation is what is being developed by Apple and Google using ultra-wideband, which is only available on some smartphones. And this is supposed to be much more secure, but there already is research showing that this can be circumvented with the right tooling.

BM (17:29):

So I'm not an expert on this topic one way or another, but it doesn't seem that this has been solved at this point.

EW (17:37):

Walking up to the car and having it open is very handy. I don't necessarily need the keys in my hand. I walk up, I just open the door, and it's all good. It's really bad when I switch cars and forget to lock the door.

EW (17:54):

But setting that aside, this seems like a battle between security and convenience, which is a constant battle. Where can the consumer or the developer make a difference in having that be truly secure?

BM (18:16):

I think it's a bit much to ask the consumer to do the security here, considering he doesn't have the knowledge. But I think this is going to be a long-term effort from the industry to try and find out ways to stop this distance-shortening attacks. ...

BM (18:34):

Because I don't think we're going to go back to not using keyless entry systems. It's kind of what people have gotten used to. So the only way we can go forward is by figuring out better ways of when these kind of attacks are being performed and preventing them.

EW (18:49):

Do you think in five years, somebody's going to go back to having a physical key that you put into a slot and call that more secure, because you can't spoof it?

BM (19:00):

You'll always see people who are trying to do their own small thing, but this won't be at scale. You might decide that you want to mod your car to do that, but I don't see any manufacturer producing new vehicles in major markets with this kind of technology.

CW (19:17):

And as I think about this, yes, there are vulnerabilities in the remote keyless entry, but as it stands now, is it a worse situation than older cars where you could walk up to them with a coat hanger or a flat thing, and stick it down the window, and pop the lock?

BM (19:37):

I don't have the numbers.

CW (19:38):

Yeah.

BM (19:38):

But I guess you deal with what you have now. And right now police are dealing with these kind of attacks mostly.

CW (19:44):

Okay.

EW (19:45):

When we have cell modems in the car, it's much easier to track them and locate them. And so doesn't that decrease stuff already without worrying about the keyless entry parts?

BM (19:58):

This is a really good question of when you figure out that the car was stolen.

EW (20:04):

Yes.

BM (20:04):

And does the car have the feature of reporting either from your app or from something to report to the police that this has happened?

CW (20:13):

Yeah. It doesn't take too long, if it's a sophisticated thief, for them to get it somewhere where they can -

BM (20:19):

Disable, yeah.

CW (20:20):

Do what they want to it, yeah.

EW (20:21):

Chris brought up the ways you can open a car if you lock yourself out, as well as if you're stealing it. And there have been incidences where you do lose your keys, or somehow lock them in the car, or have other instances where we truly do want to defeat the security.

EW (20:44):

And so we have tow trucks and locksmiths who can get into our vehicles and houses on purpose. With the new keyless entries, if my phone dies, am I not going to be able to use my car? Or if I somehow lock my phone in the car and my kid, is it going to be catastrophic?

EW (21:09):

So I assume we're not going to a point where you only have one set of keys in your phone and you don't have any keys anymore. But I guess just like today, if you could stick all your keys in a car by mistake, it might happen.

BM (21:22):

But I'm guessing you're always going to want to have that backup key, or maybe share your digital key with your partner. And that way you'll be able to get into the car one way or another. I recall that there is a way even if the phone is off possibly to open, but I'm not sure about that.

CW (21:40):

Yeah, ... I think some of them have NFC too, and you can always call the manufacturer. This has happened where -

EW (21:48):

You call the manufacturer and they unlock it for you?

CW (21:50):

Yes.

EW (21:50):

See, that's actually reassuring.

CW (21:51):

And then you social engineer them to unlock somebody else's car.

EW (21:54):

Exactly. Security comes in many forms.

BM (21:58):

I guess also some of that could be done through an online portal.

CW (22:01):

Yeah.

BM (22:02):

So as long as you have a way to log in on a PC, you might be able to do some unlocking from there.

EW (22:08):

So what are the surfaces of these attacks? I mean, what are you looking at when you're looking at the security of vehicles?

BM (22:16):

So, one thing that I've learned from the last couple of years is that looking just at the single component in the car is probably not the right way to be looking at the problem. We do work on individual ECUs.

BM (22:29):

But when we look at the functional level, when we want to have a look at a specific function, for example, we said remote locking/unlocking, if we only look at the software running on the ECU which actually unlocks the door, we're going to miss a lot of what's going on. And I mean also in the vehicle, but also on the servers.

BM (22:49):

So what we like to do, if we can, is actually research this function end-to-end, from the click that you have on your cell phone, the app itself, through the servers in the backend, which are the OEM servers, or maybe a third party server, look at what's going on there, look at what is going back out to the car, and look at the software in the car.

BM (23:12):

And when you understand this at the high level, then you can understand where it's going to go wrong.

EW (23:17):

Sure. Because it's always the cracks. It's always the places where things meet that things go wrong, because people don't understand the other piece of it, that the embedded software engineers don't understand the security on the server.

EW (23:33):

And so they leave in a back door, or they just don't check the right thing, or they allow you to somehow bypass it. And even the mobile developers going up to the server, there's a lack of communication. And those cracks are where the bugs will be found.

BM (23:52):

I agree, but it's even more than that. It's the mental model, which is different between developers. And it's also the mental model of specifically, even if you're sitting in the same room discussing, because they come from different backgrounds, and they see the function through different perspective, they might not think about the same issues.

BM (24:11):

Everyone's thinking, "Did I do my job well, compared to what I'm expected to do on my side?" And they're not asking, "Is the overall project in the right direction from a security perspective?" Again, there are cases which are different, but you get the point.

CW (24:29):

And what's the way to overcome that? That sounds like an organizational thing rather than a technology thing.

BM (24:35):

So I think this requires looking at security, through different perspectives. You have to look at the base software running on the ECU, whatever it is, if it's an operating system, or if you have a hypervisor, or Linux.

BM (24:47):

But you also have to look at things on the functional level. You have to do cross examination of these functions, wherever they're going. And you've got to figure out how you secure them, both on the individual component you're on right now, but also end-to-end.

BM (25:02):

So for example, if we're talking about cryptography, you have to have that cryptography work from the first point where the command for this remote unlock enters the system up until the vehicle. And you can't have it split up in a way where each each node is in charge only of itself.

EW (25:22):

You need a systems perspective.

BM (25:24):

Exactly.

EW (25:25):

I mean, somebody also needs to be focused on getting the pieces done, but that person probably cannot see the whole system. They can see their part of it, which is great, because they need to do their part of it. But the systems perspective, ... I mean, there are lots of system problems, cost integration, schedules, and security.

BM (25:53):

Agreed. I think this is also something where, we spoke about over-the-air updates before, but this changes everything. Because what you said now, it has been traditionally focused on hitting the startup production date for a vehicle. That's the day where everything has to be aligned. Everything has to be perfect.

BM (26:10):

And if you miss that date, you won't enter the vehicle at all. But when you have these updates, we can actually release content features, but also security upgrades later on. We don't have to take the decision that we're not going to invest in something. We can deprioritize it and get to it later over the life cycle of the vehicle.

EW (26:31):

Oh, yes. This is why any time you open something electronic, it has to do a firmware update first. I'm looking forward to unboxing my car and having three days worth of updates.

CW (26:45):

There's a risk, too, that people won't update, right?

EW (26:49):

Oh, yeah. I often click don't update. I don't want you to change everything.

BM (26:54):

Yeah. I think by law, they're required in many places to ask you if you want to upgrade, but if they entice you with new features, then you're going to do the update.

EW (27:02):

People who are, let's call them hackers, people who are intentionally trying to break the security for their own benefits, they have infinite time. They have 12 years to break the security, or however long you mentioned, or however long it is.

EW (27:21):

But as a developer, I have to hit the ship date. And as a company, investing more resources in a car that is 12 years old is just not feasible. How do you balance all of that, that the hackers have infinite time, but the developers don't?

BM (27:46):

So I think this is part of the understanding that the engineering of the vehicle doesn't end when it hits the street. And in a sense, you need to have a long-term plan of, if you really expect the vehicle to be supported for the next 12 years, you need to understand what that means.

BM (28:03):

You say, "You get five years of feature updates, and then you get another seven years of security updates. This is what we're going to commit to. This is what we can do." You've got to think about, for example, updating the Linux kernel at some point, because at some point it won't be maintained anymore.

BM (28:20):

But I don't think it's necessarily such a big deal if you, in advance, commit to producing the same vehicle or the same platform for many years, because by the time you're going to end producing that specific vehicle, it's going to be a couple years into its existence.

BM (28:38):

So it's not just that you make one batch and you're done, you're going to be producing a very similar vehicle for multiple years. So this is a worthwhile investment.

EW (28:46):

I mean, you just made the problem a million times worse by having multiple versions and being able to have a code base that can support tweaks in the hardware and firmware update for the past version of the car. I mean, this is a versioning nightmare.

BM (29:08):

So I don't think necessarily all components are going to be treated the same, but if you think about it, the same thing is true for our phones. And we're seeing gradually that our phones are getting more updates, the latest Google phones get more updates than the previous ones.

BM (29:24):

And they're doing that by figuring out which components need to be updated in which way and how they can make it easier. So I think the same thing is going to happen in vehicles. Much of the software will become standard.

BM (29:38):

And ... you don't really care what is running under the hood if it's not a user-facing feature. So as long as that code is not, business-wise, worth doing yourself, you're going to see much more standardized code.

BM (29:53):

It's going to be maintained whether by open source, or by companies, and it's going to be worthwhile, the investment of keeping it up to date, both on a functional level, but also on a security level.

CW (30:05):

What's changing in the architecture of vehicle, for lack of a better word, networks, or electronics? In the old days, say before all of -

EW (30:18):

Now? Oh.

CW (30:18):

- Bluetooth and all this stuff, it was lots of very not sophisticated processors, maybe connected via I think, CAN bus, and stuff in large complicated wiring harnesses. Is that persisting and new technology is being built on top of that, or is the entire infrastructure being replaced by things that are more modern?

BM (30:43):

So there's different takes on this and different OEMs are doing this differently. Can bus is not really going anywhere. There's actually a new standard in process now called CAN XL, which is going to be even faster. And I think you can even put full IP packets on top of it.

BM (31:00):

So that will be very different, but you also have ethernet now in cars. The physical medium is a bit different, but the actual Ethernet stack is very similar. And you have also Ethernet in the sense that it's going to be a multidrop Ethernet, which is similar to CAN bus. It's going to be a bus version of Ethernet.

BM (31:23):

Some of the OEMs are going for that. That's more of the internal network. When we look outside, we have 5G. We have vehicle communication between vehicles, vehicle-to-vehicle or vehicle-to-everything.

BM (31:36):

You also have the charging port on the electric vehicle, which is also using power line communications and vehicle-to-grid communication, which should enable plug-in charge, in the sense you can come up to the charging station, plug in the charging gun, and it will just start charging automatically since you have already some agreement with a provider, similar to your phone roaming when you go between states or countries.

CW (32:04):

Okay. So yes, that's advancing.

EW (32:07):

What is the virus in my car going to look like?

BM (32:10):

That's a great question. I personally feel that the first generation is just goin to be mostly bash scripts on Linux, because much of the way things are set up today, you don't really need to create a sophisticated C-based binary.

BM (32:29):

Maybe if you want to go to the more embedded side, you do that. But I think many of the interesting things are available on high performance ECUs. This is also maybe referring to the previous question.

BM (32:39):

So things that are changing, ... you're seeing more and more high performance chips, which are running Linux or Linux-like operating systems.

BM (32:48):

You have containers in cars. You have hypervisors in cars. You have virtual machines in cars, and at the same time, you still have the same embedded processes you've always had in order to ensure safety and real-time functionality. So that's a bit of a mixture of things. You also have new concepts of how a network is built.

BM (33:09):

It used to be more of a flat network. These days you have a gateway, which is similar to what you would think of as a router or a firewall between different buses, and you have more security boundaries than you used to.

EW (33:26):

Physical access is still going to defeat most of the security like it does in many embedded systems, or is physical access no longer as critical as wireless access?

BM (33:39):

So the question is at what level of physical access. If you're talking about just connecting to the OBD port in many cases today, that's not going to give you much, because it's locked behind some sort of security mechanism.

BM (33:53):

If you start opening up ECUs, you'll get more access, and some ECUs are more vulnerable. And others will require you to do glitching attacks, which are much more complicated, in order to get anything from them.

EW (34:08):

So there are the attacks where I'm plugged into the OBD. There are the attacks where I open the ECU, which is a big deal. There are attacks from my computer remotely. And then there are the class of attacks that are, you are near the car, possibly touching the car, but you are not in the car.

EW (34:29):

You mentioned the key security, but there are other ways to interfere with the car being near it, aren't there?

BM (34:40):

I guess it depends on the design. I guess if the Bluetooth or Wi-Fi on for some reason that you might be able to interact with those. But I'm guessing if nobody is actually in the car, most of the time, this will be off to save energy.

BM (34:55):

I guess you could also physically try to remove an exterior piece of the vehicle trying to get to an ECU which is near the boundary of the vehicle. That might be a way in.

EW (35:08):

If I'm a developer, and my company has a systems engineer who looks at security, but perhaps they have a lot to do. Can I just use the MISRA standard for compiling, and be on my way, and say, "This is secure enough," or do I have to think harder about it?

BM (35:31):

So MISRA is standard for automotive, for coding, specifically C and C++. The C standard is quite up-to-date. Last time I checked, the C++ one is not, and is actively being worked on. But this only covers coding. So this is an important step, but this doesn't cover everything we need from a security perspective from a developer.

BM (35:53):

We found a lot of bugs in code, which is 100% MISRA-compliant, but it doesn't take into account the systems perspective. So you can say the coder is only in charge of his code, but at least the way I see it is, you've got to understand the system, and you've got to protect your code from the system itself.

EW (36:11):

Are there guidelines to doing that?

BM (36:15):

So I'm not aware of anything formal, but a lot of this is about working correctly with protocols. I think most of the larger companies in this segment have their own standards and best practices. But a lot of this is just learning over time and improving as you go.

BM (36:36):

I think that would be the main thing, but you also need to have the right kind of personnel, which are proficient in each one of these cases, both at the system level, electrical engineering, microcontrollers, but also at the higher level with programming.

EW (36:53):

How can someone learn more about, or get into automotive security?

BM (36:58):

So I have some recommendations here. So first of all, there is an organization called ASRG, the automotive security research group, non-profit. They have a website. They have webinars up and a Slack channel. There's a lot of discussion going over there.

BM (37:14):

They're slowly building up a lot of open access information. If you want to just learn about what's actually going on with security incidents, so Upstream Security has very good reports on a yearly basis. And they also have some incident tracking on their website.

BM (37:32):

There is some specific research from this year, which I think is the best way to end-to-end learn about ECU reversing by Willem Melching. This was I believe, in a power steering system, which he has a four-part blog on, which I suggest people read if they want to understand how to apply their skills.

BM (37:55):

And then you also have some open source tooling and testbeds you can look at. For example, you have the ICSim simulator, which is a simulator for instrument cluster. So you can play around with CAN bus a bit. You don't need any hardware.

BM (38:13):

If you do have some time, and you want to spend some money, you can actually build yourself a car in a box. Specifically, Ian Tabor, mintynet, has a car in a box called value-pasta-auto. It's based on a design by Toyota, which he did a cost down for.

BM (38:32):

We actually built one over the last couple of weeks in the office, and we've playing with it. So I think that one is a great one if you have some money to spend and want to learn how to play with these protocols, yeah, I think those are very good resources to start with.

EW (38:47):

I'll make sure those go in the show notes. What you're saying, I mean, every time we talk to a security expert, I just get so discouraged that there's not a way I can make this work.

EW (39:02):

It's not just that it's a team sport, and I have to have a lot of people contributing and someone checking over this. It's just, it doesn't matter what I do. Someone will always hack it. Do you have any advice for getting over that sort of cynicism?

BM (39:21):

So I don't think it's necessarily the thought of, "Will it be hacked?" The question is, "What will be the impact if it's hacked, and also how easy is it to do?" And there are standard methodologies in order to quantify this, I guess, in money, which is usually the main part.

BM (39:39):

So if you get it to the point where the impact to your business is low enough that at this point, it's not worth spending more money, you're at a good point, probably. In automotive, you have these kind of processes set up in order to do this.

BM (39:54):

So you make sure that you're at the point where you want to be from the business perspective. And then I will say, good engineering goes a lot, a long way. If you cannot explain to yourself why your system operates on a functional level, then there probably will be more security issues.

BM (40:07):

If you understand very well why you're doing your software update in a reliable fashion, reliability has a good connection overall to security. In a sense, you're doing a checksum on an update. You might be doing this for reliability, but also, it's not fool-proof with security. But this is also useful for security.

EW (40:27):

And then you go from a checksum to a signature, and now you have better security and reliability. And it gives you a reason beyond, "I have to do it," to, "I want to do it."

BM (40:43):

Yes, exactly.

EW (40:45):

I have some listener questions since we're getting into the specifics. Many people asked about John Deere and bricking tractors. Should there be kill switches? Is that a good idea or a bad idea?

BM (41:01):

So I'm going to speak on a technical level. I think more level is a bit not relevant for this discussion, at least that we're having.

EW (41:07):

Sure.

BM (41:08):

I really think this can backfire. I've seen these kind of mechanisms before in vehicles, which on paper had very good cryptography associated with them and so on. But they can be circumvented and in very weird ways. And usually these mechanisms are extremely strong. As you said, you can brick a tractor.

BM (41:28):

So I wouldn't put this in a vehicle. I think it's a bad idea. I think it would be better at least on a local level, maybe to leave an interface for tracking for police with the consent of the owner of the vehicle. But I wouldn't go as far as to brick, remotely, vehicles. I think that's a bit stepping over the boundaries.

EW (41:49):

Still picking on John Deere, because they've been in the news, security, we've talked about how manufacturers would like a subscribe model like John Deere, and that goes against the right-to-repair movement.

CW (42:09):

Or ownership.

EW (42:11):

Or true ownership. And is there a way to balance the right to repair with the man getting his money?

CW (42:21):

Well, before you answer, I would rephrase that as, there's a balance, I think, between allowing the manufacturer to make changes to something you've already bought and ownership, right? Because what we're talking about is over-the-air updates, which will be necessary to keep things secure.

CW (42:38):

But they also allow the manufacturer to modify your vehicle in small and large ways. Then do you truly own it at that point? Anyway, sorry.

BM (42:49):

Sure. I think as you mentioned, the major issue here is more, IP, and, safety and the fact that you want to update the vehicle, not really security. But I would say that, I think there is an option to balance this. You've seen, over the last year, Framework do a different type of product for laptops.

BM (43:11):

I think if a manufacturer was very serious about it, they could figure out the business model which would support this. I don't think that you'll see any of the large OEMs going for this though, because there's not enough pressure yet. At some point, there will be enough pressure.

BM (43:27):

And then some small startup will come and decide that they're producing the new open car. It will be very hard to do, but they will try to do it. And it will take a couple years, and you'll see some competing options.

BM (43:39):

I just don't think this is what most people are interested in right now. That might change in the future. But I would be very happy to see that.

EW (43:48):

An open car, one that you could read all of its code?

BM (43:53):

So I don't know if all of its code, but I'd say all the relevant code for maintaining it and making sure that it works for a long time. And I also don't think this is bad for security. In a sense, a good security model shouldn't assume that the attacker doesn't know what code is running on the device.

EW (44:14):

What are some other good security model tips? I mean, A, you should assume a white box hacker, that they can see inside the code, and they know what's going on. What are some other things that are good to remember about security models?

BM (44:36):

So never trust the hardware. And you must know that from the functional side, but this is doubly true for security. Never trust the user manuals. Never trust anything which is on a specification. For some reason, things are always very nice on paper, but then they meet reality.

BM (44:57):

Reevaluate at regular intervals what is possible in the sense of what kind of capabilities attackers have? Because, for example, if we speak about passwords, over time, newer algorithms are out. Stronger computers are out.

BM (45:15):

The security advice for passwords a couple years ago is not the same as it is today. Hopefully we will be done with passwords at some point, but the point is similar for many other mechanisms.

EW (45:26):

That goes back to having to maintain these things for a decade or more. I mean, realistically two decades, by the time you start making the product, and then finish making the product, and wait 12 years after that.

EW (45:41):

How do you go to the new security models, the new security features on a car that is older than the one you have because you update it to have faster for the new security, and now you have to maintain both? Sorry, that's probably not an answerable question.

BM (46:03):

I think that's the real challenge here. And I think this is what we're going to have to figure out in the coming years of how we do this better. I guess you could design the vehicle in advance with extra performance.

BM (46:14):

You could think about, for example, if we look at quantum security, which I am not an expert at, but everyone is convinced that we should already be thinking about it today before it becomes a problem.

BM (46:26):

So I would expect that much of the question is, "What threats do we think will be relevant over the lifetime of the vehicle, and what can we do today at a reasonable point to prepare for those, so when we need to address those, we are able to?"

EW (46:42):

Reasonable is a tough question there, because it's a judgment call of, "We need to be better at security than all of our similarly priced competitors so that people steal those cars and not ours."

CW (46:56):

Well, and who's going to steal a 10-year-old car. But I think, just from a higher level point of view, security of cars versus security of other things, you have to balance that too, right?

CW (47:09):

You can't say, "Well, this needs to be as secure as an airliner that's carrying 350 people." Evaluating what the threats are that a vehicle can pose, or collection of vehicles I think, is important too.

EW (47:25):

Yeah, collection's important.

BM (47:27):

I agree. I'd also say that it's still easier to get vehicle parts than it probably is to get plane parts.

CW (47:32):

Yes. Yes. Definitely.

BM (47:33):

So that's also a large part of the story.

EW (47:37):

Going back to listener questions, ExplodingLemor asks, "How should Honda approach fixing their replay-vulnerable key fobs?"

BM (47:46):

So I'm assuming this is referring to the keyless entry systems we mentioned earlier. So I mentioned before, the industry is migrating to a newer technology called ultra-wideband. It's supposed to be more secure.

BM (47:59):

I'm not an expert on this, but there was an attack released called Ghost Peak, which has shown that there are ways to shorten the range here. The idea here, if I understood correctly, is that you measure the speed of the communication, in a sense how fast you get a response. You can get a feeling where the person is in relation to the car.

BM (48:22):

And it's supposed to be harder to spoof this compared to Bluetooth Low Energy and other RF interfaces. So I hope that over time, they'll figure this out. But I guess this is a question also for physicists and not just for security researchers.

EW (48:42):

So I'm going to restate that question a little bit. Now that Honda has a replay-vulnerable key fob, and it's wide in the marketplace, what should their approach be?

EW (48:59):

I mean, one approach is to not do anything and get sued, but what else can they do? Is it a matter of replacing every single entry with this ultra-wideband, more secure? That's going to be a very expensive retrofit.

BM (49:19):

Yeah, that doesn't sound very feasible.

EW (49:21):

No.

BM (49:22):

I'm not sure, actually. I think this really depends on the technical details here and to understand if there is something that can be done at the code level. And maybe it's possible to, maybe not avoid this, but make it harder. But it really would require some more details here to understand.

EW (49:38):

Andrie said that NXP has some CAN transceiver chips that detect spoofing and will take ECUs off the bus if they detect it. Have you seen or worked with chips like this?

BM (49:56):

So I can't really comment on what we are doing in this space, but on a personal level, I think hardware is definitely part of the solution here. There's many things that are expensive performance-wise or impossible to do in software.

BM (50:11):

So there's a bunch of vendors in this space which are trying to find hardware solutions or combined hardware and software solutions. And I think these are definitely necessary because CAN bus in itself is not very secure.

CW (50:24):

Yeah. I mean, you can put a couple wires onto your OBD port and see a whole bunch of stuff that's flying by on the CAN bus. I had to do that for work. And it was actually very eye-opening to me.

CW (50:38):

Because I hadn't really done anything like that before, all this telemetry coming across there, and I assume, other things I could inject in there to do things I probably don't want to do. But yeah, it's kind of remarkable.

EW (50:50):

But the OBD port is usually firewalled, so you can't inject that much.

BM (50:55):

So we mentioned that this depends on the generation of vehicle.

EW (50:58):

Right.

BM (50:58):

Newer vehicles tend to have a stronger firewall there, but then again, if you're going to physically access and connect yourself to a specific ECU or behind that gateway, then you will be able to do more.

EW (51:11):

Is there anything you are looking forward to in the future for how security will change?

BM (51:18):

So things are getting better overall. When I started it was much more common to find buffer overflows, simple ones and all these kind of very simple security issues. Things are becoming more complex. I think there is, this started a long time ago, but a long-term shift to open source reusing code.

BM (51:40):

We said over-the-air updates, consolidation, less reinventing of the wheel. Architecture-wise, we're seeing more Arm cores, which are a bit easier to use with tooling. So it used to be much more PowerPC, V850s, TriCore, which are very automotive-focused cores. But you're seeing more standardized Arm.

BM (52:06):

I'd say that we're getting better tools. There are much more simulation tools than there used to be. And there also are better ideas of how to do security. So slowly, a lot of ideas also are migrating from the IT world into the vehicle. And these were not possible previously.

EW (52:25):

Because the vehicles weren't computers.

BM (52:28):

I think they were computers, but they were not just as strong as they are today, not performant enough.

CW (52:34):

They were '80s computers.

EW (52:37):

And now they're '90s computers.

CW (52:39):

Well, some of them aren't.

EW (52:40):

That's true.

CW (52:40):

Some of them are pretty advanced.

EW (52:43):

You've mentioned security overflows and definitely input, and the areas between pieces of the system as good attack surfaces. Are there other good attack surfaces that I should be aware of as a developer or even be aware of as a security researcher?

BM (53:06):

So I think besides what we spoke about, the internal network, I think ... two of places which should be in focus are both the connectivity to telematics units, which has quite a large attack surface from the network coming in from the cellular network and the infotainment, because you have all of these apps.

BM (53:28):

And of course this depends on the OEM. But there's a lot of different types of apps running there. Maybe there's even a store. Maybe you can even download an app developed by a third party.

BM (53:39):

So there's quite a bit of attack surface over there, which is really worth figuring out how you secure it. Do you sandbox it? Do you review the implementation of the modem code running on your device?

EW (53:55):

I was going to say, who cares about the infotainment system. But if you made the screen flash and the radio be super loud, I would probably have a very difficult time driving. Especially if you made the screen flash.

BM (54:08):

I agree, but you'd also have a foot in the door in the vehicle. And you would be able to communicate and possibly attack more critical functions in the system.

EW (54:16):

Shouldn't there be a hard firewall between infotainment and functionality of the vehicle as a vehicle?

CW (54:25):

Yeah. The infotainment should be taped to the dash. And that should be the interface, ... tape.

EW (54:29):

I'm just going to tape my iPhone to the dash.

BM (54:32):

So there usually is at this point, a serious firewall, but you want to know, for example, if there's a seatbelt which isn't fastened or a door is open.

BM (54:41):

And many times the ECU, which will be in charge of displaying that, is no longer the dashboard, but rather the infotainment. So that information has to go there. Therefore there is some information being passed backwards and forwards.

CW (54:54):

And worse, to apps, right? Now you have a phone app -

EW (54:57):

Yeah.

CW (54:57):

- for your car, and it'll tell you if your door is locked. And you can lock it remotely or whatever. ... The scariest attack surface to me is now exposing internal functions of the car to the internet.

EW (55:12):

Well, there are more internal functions. I've wondered why we can't open and close our garage door from our phone app, because that's all part of a system to me in my head. Garage and car are the same.

CW (55:24):

You can.

EW (55:26):

Should you be able to open a garage door from somebody's phone? That seems like a -

CW (55:31):

Through their car.

EW (55:32):

Through their car, that seems like a huge security risk to your house.

BM (55:37):

I figure that it makes more sense to do that through a specific service which is supposed to tie all these things in together. And maybe then it's a bit better than just putting all your eggs in one basket, but I've got to think about it.

EW (55:50):

Everything needs the security piece, and it's easy to circumvent it when you just aren't thinking it. You're just trying to make things convenient.

CW (55:58):

Well, and I think it's hard for us as developers. And you've asked this question multiple times in different ways, and I think Benny has tried to explain that the fact of security is, it's not you've done security, and you go on vacation.

CW (56:13):

Security is an ongoing process where somebody tries to break what you've done, and you figure out how to correct for that. And then somebody comes up with a new way, and you correct for that. And that just goes on forever.

CW (56:25):

And as long as people want things, they will want to break into things. And that's just an ongoing battle. And I think that's the truth of the matter.

BM (56:35):

I'd ask you the same thing about other engineering processes. When do you know that your system is performant enough? When should you stop optimizing?

EW (56:43):

Well, when I made children's toys, we shipped a masked ROM and you couldn't ever change it again. That was the best. You never had to do maintenance when you have large, consumer, unupdatable systems. Then it's done. But for more updatable things, it often feels never quite done. There's always at least one bug.

CW (57:09):

Hey, half the time I've been on products, we've shipped them before the firmware was done. They're on the shelf. It has no firmware.

EW (57:14):

And then you have to do firmware update.

CW (57:18):

Yeah. No, you're right. That sort of model of continuous improvement has bled into other things. And I think that's just the way life is going to be for developers.

EW (57:29):

Benny, do you have any thoughts you'd like to leave us with?

BM (57:33):

So, if anyone's interested to learn more about CAN bus and automotive security, we have actually a session coming up as part of the escar conference, which is an automotive security conference. I think in general, it's very interesting to hear about some of the research going on there. I think that would be it.

EW (57:57):

Alright. I'll make sure that that is in the show notes. It has been wonderful to talk to you. Our guest has been Benny Meisels, Lead Solution Architect at CYMOTIVE Technologies.

CW (58:11):

Thanks, Benny.

BM (58:12):

Thank you for having me.

EW (58:14):

Thank you to Christopher for producing and co-hosting. And thank you for listening. You can always contact us at show@embedded.fm, or hit the contact link on embedded.fm, which is where our show notes will be as well as a transcript.

EW (58:28):

And now a quote to leave you with, from Mario Andretti. "If everything seems under control, you're just not going fast enough."