Transcript for Embedded 352: Baby's First Hydrofluoric Acid with John McMaster, Christopher White, and Elecia White.

Welcome to Embedded. I am Elecia White, alongside Christopher White. Let's talk about the inside of chips and what they look like with John McMaster.

CW (00:00:17):

Hi John. Thanks for being on the show.

JM (00:00:19):

Hi there. Thank you for inviting me.

EW (00:00:21):

Could you tell us about yourself as if we met at the Hardwear.io conference, When it was in person?

JM (00:00:29):

Sure. I would say in the community, say the embedded community in Twitter. I am mostly known for my work on computer chips. What I like to do is, I open them up, that is I decap them, and I look to try to figure out how do the circuits inside those chips work? I take microscope images, I analyze some firmware maybe I find inside them. And then I post that on Twitter, and various publications online to share with people what I've found inside these chips.

EW (00:00:58):

And I want to talk to you about the process of that and why the chips look the way they do and all sorts of things. But before we do that, we want to do lightning round, where we ask you short questions, and we want short answers. And if we're behaving ourselves, we won't ask how and why. And could you give us all the details? Are you ready?

JM (00:01:18):

I am ready. Give it to me.

CW (00:01:20):

Favorite chemical.

JM (00:01:21):

Oh, geez. Hydrofluoric acid.

CW (00:01:27):

Least favorite chemical.

JM (00:01:29):

Least favorite chemical?

JM (00:01:31):

I'm going to go with Cinnamaldehyde.

CW (00:01:35):

All right. I think I need to know what that one does.

EW (00:01:37):

Mecuryish?

JM (00:01:38):

No. It's actually very random. It's pure cinnamon extract. And the reason why it's my least favorite, is because a bottle of it broke in my lab and everything smells like cinnamon now. That was years ago, and I don't think it's going away by now.

EW (00:01:55):

Okay. Okay. We have to behave. Does a civilian owned tank require a driver's license?

JM (00:02:06):

Oh, dear. Someone did some digging. Oh, that is a whole can of worms. I don't know if that many people know about that. Short answer, no.

CW (00:02:19):

McMaster-Carr or Digi-Key?

JM (00:02:21):

McMaster-Carr of course.

EW (00:02:24):

Favorite processor?

JM (00:02:27):

Favorite processor? I guess that I would go with 68K.

EW (00:02:33):

Is that also the prettiest processor or is there a prettier one?

JM (00:02:38):

It's the prettiest because I did an involved project with it, and I have fond memories of it.

CW (00:02:43):

Complete one project or start a dozen?

JM (00:02:46):

Complete one project. That is the new marching order.

EW (00:02:50):

Do you have a tip everyone should know?

JM (00:02:53):

Take things slow, always get a perspective when you're having a difficult time.

EW (00:02:57):

Okay. According to your LinkedIn bio, you have a semiconductor failure analysis lab in your garage, including a high power optical microscope, lapping machine and scanning electron microscope. So are you building Frankenstein? What are you doing here?

JM (00:03:19):

Funny you mentioned that, because my original career choice was actually genetic engineering. But I looked into that and it was going to be a lot of schooling, 10 years maybe. And I went with computer science instead because it was a little bit more approachable, but also something that I deeply enjoyed with robots and whatnot. Not so much a Frankenstein machine in a biological sense, but sure. AI was thinking and playing with the circuits. So it's a Frankenstein machine of sorts.

EW (00:03:50):

So you got a CS degree, but you do mostly hardware now. Right?

JM (00:03:54):

I would say that, one of the things that's been really tough as an identity crisis in Silicon Valley is, I do have a computer science and computer engineering degree, but if I say that I do computer science, people think of me as a big data engineer. And that's really not my skillset. It's more about, real time embedded operating systems, maybe Verilog, I2C drivers, that sort of area. So I don't know if you would necessarily call that a computer science field or not, it's in between that and electrical engineering in the embedded space. And depending on who you talk to, I would say you get different reactions about that.

EW (00:04:37):

Well, I think for me it would be, yeah. That's what I do too, but I don't have a scanning electron microscope.

CW (00:04:44):

Well, we could.

EW (00:04:45):

We do have-

CW (00:04:46):

There's nothing stopping us.

EW (00:04:47):

We do have a good optical microscope, although I don't know that I'd call it high power.

CW (00:04:52):

I'd say, okay optic microscope.

EW (00:04:53):

And I honestly don't even know what a lapping machine is.

CW (00:04:56):

It's a dog.

JM (00:04:59):

Yeah. The dishwasher. Well, okay. We could go through those. So lapping machine, that is one method to take apart a computer chip. You may be aware that, in the old days when they were making chips, that they would maybe just sputter some metals onto a chip and if it looked about right, maybe they'd add some off and call it done. But as the number of layers increased on a chip, they found that if they didn't smooth out the layers between them using a process now that's called chemical mechanical polishing or CMP. That plan arises the different layers of a computer chip. So when they add the next layer, it goes on straight and doesn't get the crooked artifacts from the layer below. From the failure analysis perspective, which you could vaguely call the work that I do, I basically do the same thing, but in reverse. I would like to take one of these smooth layers on a chip and I will polish them off. And that's an alternative to using something like hydrofluoric acid. And so, yeah. Different ways to basically get to the layers of a computer chip.

EW (00:06:02):

So basically a very large sanding thing? Very small sanding thing. Very fine sanding thing?

CW (00:06:07):

That's precise.

EW (00:06:07):

Precise.

JM (00:06:08):

Precise, yes. Very precise sanding machine. Yes.

EW (00:06:11):

But this failure analysis is different than writing I2C drivers. Is it one of your hobby and one of your career, or are they overlapped?

JM (00:06:22):

If I had to say, I would say that embedded development, so you know, let's call it writing the I2C drivers. That's more my career and this chip stuff is a little bit closer to my hobby. And there's a whole long thing about why that's the case. But the short answer, I would say now, as I'm a consultant, maybe two thirds of it is still embedded development and maybe one third of it is chip level stuff. Just low level analysis.

EW (00:06:51):

So how did you get into the chip level stuff?

JM (00:06:54):

Oh yes. What happened was, when I was going to college, I started learning to analyze firmware. So, maybe get a binary on a Windows computer, wanted to learn a little bit more about it. And I learned a little bit of how to go to an assembly language and whatnot. But I also had this interest in robotics. And I started learning a little bit about the processor on these embedded systems and I wanted to do that same analysis, but for those chips in that system. And I quickly learned that, unlike a desktop systems where you just had all the code more or less, these embedded processors had protections and you couldn't actually get the firmware to look at. And that lead me down this route of learning, okay. I know that you can't get the firmware out, according to all the official sources. But how can you? It has to be on there somewhere.

JM (00:07:50):

They talked about transistors in classes, but I didn't really know what a transistor was. How do we get this code out. Just trying to really, truly understand how do these embedded systems work and how can I learn more about them. That was the genesis of it all.

CW (00:08:04):

When I was at a company in around 2004, 2005, we had a chip that had some cryptographic thing in it. And as we were doing, okay, how could people reverse engineer this tech to get our key or whatever? One of the people brought up, "Well, people could decap this chip and look at it and probably read out whatever's in the EEPROM." I don't remember the exact mechanism for storage, but it was protected. And back then it was, "Well, yeah. But nobody's going to do that, it would cost a million dollars." Was that wrong in 2004 or has stuff becomes so much easier in the last 15 years?

JM (00:08:44):

It really depends on a lot of factors. Maybe I could give you some examples. Even in 2004, if you're talking about a security processor, those tend to use older, less expensive processors. Just because I don't know what the state of the art was in 2004, let's just say 65 nanometer. I don't know if that's completely off or whatnot.

CW (00:09:04):

Sounds in the ballpark.

JM (00:09:06):

You might find that the security processors might use, gosh, I don't know what it'd be. Let's call it 300 nanometers or 250 or something. Something a lot larger because they're not making the state of the art Intel CPU's, they're making more cost constrained devices. And because of that, they're using the old technology that's no longer leading edge. And once you start thinking about that, and maybe another data point is, even my optical microscope, without going into the scanning electron microscope, that has a resolving power of roughly 150 nanometers. And so you start thinking about, "Okay, that's the starting point." And then, depending on the features you want to resolve, maybe the metal layers are actually a lot bigger than transistors, it becomes very plausible to start looking at these circuitry.

JM (00:09:52):

Aside from that, when you start thinking about, these million dollar attacks that you hear about in, I don't know, I want to call Hollywood, but whatever you hear about that. I don't know if there's a movie about decapping chips. Most of the time, you should do this very invasive FIB work, you're rewriting circuitry. As a very last resort. There is so much you can do just by voltage glitching a chip without opening it up, electromagnetic glitching, EM glitching, is becoming a much more popular and very powerful attack, and those don't require decapping at all.

EW (00:10:28):

Well, I think the first thing you should do is probably look for the serial port the engineers use for debugging.

JM (00:10:33):

Oh, sure, sure. Yes, yes, yes. That's fair. That's fair. I'm sorry. I should have added that, yes. You should first look to see, "Hey, maybe this isn't locked. Maybe there's a JTAG open." Yeah, sure, sure. Fair enough.

CW (00:10:51):

First check if the door is unlocked before smashing through the windows.

EW (00:10:54):

Exactly.

JM (00:10:55):

Yes. Yes. There should be a progression.

EW (00:10:59):

I remember working on masked ROMs. And that makes sense to me. It makes sense that you can see the code in that way because it's truly different things happening. But if it's flash on a modern chip, does it look any different when it's programmed? Can you really get the firmware out that way? And why does it look different?

JM (00:11:22):

Technically you can, there are some papers, I would say his name, but I think I would butcher his name. So I won't do him the dishonor. But a fellow has done some very good work showing how to directly extract the flash. As I understand it, it is possible to do that, but is very, very difficult. So what you would probably see instead, is you would understand the architecture of the chip a little bit, and you would trick it into reading out the flash rather than doing this direct microscope read out of the flash.

EW (00:11:55):

Yes. Be careful with your bootloaders.

JM (00:11:57):

Yes. And there's a lot you can do besides that. So let's say even your bootloader was 100% secure. If I was able to glitch the program counter on the chip, to go somewhere else in your code, maybe that would also unlock your chip. A number of chips are vulnerable to that because just the way that the bootloaders unlocks the chip.

EW (00:12:19):

So Chris was saying that at the time he was working on something where decapping was a possibility, it was very, very expensive.

CW (00:12:28):

At least somebody was saying it was.

EW (00:12:31):

I mean, I seem to remember it was expensive. But is it cheap now? I mean, if I had a chip, how expensive would it be to-

CW (00:12:43):

Set up a lab to do this?

EW (00:12:45):

Not to set up a lab, but to go to somebody else's lab-

CW (00:12:47):

Oh, okay.

EW (00:12:47):

... and ask them to do it.

JM (00:12:48):

Well, I want to make a very clear distinction here. If you're talking about just decapping a chip, imaging out some optically visible ROM, that's a pretty straightforward process that more or less I can get done within an hour or two, if we're talking about something older. If you're talking about something that, it's very high security and might require FIB work, I don't know about a million dollars if it's something relatively off the shelf. But it's not certainly out of the reach, if you know what you're doing. The bigger problem is that the people that know how to do this work, it's a supply and demand thing. So there's a lot of personal relationships, I would say, in this field, knowing who's good with what microcontrollers and trying to figure out how to slot things in. At least that's my impression of the industry.

EW (00:13:45):

What is FIB?

JM (00:13:47):

FIB is focused ion beam. What it's primarily intended for is, if you've made your new shiny computer chip, and you power it on after months of fabrication and all this stuff, and it doesn't work, and you're saying, "Oh, no." And do you want to know, what can we do without doing this whole process again of sending it out for wafer fabrication and getting the chips back and all that. And the solution to that is often that you use a very specialized instrument called a focused ion beam, and that can do two things. It can take material away from a chip very, very precisely. And it also can deposit material such as new traces on the chip or new insulating material. So just like you might have bodge wires on a circuit board, this is the tool that'll make bodge wires on your silicon.

CW (00:14:38):

I had no idea that was even possible. Wow, that's really cool.

EW (00:14:42):

How often does that happen? I mean, is this for really expensive chips or is this happening... I mean, all chips are pretty expensive when you have to do them yourself. But...

JM (00:14:52):

Sure. I would say that, I mean, I guess as a point of reference. I know at least two people that own FIBs in their garage. So this tool is becoming relatively accessible even to hobbyist now. In terms of how often people make mistakes, I would say EDA industry, the electronic design automation that designs these computer chips, these days go through great lengths to try to prevent having a dead chip when you get it back. That said, clock and reset circuitry can be really tricky to get right. And I've heard lots of stories about people messing chips up and needing to do this rework. So yeah, maybe not every chip that you get back, but often enough that I definitely hear about it.

EW (00:15:36):

Okay. You mentioned the lapping machine, which was the layering, and that's different from decapping, which is not decapitation, but decapsulation.

JM (00:15:49):

That's correct. Yes.

EW (00:15:52):

Is that just taking the outer shell off or is there more?

JM (00:15:57):

I would say when I think of decapping a chip, there are one of two processes that people usually mean. The first, and I would say the most common, is taking the outer layers off to basically remove the epoxy packaging. People call it a P dip, a plastic dip. In this case, plastic is actually an epoxy and a glass resin. Removing that, the lead frame, maybe the bond wires, and just ending up with a bare silicon dye with a little bit of circuits around top. I would say that's the primary thing that people mean when they say decap. A related secondary thing, which I call live decap to distinguish it from this is, where maybe you don't take it out of the package entirely, but you remove just enough packaging to see the circuitry on the chip. And this would allow you to still use the chip say on a circuit board or something like that. If you had to probe it, maybe to test something out.

EW (00:16:52):

What does it look like when it's decapped? Is it just a shiny metal square coin?

JM (00:17:01):

Yeah, I would say probably the most interesting thing is, when you see a lot of pictures of chips online, you see these shiny iridescent images of a lot of these rainbow colors. One of the things that I didn't realize until I had done this for a bit was, those images are usually under relatively specialized lighting. So if you're having maybe an older aluminum chip, yeah. It looks just like a shiny maybe silvery color, and then maybe a little bit of a black background on there. That's roughly how I describe it.

EW (00:17:36):

And then you put it under a microscope?

JM (00:17:39):

Correct, yes.

EW (00:17:41):

High power microscope is enough to see some things, everything?

JM (00:17:47):

Well, the analogy I would use is maybe the microscope that a lot of people are familiar with in the embedded world are say, soldering microscopes. Which I don't like using times zoom as a benchmark, but let's call those, I don't know. 50 times zoom, maybe something like that. 30 times. Once you start getting to these metallurgical microscopes that are able to look at things on a much finer detail, you start getting 200 times, 500 times zoom. So the amount of detail that you see is considerably higher with those types of microscopes. And I would say those type of metallurgical microscopes are fairly capable of looking at chips up until, I don't know, maybe around year 2000, maybe mid `90s, something like that. Depending on how leading edge of a chip you're looking at.

EW (00:18:40):

Have you put other things in your microscope to look at them?

JM (00:18:44):

Oh, for sure. I would say that I got a request recently and I need to follow up. Someone gave me some image intensifier tubes, which have these very intricate fiber bundles. And so maybe going to take a look at some stuff like that under there. I've looked at maybe insect parts, and minerals are a real fun thing to look at under microscopes, especially since I have a lot of polarization optics and stuff where you get some really fun effects, but primarily chips just because that's my interest, but certainly other things as well.

EW (00:19:17):

The polarization, is that how you get different colors? I mean, when you have a chip and it's decapped, and it's silvery with a little bit of black, but then the pictures that I see, they're red and green and-

JM (00:19:31):

Sure.

EW (00:19:34):

Is that from the polarization or is that some other coloring method?

JM (00:19:39):

There's two ways I can answer this. I would say the primary effect you were seeing is thin film interference. If you've ever seen bubbles, for example, right? You get those rainbowy colors, it's that same effect, but under a microscope where you have this, they call passivation layer or field oxide, which are these thin layers of silicon dioxide between the metal layers. And depending on how exactly the chip was manufactured, the thickness between layers, that can cause different beautiful rainbow colors, depending on how exactly you illuminate the chip. And that's primarily what you're seeing.

EW (00:20:17):

I mean, most of the time the ROM over here is the same color.

CW (00:20:22):

No it's not color coded. No.

EW (00:20:23):

No. Okay.

JM (00:20:24):

Oh, well, if you've got a CAD program, I mean, here's a funny bit of history there. You may see the polysilicon is represented in red. And I believe the reason why that was, was because an early chips that tended to show red and microscope images. I don't believe that polysilicon is actually red in color, I think just somehow the way the manufacturing tended to work out with that layer height, it just happened to interfere in a red color.

EW (00:20:54):

Why do different parts of the chip look different?

JM (00:20:57):

You mean like an overview image? Maybe they have some regular structure here versus there?

EW (00:21:04):

Yes, exactly.

JM (00:21:06):

I would say one question I get a lot is, how am I able to maybe look at something and say, "Oh, that's the ROM versus the RAM?" One way to think about that type of stuff is, something like a RAM, tends to be, say a six transistor arrangement in these older chips, and that tends to lead to these maybe call it our glass, where intertwined circuits tend to go in this relatively complex, but regular pattern. Versus when you look at something like these ROMs, where they're essentially isolated one bit memory cells, tend to have a much simpler regular arrangement. So one way to think about this is, thinking about, what's the design entropy? How complex is this design? And that macro level can often translate into the actual physical representation of a circuit.

EW (00:22:02):

Have you ever gotten to see the CAD? The plan for the chip and then the actual chip and gotten to compare how the circuits look different?

JM (00:22:18):

I have, actually a little funny story about that. There was a chip I was working on and I looked under the microscope and I discovered that the text was backwards on the chip. And I found this really entertaining just because the way that they displayed it on their screen, and then it didn't quite translate to the way that they thought it did on the chip. But because mirroring a physical system basically got you to the same circuit, it didn't really matter at the end of the day. So yeah. I've seen this a little bit. I wouldn't call myself a chip designer. I've done a very, very small amount of it. More on the Verilog FPGA side than the computer chip, the ASIC side. So not super involved with that, but yes, I've seen that a little bit.

EW (00:23:06):

Why are chips so pretty?

JM (00:23:11):

The rainbow interference certainly helps a lot. Personally, I also like symmetry. I think that symmetry can be very beautiful. A lot of chips have a lot of regularity, and to me, there's this beauty in engineering there, where you've got all of these intricate designs, it does something, but at the same time, it's all very tidy and very lined up and symmetrical. At least to me, that's how I see it.

CW (00:23:37):

That always hearkens back to Tron to me. These cityscapes that are weird and futuristic.

JM (00:23:46):

Racing your motorcycle.

CW (00:23:47):

Look like aerial photos of cities sometimes. Yeah.

EW (00:23:50):

They do often look like aerial photos of cities. Okay. So once I have my bottle of hydrofluoric acid-

CW (00:23:59):

Jesus.

EW (00:23:59):

... what do I do next?

JM (00:24:02):

Don't drink it. Let's start there.

EW (00:24:04):

So many things. Don't drink it, don't do it, it's dangerous, this should be done by trained professionals. But, okay.

JM (00:24:13):

Yes. And I would say that, if you're getting your baby's first hydrofluoric acid, there are low concentrations you can start with. So let's assume you're starting with something relatively benign, that you even can buy over the counter here in the US at least. So starting with something like this, what I would do for a typical project is, this is assuming I've already imaged the chip and want to get the basic high level information. I would put it into a little beaker, typically made of something non-reactive like Teflon, or maybe a polypropylene, both high quality plastics. And I would let that sit cold for, gosh, I don't know. If I'm using low concentration, maybe 30 minutes, something like that. Maybe 15 minutes. And then I would wash that chip off with water and then clean it with IPA, so Isopropyl alcohol, and blow that dry, and then inspect it under a microscope.

JM (00:25:09):

And at that point, I get a little bit of feedback about how quickly the chip is etching. Maybe at that point, for example, a metal is just starting to get exposed, it no longer has that protective layer. And so as a next processing step, maybe if it's not exposed yet, I need to put in more acid, but if it is exposed, then I may use an agent like, oversimplifying it, but hydrochloric acid and etch way all of the metal from the chip. And then that allows me to get another microscope image after that, that maybe has the metal removed, but now I can see the polysilicon and the transistors below. And so by repeating this process, and taking a series of images, I can reconstruct all the layers of the chip.

EW (00:25:53):

How many layers do chips have?

JM (00:25:56):

Oh, geez. Even for older chips, the first-generation ones, you think about maybe, a couple of different dopant masks. Well I guess related to the polysilicon you've got contacts potentially between polysilicon and the diffusion layers, you have contacts between the metal layers, you have the metal layers themselves, you have the cutouts for the bond pads. So even on older chips, you might have, I don't know. 10, 12 layers. I think maybe by the time you count it up. And certainly when you start looking at modern chips, because they do a lot, lot more of those. Gosh, I don't know the layer count on the chip that I worked on, which was maybe a 65 nanometer chip, but I think it was in the ballpark of 40 maybe, if I had to make a quick guess. So certainly if you get into higher performance chips, it can get really up there.

EW (00:26:54):

And most of these layers are planes, right? I mean, they have some things that go from plane to plane, like vias in a circuit board.

JM (00:27:02):

Yes.

EW (00:27:03):

But for the most part, you're doing 2D logic, and you're not trying to do 3D.

JM (00:27:13):

Well, when you say you, I guess there's a few things.

EW (00:27:15):

One. One.

JM (00:27:17):

Well, I guess I can maybe answer some one way, is my personal interest. I personally don't deal a lot with reconstructing the full circuitry on a chip. My personal interest tends to be more in extracting the firmware of the chip. Going back to our earlier conversation about jiggling the door before breaking down the wall or knocking on the door. The most return on investment for this chip decapping, tends to be extracting, say a bootloader firmware, mask ROMs on chips. And so for the vast majority of my serious projects, that is the only layer that I care about. Is spending time to figure out what layer is that on, and everything is 100% tuned to just target that one layer. And so in that case, it's only a very small 2D area that I'm really focused on.

EW (00:28:10):

Okay. So you take a picture of the firmware and you found the right layer and you take a picture, and then what? I mean, do you go from that to ones and zeros to object files, to reverse engineered programs?

JM (00:28:33):

Yeah. The steps that I would say that I'm most involved with is, obviously decapping a chip and getting those images. And then from there, there's also a little bit of an art of turning first of all, this image into more of an abstract computer representation. Say going from a JPEG to something where you've got a 2D matrix of all the bits that you saw in that image. And then once you've got that, which you can do either computer vision, or you can just manually type out like, "Oh, I saw a bright spot here. I saw a dark spot there." Maybe that's a one versus a zero. But once you have this 2D representation, this matrix of bits, like you said, it's not really an object file. So there's a little bit of an art then, going from that bit matrix, let's call it. Into a usable, dot bin or dot elephant, whatever you're looking for. And I would say my favorite strategy for doing that is, I know a number of common memory layout techniques, and I typically have some idea of what the architecture is that I'm looking for.

JM (00:29:46):

So say, for example, if it was an 8051, maybe it's very likely that there is a interrupt jump table at the start of the firmware. And the very first byte is probably zero two for a long jump, or possibly zero one for maybe a short jump. And I will then look for that pattern in this matrix, thinking about what I know are common memory layouts. If that doesn't work, then maybe I'll start looking at some very, very minor parts of the circuitry on the chip related to the address decoders to give some hints. And typically that information is enough to turn that into an object file.

EW (00:30:26):

Typing out ones and zeros.

JM (00:30:29):

Sure. Yeah.

EW (00:30:30):

That seems like a terrible waste of an afternoon.

JM (00:30:33):

Right. And because of that, there are several programs out there to do that automatically for you. There is rompar by Adam Laurie, which is the tool that I primarily use. And I guess I'm also the maintainer for these days. And there is also bitract by Chris Gerlinsky. And I would say that you should definitely start by one of those. And if you have very clean microscope images, you should be able to do that automatically in short order. However, a lot of times, there's maybe dust on a microscope image or something like that. And it tends to mess up these computer vision algorithms. And I would say because of that, typically there's some amount of post-processing involved, but if it's a very small ROM, maybe you just need 256 bits, sometimes there's very small ones like that. It may be quicker just to sit there for literally two minutes and just go tchootchootchoo. And type it out.

EW (00:31:33):

256 bits. I could do that. Yeah.

JM (00:31:35):

Yeah.

EW (00:31:38):

Is this hacking in a bad way? I mean, you said some companies, they need it. I get that. But when you're doing it in your garage just to look for fun, is it wrong?

JM (00:31:52):

I would say that a lot of the projects that I post, are purely for educational, nostalgic purposes. If you look, you'll notice that I actually mostly post information about older chips. And one of the reasons why that is, is I feel that posting chips that are 20 years old or more, there's really not as much invested in them or relevancy, I should say. And certainly from a legal perspective, if you look at maskright in the US, just like we have copyright, we also have maskright. That expires at the 10 year mark. So we're well beyond the legal high level. I should say I'm not a lawyer, but this is my rough interpretation. There's obviously still a lot of patents and stuff, but in any case just from the educational perspective of just looking, trying to understand your favorite computer from your childhood, how did the 6502 work and that, or something like that.

JM (00:32:56):

I really haven't seen a lot of friction against hobbyists studying these projects. Maybe if you posted some information about, "How does the latest security processor work in some console?" I think you're going to get a little bit more flack for that. So I tend to stay pretty clear away from those. But at least for the projects that I've worked on so far, I haven't had any problems.

CW (00:33:18):

It's funny that people, that it can be bad to look at something. It's breaking my brain at the moment, because... Yeah. Anyway, I'm very surprised that it copyright is only 10 years.

JM (00:33:29):

So it's maskright. And part of the reason why is, I just want to say, I'm not a lawyer, so don't take-

CW (00:33:36):

Sure sure.

JM (00:33:36):

... any of this too seriously. Let's say that you had maskright on a chip for 10 years, but you also had a patent on, I don't know. So let's say the floating point methodology on that chip. Even though I could theoretically copy or mask in 10 years, that would still violate a patent that you held.

CW (00:33:53):

Got you.

JM (00:33:54):

So it's still wouldn't be commercially viable. I think that's part of the reason for that is, there's still a lot of core IP that's being protected by other legal mechanisms.

EW (00:34:06):

How do you decide what project to do next?

JM (00:34:08):

Sure. I would say there are several mechanisms for that. Certainly, personal interest drive things. But a lot of it is, I would like to experiment with some new technique. For example, one of my side projects right now is trying to get a plasma etcher up and running. If a project came in, which I thought would be a good match for that plasma etcher, I might select that project just because it would be more interesting than for me to just decap another chip, which I've done. Gosh, I don't know. 1,000 times at this point and it's just not as exciting. So a lot of the selection is based on, what's going to challenge me a bit and get me some new technique to try.

EW (00:34:52):

What does a plasma etcher do?

JM (00:34:55):

A plasma etcher is a more modern way to basically create ICs. And the way that they do that is, by basically taking, say fluorine atoms, and launching them at an IC. And the really nice property this has versus using hydrofluoric acid to etch a chip, which is what people did traditionally, over simplifying a little bit is, that this is directional. And there's these words, they're anisotropic or something like that. But I always pronounce them wrong, so I'm not going to use them. The idea being that if you use hydrofluoric acid, for example, it may under etch a circuit that you're trying to save. Say like, a polysilicon gate that you want to look at, maybe under a microscope.

JM (00:35:42):

If you used hydrofluoric acid, it would go underneath the polysilicon and the polysilicon would eventually float off. But if you use a plasma etcher, it shoots flooring atoms at the polysilicon. The polysilicon blocks those flooring atoms, it doesn't really react too much with the polysilicon, but all of the silicon dioxide around the polysilicon gets etched away. And so you get left with this very clean, sharp polysilicon, which gives you great transistor images where otherwise you have to be very careful doing that with traditional chemical methods.

EW (00:36:18):

I'm lost in thought at Christopher telling me that when you have fluoride in your toothpaste or a mouthwash, what it actually does is replace some ion in your mouth with fluoride ions.

CW (00:36:31):

It changes a mineral from one kind that your body produces to something else that's-

EW (00:36:37):

That's stronger.

CW (00:36:37):

... stronger.

EW (00:36:37):

Okay.

JM (00:36:37):

Oh, interesting.

EW (00:36:37):

Sorry.

CW (00:36:37):

Yeah.

EW (00:36:42):

I mean, I don't know. It made me think of that. So, what project would require a plasma etcher? And would it still be on these older chips?

JM (00:36:57):

Yeah, there's a lot of reasons why I might do it. One example was, a traditional problem for me, is I would like to very clearly image a contact ROM on an old chip. There are many ways that you can encode data into a chip, maybe you do it by either creating transistors or not creating transistors. Another way to do it is if you have metal layers, you can choose to put vias essentially between the layers and that encodes whether something's a one or a zero. Those vias tend to be very large. So in theory, you could use an optical microscope to see them quite easily. The problem is the surface of these chips, that field oxide, the silicon dioxide, can sometimes be very uneven. And because it's optically clear, it also can serve as a lens and actually distort the image of the contacts below. One of the ways to correct that is to use a plasma etcher to actually remove that silicon dioxide.

JM (00:38:00):

And in theory, those ions will etch the silicon dioxide a lot quicker than they will the metal. And that could give me a very clean contact image, which otherwise would be hard to get with my traditional microscope setup. So those are the projects I'd be looking for. But the main property is, it's just a lot more even than a lot of this more traditional asset etching. So it should allow me to get more modern chips that are a little bit out of what I can currently process.

EW (00:38:30):

When you do process a chip now with the hydrofluoric acid method, do you need more than one of the chips or do you usually get it on the first try?

JM (00:38:41):

I would say if it's an older chip, that's maybe one to two layers. The current strategy is, I will take a very high resolution image of what I can see, and that typically will show you the first two metal layers, just due to the way these chips were manufactured. And by the time I strip away the metal and I'm left with just the transistor layer below, that's typically enough information that if you wanted to get the whole chip information, you could. The one bit that has traditionally been very challenging is for a time, a lot of chips use something called an implant ROM. And the important thing to note about this is, these bits were not visible under a microscope image without doing special processing.

JM (00:39:29):

I went through a lot of work to try to understand how to successfully extract those bits the first time out of a chip. I would say that's something I'm a lot better at, than I used to be, but it's still a little bit of a tricky process. If a chip doesn't have that special implant layer, I can typically get it out. If it does have that implant layer, it's still a little bit hit or miss.

EW (00:39:54):

Okay. I'm going to switch gears a little bit, because I have listener questions I want to get to, but first I want to talk about conferences.

JM (00:40:04):

Sure.

EW (00:40:04):

You gave a talk recently at Hardwear.io called "Taming Hydrofluoric Acid to Extract Firmware." I assume that's pretty much what's in the tin?

JM (00:40:16):

Oh yeah. So the talk was about the process, what I do to delayer the chips. Involves a lot of chemicals, especially on more modern chips. And this machine basically helps to apply just the right chemicals at the right times to get a higher quality images as I'm taking apart a computer chip with a lot less effort.

EW (00:40:42):

Okay. You gave a talk last year at Hardwear as well.

JM (00:40:48):

Yes. And that talk was related, but different. Earlier you asked about, the process of converting these microscope images to usable firmware. The previous talk was about post-processing the microscope images into usable firmware. Whereas the recent talk was about how to generate high quality microscope images, is another way to think about it.

EW (00:41:15):

Do you have any conferences you're planning on going to soon?

JM (00:41:18):

I don't currently have any on the docket. I've been pretty busy with work, but probably a good time to start thinking about that for the future.

EW (00:41:27):

How did you start going to the Hardwear ones?

JM (00:41:31):

Well, Hardwear.io specifically happens to be nearby me, so that made it just very accessible. I have this group Mountain View Reverse Engineering. I try to foster a local Hardwear and reverse engineering community. That was a very easy sell, hearing that there was a Hardwear reverse engineering conference nearby. I was happy to try to do what I can to support that. That's what started me speaking a little bit more at conferences. Aside from that, I've presented some things that make a fair, although others were a little crazier projects. I wouldn't say that traditionally, I have spoken a lot at conferences in part, because a lot of my work is somewhat sensitive and I have to be a little bit careful about what I say. As I've started to do a little more freelancing, it's been beneficial for me to network with people more. And that's given me a little bit more incentive to be more active in the conference community. And that's what has changed that recently.

EW (00:42:35):

That makes sense. When I first started consulting, I did a lot more conference stuff. So I totally understand that. Since I did connect with you from the Hardwear, I have folks I feel like I should say that they have an online training in January 27th to 30th of 2021. Sooner than that, there's the open source firmware conference December 1st through the 3rd. And the IOT online conference put on by what looks like UBM, but I don't think it's them, but it seems like most of the same people. That's December 8th and 9th, and I'll put all those in the show notes, as well as in the archives to the Hardwear.io, which had a ton of talks including yours.

JM (00:43:25):

Oh, thank you. Well, I'll have to check those out.

EW (00:43:28):

Okay. So now some listener questions. First I think I have to go back to lightning round. Civilian tanks. Where do you get a civilian tank and why would you want to drive it on the roads that would require you to have a driver's license?

JM (00:43:43):

So, first of all, you must have done some real digging to find that. I think I posted a picture or something. I don't know, a long time ago in my Twitter. Oh man, this could be a whole podcast episode in itself, but I'll give you the plug for it. And probably get some questions about this. I basically joined a startup incubator hacker house sort of thing. That was on part of the property of the former military vehicle technology foundation, which people called the tank museum in Palo Alto. And as part of that, there was a military vehicle that got more or less abandoned on the property that the landlord had. And so we'd drive around. So one of the perks of living there with other people was sometimes, we would drive that around. And I would say one of my funniest memories from those, I think I got a noise complaint for driving a tank late at night, which was funny.

EW (00:44:46):

Do your neighbors know what's in your garage?

JM (00:44:50):

I would say that, of all the places I've lived, no one has ever cared. It's one of those funny things where people always think, just because you have these weird things in your garage that people are going to be really nosy. I used to live in Troy, New York with Andrews Sonnenberg. It's so sketchy. We would be on the sidewalks. So we didn't really have a proper lab at the time. Just cooking chips in lab coats on the sidewalk with a hot plate and lights out there, work lights. People would walk by, police cars would go by, no one ever asked us any questions. And despite how strange that was of a thing to do. And I would say that's just been my experience. When I was in Mountain View, the landlord would come into the garage and he would complain about the cardboard on the side of the house. You maybe would think that people would ask these questions, but no one ever seems to. It just has never been the issue that you might perceive it would be.

EW (00:45:54):

Maybe you need more beakers that are filled with weirdly colored liquids that light up.

JM (00:45:59):

Yeah, it doesn't look Hollywood enough. Okay. I'll get on that.

CW (00:46:03):

Maybe a Jacobs ladder and-

EW (00:46:05):

Plasma ball.

CW (00:46:06):

... that kind of stuff. Yeah.

JM (00:46:07):

Yeah.

EW (00:46:08):

Have you ever had a lab accident in your house?

JM (00:46:12):

I would say the most popular article that I've ever written was an article titled top lab accidents and explosions. Where I go through some of those. I would say-

EW (00:46:25):

That's a podcast.

JM (00:46:27):

Yes. At least there are more since that came out. I've only ever had one that I would say had serious consequences. I certainly have tons of scars on my hands. I usually wear a fair bit of protective gear, especially since I got any reasonable budget to do these type of things. And although I've been caught in a number of explosions, a good example was, I was making lead bricks to do a gamma spectroscopy. And one of the things you have to do, if you've ever done lead casting, say, people do this for bullets a lot. You see a lot of information on this online, is you have to be very careful, never ever to get water in your old lead as you're throwing into the pot. And because I was cooling down these bricks to keep the molds going quicker, something happened where some water got in some lead.

JM (00:47:20):

And I was wearing very heavy protective gear, but there was this 20 pound pot of molten lead. And I remember I threw some lead into this pot and there was this massive lead explosion. And even some of these gray boxes I have to store materials these days, still have lead embedded in the side of them from this explosion. Now I happened to have been wearing extremely heavy gloves, jacket and all this stuff. So I got sprayed with basically molten lead. But because I was wearing so much protective gear, I didn't get any injuries at all. And I think that's a good lesson for people. It's a life philosophy of mine, I guess you could say. If you're going to do something that might be a little bit dangerous, just quantify the risk. It doesn't mean you can't do it. Just be very careful and make sure you have a backup plan in case you make a mistake.

CW (00:48:13):

Always have an exit plan.

JM (00:48:15):

Yeah.

EW (00:48:17):

So your lab is basically cinnamon and lead.

JM (00:48:21):

Cinnamon and lead. Well, I don't do as much of the radiation stuff as I used to, but certainly I have a little bit of that. But solder, whatnot. I would say lead is not too unfamiliar to a lot of people in the embedded space. Certainly a number of chemicals. I would also say I have a lot of robotic stuff. I haven't talked about it a lot, but I think some people are aware that I got some bomb disposal robots, and that's been one of my recent hobby projects, is driving those around. Similarly, I find it really funny being in Silicon Valley where there's so many robots. I've driven those around a little bit and no one has asked any questions about them. Which I thought that some people were going to give it a weird look or something, but guess not here.

EW (00:49:08):

Maybe you should have the bomb disposal robots do the lead pouring.

CW (00:49:11):

I was just thinking, yeah.

JM (00:49:15):

Yeah. I've thought about that. Definitely needs more exploration.

EW (00:49:20):

So all those questions are from Rick, but now I have some from Asmita. Who recently saw your post about the Nintendo S-PPU1 SNES posts, no. Picture processing unit.

JM (00:49:38):

Yes. Okay.

EW (00:49:40):

What is that? And can you tell us about it?

JM (00:49:44):

Basically there was some community interest to get some very high resolution pictures of this awhile back. And someone very generously collected some funds and said, "Hey, John, if we give you this money, because these are very large chips. And we need a lot of images, probably delayer them. Would you be willing to put in the time to collect these images and post them so that people can start looking through these Nintendo?" They're basically graphics cards. Is a way to think about it. This is the graphics engine of the SNES and a super Nintendo entertainment system. And so awhile back, I was allocated some funds, I used a good portion of those funds to buy basically a very high power optic. It's called an oil immersion lens, and this produces very high quality microscope images.

JM (00:50:37):

And that's a partially completed project now, where I have taken the top metal image, that is almost the chip as designed, which collect the circuit board traces of the chip. And I then posted some follow-up images where I use some hydrofluoric acid, took off a little bit of the chip, took a high resolution picture of the chip, and then repeated that process a few times. And by doing that, got a layer stack up of the chip, showing all the different parts. And now with all that image data out there, some people are now actively working to try to understand the inner workings of this chip. And fortunately, there have been some related projects using similar designs that they're able to leverage. And I think the community is already moving to understand some things out of that chip.

EW (00:51:28):

Okay. That was a new microscope. How many microscopes do you have?

JM (00:51:32):

Well, that wasn't a new microscope per se. It was a new optic on an existing microscope.

EW (00:51:39):

Okay. And yet the question stands, how many microscopes do you have?

JM (00:51:44):

Well, okay. Let's count them off. Now, mind you, there's only two microscopes that I use heavily. The two microscopes that I use very heavily are my main metallurgical microscope and my soldering inspection microscope. So those are definitely the two favorites. Aside from that, I also have a laser probe station. I also have another metallurgical microscope, which was a Craigslist impulse buy. It was $300, and for that caliber microscope, I was like, "Okay, I can't resist $300 microscope." Another one is, I'll have an infrared microscope. I also have a scanning electron microscope, and I have a confocal microscope. I think that would be the list.

EW (00:52:36):

What's a confocal microscope?

CW (00:52:36):

Those are super cool.

JM (00:52:37):

Yeah. Confocal microscope, the high-level idea is you eliminate out of focus artifacts in the background of an image. The idea of being that, instead of looking at the out of focus parts of an image, just get the very crisp in focus. And extrapolating this a bit, what you can even do is, you can get it so that different focal planes of an image are encoded in different colors. And so the end result is, you tend to get these very high contrast images that show layers and different colors and at very high resolution. And they're very useful. For example, if you wanted high contrast optical images to reverse engineer a chip, it allows you to do that much easier than a conventional metallurgical microscope would let you do.

EW (00:53:27):

Is that the camera that Phil and Rob worked on?

CW (00:53:30):

Not really.

EW (00:53:32):

What's the name of the camera I'm thinking?

CW (00:53:33):

Lytro.

EW (00:53:33):

Lytro. Okay.

JM (00:53:35):

Lytro, oh yeah. Lytro. The way that mine works, maybe this will give you a little bit more idea. Mine I think they call it a Nipkov disc or something like that. The basic idea is, it's almost like you have a pinhole where, if you had some light coming out of that pinhole, it focuses on an object. And if it is in focus, it will come back through the pinhole. But if it's out of focus, it'll miss that pinhole. And basically you have one of these pinholes for every pixel. And the way that they do that, is by putting a bunch of pinholes on a disc and then spinning that disc very quickly.

CW (00:54:14):

Yeah. We had one of those at Avenger, and we used it for looking at biological samples and stuff to look at various layers and tissue. Because it almost was a thing where you could scan through and look at different layers, especially something that's translucent.

JM (00:54:33):

If I had to guess, the biological ones tend to be laser-based.

CW (00:54:36):

Yes.

JM (00:54:37):

Rather than disk-based. Yeah. Similar concept, different implementation.

EW (00:54:41):

Okay. That covers the listener questions. So I want to go back to one other thing. You have a huge Wiki based website that tells people how to do all this. Why? I mean, why did you do that?

JM (00:55:01):

It's the Linus Torvalds approach to something. I would say that what the Wiki really is, it's me working on a project and then posting my notes of what I did so that when someone asks, "How did you do that?" I just share the notes of what I learned from the last time I did on that. And I use that then to answer emails, by pointing people to a page. And I also encourage collaboration from others that if they're working on similar things, they can also share their experiences on there so that it will save me time, next time I need to work on something.

EW (00:55:38):

It seemed like a good portion of that site got turned into a college course.

JM (00:55:44):

Yeah. And this goes back to, I mentioned Andrew Sonnenberg, who I was cooking chips on the sidewalk with at RPI. He stayed at RPI a bit longer than me, and we collaborated a lot, especially at that time on our projects. Because we had a shared lab space up in New York by Albany. And he eventually got permission to teach a course, basically, alongside a professor. And that was a core interest of his. So yeah. He took a lot of our shared experiences, projects we had worked on and used that to create a course over there.

EW (00:56:24):

I was really impressed by both the website and the course because it just laid things out so beautifully and it was all there. It wasn't like I needed to watch videos or anything, the slides-

CW (00:56:36):

Videos.

EW (00:56:36):

... were nice. I'm so bad at watching videos.

JM (00:56:39):

It's funny you mentioned that, because a comment that I've repeatedly got is, "Why do you write things as text? Why don't you make more videos?"

EW (00:56:49):

Because text is searchable.

JM (00:56:52):

There's a lot of reasons. But at the end of the day, yeah, my preferred medium for communicating technical information today, is text and pictures.

EW (00:57:01):

But that's what I prefer. Although, I know people prefer podcasts and I know people prefer videos. And-

CW (00:57:12):

Podcasts are terrible.

EW (00:57:13):

... everybody-

CW (00:57:13):

People should not listen to those.

JM (00:57:14):

No. I don't know if you'd go to a podcast maybe, to learn about some car you're very passionate. I don't know if I would go to a podcast to learn how to change the oil pan on that car. I feel like it's a different mindset.

CW (00:57:28):

Put some visual things that require [crosstalk 00:57:30].

EW (00:57:30):

Yeah, maybe.

JM (00:57:30):

Yeah.

CW (00:57:33):

So I had one question before we wrap up. You talked about the processes you used to examine these chips and that some of them are quite a bit easier with larger feature sizes and older parts. Where do you see this going in a decade when the older parts are now 14 nanometer and 10 nanometer, and things like that? Are you going to be able to step up your techniques to be able to probe those or is there some wall eventually?

JM (00:58:02):

Oh, for sure. As I mentioned, the trend is, the failure analysis equipment has to keep up so that when Intel makes these new parts, that they can actually debug them when they have problems. And then over time, that failure analysis equipment trickles down next to second tier fabs and then to corporations, and then finally to the hobbyist market. I know two people today that have these focused ion beams, and they're basically in their garage. These very high quality instruments. And I expect to just see higher quality microscopes in people's garages, better equipment. I think that it will require a little more involvement maybe than we have today, but it's not going to be out of reach. Because more and more this equipment is going to filter down.

EW (00:58:53):

You started your career with computer science, computer engineering, and you've gone along embedded and hardware and deep into the chips. I mean, I've done a lot of embedded and I've never gone deep into the chips like this. If somebody wanted to do similar things with their career, do you have any advice?

JM (00:59:16):

Well, my biggest piece of advice is always, follow your passions. And at least for me, the way that I've structured everything is finding these passion projects and with a goal in mind, pursue that. I would say, if you wanted to learn for example, about chip security, and there was a current chip that you wanted to learn about, maybe instead of decapping it, maybe start with something like fault injection, because that's going to apply to your passion, it's going to teach you a lot about how these chips work, and it's going to be a lot more approachable.

EW (00:59:49):

That sounds like very good advice. Do you have any thoughts you'd like to leave us with?

JM (00:59:54):

I wouldn't say any thoughts at this time, but thank you very much for having me on the podcast.

EW (00:59:58):

Our guest has been John McMaster, embedded engineer and president of McMaster Consulting.

CW (01:00:05):

Thanks, John. This has been really interesting.

JM (01:00:07):

All right. Thank you. Good chatting.

EW (01:00:08):

Thank you too Christopher for producing and co-hosting. Thank you to Sparsh from Hardwear.io for pointing me in the direction of John. Thank you to Rick and Asmita for questions, and to our Patreon supporters for his mic, which arrived DOA, but that's not part of it. It was fine.

CW (01:00:26):

It's not your fault Patreon, of course.

EW (01:00:27):

It's not your fault Patreon, no.

JM (01:00:29):

I will try to do a post-mortem on it. I'll let you know.

EW (01:00:33):

You can always contact us at, show@embedded.fm or at the contact link on embedded.fm. And now a thought to leave you with.

CW (01:00:45):

You look like you don't have one.

EW (01:00:46):

I don't actually have one.