Transcript from 521: Are You the Tiny Domino? with Kenneth Finnegan, Christopher White, and Elecia White.

Welcome to Embedded. I am Elecia White, here with Christopher White. Our guest this week is the internet. Wait. Our guest this week is Kenneth Finnegan. I think we are going to talk about the internet?

KF (00:00:20):

Yes, I am from the internet.

CW (00:00:23):

Hello, Kenneth. Welcome to the show, finally.

KF (00:00:25):

Hi, Chris. How is it going?

CW (00:00:27):

I am doing all right.

EW (00:00:29):

Could you tell us about yourself, as if we met at a Supercon lunch table?

KF (00:00:37):

Yeah. I started life as a mechanical engineer. That lasted until about the second week of my master's program, when I had my first meeting with my thesis advisor. That managed to annoy me enough that I switched to electrical engineering.

(00:00:53):

I got a job at a solar cell startup, testing solar cells for satellites. Then moved over into actual electrical engineering, building semiconductor wafer plasma etch equipment at Lam Research.

(00:01:11):

Then accidentally created a whole bunch of load-bearing parts of the internet. I worked at Arista in their technical assistance center, helping ISPs fix their own internet, as their support helpline. Now I work at Nvidia, building the AI networks, that Chris really loves to think about and talk about constantly.

(00:01:39):

On the side, I just have all sorts of wacky hobbies that get wildly out of control.

EW (00:01:45):

So there is the xkcd comic where there are a bunch of pieces, blocks all put upon each other, and then there is one little...

CW (00:01:56):

Tiny domino at the bottom.

EW (00:01:57):

Tiny domino that everything really depends on.

CW (00:02:03):

Are you the tiny domino?

EW (00:02:03):

Are you the tiny domino?

KF (00:02:05):

I am one of the many tiny dominoes.

CW (00:02:07):

Should you have a security detail?

KF (00:02:09):

I do not. That is the scary part is- The most misleading part about that comic is that everyone is like, "Oh yeah, there is that one small project that everyone relies on." But that is a single small project that everyone relies on, in every single different dimension.

(00:02:25):

So I am one of- I would describe myself as one of the 250, 500 dominoes, that keep the internet running. I am in no way the sole load-bearing part of any one part of the internet, but it is terrifying how small the pieces get at the bottom.

EW (00:02:45):

It really is. Okay. But we are going to do lightning round. You are familiar with the show, so I will not tell you the rules. You will just have to remember them for yourself. Are you ready?

KF (00:02:54):

Yes. Yes.

CW (00:02:54):

Would you rather fight ten half dollar coins, or 100 nickels?

KF (00:02:59):

Oh, I carried nine half dollar coins with me on a daily basis. So hands down, I would fight ten half dollar coins. Because I think I can just take on one of them myself.

CW (00:03:07):

Okay.

EW (00:03:07):

What is the worst internet protocol?

KF (00:03:12):

The Border Gateway Protocol.

CW (00:03:13):

Oh, thank God.

KF (00:03:14):

BGP is the worst internet protocol. It is also the only internet protocol we have, that is useful.

CW (00:03:20):

I did say "routing protocol" in the question.

KF (00:03:23):

Well, there are other routing protocols, other than the internet-

CW (00:03:25):

Okay. What is the worst interior routing protocol?

KF (00:03:28):

RIP. And I can tell you for a fact that RIP is still used in many production, very large, national networks.

CW (00:03:35):

I just did not want you to be mad at OSPF, because I implemented OSPF once, and I just...

KF (00:03:38):

Oh. No, OSPF is great. OSPF is probably my favorite routing protocol for underlays, so no complaints there.

EW (00:03:46):

Well, for all the people who do not currently understand what the questions are about right now, let me ask this one. 0-4-2, 4-4-0, 4-6-0, or other?

KF (00:03:59):

It is definitely other. My favorite configuration is B-B.

EW (00:04:03):

Seriously, I have no idea what is going on either. It is not just you.

KF (00:04:07):

<laugh> So what you are asking me about is the wheel configurations of steam locomotives. I am not at all a fan of steam locomotives. I am much more a fan of diesel locomotives.

(00:04:19):

When you look at a diesel locomotive, the number of axles on each group of wheels, will dictate the configuration of drive for it. And so "B-B" indicates a two axle at the front and a two axle at the back. Where many modern locomotives that you will see, will actually have three axles on the front and three axles on the back. So it will be a CC configuration.

CW (00:04:46):

Do you own a conductor's hat?

KF (00:04:48):

I do not own a conductor's hat. My head is a little awkwardly large, shockingly. And I spend most of my time in rail yards wearing a hard hat. So I do not really have much need for-

CW (00:05:00):

Probably what they make you wear these days, is it not?

KF (00:05:03):

Well, yes.

EW (00:05:04):

How do you most effectively convince someone they need to own two miles of telegraph wire?

KF (00:05:12):

Oh, so this is a very fun game that I love to play.

EW (00:05:16):

<laugh>

KF (00:05:22):

In the Bay Area, we have all sorts of electronic swap meets, and salvage surplus stores that at this point are mostly gone.

(00:05:29):

The game that I loved playing with all of my friends, as we were hanging out at Halted, or Weird Stuff, or at the Silicon Valley Electronics Flea Market, is we would see something and I would try and convince my friends to spend their money to get it.

(00:05:43):

My crowning achievement in this whole game, was that one time I managed to convince one of my amateur radio friends, as we were standing there in a parking lot looking at a vendor trying to sell two miles of rusty telegraph wire still in the original case, that he would definitely use that to build dipole antennas and long wire antennas and stuff.

CW (00:06:07):

<laugh>

KF (00:06:08):

So I got my friend to lay 20 bucks on the table and carry two miles of rusty telegraph wire back to his car, drive it home, and then sit in his garage for five years, before he finally came to his senses and threw it away.

EW (00:06:21):

Did they know that they were playing a game?

CW (00:06:23):

<laugh>

KF (00:06:25):

Yes. My friends have regularly accused me of, "Oh dang, you are doing that thing again, where you are trying to get me to spend my own money. Are you not?"

CW (00:06:34):

Have you ever networked a locomotive?

KF (00:06:36):

I actually have. Locomotives have a very standard- Oh, is it? It is 68 or 72-pin network connector on the front and the back of them. It is this big cable that looks like- It is about the same diameter as a gas pump hose. You can network two locomotives together, so you can drive both of them from just one of them.

EW (00:07:00):

Do you want to ask the follow-up?

CW (00:07:03):

Have you ever locomotived a network?

KF (00:07:08):

I do not think I would say I have locomotived a network.

CW (00:07:10):

Okay. All right.

KF (00:07:10):

No, I have-

EW (00:07:13):

Have you ever steam powered a network?

KF (00:07:14):

Oh, heck no. No. Steam power is the worst.

EW (00:07:19):

Okay. Complete one project, or ask a dozen?

KF (00:07:23):

Ask a dozen?

EW (00:07:23):

Complete one project, or start a dozen?

CW (00:07:26):

<laugh> "Or ask a dozen?"

KF (00:07:29):

<laugh> My brain works in this wonderful way, where I get the best of both worlds, in that I love to spend years architecting projects in my head and on paper.

(00:07:41):

So I have a dozen projects that are in the block diagram and pipeline. Then when I actually physically start materially building one of them, I can focus on it pretty well, and get it across the finish line or give up on it pretty quickly.

CW (00:07:57):

All right. I am going to have to ask. Favorite fictional robot?

KF (00:08:00):

Favorite fictional robot is WALL-E, from the fact that WALL-E was a self-healing robot that was so self-sufficient, which a lot of robots- They really portrayed it in a way that a lot of robots is like, "This robot would never sustain itself in any reasonable or useful way."

EW (00:08:21):

And do you have a tip everyone should know?

KF (00:08:23):

Yeah. Experts love to talk about whatever they are experts at. So people need to be less afraid to reach out and say like, "Hey, I saw that you wrote this paper, or this blog post. Or, you have been posting about this. And, I have these specific and constrained questions. So can you answer these, or can we jump on a call and just talk about it for 30 minutes?"

(00:08:44):

People are much happier to talk to you about things, than you would think they are.

EW (00:08:50):

I often hold myself back from talking about things, because I do not know at what point people get bored.

(00:08:56):

So yes, if you ask me to talk about something and you truly want me to talk about it, you may have to ask twice. But after that, you might as well just get a cup of coffee and sit back, because I will be forever. Just ask me about origami. It is very embarrassing.

KF (00:09:11):

It is important to ask it in a way, that it does not make it insurmountable. So, yeah. I get a lot of email questions where people are like, "Hey. Can you tell me how to get into networking?" And I am like, "No." <laugh>

CW (00:09:21):

"So science. Tell me about science."

KF (00:09:24):

Yeah. Science. Exactly. Right. So you need to come to them with an attainable ask of like, "Hey, can I just pick your brain for 30 minutes?" And not in a skeezy way, where we are trying to pay you $400 to disclose industry secrets. But, "I am just earnestly interested in this thing, and want to talk to you."

EW (00:09:42):

So networking is hard, and it keeps getting harder. What do embedded developers need to know about networking, and how can they acquire that knowledge?

CW (00:09:51):

<laugh> That sounds like, "Tell me about networking."

EW (00:09:52):

<laugh> It sounds a lot like, "Tell me about networking," does it not?

KF (00:09:56):

No. Networking is very hard. The thing that- The subsystem embedded level that I see a lot of people struggle with the most, is that people really need to start- People need to be much more familiar and comfortable with the vocabulary, and the risks, around the field of study of distributed systems. Because everything is a network.

(00:10:22):

Obviously, you have things that look like networks that are- You have got cellular modems, or you have got ethernet links, or Wi-Fi. Then you have got smaller things that do not look like networks, like I2C or SPI, which have a lot of the same failure characteristics and same dynamics as larger networks.

(00:10:46):

So people need to really get more comfortable with problems like the Two Generals' Problem, and consensus protocols like Raft. And really appreciate like, "When the network fails, whatever scale of network that is, what is my fallback positions? How should my system respond appropriately to it, and handle it gracefully?"

EW (00:11:13):

So basically, we should be checking the return codes on our SPI calls.

KF (00:11:21):

Yes and no. Yes, you should be checking the return codes on your SPI calls. But then you should also be asking yourselves questions like, "Well, what if it comes back with a bad answer?" And, "What is the failure?"

(00:11:35):

The one that we see constantly, or that I have to interact with a lot, is that a lot of people do not appreciate that temperature sensors, like those little I2C temperature sensors, have a really bad habit. That if you manage to ask them at just the right moment, they will come back and tell you that whatever system they are currently measuring is 64,000 degrees.

EW (00:12:02):

Mmm. So warm.

KF (00:12:04):

It is like, "That is a little hot for Irvine. Yeah."

CW (00:12:09):

<laugh>

KF (00:12:10):

So it is one of those things that it is like, not only do you need to be checking the return codes. But you need to be asking yourself, "Well, what if I get bad information off the network?" Or, "What if I make an SPI call and it never comes back?"

(00:12:21):

There are all sorts of these sorts of problems of, what is every single different failure mode, that when I am inside of the microcontroller, trying to talk to anything else, that I need to handle in some graceful way. Because saying that your system drops out of cellular reception and just absolutely nothing works anymore, does not make for a very robust project.

EW (00:12:50):

There is how the application works with the internet, or works with a degraded network, or a non-existent network. And then there is the, "I do not know how to get started with networking."

(00:13:06):

It is big, and setting up a Raspberry Pi for the first time in your home network, is not trivial. Like, "What is the IP address? What is an IP address?" All of these features, but that is not what we talk about when we talk about networking. Do you have a good resource for how people can just wrap their head around-

CW (00:13:33):

Internet addressing, to start with.

EW (00:13:35):

Internet addressing, to start with.

CW (00:13:36):

MAC addresses, and all of that stuff. Yeah.

EW (00:13:38):

The big problems of networking. Not the specifics of like, why OSPF is really good for when you want to multicast. And "multicast" means you want to send lots of people the same-

CW (00:13:49):

OSPF is not good for multicasting.

EW (00:13:50):

Whatever.

KF (00:13:50):

Not at all good for-[anything]

EW (00:13:54):

But there are certain protocols that are better for certain things. I do not want that information, clearly, because I do not have it.

CW (00:13:59):

Right. There is tactical information for everyday people who encounter networking, and then there is operator information for people who are running networks. Which are different domains sort of, with some overlap.

KF (00:14:11):

Yeah. I would say that we are really in this- We are on the tail end of the sweet spot, as far as on YouTube there are tremendously valuable resources. Like [ Network Chuck ], and these channels that will sit there and talk to you about the basics of like, "This is what a router is. This is what a switch is. This is what a Wi-Fi access point is."

(00:14:33):

I am really- I have never had good answers for like, "Hey, how did you get into xyz?" Because usually my answer is, "Well, I downloaded the IEEE spec for it, and just read it." Or, "I sat down in a coffee shop with all the RFCs printed out, and read through them and highlighted them and put flags on them."

(00:14:56):

The way that I approach information I think is not a very kind or accessible manner, for most people. But I would say there are so many good resources out there right now, that have not quite been overrun with the AI slop, that is trying to remonetize the same venues.

CW (00:15:11):

Yeah. I do not think, "Get a summer job at Cisco in 1995, and have somebody hand you a pile of RFCs," is really a good answer for most people.

KF (00:15:18):

No, it really is not.

EW (00:15:20):

The man page is still one of the best places to get information.

KF (00:15:23):

Oh. Yeah.

EW (00:15:25):

I do not think people even- I think there are a whole bunch of people who have no idea that man pages exist.

KF (00:15:31):

Yeah. Man pages-

CW (00:15:33):

Well, first step is a terminal.

EW (00:15:33):

<laugh>

KF (00:15:34):

Oh yes. Right. You have to learn the terms. But not only just man pages. But it is so easy now to be able to build something that you can then just look at it, right? The burden of you get me a Raspberry Pi, get me a $60 travel router, and build an entirely completely zero stakes network on my desk.

CW (00:15:54):

Right.

KF (00:15:54):

If I need to learn about, "What does this packet look like?" I can set it up and do a packet capture. I can look at that packet and then I can change something and see how that changes the packet. Where it is like now for-

(00:16:09):

So for a couple hundred dollars, you can build an entire lab environment, that lets you play around with any of this sort of networking stuff at just smaller and slower speeds than exactly what everyone else is doing, right? There is no reason that you cannot have BGP running between two virtual machines on your laptop.

CW (00:16:27):

Yeah.

EW (00:16:28):

But you have to know what BGP is, which I do not. So could you explain?

KF (00:16:34):

Yes. So BGP is the Border Gateway Protocol. This is the routing protocol that is used between every separate network of the internet. So when you think about the internet, you are looking at it from the perspective of you are a customer connected to one internet service provider at your home, where they hand you one cable and they say, "The entire internet is this way."

EW (00:17:04):

Right. It is a vast ocean.

KF (00:17:06):

Yes. It is just, "That way." But once you move from your leaf right on the very end of the tree, and you start moving back towards the branches, to the larger branches, to the trunks, the answer of which way is the internet becomes much more complicated.

(00:17:26):

Your service provider is going to be connected to anywhere from two to hundreds of other service providers, or service providers who are then also connected to other ones. There are ultimately about 60,000 unique entities that make up the internet, that all are connected to each other in certain places.

(00:17:50):

BGP is how those two service providers, wherever they meet, can exchange information between their two routers on the edges about, "Hey, if you want to reach this block of IP addresses, I can get it to you via this path, at these sorts of cost metrics. And if you want to go to these blocks of IP addresses, I can get you there or I cannot get you there."

EW (00:18:17):

So this is kind of like airlines. You used the tree, which I think is great, but it is hard for me to visualize how you do to cost metrics with a tree.

KF (00:18:31):

Yeah. Airlines would be a fantastic analogy as far as like, yeah, if you are in some little town like San Luis Obispo that technically has an airport that has one flight into SFO a day. Once you get to SFO, you then have many different directions that you can go on many different flights to all sorts of different places. That may not necessarily get you exactly where you are going in one hop, but will get you closer in that direction.

EW (00:18:58):

And multi-hop might be cheaper, but it might take longer. There are all of these different cost metrics that you have to balance. Honestly, you as a packet, or you as a person, do not really get those. It is your ISP manages that.

CW (00:19:16):

And the packet- All that stuff gets programmed into hardware. So there is just- Those are locked in unless something changes.

KF (00:19:24):

Yeah. Unfortunately the things change constantly. Right. Something that people do not really appreciate is-

(00:19:29):

Well, to finish the thought- The cost metrics, which is like the AS path and lots of other things. A lot of the packet routing decisions between the service providers is not really a, "What is the optimal way to route this packet for the best experience?" It is often a, "Well, I like this airline more than that airline." Or, "I want to earn points over here versus over there."

CW (00:19:57):

"I have a service agreement with this one."

KF (00:19:59):

Yeah. So the BGP is often called a "routing policy expression protocol" that also happens to forward packets. But it is much closer tied to the actual business objectives of how does our service provider want to interact with that service provider, in not optimal lowest latency, highest bandwidth criteria always.

EW (00:20:31):

If I look at the, I want to say, Lego website that is located in Denmark. How does anybody get paid for that?

KF (00:20:46):

Yeah, so...

CW (00:20:49):

For the traffic.

KF (00:20:49):

Right. So to finish the- Pop the stack, just one- Sorry, we are going to be all over the place here.

CW (00:20:57):

Yes. That is fine.

KF (00:20:58):

The internet is constantly broken. So when you are looking at a website in Denmark, you are very likely not actually looking at a website in Denmark. You are looking at a website that originally came from Denmark. But you are probably connected to a server that is much, much closer to you.

(00:21:17):

Because the burden of actually getting- Like you clicking on a link, and getting a packet from you to your service provider, to their transit provider, from that transit provider to another different transit provider, then going through a cable underneath the Atlantic Ocean, gets very difficult.

(00:21:36):

So most large websites will have content distribution networks, acting as their front doorstep into their website, in a hundred or 200 locations throughout the world. So even if the website is nominally from Denmark, you are probably connecting to a server a couple dozen miles away from you; it will have some parts saved already of the website. And for other parts, it will reach back and find the best way to get back there.

(00:22:10):

So from a payment and money perspective, you are paying money to your internet service provider, to give you internet access in your home. They are then paying some other, a tier two transit provider, to give them access to the whole internet.

(00:22:31):

Then the tier two and tier one transit providers, which are progressively larger and larger networks from a geographical perspective, they interchange packets with each other on a what is called a "settlement free peering session."

(00:22:50):

Where they both say, "Hey, I can reach about a quarter of the internet, and you can reach about a quarter to the internet. So if we just equally trade about the same amount of traffic, it works out in the end that both of us can just trade traffic with each other, and our customers can all reach half of the internet without us having to pay anyone else."

EW (00:23:11):

And then someone lays a new cable across a giant ocean. Is it the tier three people who do that?

KF (00:23:19):

It is anyone who needs it. A lot of cables will be cooperatives between like these five very large tier one service providers all need extra capacity between these two places. So they will form a new holding company to go lay one specific cable for a project, and then they will share bandwidth capacity across it.

(00:23:43):

It is not the historic idealistic world where it is like you have these nice, neat tier one networks at the top and then tier two networks and then tier three networks. If that ever really did exist, it definitely does not exist anymore.

(00:24:00):

You have this much messier, well, Facebook and Google are bigger networks than the big networks, but they are not really transit providers so much as end content providers. But they are laying their own cables, because they have larger bandwidth demands internal to their network, than anyone has external to it.

(00:24:21):

So the not useful answer is, "Anyone will do it, depending on what they need. And it depends."

EW (00:24:32):

That makes sense. It seems like the sort of thing that would go through phases of vertical integration, and then specialization, and then vertical integration, specialization.

(00:24:40):

What you want when you are vertically integrated and want to handle all your own traffic and charge other people to do traffic, is different than I just want to buy traffic. I just want to be in charge of what I am doing. I do not want to worry about the rest of the world.

KF (00:24:58):

Yeah. Then inevitably, geopolitics comes in and plays lots of parts as far as like, "Oh, you want fiber connectivity from this country to this other country." And, "Oh, you want to lay fiber down through the middle of what is otherwise a relatively unstable area."

(00:25:16):

And so you go look at places like South America, where it is often easier to lay subsea cables from your country out and then back into your own country, versus having to go forge roads through the middle of your own country.

EW (00:25:34):

That is such an odd concept. Of course, laying cables in an ocean is going to be super expensive. There is the physical. Not all two mile cables cost 20 bucks.

CW (00:25:47):

<laugh>

KF (00:25:47):

<laugh>

EW (00:25:51):

And then there is the personnel and the maintenance, and the boat dropping all the- It is going to be expensive. And the cables get cut. Or bitten by sharks, which is of course my favorite.

CW (00:26:07):

Bitten by sharks?

EW (00:26:09):

Yeah. A bunch of them-

KF (00:26:09):

Yeah?

EW (00:26:09):

Until they stopped making some of the cable stuff, the coating, out of shark deliciousness.

CW (00:26:16):

I thought they were relatively thick. Okay.

KF (00:26:17):

Well, my understanding is- The issue is that- When you have a thousand kilometer strand of glass that is as big as your human hair, that is going from one country to another, you cannot possibly pump enough light into it on one end that it comes out the other side usable.

(00:26:35):

So in the middle at intervals of this cable that you are just dropping on the bottom of the ocean, you are putting amplifiers in there to regenerate the signal. Unfortunately, those amplifiers need power from someplace. So part of this cable you are laying down is a power rail to energize these amplifiers.

(00:26:54):

Power unfortunately gives off magnetics. I guess sharks are somehow sensitive to magnetic signals.

EW (00:27:04):

All kinds of electrical signals. They hunt using the electrical signals of the fish, because the-

CW (00:27:09):

Oh, here we go.

EW (00:27:10):

Never mind.

CW (00:27:10):

<laugh>

KF (00:27:10):

Yes.

EW (00:27:10):

Sorry.

KF (00:27:12):

Long story short. Once you even lay the cables, they tend to break, be it either by sharks or quote accidentally broken unquote by nation state fishing vessels.

CW (00:27:25):

I do think I want to go back to one thing you mentioned about the caching stuff. Because I do not think people realize just how much of the internet is Cloudflare and AWS.

KF (00:27:34):

Yes.

CW (00:27:34):

Most of the time you go to a website, you are hitting one of those two things.

KF (00:27:38):

Yeah. And that really comes down to the fact that the speed of light is painfully slow. So the concept of everyone and everywhere in the world being able to access your website, coming from one server in one place. A, that is a very high stakes single box you have sitting there.

(00:27:57):

But secondly, the speed of light means that as you move physically farther and farther away from that one place, the user experience on even the human timescale starts getting noticeably and noticeably worse.

(00:28:13):

A lot of the modern good high-end internet experience, where you are like, yeah, you expect to be able to click on something and the website immediately opens. A lot of that relies on this fact that you are, as a website host, if you set up your own little website, you are then paying some other organization. Or you have to grow your organization to build out this infrastructure.

(00:28:36):

So that when a user clicks on a link, they are accessing a copy of your website that is geographically very close to them. Which is hopefully more or less representative of what your website actually looks like back on the source of truth, ultimately.

CW (00:28:54):

It is interesting to me how both different and the same the internet is, from when it was first getting going in the nineties. BGP existed then. All these other routing protocols, the basic fundamentals of it, were all the same. DNS existed. We built networks out of all of this stuff.

EW (00:29:12):

But the big routers now, are very different than big routers then.

KF (00:29:13):

<laugh>

CW (00:29:15):

Well, the big routers are big routers, but they are still running BGP.

(00:29:16):

But we are doing everything differently with content distribution networks and caching and all of this stuff. It used to be that yes, you did go from your PC to a server in Denmark. That was the way it worked. But we have taken these tools that existed that are pretty old and crufty now in some ways, and rebuilt them in a completely different way, which is kind of fascinating to me.

EW (00:29:42):

IPv4.

CW (00:29:43):

Yeah, well. <laugh> It is a good segue.

KF (00:29:47):

Yeah. IPv4 is not going anywhere fast.

CW (00:29:51):

No, no. I was assured in 1997, that IPv6 was going to be deployed pretty widely by 1998.

EW (00:29:58):

Fully accepted by 2000.

KF (00:30:02):

Yes. Absolutely.

CW (00:30:03):

Not for all these damn ISPs using ATM. Never mind.

KF (00:30:10):

<laugh> Yes. Thankfully ATM is mostly gone.

CW (00:30:15):

Oh God. It is not completely gone?

KF (00:30:17):

No, no. The Lord, no.

CW (00:30:18):

<laugh>

EW (00:30:19):

Nothing ever dies on the internet.

CW (00:30:20):

I know.

EW (00:30:21):

Okay, so we have talked about when these tier one networks meet like slime molds fighting on the edges.

CW (00:30:33):

With suits.

EW (00:30:34):

With suits.

KF (00:30:34):

Yes. With suits.

EW (00:30:36):

Slime molds in suits. Does everybody like that for a title? All right.

(00:30:42):

So they meet and they exchange traffic through BGP. Is this an internet exchange?

KF (00:30:55):

It can be. Right. When you are two extremely large networks, and you specifically want to trade traffic with each other, it is pretty easy to say like, "Hey, I have 300 gigabits per second of traffic I want to send to you. You have 250 gigabits per second of traffic you want to send to me. It is probably worth it for both of us to build our networks into one building, and have a fiber physically running from my router to your router."

(00:31:31):

That cross connect between your two networks often happens in what is called a "carrier hotel." These are just sometimes deliberate, sometimes accidental buildings that end up being these kind of sweet spots, where lots of different internet service providers all build into the same building. They can then physically directly connect to each other.

(00:31:54):

These buildings are often nondescript. So if you are driving- Local to the Bay Area, not that this will help a lot of your remote listeners, but if you are driving up 101 and you take the 85 off-ramp, right on the right side, you will see this complex of four big buildings. That is the Equinix SV 1, 5, 10, 11 campus, which is where an extremely large portion of the internet traffic flows for all of the west of the United States.

(00:32:29):

When you are extremely large, you can justify the cost expense of like, "I need a 400 giga port from my router to this other network's router." It is expensive in that you have to have a physical port on your router. But it is also expensive in that whatever carrier hotel you are in, will often charge you a couple hundred dollars a month for the rental space for the two millimeter cable in their tray running between your two networks.

EW (00:32:54):

<laugh> That is ridiculous.

KF (00:32:55):

Yes. Right. The most expensive four square millimeters of real estate that you are going to pay for, is that cable up in the cable tray.

(00:33:03):

When you go down in size from the largest 15 networks, down to the other 60,000 networks, this idea of cross connecting with each other individual network starts getting problematic. Because you have got an N squared problem there. That densely connecting with 120 other networks, requires 120 router ports on my router, and 120 cross connect bills from the data center.

(00:33:35):

Internet exchanges are this service provider for the service providers. Where you essentially just have an ethernet switch in one rack, with a bunch of ports on it. Then anyone who wants to trade traffic with everyone else plugged into that switch, can come to the internet exchange and say, "Hey. Can I get one port on your switch to get effectively direct connectivity to a hundred other networks."

(00:34:03):

"That for any one of them, I may be only trading a few megabits of traffic. But aggregated together, I have got a couple gigs worth of traffic that I can send to the switch of the internet exchange. Which then end to end scatters that traffic to where it actually needs to go, to get to the rest of the internet."

EW (00:34:25):

What is the Fremont Cabal Internet Exchange?

KF (00:34:32):

So the Fremont Cabal Internet Exchange or FCIX, or as it is properly pronounced, Fuckx-

EW (00:34:37):

<laugh>

CW (00:34:40):

I am not even sure I heard that.

KF (00:34:43):

Sorry. This is an internet exchange that a couple of friends and I originally started as a joke. Accidentally it became successful and a load-bearing part of the internet.

EW (00:34:58):

Okay. You told me that it is very expensive to house, hotel, to have guests at, vacation, at internet exchange hotels. How did you accidentally do this?

KF (00:35:19):

There is a very long backstory to this, that we are not going to have time to get into. About how one of my friends challenged me to set up my own autonomous system. Or, to be one of those 60,000 autonomous entities on the internet. Because it was going to be an impressive operation, and I needed a rack and a data center for lots of other reasons.

(00:35:48):

We eventually ended up in this situation, where in this one aisle of one suite of a data center, we ended up with about five of us immediately adjacent to each other, each with our own autonomous systems in one rack.

(00:36:04):

And it is like, "Well, I have got an autonomous system. You have got an autonomous system. Clearly we should peer with each other, to be able to trade any traffic that needs to go from my web server to your web server."

(00:36:17):

Being right next to each other, we just pulled an ethernet cable up and over the wall between our two racks. Which once or twice you can kind of get away with. But when you have got six racks next to each other, that is 25 cables between your six racks.

(00:36:33):

The data center eventually notices and says, "Hey guys. We did not pull any of these cables, and you did not pay us for any of these, so you need to knock this off." So we went, "Okay, okay. All right. You caught us."

(00:36:49):

"Kenneth's rack happens to be in the middle of this block of racks. And so on the router in his rack, we will set up one segment for this peering exchange. Everyone will just run one cable into that rack. We will rip out the other 20 cables." Our N squared problems becomes a crossbar in the Cisco 6506 router I was running in my rack.

(00:37:25):

We were having dinner with the owner of the data center later. Someone casually mentioned, "Oh hey. You guys were complaining about all those extra cables we had. We fixed it. We started an internet exchange." Hurricane Electric, who was the data center that we were in, the owner of it, Mike Leber is one of the largest advocates for free and open peering.

(00:37:47):

So when he hears that a couple of smart guys in his data center has started their own internet exchange, he got very excited about this. And went to all of the salespeople for the carrier and the data center and said, "Position and sell this internet exchange in our building, as one of the perks for other networks to build into our building."

EW (00:38:13):

Want to back up just a little bit. You and some friends decided to have your computers talk to each other.

CW (00:38:21):

<laugh>

KF (00:38:21):

Yes.

EW (00:38:24):

Due to an overgrowth of cables, you decided everyone would talk to your computer, and you would just pass messages like you were in fifth grade.

KF (00:38:34):

Yeah, exactly.

EW (00:38:34):

But you are really only talking to each other at this point.

KF (00:38:37):

Yes.

EW (00:38:38):

Then you have dinner, and some guy is like, "Oh, you know what? That is really cool. It is free and it is open source. We will make this sexy and let everybody else talk to Kenneth's router."

KF (00:38:50):

Yes. The data center started pitching as part of the value add services in their building, of not only do we provide you power and cooling and security, and access to these X, Y, Z service providers that give you the whole internet. You can also connect to this random guy's router in one rack, and from there, talk to anyone else in the same building for free.

CW (00:39:15):

Let me just stop you. I was there 3,000 years ago-

EW (00:39:18):

<laugh>

CW (00:39:20):

When the 6500s started coming out. They are not that fast. How much traffic are you exchanging?

KF (00:39:26):

So the 65-

EW (00:39:29):

It was just the five guys talking to each other.

CW (00:39:30):

It was!

KF (00:39:30):

Just the five guys talking to each other. The 6500 did not last very long.

CW (00:39:35):

Okay. <laugh>

KF (00:39:36):

Because once you have got real actual networks coming in and saying, "Hey, we would like a 10 gig port on your exchange, please-"

CW (00:39:44):

Exactly.

KF (00:39:44):

For anyone who does not know, the 6500 is about the size of a very large suitcase. A high density line card on it has eight 10 gig ports on a single line card, which is about the size of a pizza box. Very rapidly, we were faced with this problem of, "Oh my God, we need better switch hardware than the 6500, because buying this thing on eBay for $400 did not get us where we need to be here."

(00:40:17):

At that point, I was posting on Twitter, rest in peace, about this. And posting on Facebook about some of these shenanigans. My godfather's son, the son of my godfather-

CW (00:40:32):

<laugh> Your godbrother.

KF (00:40:34):

Family friend. I have known him the whole time I was growing up. He reaches out and was like, "Hey. I see that you are starting an internet exchange. I work at Arista-" Which is one of Cisco's big competitors, building huge 50 terabit ethernet switches. "Would you like us to sponsor you, and give you the switches you need to make the internet exchange successful?"

CW (00:40:56):

I love this. I love this.

EW (00:40:58):

There is no way you would say, "No," to that, because it would be so much fun to get the good gear. Somebody offers me a nice O-scope, I am like, "Yes, please. Okay. Yeah."

KF (00:41:08):

Yes. Exactly. I met up for lunch with him. He handed me a pile of end of life obsolete switches to them, that were still 64 ports of 10 gig. So it was able to move the traffic we needed, absolutely no problem.

(00:41:27):

On Twitter, Flexoptics reached out and said, "Hey, we would love to be the optical transceiver sponsor for your internet exchange." They sent me an entire case of the optical transceivers to put into the switches.

(00:41:39):

Hurricane Electric said, "Well, you are an internet exchange in our building, so the first cross connect into the internet exchange for every single member, is also free."

CW (00:41:49):

I feel like this is one of those stories where you start with a paperclip, and ended up with a house.

EW (00:41:51):

Yeah.

KF (00:41:52):

Exactly. Exactly. It started as a joke between the five or six of us in the one aisle. The right people hear about it, and the right people know of the types of shenanigans I get into.

(00:42:05):

Eventually I end up with a hundred thousand dollars worth of networking gear given to me for free. And an internet exchange with an annual opex cost of $260, and I am moving 50 to 80 gigabits per second of the internet.

EW (00:42:26):

Where is that now? Do you still- Where is it now? Yes.

KF (00:42:32):

It is still running. We are up to about 120 networks in the building. So we have about 120 networks that are tied into this fabric. Which the peering fabric has now grown to the point where it is about seven switches, all just in a stack.

(00:42:47):

A good way to visualize this is imagine a bowl of spaghetti, and then just look down straight into a bowl of spaghetti. Just yellow fiber running everywhere. It is absolute chaos.

EW (00:43:02):

But are you still running off of donations? And enough- You said, "260," which is- I would cover that, because that is amusement money.

CW (00:43:12):

What if a line card goes down? That is an expensive-

KF (00:43:15):

Yeah. Yes. It is totally volunteer run. A lot of- Every year we have new members that come and say, "Hey, this is great. You are adding a lot of value to our network. I am getting a lot of value out of this. Can I just give you a couple thousand dollars?"

(00:43:34):

So when we have some sort of hardware failure that I actually have to open my wallet and go buy replacements for it, I just have this floating pool of- Enough people have given me somewhere between small and extremely generous donations, so that we are able to cover the costs of this thing.

(00:43:55):

It is really just seven ethernet switches, and a couple little virtual machines running on my own network. The work and effort on this thing is connecting a new member and troubleshooting that initial link up. Once they are on it, it just sits there and moves traffic for the internet.

(00:44:14):

We do not have 24/7 on call, because it is just three of us. So if the internet exchange goes down in the middle of the night, it is just going to stay broken until one of us wakes up.

EW (00:44:24):

But you are passing like a quarter of the download capacity of North America.

CW (00:44:28):

No. <laugh>

KF (00:44:30):

No. No, no, no. We are only moving 50 to 70 gigabits per second. The local carrier hotel facilities near us in South San Jose, it is moving like a quarter of the internet. They do have a 24/7 on call for some of their exchanges. Yeah.

EW (00:44:48):

All right. Good, good. Okay.

CW (00:44:48):

Armed guards. That sort of thing.

EW (00:44:51):

Okay. You accidentally did this. But you also made a DNS hosting service, which we do not quite have enough time to talk about. But it is another of these internet things, that you can start small and suddenly it becomes very important. You did that there too. Do you think there is a trend?

KF (00:45:12):

Yeah.

CW (00:45:12):

Are you going to start your own caching service? <laugh>

KF (00:45:16):

I have, right?

EW (00:45:16):

He did. He owns a CDN.

CW (00:45:18):

Well, yes. Yes. Right, right. I guess it is. Yes.

KF (00:45:21):

The FCIX internet exchange then grew into the Micro Mirror project, where we are providing caching services for free software. NS-Global was this DNS service that I built, that then became successful.

(00:45:34):

All of it is really just a demonstration of- When people talk about networking. When you are in college, you need to work on networking. It is not the awkward go to the career fair and glad hand people and trade business cards. It is really a lot of finding and getting to know the right mentors and the right people.

(00:45:59):

And then demonstrating to them that you are reliable, and that you are serious about learning what it is that you want to achieve.

(00:46:08):

Having that sort of credibility, makes it so that I am able to go to friends of friends' service providers and say, "Hey, would you be willing to give me a virtual machine in Europe to set up this DNS service?" I was able to build the NS-Global DNS authoritative service, entirely from free donations of virtual machines in just various places from friends, or friends of friends.

(00:46:39):

That is where it is having something like Twitter, rest in peace, or my Mastodon, of being able to just post about, "Hey, I have got this predicament. Who has the solutions to this just kind of sitting around, and is willing to toss it in my direction for the amusement of watching to see what happens?"

CW (00:46:56):

Now, after all of that, if you download a VLC, there is a good chance you are getting from your mirror, right?

KF (00:47:01):

Yes. If you download VLC in North America, there is about a 70% chance that it is going to come from one of my servers.

CW (00:47:10):

Many times I have needed to get an image for Ubuntu. I have just gone straight to your servers, because I cannot be bothered with the official servers. They are too slow.

KF (00:47:18):

Right. Well, so, yeah. When you go to ubuntu.com and you click the download link, they are routing you to one of these caching mirrors for them. We are about, depending on how you measure it, about 20% of the download capacity for Ubuntu. You might land on one of our servers that is pretty fast. You might land on someone else's server that is not quite as fast.

(00:47:41):

Yeah, a lot of people have realized that Kenneth runs pretty good download mirrors, and so they will just go directly to one of those Micro Mirror sites. If you ever see an mm.fcix.net URL while you are downloading free software, that is coming from the same project.

EW (00:47:59):

That is coming from Kenneth's server.

KF (00:48:01):

Yes. Servers. There are about 40 of them.

EW (00:48:05):

So these turned out to be a little bit bigger, more successful, more work than perhaps originally anticipated.

KF (00:48:12):

Certainly bigger. I would not say that- A very important part of these sorts of projects, is focusing on how do I not make it a lot of work. Things like robust CI/CD pipelines, and hands-off provisioning, and playbooks and checklists, so that I do not have- So it is like it is not a lot of work.

(00:48:37):

It is a lot of, "Okay, I need to do the same thing that I have already done 15 times." I just plug the programmer in, and click go on my build pipeline. These projects have been a harsh lesson in becoming acutely aware of exactly where are the manual steps in any sort of project, and how do you eliminate them.

EW (00:49:01):

That is actually- There are a lot of times where that piece of information- Do you identify the manual steps because you are like, "Oh, I have already done this"? Or can you pre-identify them?

KF (00:49:15):

It took a lot of experience and a lot of pain, to get to a point where you are able to look on paper and systematically identify that. So a lot of it was just the hands-on experience of, "Wow. I have already done this six times, and I accidentally did it five slightly different ways." And, "I would love this to not take 20 minutes anymore. But I would love it to take two minutes."

(00:49:38):

So I go spend 150 hours developing a solution, to save me 15 minutes every single time I go to do something. It is not an easy thing to just look on paper and say, "Oh, that is where the bottleneck is going to be." Or, "That is where all the human errors are going to be."

CW (00:49:55):

And things change. It is a scaling thing, right. "Uh, I do this three or four times a year. I am just going to forget how to do it, because I can relearn it once a quarter." And then suddenly something happens, and you need to do it 30 times a year. Well, now that becomes much more of a, "Okay, I need to either become an expert in this, or automate it some way."

KF (00:50:13):

Exactly.

EW (00:50:16):

Why did you not sell out?

KF (00:50:16):

<laugh>

EW (00:50:22):

There are a lot of people making a lot of money, doing what you are as a hobby.

KF (00:50:28):

Yes. I get a lot of offers. I will regularly get offers to buy various little projects off of me. Nominally owning a lot of the motivation for not selling out is that, like updateyodell.net is a project that I am standing behind with my credibility.

(00:50:51):

So if I sell it to some sketchy organization that is going to continue serving updates for end of life Dell servers, I do not know if they are going to start serving malware off of it at some point. I just do not want to ever run the risk of being associated with that.

(00:51:08):

I have the luxury and the blessing of having a job in tech, so that paying my rent this month is not something that is top of mind for me. So I do not necessarily need it. In some ways I arguably have sold out. I got a job at Nvidia, selling shovels in the Gold Rush that is AI. So I feel like in some ways I have sold out a little bit.

EW (00:51:34):

But you have not. You have made money off of the skills you have learned.

CW (00:51:38):

Yeah, I am not going to call that selling.

EW (00:51:40):

I am not going to call that selling.

KF (00:51:41):

Okay. Well. I really enjoy it. I just love the infrastructure, and how does things actually get moved from here to there. It is just such a satisfying and gratifying like looking behind the curtains and understanding it enough to build it.

(00:51:54):

That it is like people love to take apart- When you are kids, you love to take apart your printer or something to figure out how it works. But what if you go take apart the whole internet, and figure out how it works and put it back together. And get to futz around with your own little pieces of it.

EW (00:52:10):

Are there projects you are working on now that...

CW (00:52:15):

Are in danger of becoming one of these?

EW (00:52:17):

I was trying to figure out how to phrase that.

KF (00:52:19):

No. It is the running joke is like, "Oh, someone offered to hold Kenneth's beer. He is going to go do it again."

CW (00:52:25):

<laugh>

EW (00:52:25):

<laugh>

KF (00:52:25):

At this point, I am really trying to restrain myself. My dance ticket is pretty full. Because each one of these little projects, yeah, I get it mostly automated and it is mostly hands-free. But there is a little bit of background chatter. They start adding up by a death of a hundred paper cuts there.

(00:52:48):

So I have been trying to not think about any other world problems on the internet, that with a little bit of technical skill I can really start putting to rest. The network time protocol is the one that I keep not being able to stop thinking about, but I-

EW (00:53:10):

Ahh. Please fix that.

KF (00:53:12):

I am really trying not to.

CW (00:53:13):

Never work on time. Never work on time.

KF (00:53:15):

Yeah. No.

EW (00:53:15):

<laugh>

KF (00:53:17):

No one has got time for that. I have been trying to avoid that sort of thing. This year a relatively new hobby for me is I got into coin collecting. That refreshingly has absolutely nothing to do with the internet at all.

CW (00:53:34):

But given your history, there is a 1% chance you are going to start your own mint.

KF (00:53:38):

<sigh> Okay. Well actually-

EW (00:53:40):

<laugh> Well, actually.

KF (00:53:41):

Not entirely incorrect.

CW (00:53:43):

<laugh>

KF (00:53:43):

Because I actually have minted my own coins before. That went back to when "Make: Magazine" came out with that awful Make: crypto coin.

CW (00:53:54):

Right. Right, I remember now.

KF (00:53:56):

Yeah. Dale comes out with this Make: crypto coin, and everyone in the maker community unanimously went like, "What the hell are you doing? Crypto is the antithesis of everything Make:"

(00:54:08):

I sat down that weekend and went on Alibaba, and found a custom arcade token vendor. I drew up the artwork for a, "Hey, make something awesome." It has the word "make" on the front of it. It has got a MIG welder, and a pair of knitting needles on the back of it, as the obverse and the reverse designs for it.

(00:54:32):

It turns out that if you want to press your own custom arcade tokens, the tooling dye cost is like 60 bucks, and the minimum order quantity is 5,000 tokens. So for $600 all out the door, I managed to get 5,000 of these custom brass tokens made and shipped to me.

(00:54:57):

I was handing those things out like candy for a couple years, and I was selling them off. So I managed to demonstrate, look "Make: Magazine" who is nominally focused on making things, put out this scammy crypto token thing that ultimately ended up going nowhere.

(00:55:12):

And one guy with a couple hundred bucks, managed to actually make something that physically is substantiated. Numerous people will trade these things back and forth at Maker beat up and events, for like, "Hey, you made something awesome!" I want to do John Wick style show appreciation for what you have been achieving.

CW (00:55:32):

That is pretty cool.

EW (00:55:36):

That is very cool. I am supposed to ask you about trains.

KF (00:55:42):

Yes.

EW (00:55:44):

What should I ask you?

CW (00:55:45):

<laugh>

KF (00:55:47):

<laugh> Trains are what originally prompted me to become a mechanical engineer, which did not last very long. Well, the mechanical engineering did not last very long. The trains are forever. Trains are awesome and amazing.

(00:55:59):

The trains started while I was growing up. My dad has always been very large rail fan, so we would- During the nineties, Congress was always talking about killing off Amtrak. And so we made a point of doing a lot of family trips, that were taking Amtrak long-haul train rides up to Washington or out to Denver.

(00:56:26):

Then we would go to the railroad museums, like the one in Sacramento. Back in the nineties, all the museums and stuff would have these racks of tri-fold pamphlets of other tourist destinations.

(00:56:42):

My dad found one of these pamphlets that was for the Western Pacific Railroad Museum in Portola, which is very, very far north up in the Truckee Sierra Nevada area, that has a locomotive rental program.

CW (00:57:01):

What?

KF (00:57:04):

If you want to learn how to drive a locomotive, they have this program where you can go and rent one of their locomotives for an hour, and drive it around the museum property. And then you are done.

CW (00:57:20):

And then you get a certificate and you can drive it wherever you want after that, right?

KF (00:57:22):

You get a little certificate and you get a picture.

EW (00:57:24):

Does the certificate say, "I am a good conductor"?

CW (00:57:25):

Oh. Come on.

EW (00:57:26):

Because I need it for some of my jokes.

KF (00:57:29):

Unfortunately not, because a conductor and an engineer are two very separate roles.

EW (00:57:34):

Yes.

KF (00:57:36):

To be a good conductor, you have to join the society and attend about 150 hours of training and hands-on experience, to get your brakeman certification, and then your conductor certification. Then you actually will be a certified conductor.

EW (00:57:54):

But you still will not be able to pass that much electricity through your body.

KF (00:57:58):

Exactly. Long story short, my dad gets involved with this railroad museum and drags me along with him.

(00:58:05):

Since the museum is a live operating full-scale historic preservation of the Western Pacific and the equipment that operated on the Western Pacific Railroad, the museum falls under the FRA safety guidelines and GCOR. As a 12-year-old kid, I was not really allowed to do much on the hands-on operating 12 inches to the foot model railroads that much.

(00:58:37):

But thankfully, there are a lot of adjacent activities to running locomotives and trains, in that you have to maintain locomotives and trains. So my summers growing up were spent at this railroad museum, with these crufty old mechanic guys mentoring me and teaching me how to weld.

(00:58:56):

And how to take apart locomotives and rebuild them. And service an engine that has 150 gallons of crank case oil, as opposed to the five liters in your car. I got to spend all of this time at what is a really fantastic hands-on museum, that also-

(00:59:18):

We fell in love with the place because of this rental program. It is called the "Run-A-Locomotive" program. And then I have just always been involved in that place, driving trains.

CW (00:59:31):

That is extremely cool. How many times are you allowed to blow the train horn, when you are renting?

EW (00:59:40):

<laugh>

KF (00:59:42):

We do have a public crossing on the typical route. You need to blow the horn every single time that you are crossing the road crossing. So you will get to blow it many times.

EW (01:00:01):

That is museum.wplives.org?

KF (01:00:07):

Yeah, wplives.org. The museum.wplives.org is the main landing page for you want to come here and see what it is at.

EW (01:00:19):

Book now. Okay. I think that is all. I have other things to do now. No- <laugh>

CW (01:00:24):

<laugh>

EW (01:00:24):

I have so many more questions for you, but we are almost out of time. Chris, do you have anything outstanding, before I close up?

CW (01:00:35):

No, no. I think we will have to leave that for another day.

KF (01:00:38):

Absolutely.

CW (01:00:38):

Given how much other things we could talk about.

EW (01:00:43):

Kenneth, do you have any thoughts you would like to leave us with?

KF (01:00:47):

Yeah. I think that something that everyone should really take away and gnaw on, is there are a lot of things in the world that you take for granted and do not engage with.

(01:01:01):

Coins are something that I have just become infatuated with recently. It is one of the things that you handle on a day-to-day basis. But stop and think about, "What does every single symbol on these things mean?" Or, "How do traffic signals work?" Or, "What is the logistics of where my coffee comes from at work?"

(01:01:24):

The entire world is so fascinating and has so many neat things going on with it, that other people are making happen, just because it is a mundane part of their worlds. I feel like particularly now, in this day of people getting into these smaller and smaller silos of understanding and reality.

(01:01:46):

Make a point of really noticing what you take for granted. And what just happens that you do not think about. Because it is such a neat world out there, that has so many cool little things, that become stupid little fun facts that you can whip out at maker parties and events.

EW (01:02:06):

Our guest has been Kenneth Finnegan, Senior Solutions Architect at Nvidia, and Technical Director at Fremont Cabal Internet Exchange.

CW (01:02:16):

Thanks, Kenneth. It was good to talk to you.

KF (01:02:17):

Thanks guys.

EW (01:02:19):

Thank you to Christopher for producing and co-hosting. Thank you to our Patreon supporters and Ko-fi supporters. Thank you to our Slack group for their questions. And of course, thank you for listening. You can always contact us at show@embedded.fm or hit the contact link on the embedded.fm website.

(01:02:36):

Now quote to leave you with. This is Susan Sontag saying something very close to what Kenneth just said. "Do stuff. Be clenched, curious, not waiting for inspiration's shove or society's kiss on your forehead. Pay attention. It is all about paying attention. Attention is vitality. It connects you with others. It makes you eager. Stay eager."