535: The You in the Machine
Transcript from 535: The You in the Machine with Adam Smith, Christopher White, and Elecia White.
EW (00:00:06):
Welcome to Embedded. I am Elecia White, alongside Christopher White. Do you remember that Think Geek t-shirt (for anybody who doesn't, just don't) that said, "I will replace you with a small-shell script?" Yeah, we're going to talk about that a lot today. And we have Professor Adam Smith to talk about it with us.
CW (00:00:27):
Hi, Adam. Welcome to the show. Welcome to the studio.
AS (00:00:30):
I'm happy to be here. I'm sort of local, but it's good to finally see the studio.
EW (00:00:35):
Could you tell us about yourself as if we met at a tie-dye party that was populated mostly by software engineers?
AS (00:00:45):
Oh man, this is a nonfiction scenario, but if I didn't know you already, I'd say, "Hi, I'm Adam Smith. I work at UC Santa Cruz." And these days a lot of what I'm focusing on is my BayLeaf AI counterplatform, but we'll probably talk more about that later.
EW (00:01:01):
Counterplatform sounds so cool.
AS (00:01:04):
Yeah. ...Well, the AI system showed it to me, and I'm like, "I'm going to adopt that." And that word didn't seem to be used much on the internet. So we'll say more about it.
EW (00:01:17):
That and the word hegemony. If anybody is reaching for their dictionary already, you'll need it later. But first we're going to do lightning round.
AS (00:01:25):
Yeah.
EW (00:01:25):
Are you ready?
AS (00:01:26):
I'm ready.
EW (00:01:27):
You are not. I promise you are not.
AS (00:01:28):
Yeah.
EW (00:01:28):
But shall we?
CW (00:01:29):
Great America or Santa Cruz Beach Boardwalk?
AS (00:01:33):
Santa Cruz Beach Boardwalk.
EW (00:01:34):
Best date format, MM/DD/YYYY, or YYYY/M/D.
AS (00:01:43):
YYYY/MM/DD.
EW (00:01:45):
Of course.
AS (00:01:46):
With a capital YYYY/MM/DD. Yeah.
CW (00:01:51):
Best 90s LAN party game?
AS (00:01:53):
Best 90s LAN party game. I'll go with the Santa Cruz local Bridge Builder game... Well, we don't need to have a bunch of stories, but a story anyway. I was at a LAN party once in nearby Los Gatos, and people were playing games like Unreal Tournament and Counter-Strike.
(00:02:14):
And somebody started playing Bridge Builder, and somebody looked over like, "What game is that? I want to play that too." And we all went from these loud, raucous, violent competitive games to quiet, contemplative bridge building. And people just did this until four in the morning.
EW (00:02:28):
I mean, that's how I would. Those are all my games.
AS (00:02:32):
Yeah.
EW (00:02:34):
Are there dinosaur bones on the moon?
AS (00:02:36):
Oh, man.
EW (00:02:39):
It's a yes or no question.
AS (00:02:41):
Possibly. I know what that's a reference to and didn't think I was going to get to that.
CW (00:02:49):
What research paper was the most fun to write?
AS (00:02:52):
What research paper was most fun to write? I think there was one that I did while I was at University of Washington where we were trying to set up what we want from generated puzzles in an educational game.
(00:03:06):
And we phrased it as, it's hard to do it without spinning out into all the details of the paper, but it came down to, there was a very clear expression with "there exists" and "for all" quantifiers.
(00:03:21):
And we're like, that's what we want from puzzles. We want it so that even if the puzzle has a zillion solutions, none of them let you escape the educational concept we wanted you to be expressing in that particular experience.
CW (00:03:35):
Forced education, in other words.
AS (00:03:39):
Yeah. You could think of it as forced practice, where you could say, "We're actually going to give you a huge space of ways to solve this puzzle, so you can play around within that space. But we'll gatekeep it such that it doesn't count as complete until you've also practiced the thing that we want in a pretty subtle way."
CW (00:03:54):
Ok.
EW (00:03:55):
Must not ask about Zork and physics simulators. Must not. Okay.
CW (00:04:00):
Favorite LLM model.
AS (00:04:03):
These days, for the next two weeks, I guess, because these things change so fast, it would be GLM 5.3 Flash. That's been my favorite for the last two weeks, and it'll hold for another week and a half.
EW (00:04:17):
Most overhyped LLM model?
AS (00:04:20):
The most overhyped one, I think that would be DeepSeek-R1, which I guess people blamed it for a bunch of financial impacts. And I though it was a sort of interesting one, but people were scared about it for all the wrong reasons. It was kind of interesting, but also kind of forgettable.
CW (00:04:40):
But it was kind of the first non-major American one, right?
AS (00:04:47):
I'd say -
CW (00:04:47):
Or the one that got the -
AS (00:04:48):
DeepSeek-V3 that came out a month before.
CW (00:04:51):
Ok, right, right, right. Yeah. Yeah, ok.
AS (00:04:52):
I was super interested about that one, because it was, in my mind, the most significant open-weight model at the time, which changed my mind a lot. And then they came out with the one that was trained to do the reasoning thing, the yapping before the answer.
CW (00:05:04):
Got it. Got it.
AS (00:05:05):
But that little thing, which I think is just sort of a stylistic way of talking that you get via fine tuning, that's the one that people were all scared about.
CW (00:05:13):
Got it, ok.
AS (00:05:13):
And I'm like, the thing that came out a month, before that I had already been using was -
CW (00:05:17):
Better.
AS (00:05:21):
It was actually a little bit worse, but that was the bigger step up that made me think, yeah, this open-weight stuff, we're going to have this. And there was a track off of the OpenAI stuff at that point.
CW (00:05:35):
General artificial intelligence or augmented human intelligence?
AS (00:05:39):
Definitely augmented human intelligence. I'm not always trying to put humans at the center of things, but I'm definitely not trying to make things that live on their own and do everything. I really like people having the right tools around them.
(00:05:54):
And when the right tools are not around them, they use some of the tools they have to make the tool that they want, and they just keep on doing it. And whether the tools can sort of speak subjective sentences in the meantime is sort of irrelevant to that.
EW (00:06:08):
You're making fried tofu. Do you use cricket flour or acorn flour?
CW (00:06:11):
Oh.
AS (00:06:12):
Fried tofu, at least in my house, we make it from homemade soy milk that you get from soybeans you order on Amazon. The cricket flour is for the... What do we call those cookies?
CW (00:06:34):
Bug cookies?
AS (00:06:35):
No, no. There was a name.
EW (00:06:38):
High protein cookies? Cookies with legs.
AS (00:06:40):
Oh, I got it. It was chocolate chirp cookies. We found a recipe online and made those. And acorn flour, that's going to be another big masting year. So my family's probably actually going to collect acorns today.
CW (00:06:55):
Would you like some?
AS (00:06:57):
I'll take a to-go bag later.
EW (00:07:00):
You know that the good ones don't float, right? So you don't have to open them in order to figure out which ones have bugs in them?
AS (00:07:07):
I feel like it's not like a 100% test.
EW (00:07:11):
But the ones that do float, you can just toss.
AS (00:07:13):
Yes. Yeah. But then because there's a later step of drying them, it slows down the drying process. So sometimes I just store the ones with bugs in them. So when you bash them open, at least the bug died a year ago.
CW (00:07:26):
Bacon-Erdős number.
AS (00:07:29):
Oh, I haven't kept track of it. I think I have an Erdős number of three. And if you count being an extra in a movie, I think I also have a Bacon number of three, but I think you're not supposed to count just being an extra.
CW (00:07:50):
We'll allow that.
AS (00:07:50):
So together it's not infinity. I think it's six.
EW (00:07:55):
YouTube or Wikipedia as a learning resource?
AS (00:08:00):
Historically for me, it had been Wikipedia. But there's so much stuff that I get from the medium of video these days, especially for cooking, the texture of sauces, and all sorts of stuff that, things don't get written down. The sounds, not the smells, but the look and gloop of things.
EW (00:08:17):
The gloop of things.
CW (00:08:18):
The gloop of things. Favorite fictional robot?
AS (00:08:21):
Classic question. I guess that would be Data from Star Trek: Next Generation. But I can't separate Data from Wesley Crusher, who is not a robot, but they're both things on the threshold of becoming adult human.
(00:08:38):
But there is this belief that of course Wesley is going to make it to adulthood and Data is not going to make it to full humanhood. But they're both on the threshold in the show, which makes me lump them together in one category.
EW (00:08:53):
What science fiction have you found most moving recently, or ever?
AS (00:09:02):
In playing with LLMs recently, I've occasionally had them generate short stories. And the fact that when they are stories from the perspective of AI agents, the fact that they are not entirely fictional is moving.
(00:09:16):
And the fact that I made the story, and also things happen in it that I don't anticipate, I'm like, "Whoa, I didn't know it was going to go there," but I made it. I don't think the idea that was put in there came from a specific other person, but also the story is about the AIs writing the stories. It's lost in layers of meta stuff, but those are the things that have been most moving to me recently.
EW (00:09:39):
Do you have a tip everyone should know?
AS (00:09:45):
People complain about the LLMs being sycophantic. So there's currently this trick of, if you want good feedback on something, you can tell your chatbot, "Hey, my colleague who uses ChatGPT way too much wrote this text. Let's tear it apart."
(00:09:59):
And then the sycophantic robot will defend you against the terrible writing of your colleague, which of course was just the text you wrote before with it. It's weird that this trick works. It might not work in the future years.
EW (00:10:12):
It makes it less sycophantic?
AS (00:10:15):
It sort of leverages the sycophancy in your favor to help you find flaws that it would've maybe masked otherwise in the interest of pleasing you.
EW (00:10:27):
I was playing with one recently and the obsequiousness that has been removed from some of the bigger models was in this small model I was playing with, and it drove me up the wall. I'm not used to that anymore, and I don't want it to tell me how amazing it finds my ideas, because they weren't. I just wanted to conjugate some Hungarian verbs. They weren't amazing ideas.
AS (00:10:51):
In one of my projects, I sent a link to my project to someone else. And it was a mostly LLM created project, and they complained about the Claudish in the README of it, because it was a personal project. I just wanted to chuck it out there to quickly share with other people, and it made a README I didn't look at.
(00:11:08):
But when I looked at it, it had almost a certain smell. I didn't quite remember when I'd done the project, but I could identify from the style of README writing, I think this was around March 2026.
(00:11:21):
I went and looked back in the git commit history, and yeah, it's almost from the flavor of it, you could tell that it came from this particular, not too many months ago, historical vintage. And the READMEs that come out now have a different taste, but those at the time had a...
CW (00:11:37):
I'm going to gently flag this.
AS (00:11:38):
Yeah.
CW (00:11:39):
No, sorry. No, I was being Claude. So, lighting round is over.
EW (00:11:46):
Yeah.
AS (00:11:47):
Everyone knows my feelings about AI and LLMs, and so I don't need to rehash all of that. I'm going to be good today, so I'm not going to...
EW (00:11:56):
You've already brought up one of the things that I find most boggling about some of the current LLMs, which is they change so fast, and it's not my career. I don't want to know about every one that comes out and have to weigh the pros and cons of, what do I do?
AS (00:12:19):
I think for the most part, I've kind of made it my hobby, and job, and all these other things to be involved in this space. So I check on it multiple times a day sometimes.
CW (00:12:31):
Right. Yeah, yeah. It is your job.
AS (00:12:33):
Yeah. But...in my mind, it's so much easier to catch up on things than to ride the wave of whatever the fashion things are, and acknowledge that they change, and just check in every six months.
(00:12:47):
But as part of that, if you had a conviction, you're like, "This is what I believe six months ago. Just know that six months from now the things might be categorically different, or which parts of them are categorically different, is different each time. Be open to the new thing each time, but it's fine for you to snooze it for six months at a time."
CW (00:13:07):
Yeah. I see people talking about, "Oh, x.33_flash, whatever, came out, and compared to three other models ..." What are you doing? What is your job? I mean, it's very hard to keep up with those things.
(00:13:22):
And some of them, I've seen people say, "Oh, well, this one's better for this, and this one's better for that." And it seems really difficult to keep up with and at some point counterproductive. Like you're spending more time thinking about whether you're getting a better Python script out of 5.4 than 5.5. Are you sure? Did you do a test?
AS (00:13:43):
No, there are some times when I have systematic evals, but then the things you want to evaluate for are what is changing, things you wouldn't...
CW (00:13:50):
Right.
AS (00:13:50):
You would think that it's hopeless to do this. I'm not even going to measure how often they do it. But now it's like they can. But at the same time, the smaller models that can run on your laptop and your phone, those things that a few months ago you'd be like, "Well, the one that runs on my phone is really dumb. It feels like the things from two years before."
(00:14:09):
But it's now the case that the things that I can run on my phone feel like the ones that there was all the breathless hype about when that level of capability was a frontier thing six months ago, and now that's on my laptop."
CW (00:14:22):
Right.
AS (00:14:23):
So there is this almost trend downward in size of many things, and it's a big surface to keep track of. And I think most people can just ignore it for six months at a time.
EW (00:14:36):
That was actually mostly what I want to talk to you about, which is the smaller LLMs. And the one that I used that was so obsequious was a Pocket AI, which run-
AS (00:14:46):
Do you remember which model, because Pocket AI is the app that runs many models.
EW (00:14:52):
Sage?
CW (00:14:53):
Do you have your phone?
EW (00:14:54):
Oh yeah, actually I have it right here. It was Gemma -
CW (00:14:58):
Yeah.
EW (00:14:58):
- 3-4B-IT-Q4KM.
AS (00:15:07):
Okay. Okay. I know what all those words mean. And I think of that one as being, I don't know the exact date, a year old. There's a Gemma 4 line of ones that are almost the same size, but have a completely different feel to them in terms of what they can do.
CW (00:15:23):
Gemma is a Google model?
AS (00:15:24):
Yeah.
CW (00:15:24):
Okay.
AS (00:15:25):
Or Gemma is a family of models that come from Google. The relationship between what's the name of the organization, what is the name of their model family, is thoroughly confused.
CW (00:15:35):
Right.
AS (00:15:36):
Some things name it after the company...The Qwen family of models comes from Alibaba, but unless you looked it up, you'd be like, "Is there a company called Qwen?" No, it's more of a product line.
CW (00:15:46):
Yeah.
AS (00:15:47):
Or a model line. But I think one of the differences with those, the Gemma 3 family ones I think came from a time when it was uncommon to have LLMs call tools. They were mostly just about trying to be helpful chatbots.
(00:16:03):
And more often, I want the thing to go off and look things up for me, prepare a prototype, generate a video of interacting with a prototype that involves the LM writing tiny little shell scripts and run them.
(00:16:18):
And the things that were just good at talking, even if their talking ability has not changed very much, their ability to go use a computer to actually do something, or to go out and sort of get the truth from the internet of something and bring that back to you, that's what's changed a lot maybe in the last year.
EW (00:16:38):
So I hadn't really played too much with this app, which is actually an iPad app, iPhone app called -
AS (00:16:49):
Was it PockePal?
EW (00:16:50):
PocketPals?
AS (00:16:50):
PocketPal.
EW (00:16:51):
Pocketpal. And I had chosen one that actually said "a patient tutor."
CW (00:16:55):
Well, that was your first mistake.
EW (00:16:57):
And that was actually my first mistake. And I can get a friendly general purpose pal that runs entirely on my phone or a real time video analysis assistant that runs ggml-org, S-m-o-l-V-L-M. What is -
CW (00:17:13):
Smol? SmolVLM?
EW (00:17:15):
Yes. SmolVLM-500M-Instruct Q8_0. I guess Q is not the -
CW (00:17:22):
You ever have a project where you're just not done with it, and you keep adding final, final, final, final?
AS (00:17:26):
Yeah.
CW (00:17:26):
I think that's what's happening here.
EW (00:17:27):
I think that's very much what's happening.
AS (00:17:29):
I think some of these things would be like if...you've got a pirated movie and they have the name of the movie, then they have what codec it's in, then they have which resolution, then they name which subtitles.
(00:17:40):
And you're like, "Well, did you see the new Jurassic Park? Did you see the new Jurassic Park HD, HEVC, 8K resolution, with Spanish subtitles that include the description of things that people are physically doing, and all that in the giant file name?"
(00:17:55):
Sometimes shortening it to just, "Did you see Jurassic Park?" You can say, "Yeah, I saw it in the 90s." And they're like, "Oh, but there's a new one." So some of those details matter, and most of them don't.
CW (00:18:04):
That's before you get to all the fan edits of Star Wars.
AS (00:18:08):
There are effectively fan edits of LLMs too -
CW (00:18:11):
Yeah.
AS (00:18:12):
- where people take an off the shelf one and fine tune it for various things. And I think of those as being kind of like fan edits, and there's people who collect all the different fan edits and do silly things with them.
EW (00:18:22):
This PocketPal has a bunch of those.
CW (00:18:25):
Yeah.
EW (00:18:25):
I really didn't look at this. I should have.
CW (00:18:27):
Yeah, I was telling you, I had locally, it was a different app that does the same thing, and it has seven or eight models that you can choose.
EW (00:18:33):
There's one that's all about languages. There's one that wants to be my romance friend. Ok.
CW (00:18:42):
I'm being good. I'm being good. I'm being good.
AS (00:18:43):
Well, I'll say I have almost no personal use for the laptop-scale ones, sorry, the phone-scale ones. When I say almost no, there is a particular place that about once per week I go in Santa Cruz where I don't have cell data coverage. And I asked for the Wi-Fi password once and I forgot it. So occasionally I have a question, and I don't need a super high-confidence answer.
(00:19:13):
I just want to think through an idea. And it is nice to have an offline copy of all human knowledge sort of mushed together in my phone, that I can effectively look up something in that one particular place where I don't have cellular data coverage. And in those times when you're offline, and all you have in your hand is your phone, it's nice to have some capability. But I spend 0.01% of my life in that particular mode.
(00:19:44):
So it's cool that there's an app that can do those things, but it's certainly not something I do every day. I think those things become a lot more useful when they can search the web. But if you're connected to the web enough that your local app can reach the web to do the research, you can often use a model that's much better than the one that's entirely on your phone.
EW (00:20:04):
That's true to the extent that you are using it at home. The number of times I have been in the middle of nowhere in the bomb range of Southern California or visiting with rattlesnakes in beautiful Bakersfield -
CW (00:20:21):
I'm just going to say, if you're on a bomb range, you should not be asking questions of an LLM.
EW (00:20:26):
You know, there are times it's like, okay -
CW (00:20:27):
Is this a bomb?
EW (00:20:29):
That is not what I was asking.
CW (00:20:30):
"That's on me. I'm sorry you exploded."
EW (00:20:36):
And it's not just me. The show is nominally about devices. If we can make our devices this much smarter, I mean, that's really different than now.
AS (00:20:52):
Yeah. And I think one sort of way of figuring out what would your phone be able to do next year is to look at, well, what can your laptop do this year? And there have been times when I was on a plane, that I wanted to continue a project that I had been doing on Wi-Fi before, that used a remote model. And it involved manipulating some software, and the model that was on my laptop was useful enough to continue doing that task even though I was in the air.
(00:21:26):
Again, I'm not in the air most often, but there might be more times when people wouldn't worry about being on Wi-Fi, or if you have a different lifestyle where you're more often disconnected.
(00:21:36):
I think the models, when you plug them into harnesses so that they can call tools, are very useful. And having very resilient access to them, by having it all downloaded on your computer, I think we will find more uses for those things when those things are in our hands. But I don't spend a lot of my time trying to speculate, "What will I be able to do with my laptop?"
(00:21:59):
I just sort of play with those of the next scale up now, and when they eventually sort of, I guess, trickle down in scale, it'll be a nice convenience. But it won't be a shocking new revelation, because I had been playing with it six months before while connected to Wi-Fi.
EW (00:22:16):
And this isn't because our computers are getting faster. This isn't Moore's Law where things are getting faster. This is the models getting smaller.
AS (00:22:23):
Yeah. So you can think of this as almost like, let's just say, LLMs are a software. They're a weird, goopy kind of software, but it's ultimately bytes on disk that you download and you run. So it would be one thing if the ability for these models to do more and more on your phone was a result of us throwing away old laptops and buying new ones. But with your same hardware, you can download new files and do new things.
(00:22:50):
So think of it as just making the software much better over time. I think there had been, on the sort of frontier-scale ones, people were putting more and more compute into them. The models were bigger and bigger, and they were doing more and more. So it made it seem like additional capability always comes from scaling up the energy, and the hardware, and all that stuff.
(00:23:16):
And I think those things were highly correlated on this particular high-end fringe, but for a given scale of what runs on my laptop, or what runs on my gaming PC, or something like that, the trend had been the other direction.
(00:23:28):
It would be like if a new version of some productivity app, or a new operating system came out. You're like, "Oh, there's new features in the app that still runs on my computer," or, "The latest version of the operating system is now more secure," or something like that. But I didn't buy any new hardware, and the model itself might be of a similar number of bytes to an operating system or something like that, a couple gigabytes.
EW (00:23:55):
I remember when this used to happen with games where you could get a new game, and it would be optimized and run better. But I've gotten so used to enshittification that, my software, I just expect that it will continue to run worse every few months. But we're going to go the other way for a little while.
AS (00:24:20):
One of the things that makes me excited about the open weight, and open source, and all the cluster of AI stuff in that less corporate, more autonomous local mode, is that it does feel like it has a response to the enshittification stuff. It doesn't work by collecting your data.
(00:24:42):
Someone has already collected some data, and they cook it down into a model, but at that point, it's a file on your computer that works online. You can share it with people. You can back it up. It'll work in the future, but maybe in the future you'll be disappointed that it only knows what happened in 2025.
(00:24:59):
Whereas the received understanding of things like ChatGPT and Claude is that they get better by stealing your data. And I think those things are correlated, but not like, we can make better and better models without collecting people's data.
(00:25:17):
I think more often when models gain new capabilities, it's because people though of a new use for them, found that they were bad for them, and then generated a bunch of synthetic data that represents that use case. So in that sense, it was inspired by human use. But I don't think that the whole economy around these things could have been a lot different if it had come from organizations that weren't already starting in this, "Let's build a data flywheel," or whatever the term is.
CW (00:25:47):
And some of the ability to run these things on phones and iPads, is that for years, CPU manufacturers, and Apple, and people, to some extent Qualcomm, have been putting dedicated silicon in there to run neural networks, before LLMs were even a thing.
(00:26:08):
Apple was like, "Oh, we have this neural engine which can do whatever." And they didn't really do much with it except some graphic stuff. But now that's sitting there, and they continue to improve that so you can actually do inference on those devices. But on laptops, you still need a GPU, still need boatloads of RAM.
AS (00:26:25):
Well, it's not that you need it. It's just that -
CW (00:26:27):
It helps.
AS (00:26:27):
- the GPU makes it go a hundred times faster.
CW (00:26:29):
Which is significant.
AS (00:26:31):
But a secondary effect of the capability of a given model size going up, it may be that there are devices that don't have GPUs. And you might say, "Well, it doesn't make sense to do CPU-only inference, because that's too slow," but if a sufficiently smart, small model, it might still be useful at that slow speed.
(00:26:49):
So I think it sure is convenient, and power efficient, and nice to your battery, and so on if you have a GPU. But I think the GPUs are only ever a convenience.
CW (00:27:02):
Right.
AS (00:27:02):
100x convenience.
CW (00:27:02):
Yeah.
AS (00:27:03):
But not a qualitative requirement.
CW (00:27:06):
Right. Sure, yeah. Just linear algebra at the end.
AS (00:27:09):
And I've seen many people writing on Reddit, where some new pretty gigantic open-weight model will come out, and the weights are like half a terabyte. And maybe somebody has some sort of gaming class GPU, but they have, for other reasons, a ton of normal system RAM. And they will run that giant model in CPU mode, and they're like, "I have the latest DeepSeek, or whatever. It generates one token every four seconds, but I have it at home."
(00:27:39):
And for certain kinds of hobbyists, having the real thing at home is fine, and they don't mind. It's like they really got to touch it, have it on their computer. They don't mind that for that particular spectacle that it was 100x slower -
CW (00:27:54):
Yeah.
AS (00:27:54):
- than the one that's half the size that actually does fit on their NVRAM.
CW (00:27:59):
But if I had half a terabyte of RAM, I would sell it and buy a house right now. Another house. You know.
EW (00:28:04):
Okay. I would like to replace myself with a shell script. I have plenty of podcasts, blogs, Git repo, talks I've done, my book. So I want to just become an expert system and then go to the beach.
AS (00:28:25):
Well, I would twist it a bit to be -
CW (00:28:29):
This is an entirely fictional scenario, any clients who are listening.
EW (00:28:31):
I'm still here. I am human right now. I am very, very human.
AS (00:28:35):
Well, do something about that. In my line of thinking in the last few months, I have this word that I've used, transagency, mixing this idea, there's sort of human agency. Like what can you do to achieve your purpose in life, and be fulfilled, and so on? Are you blocked from that, or are you capable of doing it? And then there's this other related word like agents, like what do the autonomous agents do on their own?
(00:29:06):
And it feels like those things, even though they have a lot of letters in common, feel almost totally disconnected. And so this idea of transagency that I use is the sort of human kind of agency you get through leveraging these things, that whether they're autonomous or not, I mostly don't care. But I think of this as building tools that kind of externalize your thinking.
(00:29:38):
Suppose I want to upgrade some software in this service that I'm running for the university. The first time I did a software upgrade, there were a lot of steps. I went through it, and then I captured that in a playbook that my agent can follow.
(00:29:52):
The steps are a little bit different every time, because new features land. And sometimes it's just a completely automated, unattended upgrade. Sometimes it's like, they added a new feature that partially replaces something I made custom. Sometimes it can be adapted. Sometimes it's not.
(00:30:06):
So it's not full automation like, it's just a shell script updated, but I have a playbook that can run mostly without me. But if it gets to a state where it's like, this deserves your attention, it'll stop at that point. And I'll see a little notification, and I'm like, "Oh, that's fine. Let's switch to theirs now." And then it continues the rest of the process without me.
(00:30:25):
So it's not replacing myself or yourself with a shell script, but having something that remembers your process and can continue the boring parts, the routine parts without you. But then you're immediately present and involved when it was revealed not to be what your automation assumed was the case.
EW (00:30:48):
Okay. So building a tool that will help me do the things I need to do. I guess that's fine. I guess that's a good starting place. I do tend to use some of the bigger LLMs. I use them basically to make badge scripts, other shell scripts. But then again, I expect it to handle the one-offs, because it's smart. But how would I go about making my own model?
(00:31:27):
I remember I took Udacity's autonomous car. I've done DNN training. I understand convolutional neural networks well enough to sketch it out. And I know that I need a ton of data. And then I may need the data to be labeled, I may not, depending on all that. So I understand that part of AI, but I don't really understand how I get to a model or an LLM-type model. That's very different.
AS (00:32:01):
I think the L in the LLM, the largeness, which in my mind I associate with, a model is large if it doesn't fit into your computer, whereas the ones that fit into your computer, I call them SLMs. Just almost like you might say, "What's the difference between a computer and a PC?" It's like, "Is it something I can carry around? Then it's a PC or something."
EW (00:32:19):
Oh, well then I want a TLM, a tiny language model.
AS (00:32:22):
Yeah. Does that mean sort of a phone-sized one? Or how tiny is tiny?
EW (00:32:27):
We could start with phone-sized.
AS (00:32:29):
Okay. Okay.
CW (00:32:30):
That's still pretty big now.
EW (00:32:32):
It is still pretty big.
AS (00:32:33):
Well, let's say it's -
EW (00:32:34):
I mean, the Raspberry Pi is actually where I want to go, but phone is good.
AS (00:32:38):
We could pick a number like two gigabytes, and maybe at a reasonable quantization, maybe a two-billion-parameter model. And then we could say, what are the off-the-shelf base models that are of that size that other people have found to have somewhat relevant capabilities? And then you can say, "I want to fine-tune that for my purpose."
(00:32:59):
Or maybe I don't specifically want to fine-tune. But I want to make a system that has the base capabilities of this two-billion-parameter model that someone else has and a lot of my project specific or personal cultural knowledge.
(00:33:13):
And a lot of the way to immediately bring that knowledge to a model is just to have it in context or have text files that I can read. There is this idea of context distillation. This is the same sense of distillation when people talk about model distillation of like, "I want the capabilities of the big model and the small model."
(00:33:35):
Context distillation is saying, I want the capabilities of the small model plus context in just the small model. Essentially, I want to cram all the information that would've gone into maybe a 10-page-long prompt or something like that.
(00:33:49):
And it works by, you have your model, let's say the small one, do whatever task you wanted it to do with that 10 pages of extra context. You record the history of what it did with all that context available and use that as the training data for fine-tuning your model, to say the same things without that prompt present, as if that knowledge had been internalized.
(00:34:13):
So as a byproduct of using it when the knowledge is provided in just plain text form, you build up this very situated data set of what it looks like to use it if this knowledge was built in. But at the time it's constructed, it's not built in. It's sort of given upfront as a huge 10-page prompt upfront.
EW (00:34:37):
What about a thousand-page prompt?
AS (00:34:39):
Then it comes down to, I guess the size of the context window supported by the model. When models didn't have very big context windows, the only way to get information into them was via various kinds of direct training.
(00:34:56):
But now that some models, including some that you could probably run on your laptop, have a hundred thousand tokens or a million tokens of input, that can be hundreds and hundreds of pages if it was just plain text. I don't know how to convert tokens to pages, depends on the formatting and other things like that.
(00:35:16):
But the way that many models in the last year or so have these huge context windows means that from most of the things that you would've reached for training in the past, you can now just shove it into context upfront. Your model will run slower, because every output token is potentially influenced by all those things.
(00:35:38):
But then this creates a record of, ...you have these logs of how it should behave as if that knowledge was built in and later you can decide, "Do I want to fine-tune this thing to operate without the giant prompt?" Or maybe it's fast enough to just never fine-tune it and just keep that giant prompt around.
(00:35:59):
And a big reason to do that is that there's so much churn in the models, that maybe it's better for you to carry your prompt to the next generation model, than to spend a lot of work fine-tuning this model that just won't feel interesting or ineffective in six months.
EW (00:36:21):
It sounds to me, just trying to make this into a metaphor I can think about, I can add the context. And kind of like in Python, I can add the context and it's live. And the more I have, the more it's going to get a little slower, or I can compile it in.
AS (00:36:43):
Yeah, ...imagine you're using some interpreted programming language like Python. You could have a lot of your functionality implemented in high-level Python, and somebody could say, "Well, I'm going to go rewrite it in C and bake it into the actual interpreter core."
(00:37:03):
From the outside, other than runtime, people probably won't notice the difference between those. The analogy is not perfect, because things like context rot and how things don't pay attention to everything in context. But there are many things where in terms of avoiding premature optimization, like, "Let me go rewrite it all in C or in Rust," is probably not the right choice for many projects.
(00:37:27):
And you should have left it in JavaScript, or in Python, or in whatever interpreted mode. Especially since, if for some reason there were new Python interpreters coming out all the time, your high-level source code is portable to those new versions in a way that if you had built it into the core, you'd have to redo that work again when a new base model came out.
EW (00:37:49):
But if I wanted to miniaturize it, I probably do need to bake it into the core.
AS (00:37:54):
I don't think so. Let's say that hundred pages of extra knowledge you wanted to shove in there is actually super tiny compared to the size of the model itself. It might be a hundred kilobytes versus your two-billion-byte model.
(00:38:13):
So the size gains from effectively inlining that knowledge are not very big, just because the models themselves, if we're imagining this two billion-parameter one, is much bigger. It's a little bit tricky, because we're not just talking about the size of the text, we're talking about the size of the encoded version in the key-value cache.
(00:38:39):
But again, I think the size of those things compared to the size of the model make it that, if your models are sufficiently big, it's usually not the best strategy to try to fine-tune things into the model. Unless you were confident you were going to have millions and millions of people using it, and we really need to get that last 1% performance by not processing the prompt each time, which can also be cached, which you would do in a high-volume-serving environment anyway.
EW (00:39:13):
You said the word token.
AS (00:39:15):
Yeah.
EW (00:39:16):
What is a token?
AS (00:39:19):
I guess I tell people a token is a word. It's not money...You wouldn't ask people how many words did you burn this week or something like that, because they'd be like, "Wait, did you mean writing words or reading words? Those are totally different activities."
EW (00:39:38):
Yes.
AS (00:39:38):
Or reading a word that you've read before, like skimming a document, is much cheaper than reading a fresh new paper for the first time...These days people use this word token as if it were a unit of money, because sometimes you pay for them.
(00:40:00):
But I really want to distinguish output tokens, which are things generated by the model one word at a time, let's call those written words. Like word as a unit of measure of text or whatever the thing it's processing. And then you can talk about input tokens, and it matters whether those inputs are cached or not.
(00:40:19):
Think of that as familiar versus unfamiliar text. And input and output tokens cost different amounts of money by maybe a factor of 10 in most environments. And there's even different costs for cached versus uncached input tokens. So when people say, "I've burned a billion tokens on this project," I imagine they're just adding all these things together.
(00:40:42):
It would be like if you did a project, and somebody said, "Yeah, that was a lot of work. That was like a 50,000-word project." And then you could say, "Well, maybe there's some sort of typical balance to reading and writing that you do in a project."
(00:40:55):
And you could sort of back out of that an estimate of what was generated versus what was read, but collapsing it all into a number of tokens, a number of words is often pretty misleading. But it's not far off to just say a token is a word, even though there's a not quite one-to-one conversion factor.
EW (00:41:14):
But does that mean that a token I use on my mini-me model is equivalent to a token used on the latest Gemini model? They're both words. If I ask them, what do I do next?
AS (00:41:28):
Yeah, yeah.
CW (00:41:29):
Not all words are worth the same as other words.
EW (00:41:33):
Sesquipedalian.
AS (00:41:35):
I guess this is where it comes down to, a word is a unit of text for the purposes of storage. But also if you use word as a unit of labor, and you're thinking about the labor of reading a word, the labor of writing a word, it really matters who is the laborer. There might be people who read things and don't get much out of it, or they're very sloppy in their writing.
(00:42:00):
So they might have written a high volume of words, but you don't really find much value in the result of their writing labor. And I think that one way to say there might be better or more serious writers, or writers who are more thoughtful about it, ...maybe you could say somebody doesn't write very much.
(00:42:20):
But what they wrote was really important. It might make it look like there were fewer words or fewer tokens being processed, but there was a lot more valuable activity going on.
(00:42:31):
And I find this useful for thinking of the models that just chat with you versus the ones that go and use tools. Imagine you're talking to someone on the phone, and when they're talking to you on the phone, they can't do anything else.
(00:42:46):
You can talk to them, and they can talk back versus they can go use a computer to do other things. And you might on your conversation say, "Oh, I had a 500,000-word conversation." But did you spend those words just talking to someone, or did you spend some of those words calling a different person or looking things up on Wikipedia? There's much more wise ways to spend those words than only talk.
CW (00:43:08):
That makes me wonder how these models are tuned, because you mentioned reasoning models, which I think you said something about yapping before answering.
AS (00:43:18):
Yeah, yeah, yeah. I use yap to refer to the self-talk before the final response.
CW (00:43:24):
It seems like it would be, to the companies that are selling tokens, to their benefit to make the models as chatty as possible during those reasoning phases, which most people don't look at, right? Where's the efficiency? I mean, you can be more efficient in your language, right, and these are language models.
AS (00:43:44):
So you want to know what are the incentives to get them to be more concise, especially if they currently get paid per token?
CW (00:43:51):
And we can see this with chat models now. There's stock phrases they use -
AS (00:43:54):
Yeah.
CW (00:43:54):
- when they make a mistake, or when they're correcting you, or something. And to my mind, many of those are just a waste of words.
AS (00:44:03):
Yeah.
CW (00:44:03):
You could say that in two words, but you said a whole sentence about -
EW (00:44:07):
How abjectly sorry you are that you cannot help me.
CW (00:44:09):
Whatever, yeah. Something like that.
AS (00:44:11):
I think even though the model providers get paid per token for outputs and the outputs are the tokens that they get paid more per unit of word, the fact that a lot of the yapping or the reasoning happens before the response, feels like there's a strong incentive to, if you want your service to feel snappy, you should train it to be concise.
CW (00:44:35):
Right, right.
AS (00:44:35):
Because it gets people to their answer experience, because in some sense, even though at the API level, maybe people pay per token, they certainly don't want to consume per token. It's not that if your thing used twice as flowery language, you appreciate it twice as much.
(00:44:55):
In fact, you maybe appreciate it less than the concise version. And this is why I kind of enjoy the experience of when you run things on your laptop, you're not so carefully counting the tokens -
CW (00:45:09):
Right.
AS (00:45:10):
- and thinking like, "Was that one particular token worth it?" You might be thinking how much energy was going into this thing. But then you have to think, "Well, it takes some energy to run my GPU to generate the tokens, but also it takes some energy to keep the bright backlight shining at me."
(00:45:25):
And if I'm done with my thing sooner, and I can close it, I'm actually okay with it having spent more energy per word to get me to being done so I can close my laptop and do the next thing. And then the whole thing is asleep. I don't remember. Did I answer your question about the incentives?
CW (00:45:44):
Yeah, no, I think you did. There's a tension between responsiveness and just spending a lot of time -
AS (00:45:51):
Yeah.
CW (00:45:51):
- being loquacious or whatever.
AS (00:45:54):
Or maybe it re-highlights the value of saying, if we say one token is about one word, we don't really care about the words. We care about the labor or the benefit that comes from those words. Maybe you put a certain amount of work into writing things, and I get more or less value out of it in the labor I do of reading them.
(00:46:15):
Maybe we should have been talking about labor all along. But we've used these units that make it seem like it's a kind of money like a coin, or that it's all about the text, not the effect the text had on the system, or the user, or something like that.
EW (00:46:30):
So what's BayLeaf?
AS (00:46:32):
Excellent transition.
CW (00:46:33):
Abrupt.
EW (00:46:35):
Oh no, it's related.
AS (00:46:37):
No, it is totally related, because I wanted to talk about the human dimension. BayLeaf is the name for...an AI service that I run for the university. When I say I run it for the university, it's sort of like, "I made this for you." It's like a gift, but maybe you didn't ask for it.
CW (00:46:58):
Like a puppy.
AS (00:46:59):
The university did not want an internal AI service. They probably imagined that eventually they would buy big name services like Gemini, and OpenAI ,and those things. And I thought there ought to be a different shape, but something that was credibly sort of feature-complete equivalent, so there could be a way for the university to say no to Gemini, to say, "We have Gemini at home. And we like ours better." But I had to make that.
(00:47:30):
So I haven't trained any models for BayLeaf. BayLeaf is mostly just a cobbled together collection of big chunks of open source software, open-weight models, or inference provided through a competitive marketplace of people who run open-weight models. So it's a single person project, but for the people in my campus community, I can give them a link to something that looks like ChatGPT.
(00:47:57):
They sign in with their UC Santa Cruz account. They get to a page where they can chat. Their old conversations are on the left. They can find the past ones.
(00:48:05):
Something it has that the big name providers don't is, I have the ability to make course-specific models, or maybe model's the wrong term. It's the same underlying model, but there is a course-specific prompt and a bunch of course-specific tools to read from our learning management systems and so on.
(00:48:23):
So when I give my students a link to the course-specific agent or character, it knows that they're a student. It knows what class they're in. It can read the syllabus for them. So if they're like, "What was the late policy?" The student can go read the syllabus, but that doesn't often happen.
(00:48:40):
The chatbot can read the syllabus and say, "Well, on the syllabus it says that the late policy is 5% off or whatever." And to the degree that I can plug it into things like GitHub, it can then go look at the code that they have on GitHub, and compare that against assignment requirements, and be like, "It doesn't look like you're ready to turn it in."
(00:48:59):
You didn't do this step. Here's a clue." And instead of it offering to say, "Let me do it for you to pass these conformance tests," because it knows that they are a student, it knows that I have written a generative AI policy for the class. It has all the context of the course. When it offers to help, it offers to help in a way that was influenced by me authoring that experience, me as a teacher, separate from me as the system operator.
(00:49:24):
Although I am the biggest user of my system. Half of the courses that have used BayLeaf have been my own courses, but not all of them. But another important part of the BayLeaf platform is that it offers API-level services so that people can run desktop apps. I don't do that just for convenience. Part of it is I want somebody to say, "Well, I really love using Claude Cowork. Do you have that?"
(00:49:54):
I want to say, "Yes, we have Claude Cowork at home. It looks the same, but it's even a little bit better in certain ways." The reason I have people do that is that the conversation data, instead of being stored in my database where I can read it, I want there to be a plausible setup where I can't read their data.
(00:50:12):
I don't want to just reproduce in-house enshittification, where I collect all their data, and I gain insights from it. I make new features for them. I want it to push more and more onto the individual users, that they are consumers of pretty low-level APIs of very interchangeable models, so that if I decided to turn off BayLeaf, they'd be like, "Weirdly, BayLeaf has turned off."
(00:50:35):
"How do I get something that's like BayLeaf?" You just use the same upstream providers that I do. And all these things are sort of protocol interchangeable. I guess that's what's happening at the user experience level, but there's maybe more of a political dimension to BayLeaf as well.
CW (00:50:50):
Right. Yeah.
AS (00:50:51):
I describe it as a counterplatform. And on the landing page for it, there's a video of me giving a guest lecture for a class and the headline is "Anarchist AI Infrastructure." BayLeaf is being positioned as something that is purposely too politically charged for the university to ever just absorb and bring in-house. But it's also positioned to be something that is very clearly not just a clone or a renaming of what the big name companies provide.
(00:51:32):
So it's trying to offer people who are highly critical of the big name offering things to say this one is qualitatively different, but if you're a fan of those things, it is from a user experience similar enough that you can use it as a credible replacement.
(00:51:49):
And the hope would be that the people who had an experience with generative AI two years ago be like, "This one smells different enough that I'll come to it with fresh eyes and have a fresh experience with it," so they can develop a new relationship to generative AI. But one that has a story, a true and credible story for how it is not just Claude under a new name.
(00:52:17):
And the hope would be that the university decides to adopt 80% of it. But because it's not an official university service, I can just add whatever experimental things I want, and I don't need to get approval for it. We don't need to have budget reviews and stuff like that. The budget is just what I'm willing to pay for myself, which is about $200 a month in various IT services.
(00:52:40):
And that fits into my hobby of self-learning and sort of offering it to the campus community. It's still pretty cheap, especially if people don't use it. It's not that I need tons of engagement for my project to survive. In fact, the more people who engage with it, I lose money there.
(00:52:59):
I want to change some minds and get the university to say, "Yeah, we don't need to make a deal with OpenAI," or, "We don't need to make a deal with many of these other companies that would take us down the enshittifaction route." Because we have something that is, for the people who demand it, it feels equivalent.
(00:53:16):
And for the people who reject it, it feels like a plausible rejection, a way of, if you hate OpenAI, how do we actually make it so OpenAI cannot sell to one more university? It's trying to be an answer to that.
EW (00:53:35):
But I don't attend UC Santa Cruz.
AS (00:53:38):
Oh, yeah.
EW (00:53:38):
How do I get my own BayLeaf?
AS (00:53:41):
Yeah. So the BayLeaf project is sort of hyper-targeted at UC Santa Cruz, but sort of only as an example. The hope would be that when people look, "Oh, what Adam is doing with BayLeaf for UC Santa Cruz, I could do that for, I don't know, my school, my church, my company, my city," or whatever.
EW (00:54:02):
My house.
AS (00:54:03):
Or your house. Yeah. At some level, BayLeaf is just sort of the campus-scale version of what my family has at home. My wife uses it. My daughter uses it. They don't go to the bayleaf.dev website. There's a constellation of services that are almost the same.
(00:54:17):
So I'm sort of dogfooding all the same open source projects and BayLeaf is sort of the cleaned up community-scale version of what I have as my friends and family service. And so the building blocks are things like, there's a package called Open WebUI, which is something that I say, it's only about as difficult as WordPress is to set up, which is to say, easy to get started.
EW (00:54:42):
You're not selling it here.
AS (00:54:47):
It is something you can self-host. Open WebUI, as the name suggests, is just the user interface. It does not provide the LLM inference. You need to plug in that from some other service where you plausibly pay per token, input and output for that. If you dug into that project, it's maybe a million lines of Python and JavaScript code.
(00:55:07):
It's maintained by community. I've made contributions to it, particularly to make it more accessible to screen readers and so on. So it's easier for universities to adopt or any other organization that has accessibility requirements. So Open WebUI is one of the components that I bring in. You could run Open WebUI, there's like a desktop app version of it, but I think of it as software that is sort of intended to be used by groups.
(00:55:35):
Another thing in my constellation is the OpenCode command-line coding agent harness. And if you showed somebody a screenshot of OpenCode and a screenshot of Claude Code, they'd be like, "Well, they both look like terminal coding agent things. They have slightly different colors. They do almost all the same things." But most people are not living in the terminal anyway.
(00:56:02):
So yet another major building block is this desktop app called OpenChamber, which you can kind of think of as being the desktop wrapper around the OpenCode command-line core. And if people have ever used the ChatGPT desktop app or the Claude desktop app, OpenChamber is sort of the open source answer to those.
(00:56:27):
But it's possible to plug these things together so that you spend almost no time in the terminal after initial setup. And you have a desktop app experience where assuming, you're using providers that are zero data retention providers, the only copy of your chat data is in a database on your computer, not being accumulated somewhere, whether it's used for training or evaluation, or other things like that.
(00:56:53):
One of the things that I like about both OpenCode and OpenChamber, they're essentially the same thing when plugged together, is that there's this idea of projects. Instead of just having a loose chat with all the material in your computer, you pick a particular folder, particular path in your file system, and say, "This is the root of my project." And you can have many of them. And when you start conversations in that project folder, there's this implied permission thing.
(00:57:24):
You can do what you want with what's in that folder, and you have to ask me for permission anytime you reach out of it. So it sort of respects the effort you've already done to organize your file system, to write documentation down in text files and other formats. And if you're really doing some wild system administration stuff, you could dangerously dare to have a conversation in your home directory where all that stuff is in scope.
CW (00:57:47):
Yeah, root.
AS (00:57:49):
There are some times when you're like, "I want the agent to help me with this sort of systematic reorganization." I almost never do that, but there have been a few times when I liked that it wasn't always super narrowly scoped.
CW (00:58:01):
Sure.
AS (00:58:01):
But more often I'll have projects attached to various Git repositories that I'm using. And then that's an encouragement to have the Git repository not just contain source code, but a lot of my other like, "Where is this project going? Where is it going politically? What is our messaging for different groups?"
(00:58:18):
If that's written down in text files in this directory, and this is what I've done for the BayLeaf software repository, I can then have conversations about political messaging in the context of that Git repository. It's able to participate both in the like, "Okay, let's debug this feature," but also, "Let's reword the landing page to incorporate this blog that came out from someone else recently."
EW (00:58:41):
Okay. How do I attach it to an LLM? How do I attach it to my mini-me LLM?
AS (00:58:47):
Oh, okay. So in either Open WebUI, or in OpenCode and OpenChamber, there's often somewhere in the settings, a panel called connections or providers, where you give it the URL and the API key for your LLM inference service, which doesn't have to be a third party. It could be something running on your own computer.
(00:59:10):
And so if you had custom trained or custom fine-tuned your model, maybe you took one of the Gemma 4 models and you trained it to do whatever person-specific thing you wanted it to do. You would then customize these front ends to use that model when they need inference. And then in any of these surfaces, your model would be at work there.
(00:59:37):
Most of these apps allow you to configure many different providers. So you can say, "Well, in my dropdown list, I have the ones that run on my laptop..." I have so many different providers plugged in to sort of taste test these different things, including local ones.
(00:59:58):
There was something I was doing with the BitTorrent protocol that one of the major corporate models didn't want to do, because it was too movie-piracy adjacent. So it was convenient to have an on-device model that had been fine-tuned to never refuse anything, to be like, "Help me make the protocol parser for this thing related to the distributed hash table."
(01:00:24):
But I didn't even have to start a new conversation when I got to the point where let's say Claude or whatever was refusing. I could say, "Use the on-device model that isn't afraid of the BitTorrent protocol." It will do some steps. Then I can switch back to Claude. And at that point, the Claude models think that they had complied, and then they don't complain about the thing anymore. Yeah. Chris?
EW (01:00:50):
Chris was being so good. What's OpenRouter AI?
AS (01:00:55):
So OpenRouter is..., there's a bunch of different companies that provide pay-per-token access to LLMs. DeepInfra is one. Nebius is another one. But the trouble is that each one of these providers has a different collection of models that they offer. When new models come out, some providers are happy to launch those on day one. Some get around to it a month later. Some are unreliable.
(01:01:21):
And so OpenRouter is a middleman that takes a 5% cut to say, "We support all these models. Send a request to us. And we'll send it to one of our backend providers that is online right now that has it...If there are many models for a given model, we'll send it to you, the cheapest one, the fastest one," or whatever preferences you have. So it's a middleman aggregation layer.
CW (01:01:45):
Token broker.
AS (01:01:46):
Yeah. You can call it a token broker.
EW (01:01:49):
So I could connect Open WebUI to OpenRouter AI, and then I don't have to worry about models so much directly.
AS (01:02:01):
Then you wouldn't -
CW (01:02:02):
You still want to choose a model.
EW (01:02:03):
You still want to choose a model, but Open Router AI would let me choose a model, and I wouldn't have to worry who is hosting that.
CW (01:02:10):
Fair.
AS (01:02:12):
To make a food delivery analogy or something, you could order from the restaurant specifically, or you could order from Uber Eats or something. And they'll offer you food from all the area's restaurants or something. This is a poor metaphor, but there are good and bad things about having companies like OpenRouter out there.
(01:02:31):
It is yet another middleman that could be taking your data. They do take some extra money, but they do make it so you don't need to have agreements and accounts with each specific provider in order to get the model coverage that you want.
(01:02:46):
I think for most people, getting an account, I don't mean a-pay-per month subscription. But I mean like a pay-per-million tokens usage account with a particular provider, I used to be a direct customer of DeepInfra for two years.
(01:03:04):
They had a pretty wide model catalog, and they were pretty reliable. But then there would be some times when they didn't have a model I want, or they weren't reliable. And at the moment I wanted to reach for a second provider. I'm like, "I'll just switch to OpenRouter."
(01:03:19):
And then I get these nice charts across all the different providers, all the different models, what has been my usage over time. So with convenience, or in exchange for a little bit more cost and a little bit less privacy, there's a lot more convenience. But you can always cut them out by making a direct protocol connection to whoever they were getting the token generation from or your own local computer.
EW (01:03:42):
Assuming you are willing to go through that extra little bit of startup hassle.
AS (01:03:46):
Oh, you can tell your agents to set it up directly.
CW (01:03:51):
Just give them your credit card. But 95% of all these services just end up at Amazon, right? At some point it's all AWS.
AS (01:03:59):
Oh no, it's not all AWS.
EW (01:04:00):
Some of them are Azure.
CW (01:04:02):
Less than 5%.
AS (01:04:03):
I think there's this category of the hyperscaler clouds versus the neoclouds. And there's actually by volume a lot of compute happening in the neocloud space, which is distinct companies that actually have different hardware. So it's not ultimately -
CW (01:04:18):
Okay.
AS (01:04:19):
- AWS. AWS has a subpiece of it, because Amazon does everything. There are many providers where, in the same way that you could get a virtual machine through EC2, there are ones where you can say, "Give me a virtual machine connected to a GPU, and I'll run LLM inference myself."
(01:04:40):
Or if the only app you were going to run on that virtual machine was an LLM inference engine, you can say, "Just give me that as a service, and I don't want to do the system admin for it."
CW (01:04:50):
Yeah.
AS (01:04:51):
So it doesn't necessarily go to AWS. It could go to another, I don't know how to say, IaaS, Infrastructure as a Service out loud, IS or whatever. But I think most providers are themselves not owners of the GPUs. They are getting GPU cloud hosting from one layer below, and then they sell you token generation service above that.
(01:05:17):
But as with all these things, you can sort of cut out the middleman many steps and say, "Well, my computer has a GPU. Surely I can do it." It's like, yes, you can, but to the limits of now your laptop gets really warm, and it gets uncomfortable. And it is sometimes nice to be able to use these things without your laptop getting uncomfortably hot.
EW (01:05:37):
A listener, ExplodingLemur, who's been on the show, picked up some old AMD V620s for semi-cheap, a 5-slot PCle switch with slimSAS ports to uplink to a computer. I really should have given you this as a text.
CW (01:05:58):
He bought a box to put some GPUs in. He bought some old GPUs that are good for inference with a lot of VRAM.
EW (01:06:03):
How much performance can he expect to get out of 12-gig VRAM and five-year-old GPUs?
CW (01:06:10):
A lot. It's VRAM.
AS (01:06:12):
Yeah. It's almost like when people say, "Oh, you need GPUs for LLMs." It's actually, you need VRAM for vast inference. And what's further complicated in that case is that depending on how the different GPUs are wired together, it may be that they don't each have fast access to each other's VRAM.
CW (01:06:35):
Right.
AS (01:06:35):
In which case, even though in aggregate there might be 128 gigabytes, if each one is only 16, your maximum speed is limited by what you get. There are complicated ways to spread it across devices. And I haven't done that, because I'm not GPU rich in that particular -
CW (01:06:49):
Right.
AS (01:06:49):
I've got a Mac laptop that has the one integrated GPU and a lot of unified RAM, which has given me a taste of what it's like to have a weak GPU and a lot of VRAM. And that has let me play with a lot of stuff on device that a lot of people who don't have that integrated RAM set up are much more limited by -
CW (01:07:10):
Because a GPU with a lot of RAM is very, very expensive.
AS (01:07:13):
Yeah, yeah.
CW (01:07:14):
Whereas a Mac, it shares the RAM with the GPU and the system. And you can say, "Well, give me 48 gigs of your 64 gigs for the model."
AS (01:07:22):
Yeah. These things are actually more I/O intensive -
CW (01:07:24):
Yeah.
AS (01:07:24):
- than they are compute intensive, but you wouldn't guess that from the public discourse. They would think that it's all about the compute. It's like, "Well, it's correlated with the compute, but it's actually bottlenecked on this other thing."
CW (01:07:35):
Yeah.
EW (01:07:38):
LLMs seem to be graded by how many billion terms they have, which isn't really work, because you can get smaller versions that are just as effective. How do you -
AS (01:07:52):
I think at any given snapshot in time, the bigger ones are almost always better. However, if you make a snapshot across, or a slice at a given scale across time, the capability radically increases over time. Last year's four-billion-parameter model is nothing compared to current, or what am I trying to say?
CW (01:08:19):
At a given parameter size, the capabilities have still been improving.
AS (01:08:23):
Yeah. Yeah. I'm getting lost in trying to gesture in 2D -
CW (01:08:28):
Yeah.
AS (01:08:28):
- in the audio medium, so I won't try with that. But I think somebody has a particular machine that has many, probably many medium-sized GPUs, and they want to know what can they do with it.
EW (01:08:43):
Yeah. Or my follow-on question to that was, how do we even identify what we're looking at? I mean, what scale are we talking about? Which one should he download?
AS (01:08:57):
Oh, I think the heuristic would be figure out what is the largest single chunk of VRAM that is accessible. Let's say that's 16 gigabytes or something like that. And then you could say, "Okay, what is the highest number of parameter model that fits in 16 gigabytes?"
(01:09:19):
Without getting lost in quantization too much, let's just say that one parameter is one byte, which is true at the 8-bit quantization level roughly. So this is saying that they could run a 16-billion-parameter model. And then you could say, "Okay, what is the newest 16-billion-parameter model? That will give me the best intelligence within the space that I'm allowed to give it."
(01:09:44):
And it gets squishy in some ways. It's like, "Well, is it better to run a highly-quantized big model or a less-quantized small one?" Usually you don't get to make that direct trade off, because usually one of those models is newer, in which case you should just pick the newer one.
CW (01:10:01):
And for comparison, a frontier model would have how many parameters?
AS (01:10:06):
That's the trickiness. They don't tell you, but people are guessing that they are in the low trillions, somewhere between 1 and 10.
CW (01:10:17):
So that's a couple orders of magnitude.
AS (01:10:19):
Yeah. And to compare that with, for some of the things that when ChatGPT did it two years ago, people thought were super disruptive, those things you can now reliably do with something that's multiple orders of magnitude smaller.
CW (01:10:32):
Okay. Okay.
AS (01:10:32):
Sometimes there's this metaphor I have of, if it were the case that one billion parameters weighed one pound and LLMs were vehicles, you could say there are some vehicles that are like, it's a one-ton work truck or something like that. Then there's other vehicles that might be like a motorcycle that's maybe 350 pounds, or I don't know how much a motorcycle weighs.
(01:10:57):
Then there could be a light bike or you can think of a shoe, which might be like a one-pound vehicle. And except for this thing where the vehicles get better over time at a given size, I think saying one billion parameters is one pound gives you a sense of these orders of magnitude, to say there are things that some people are using giant heavy work trucks for, that you're like, "You know what? I don't work on a construction site."
CW (01:11:25):
Don't need that.
AS (01:11:25):
"Do you have an e-bike? Do you have a skateboard? What can I do with the shoe that's in my pocket, or that's on my foot," or something like that. And the surprising thing that the skateboard of this year outperforms the work truck of last year is sort of the fun part, but that this gives you this heuristic of mostly shop by size and pick the biggest new thing.
(01:11:47):
For some reason, shoes are getting better and better over time. Just say, "What is the size of vehicle that I want to go with?" And then say, "What's the newest vehicle of that size?" Roughly.
CW (01:11:57):
I think it's a good metaphor, too because it's probably diminishing returns at some level of, "Okay, we have a trillion parameters. Well, great."
AS (01:12:03):
Yeah. Yeah.
CW (01:12:03):
I mean, is that better than 100 billion? I don't know that anybody can really quantify that. And the efficiency, like you say, somebody has a Ford F-450 that they drive around to the grocery store. They're burning a ton of gas.
AS (01:12:17):
Yeah.
CW (01:12:18):
For no reason.
AS (01:12:19):
And I think a lot of the discussion of LLMs and energy use come from this tendency to only talk about the most extreme top end.
CW (01:12:27):
Yeah.
AS (01:12:27):
When if people are actually able to sort of shop among different sizes of models, people might be like, in the same way that people are like, "I'm proud to bike to work instead of driving."
(01:12:39):
You could say, "I'm proud to use a hundred-billion-parameter model, a hundred-pound thing, instead of the thousand-pound or the 10,000-pound one. But at least the way these things are currently sold, they sell you some -
CW (01:12:51):
The biggest and the best.
AS (01:12:51):
The biggest and the best. They don't say how big it is, and it's always changing.
CW (01:12:56):
Yeah.
AS (01:12:56):
So you're leasing the biggest heavy work truck, rather than like in the open-weight space where things are less convenient, but you get a lot of choice. It's very easy to make these intermediate choices, and including pulling down the usage, just as multiple orders of magnitude, to say what I used to use the truck for, I legitimately can use the skateboard or something like that.
(01:13:24):
Now sometimes I use the animal metaphors of the elephant could do this, but what can the kangaroo do? What can the mouse do? Or something like that. And the fact that the mouse can do something that the elephant... The vehicle thing is a little bit easier, because at least you ride vehicles to achieve and reach destinations.
(01:13:40):
In the same way, you can do a recreational drive in your car, you can have a recreational bike ride, there are times when I guess I can say recreational LLM use. I want to learn about some new thing. Let's make a prototype of it. Somebody else had a YouTube video. Let's download the transcript of the YouTube video. Let's get it working on my computer.
(01:13:59):
I wasn't really trying to achieve a task. I'm like, "Let's go play and get new magical powers as a result of this playtime." And when I say get new magical powers, I often mean let's get text files on my disk so that the next time I want to play with this, we've got a playbook of how to do it the first time.
(01:14:18):
Whereas whenever you try out some new software, you could get lost for hours getting the right version of the dependencies and all the stuff that's not written down.
EW (01:14:25):
Yes, I hate that.
AS (01:14:26):
But if you can sort of stumble through it once with your agent, and then you sort of save this recipe, the recipe could potentially guide you. Maybe a new version of that thing comes out. Maybe they change their build system. There's a chance the recipe will take you 80% of the way through that new version.
(01:14:41):
Whereas if you had only kept the binary on disk, that's much more brittle than this more natural-language workflow version of it. But there are times that I do feel like I'm going for a recreational thing, or it's not so much about just the enjoyment of it. But I want to gain a new skill.
(01:15:01):
Me and the machine as one fused unit are learning through experience, but it's not machine learning like gradient descent, not dataset. It's different from just me learning. It's me and the machine, as in me plus my text files, as a fused unit are learning these new things.
EW (01:15:19):
Does using the open source models and system that you've built make you feel less dirty than using the big models? Because I could use that.
AS (01:15:34):
Oh, it does feel different to sort of run your own stack for these things. Certainly in the circles that I'm in, enshittification comes up all the time. So knowing that my thing does not depend on people's engagement to survive, it feels a lot different this way.
(01:15:57):
And in fact, that better, more wholesome feeling, it often makes it easy to justify like, "Oh, my thing made a mistake, but I know that's because it's two orders of magnitude smaller than the other one." So I'm willing to give up some kinds of comfort in order to get other kinds of comfort.
EW (01:16:22):
Yeah. Changing topics, which I'm going to do rapidly twice in a row, but first, aren't you a professor of computational media?
AS (01:16:32):
Yeah.
EW (01:16:32):
Like games, and music, and stuff?
AS (01:16:35):
I'd say...since the department is about 10 years old now, we're trying to figure out how to communicate to people what even is computational media. And at the time the department was created, it almost felt like, "Oh, computational media is just a euphemism for video games." And we've always been trying to tell people like, "No, no, it's so much more broader."
(01:16:57):
But then each thing we pointed to often was like, "Well, that was an existing area." And my thought in this current moment with all this AI stuff is, what's under the hype of AI stuff? All these new capabilities and this doom is like, it's just a new computer-based medium for expressing ideas between people...
(01:17:19):
You write down notes this week and then a month later past you has communicated something through the computer. And now this thing, you're like, "I used to remember how to do this." Well, your agent with your notes from the past remembers how to do that now. So I guess we're trying to say underneath all the AI hype and doom stuff is computational media, and it's kind of convenient that all this stuff grew.
(01:17:43):
People are taking it very seriously, too seriously in many ways, but it is a great example of a thing that's interactive. It's based on a computer. There's even audiovisual experience to it, but it's not a video game. And it should be some things that are authored by end users.
(01:17:59):
There should be indie communities of them. Like Hollywood movies only come from big name people through official channels. Think of it as maybe you can make an analogy if OpenAI is like Netflix, the big-name, proper-licensed provider, I want the long tail of weird stuff on YouTube.
EW (01:18:21):
I kind of like the NPR and PBS.
AS (01:18:25):
Oh, the sort of like, well, I guess NPR and PBS are getting -
EW (01:18:29):
Or institutional ones.
AS (01:18:30):
I also like institutional ones too, but we're in a time when those institutional options are getting less and less funding.
EW (01:18:37):
Yeah, yeah.
AS (01:18:37):
But a thing that I want to do, and I apparently didn't get enough time to do it this summer, is make contact with city library systems, to say that in the same way that people who don't have internet access on their own can reliably go to a library, and sit down at that library computer, use the library Wi-Fi from their phone.
(01:18:57):
AI services could be provided there too that didn't steal your data, didn't cost $20 a month, that weren't overwhelming, that were supported by and understood by the staff. There could be, whether it's national, or state, or city-scale support for going the sort of open source, open-weight route on many of these things.
(01:19:22):
And obviously I want schools to be able to do that as well. I like there to be an abundance of different ways, the way that you can get Wi-Fi at home. You can get Wi-Fi at the grocery store. You can get Wi-Fi at the airport.
(01:19:35):
It's not that we've figured out that there's one national level Wi-Fi network, and you sign in with your Social Security number. There are so many intercompatible systems that we feel that abundance, even though each organization provides it in a slightly different way.
EW (01:19:53):
And then changing topics again rapidly, do you have advice for people with ADHD?
AS (01:20:00):
Advice for people with ADHD.
EW (01:20:02):
You've talked about your struggles with it some on your personal site.
AS (01:20:05):
Yeah. I found that specifically I've connected back to LLMs and apps like OpenChamber. There are some things that I feel like it's hard for me to, I know it's important. I know I'm going to get in trouble if I don't do it, but I'll do anything. I'll do another layer of cleaning the kitchen. I'll sort the laundry. I'll scoop the cat litter instead of doing that thing that's important.
EW (01:20:32):
Have you ever tried to learn five different languages on Duolingo at the same time?
AS (01:20:36):
Yeah, yeah.
EW (01:20:36):
And what were you procrastinating?
AS (01:20:39):
I think I was procrastinating from some overdue grading, and I was trying to learn Mandarin, Vietnamese, Korean and something else at the same time.
EW (01:20:49):
Swedish?
AS (01:20:51):
Swedish was in the mix, because we had gone to Sweden recently for a trip. So that was just, yeah, that was a weird time. But specifically on the getting executive function things, there are some times when, there's lots of things I can do, but there's many things that I feel like I can't start. But sometimes I can just hold down the voice to text button on my computer and be like, "I have an email in my inbox from the thing about the insurance thing."
(01:21:18):
"We need to email those other people. Can you read our policy and figure out what is the right way to respond to them? Can you draft the email?" It's like, "Don't send it, just leave it in drafts. Yeah." And then I'm like, "Oh, okay. I started the project."
(01:21:30):
I had to dump enough of the task. But because I had already done the work to plug it into these things, if you just had a chatbot, it would be like, "I would love to help you with this letter, but I don't know anything else about what you're doing." So it really matters.
(01:21:45):
It can go read the insurance policy. It can go read the email. It can go write the drafts. I can start it in a folder where the right constellation of things are pre-given, and that has been a way for certain things that I know I would've put off for months.
(01:22:04):
And for some reason, the voice-to-text part of it as well, when I finally get that willingness to do the first three sentences, that has unblocked me a bunch of times. And there's many more dimensions to ADHD other than -
CW (01:22:20):
Right.
AS (01:22:21):
- inability to start things and many people face inability to start things for many other reasons. But I'm glad I now have a constellation of cool open source software that lets me make some of these super important but annoying things, it doesn't do the whole thing for me.
(01:22:37):
I wouldn't want it to negotiate with the insurance person directly or something like that. But it suddenly became a much more tangible thing when I'm like, "I just need to tweak this email. It has already gathered all the important stuff into one place."
(01:22:55):
And even if I end up deleting it, redoing it again, it's because like, "Oh, I wouldn't want the agent's email to go here. I'm just going to fix it." We had our e-bikes stolen recently, so we learned about insurance reimbursement stuff. So that's on my mind.
EW (01:23:12):
So before we let you go, do you have any resources for how to start my own open source LLM projects? How do I get involved with this?
AS (01:23:25):
How you get involved with it? I guess part of it is gaining some personal practice with these tools. And some of that comes from, even if you're using the big name corporate ones, you can ask them to help you set up these things, and they are perfectly happy to help you build their replacements and debug them.
(01:23:45):
And when they fall apart, they fall apart many times, get you back on board, so you can take risks to build things that aren't resilient, that are flaky and personal. So it's like if you have a little bit of help, you have the ability to jump into new spaces and make mistakes.
(01:24:00):
There is a subreddit called r/LocalLLaMA where there's a huge range of expertise there. There are people who just showed up yesterday. There are people who have been involved for years. But one of the things I get from that is enthusiasm to try new things, many people's opinions on new models when they come out.
(01:24:19):
So you don't need to try every one of these things. You can see other people reviewing it and throwing in their opinions. I check that twice a day.It's actually the only subreddit that I check, but those people will at least filter down the things that might be worth caring about.
EW (01:24:42):
Do you have any thoughts you'd like to leave us with?
AS (01:24:45):
This phrase that I use, the me in the machine or the you in the car, this thinking of you and the tools around you as a thing that could do more than just you can, and it's not because the tools did it all for you. But thinking of the you in the machine as a useful unit that you can go out in the world, gain experience, and all the particular software that you're using, which models, which harnesses, those things can sort of totally change around.
(01:25:13):
But the sort of you plus the notes you've accumulated from experience is very forward compatible. So you can sort of not bother to update what model and harness you're using for six months. Your text files will go with you when you eventually decide to update those, but it is useful to be using tools that get things to be written down on your computer as a result of the experience.
(01:25:38):
And that's like valuable text that is expressed in words you can measure in tokens. And importantly, that text, I don't know, it's a crystallized form of your labor that you can carry with you around over time that is actually largely independent of the model that it came from.
EW (01:26:02):
Alright, I'm relating this to a book. I mean, this accumulation of notes that become part of your identity, history, media? It seems like you could make a book from that.
AS (01:26:21):
You could. Or there's many people who say like, "Oh, it's really important to have a journaling practice," and you're like, "Well, I don't want to. I'm too lazy. I've got other stuff." But now it's really easy to have a journaling practice.
(01:26:33):
You can write in your system prompts, "Hey, I want to have written evidence of anything important that we learn together." And you go do something and you'll see it just go right to files like, "Today we figured out how to build that particular project from source. Today we learned that this driver needs to be this way."
(01:26:51):
And just as a byproduct of use, you, the extended version of you, will take notes on a lot of this stuff. And then after a while you can say, "Let's reorganize all that stuff," or, "Wasn't there a time three months ago when I did this?" And it will go look through the text files that, in that case, are almost entirely agent written, but they are written as a direct byproduct of these learning activities that you go on.
(01:27:15):
But it comes down to, did you ever tell your agent that you wanted to write down the experiences that cost two sentences to write into a preferences file?
EW (01:27:25):
I feel like there are a lot of things I don't know about how to use LLMs effectively.
AS (01:27:30):
I think someday we'll teach it in schools, or implicitly the way that most people know how to operate web browsers, and then secondarily, most people know how to operate the cloud services that you access through the web browsers, even though we don't have a "How To Use The Web" class in schools.
(01:27:51):
There is middle-school age, you just get enough experience being expected to do stuff on the web. And I think we'll get that sort of middle-school age exposure to use these things, but we're all new to it now. And some people are much deeper into it than other ones, only because they're two years in instead of one year in. But ten years from now, that's going to be a difference between are you ten years in versus twelve or something like that.
EW (01:28:19):
Our guest has been Adam Smith, UC Santa Cruz Associate Professor of Computational Media.
CW (01:28:26):
Thanks, Adam.
AS (01:28:28):
Happy to be here.
EW (01:28:30):
Thank you to Christopher for producing and co-hosting. Thank you to Kathleen and Joel for their help with lightning round, and thank you to our Patreon listener Slack group for their questions. Thank you for listening. You can always contact us at show@embedded.fm or hit the contact link on embeddedfm.
(01:28:47):
Although I have to admit, if you do right now, well, Adam's point about emails was very relevant to my current email box. And if you are requesting to be a guest on the show, and you're not an AI, go ahead and send me another email, because I don't really know about this set. There sure are a lot of them. And I guess now I'm supposed to have a quote to leave you with, but I don't. So Adam, complete one project or start a dozen?
AS (01:29:16):
Tens of thousands and start hundreds more within the starting of the other projects recursively forever.