380: Trending Toward Telepathy
Transcript from 380: Trending Toward Telepathy with Adelle Lin, Elecia White, and Christopher White.
EW (00:00:06):
Welcome to Embedded. I am Elecia White, alongside Christopher White. We're going to talk about art and technology today with Adelle Lin.
CW (00:00:15):
Hi, Adelle. Thanks for joining us.
AL (00:00:17):
Hi. How are you?
EW (00:00:20):
We're good. Could you tell us about yourself as though we met, I don't know, maybe at a Supercon-like thing.
AL (00:00:28):
Hi everyone. My name is Adelle Lin. I'm someone who is passionate about integrating the digital and the physical. I'm originally from Malaysia. I guess I would call myself a designer and an engineer. I've lived in Australia before moving to the U.S.
AL (00:00:43):
Professionally, I work on software development and machine learning projects at a startup. But with my own work, I try to draw on more personal experience of having negotiated diverse systems to create projects and artwork to help connect people with themselves and the spaces that they occupy.
EW (00:00:58):
Cool. We're going to do lightning round, where we'll ask you short questions, and we want short answers. And we may ask you "Why?" And "How?" But we're not supposed to. Are you ready?
AL (00:01:09):
Yes. And I can pass on some of the these, right?
CW (00:01:11):
Oh, yeah.
EW (00:01:12):
Oh, yeah. Always.
CW (00:01:14):
What is your favorite type of art?
AL (00:01:17):
I think, that's a hard one. Short answers.
EW (00:01:23):
It doesn't have to be. Nobody enforces the rules here.
AL (00:01:28):
I like art that's provocative, I guess, making stuff that makes me think about something I haven't thought about before, or exposes me to new ideas.
EW (00:01:37):
Best Animal Crossing villager? I hope they don't say one of the Jocks.
AL (00:01:43):
You are testing me there.
EW (00:01:46):
I mean, you could just list five. We're not going to be mad.
AL (00:01:50):
There's a penguin one, which is my favorite, but I've just forgotten their name all of a sudden. Friga.
CW (00:01:57):
What's your favorite communication method?
AL (00:02:00):
Face-to-face.
EW (00:02:03):
What is your favorite wireless communication method for processors?
CW (00:02:08):
That's also face-to-face. Usually.
AL (00:02:13):
I like the 900 megahertz radios.
CW (00:02:19):
I'll ask an easier one. Complete one project, or start a dozen?
AL (00:02:23):
Start a dozen, complete one.
EW (00:02:26):
Favorite fictional robot?
AL (00:02:29):
...I have a few. Major Motoko Kusanagi from Ghost in the Shell. Alita. The Dolls from the Dollhouse. I think I really like stories that explore the line between humanity and programming.
EW (00:02:46):
So you've worked in many different industries, investment banking, architecture, and now machine learning.
AL (00:02:57):
Yes.
EW (00:02:57):
Is that right? How do you choose which industries, and how do you go from there to here?
AL (00:03:05):
Yeah. I often ask myself that question. I guess I do think of life as a little bit of a choose your own adventure. And I learn by doing, so sometimes I get really curious about an industry, and I guess I take the time to immersively investigate it.
AL (00:03:29):
So I would say I kind of started from going from high level to a low level of working with things. And I think initially what I was doing with banking and consulting was...I felt more from a high level, already kind of working with businesses that already have been built.
AL (00:03:51):
And I was very curious about how things were made. And...I felt like at a young age, it was very hard to give advice to businesses before really even knowing that much about it myself. And so I started getting into design.
AL (00:04:06):
From design, I got really fascinated about kind of lower-level and learning how to actually build things. And I sort of ended up kind of teaching myself software and electronics, and I basically professionally found myself there. Right here. Right now.
EW (00:04:26):
You got a master's degree a few years ago, going back to school after not being in school for awhile. What was that like?
AL (00:04:34):
I've actually gone to school a few times. I have an undergrad. My undergrad was in actuarial studies, so it was math and statistics-based. And I have another undergrad in design.
AL (00:04:46):
And the masters that I did was at the Tandon School of Engineering, and it was kind of a hybrid of art and engineering sort of course. And...I think at the time it was sort of build your own kind of course curriculum. And I took subjects in data science programming.
AL (00:05:05):
And what I ended up doing my thesis in was looking at mixed reality interfaces and kind of designing for, I guess, sort of 3D interfaces and sort of virtual slash mixed hybrid worlds. And so I felt like that was sort of bringing bits of all my different backgrounds together.
EW (00:05:30):
Mixed reality like augmented reality or something else?
AL (00:05:34):
Yeah, I guess mixed reality was the term at the time, but it was using the headsets, which had an augmented reality layer on top of it. So you could still see your environment, but it spatially recognized everything.
AL (00:05:51):
So your virtual elements sat in sort of particular spatial coordinates or sometimes augmented reality. Well, augmented reality at the time was not necessarily tied to 3D spatial coordinates.
CW (00:06:06):
Oh, okay.
AL (00:06:07):
But...I feel like at the moment, everything I think has now been called extended reality or XR and everything's kind of hybridized.
EW (00:06:19):
When are they going to start calling it unreality or surreality?
CW (00:06:23):
Well, when they decide that they need another term. So the Microsoft HoloLens -
AL (00:06:28):
Yeah.
CW (00:06:28):
- kind of thing would be an example of what you're talking about? Okay.
AL (00:06:30):
Yeah. In particular, I worked at the HoloLens for my thesis project.
EW (00:06:35):
I've used a few of the headsets, and I was super, super excited about them when we got our HTC Vive. And then we got the Quest, and that was really cool, but I don't feel like it's changed much.
EW (00:06:51):
I had all of these images of how it would change the world. And now it's still exercise games and dancing about. Is that just because I've limited myself or is there a lack of progress?
AL (00:07:05):
Well,...I mean, there's been progress from a technological standpoint, definitely. The processing power has helped bring a lot of realism into these environments. And I think the last few years it's definitely been that sort of leap from the first Oculus was almost unusable in a way.
AL (00:07:28):
Sophi Kravitz and I built a, how do you call it, it was a game called WobbleWonder. And we had hooked up a balance board for the headset, and she'd made these fans that would blow at you to try to simulate being in the environment.
AL (00:07:42):
And I kind of designed this sort of Alice in Wonderland-like terrain that would move through, but you really couldn't be in that for more than a few minutes. And even building stuff in it, I couldn't even test the set for more than five minutes without feeling super nauseous.
AL (00:08:00):
And I think that was a big hurdle, really,...which the last few years have really helped us get through. And I think you're right. Now the question is, what happens with it, and what can we do with it?
AL (00:08:15):
And I think in many ways, especially over COVID, and quarantine, and the pandemic, it has changed things for a subset of people. And even myself, I wasn't really so much into virtual reality. That's why I was really focusing on mixed reality and anything that had kind of a real world tie to it.
AL (00:08:39):
But for virtual reality, what sort of really got me was the social VR side of it. Last year was really a way to help connect with communities that I hadn't been able to be a part of for those reasons. And I've heard other friends independently have been spending time in social VR spaces and attending dance parties.
AL (00:09:07):
And it's kind of helped them reconnect with parts of life that they weren't able to. So I think we've made some strides there, but like I said, I love face-to-face connection. And so I think it's a nice supplement, but I don't necessarily think it's the main way we need to be interacting.
EW (00:09:29):
Do you think it will become more a part of our toolkit as engineers and designers? Or do you think it will remain, I don't want to say a toy, but -
CW (00:09:41):
A curiosity.
EW (00:09:42):
Fine. A curiosity.
AL (00:09:45):
So one of the things I'm working towards this year is Virtual Burning Man. Burning Man isn't happening again this year. It's the 60, 70,000 person festival in the, I guess, city that's built in the desert of Nevada, and I've been going for...many years.
AL (00:10:07):
And when it wasn't happening last year, I started participating in the the BRCvr, which is the virtual reality version. And there are multiple platforms that you can engage with. And they're reviving it again this year, because it's not officially happening again.
AL (00:10:26):
So, I'm kind of working on an art piece for that. And part of my motivations for that is that it is kind of a nice speculative design tool in a way too. Because you can kind of build something that you dream of, have people come and interact with it and not necessarily have to worry about how is it going to work yet.
AL (00:10:54):
And maybe through that process, learn what people enjoy about it, and then bring that back to engineering and "How would I actually build that in real life?" That can be a tool. And I think in architecture, people were using VR pretty heavily for client walkthroughs, for being in the space, being able to imagine something before you build it.
AL (00:11:19):
I think it's a very powerful tool or toy in that sense, but I'm curious, what have you experienced with it? What have you played with, or tried, or thought?
EW (00:11:34):
I do like the dance games. But in the last year I haven't been doing much of those, because I need to be in this reality. It's too easy to go there and not want to come back. That's my own brain quirk. I really liked the Google Draw.
AL (00:11:55):
Yes.
EW (00:11:55):
Doodle draw. That was pretty magical, but I didn't -
CW (00:11:58):
What was that called?
AL (00:11:59):
It was Tilt Brush.
EW (00:12:01):
And it was pretty neat for being able to build things. But I couldn't figure out how to build things and then export them so I could 3D print them. Or I could share them with one person, but I couldn't share them with a group. That kind of thing. It was too much me looking at me, and I wanted to go look at other people.
CW (00:12:27):
I think you stopped doing stuff with just before the kind of before the Quest came out and the Quest really has driven a lot more things, because it's so inexpensive. So I think it's moved on a little bit from solitary kind of experiences.
CW (00:12:42):
And I've seen some cool things that I wanted to try out recently. They had a whole canvas painting app. It wasn't as complicated as Google -
AL (00:12:51):
Tilt Brush.
EW (00:12:51):
Tilt Brush.
CW (00:12:51):
- Tilt Brush, because it wasn't 3D, but you had the whole palette and you can mix paints. And I saw somebody demonstrating, doing a whole painting, and it was like, "Oh, this is like painting."
CW (00:13:00):
So I think things like that are getting more prevalent. And I have to admit, I don't think either of us have really tried any of the social stuff, because we're not inherently social people.
AL (00:13:12):
I would really encourage you to come to the Virtual Burning Man, the BRCvr. I can send a link to you guys. It's really quite incredible...There were some of the camps that had Tilt Brush art in it...Last year someone had made this beautiful whale from Tilt Brush.
AL (00:13:33):
And you could kind of walk inside of it and see the sort of sketch work up close, and then kind of step away and look at it from afar. But that was sitting in this sort of desert scene landscape. So it was very surreal and surreality...Sort of what you were saying before.
EW (00:13:59):
Is it like Gather Town, which...one of the new apps where you, in 2D, have a -
AL (00:14:07):
Yeah.
EW (00:14:07):
- space, and you go and see people. And you can identify them, and then you end up Zooming with them, and then you can walk away. It's kind of parties online, but it's also kind of -
AL (00:14:23):
Yeah, it's very similar in terms of kind of, I guess, maybe like...having audio fall off when you move away from someone and kind of approaching conversations. And I think, yeah, the 2D applications like Gather Town have done that pretty well.
AL (00:14:46):
And...I've actually tried quite a few different ones. Some of them fade in your video as you get closer, fade out, and that sort of thing. The VR, the virtual reality ones are a little bit different in the sense that you don't have somebody's video, and they are an avatar.
AL (00:15:06):
And it depends on the platform or the engine as to...what these avatars look like and how realistic they are. But if you do have a handset,...you get the head movement, and you also get the hand movement. So it's a little bit more expressive in that way.
AL (00:15:23):
And it does feel a little bit more like real life. And...walking up and walking away from a conversation feels a little bit more realistic in that sense, but it's very much the same concept.
EW (00:15:39):
And are there servers or platforms I should look for?
AL (00:15:44):
Yeah,...this brings up an interesting conversation that I've had with my group..., we've been building these server worlds together, is that most of the servers and platforms that work really well, they're all corporate owned.
AL (00:16:03):
So...the Virtual Burning Man is hosted on Altspace, which is now owned by Microsoft. There was another one that I was using where, our hackerspace, NYC Resistor, a few of us built a virtual version of the NYC Resistor Clubhouse.
AL (00:16:25):
And that was on Mozilla, Mozilla Hubs, and they had their own sort of proprietary tools, which you could use to upload your models. And they are hosted on AWS. We actually hosted our own. So we didn't use a Mozilla-hosted one. We hosted it on our own AWS server, but they're, I think, reducing support for that.
AL (00:16:56):
And so now we're trying to figure out if we can recreate those servers on our own, but I think the AWS system is pretty convoluted. And what Mozilla did to get that hosted, we haven't been able to sort of reverse-engineer that.
AL (00:17:12):
A friend of mine, [Trammell Hudson], is trying to now build his own server to post another game that that we can play. And maybe we'll try to use that model to move our own worlds onto. But what is nice about, I guess, these hosted platforms is the connectedness.
AL (00:17:34):
So if you're on Altspace, for example, you can really move from one world to another...So there's a little portal thing..., so if you have been to a previous world, you can choose that world, grab the 10 people maybe around you, and you can sort of teleport to that world.
AL (00:17:55):
And so very quickly you can move through all these different universes that different creators have built. And it's really quite wonderful. And it kind of opens up from that really isolating experience in general you associate with VR, where you're in that one application that you've selected, and you're kind of just playing that until you quit.
AL (00:18:17):
And sometimes I can't even figure out how to quit games without restarting the headset. But yeah, I would definitely take a look at Altspace and Hubs. Hubs has a lot of interesting world spaces too. And what's really nice about Hubs is you can also connect like hyperlink to websites and stuff like that.
AL (00:18:39):
So you can click on an object, and it could take you to a website, and...anything that's browser available, you could do on that.
EW (00:18:50):
I know one of your other interests is wearable devices. Does that fit in with mixed reality or is that separate?
AL (00:18:58):
I think there is a connection there. I'm still, in a way, trying to kind of work out how my interest in these sort of more tactile, in-person technologies relate to these virtual ones where you're kind of in one place.
AL (00:19:15):
And one of the things I really enjoyed about wearables, from an early perspective, it was more from a self-expression side, kind of using wearables...initially as an add-on to what you're wearing,...almost in a kind of costume-y sort of way, adding to what you're saying to the world about yourself and your personality.
AL (00:19:45):
And I then started getting more interested in embedding some of the sensor technologies in, and then where I kind of really got stuck into it is the connected side of it. And I think for me, my interest in tech kind of had a lot to do with when I was studying architecture and interior design.
AL (00:20:11):
I was thinking about technology as an additional layer, as a material that we needed to kind of understand in this new world.
AL (00:20:19):
So just as much as you would understand wood and metal, technology is something that sort of permeates as this additional layer, and how do we use that to connect people, and how do we draw connections between each other, and stuff like that.
AL (00:20:40):
And I think I'm often an awkward person. I come across as very sociable, but I'm a little awkward. And I think a lot of us in this space are, and so sometimes playful technology can help. And the same with with wearables.
AL (00:20:58):
It can help strike up an interesting conversation,...and where the wearables got interesting for me was, I started this project called Tiny Gang, and it was this idea of sort of basically like a gang. You have your jacket, and it kind of means you're part of something, and with assorted radios where anybody could kind of be on that frequency.
AL (00:21:25):
There was this kind of romantic idea that...anybody could buy the radios, use our code, put it on something that they were wearing, and if we met in a space, we may not know each other.
AL (00:21:39):
But our jackets are suddenly a connected thread, and we suddenly had that connection. And I think that's where some of the magic of tech has been for me.
EW (00:21:54):
This Tiny Gang, as I look at it on your website, it's jackets that have LEDs.
AL (00:22:01):
[Affirmative].
EW (00:22:01):
And, I suspect the LEDs light when I'm close to somebody else who has the jacket, as you said.
AL (00:22:09):
Yes.
EW (00:22:10):
But wouldn't I know if they had a jacket with a patch on it that said, "I'm a huge nerd." Not that that's what this -
CW (00:22:19):
Would you wear that?
EW (00:22:19):
Realistically, yes.
AL (00:22:21):
Yeah, totally. I mean,...it's kind of derived from that concept, right? That you can kind of where the symbol of your gang or something. And the idea would be that people would contribute content to this project. And so what we do do with this is they light up...You do send information about your yourself to the other person.
AL (00:22:49):
And so, at a base level, color is a very easy one. So I pick a color, another person picks a different color. And so when I'm just on my own, my jacket is this one color, let's say pink. I'm often in pink color, and my friend is in turquoise.
AL (00:23:13):
And then when we're together,...my jacket starts making turquoise patterns, and their jacket starts making pink patterns. So when you're in a group, you become more and more colorful, because of the different people that are around. And so that's one of the things that we transmit.
AL (00:23:32):
I had a version which had an accelerometer, and we were using that. We were transmitting accelerometer data across, I played with the heartbeat one, haven't used that in a group yet. I think it's harder to get a good heartbeat sensor in a comfortable, but yeah, so that was the initial concept.
EW (00:23:55):
Are you going to productize it or is it do-it-yourself for other people?
AL (00:24:00):
It's in a do-it yourself phase at the moment.
EW (00:24:05):
Cool.
AL (00:24:05):
But I have found that it is very difficult...The most successful version we had of this was, a friend, [Elliot], he had come up with this design putting LEDs in an Adidas jacket, and he kind of had a really good kind how-to of how to do that.
AL (00:24:29):
And there were about ten of us that were really excited, and we made ten similar jackets. And me and my partner, Matt Pinner, we made a set of ESP32 pre-programmed boards, and sent those out with instructions of how to connect those to the jackets.
AL (00:24:51):
And that ended up quite successful, because everybody had the jacket, and then they had the electronics. And I think the tricky part has often been people seeing what we've got, and we often use leather jackets, and studs, and things like that, often get stuck with the creating of the jacket with the studs and that kind of thing.
AL (00:25:15):
So actually wearable clothing can be quite labor intensive, unless you're kind of already doing that. So I think that side of things, we're trying to figure out how to make that a little bit easier, so that people can kind of pick that up in a more off-the-shelf kind of way. I know you've worked on some wearables.
EW (00:25:39):
I have. Chris too.
AL (00:25:41):
Yeah.
EW (00:25:41):
We both worked for Fitbit.
AL (00:25:44):
[Mhmm].
EW (00:25:44):
I worked a little bit on Jewelbots, which reminds me of what you're saying so much, because it was friendship bracelets that would glow when you were near another person who had one. And one of the features they wanted was to be able to send secret messages. And I was just like, "What if they're in class?"
CW (00:26:08):
You don't want to make...the class notes feature?
EW (00:26:14):
I am such a goody two shoes, but the idea of being able to bond through the technology and being able to meet people who have similar interests so easily was very nice. They used BLE and a mesh network, which technology-wise at the time was new and tough. Even now, I think it's pretty tough.
AL (00:26:42):
Yeah.
CW (00:26:46):
I like all of this because it's kind of trending toward telepathy without being telepathy, but...it's a bonding thing that doesn't have to be verbal, and I appreciate that. But it's also a discovery mechanism, right?
AL (00:27:03):
Yeah.
CW (00:27:03):
That's the intent?
AL (00:27:05):
A big part for me was the non-verbal, and often you can connect with people without having to, and especially if you don't speak the same language, what are ways you can have something in common.
CW (00:27:19):
Right.
AL (00:27:19):
But I'm actually kind of curious, I guess, because I haven't productized hardware before. I'm really curious, what were the challenges you guys feel in each of these, and especially from the connected perspective?
EW (00:27:36):
The technology itself. BLE changes quickly. And while it is mostly backward compatible, it does require you to have a smartphone, and not everybody does.
CW (00:27:51):
And not every smartphone works very well with Bluetooth, and you have to make them all work somehow.
EW (00:27:57):
And the battery situation..., I mean, for any wearable, that's got to be a consideration. Sure you can put on a jacket, you can put a battery pack, on the lower back and...it's not that heavy.
EW (00:28:12):
It probably isn't that hot, but if you're doing something wrist-based, you have to be very conscious of weight, and that is batteries. And then that affects what you can do. And it's all a horrible downward spiral.
CW (00:28:28):
Many of the challenges are more mundane that I recall, like user interface things like what's comfortable to interact with in a small touch screen, how many buttons should there be.
CW (00:28:40):
And is one button okay if we make it so that you have to hold it or tap it multiple times? And trying to get things that work for...a diverse group of people, that was also a challenge, especially with sensors, and just with fit kind of things.
CW (00:28:59):
Some people don't like having things on their wrist, and some people don't like certain materials on their wrist which, okay, so do you just ignore those people, or do you try to accommodate them? And it's just a lot of issues, especially when you're trying to reach a broad customer base.
EW (00:29:17):
I was always mad at Fitbit for their charging strategies, because -
CW (00:29:22):
Well, it was different every product.
EW (00:29:23):
It was different every product, but every product you had to take it out of its mounting -
CW (00:29:28):
Well, with the watches, yeah.
EW (00:29:28):
- for all the little ones. And so I wanted to build them into jewelry, but I had to take it entirely out of whatever it was I put it in, which meant I couldn't use polymer or anything that wasn't rubberized.
CW (00:29:43):
Yeah, I think that's just kind of thing that happens with real miniaturization where you're trying to make something tiny. It's, "Okay. Well, it has to be tiny. So we're just going to have to force a charging thing in some way."
EW (00:29:52):
Yeah.
CW (00:29:52):
And it's going to be off in this corner, because that's the only place the electronics fit...Yeah, that's another thing with productization, right?
CW (00:29:58):
It's, "We need to make something that's as small as possible, the battery lasts as long as possible. It's as cheap to make as possible." And then all sorts of things just kind of fly out the window at that point.
EW (00:30:08):
And when you're doing a distributed system like this, do you have a single eye in the sky or do they really talk to each other? And that's a pretty big difference.
EW (00:30:19):
Having them all talk to each other requires more power, because they have to be on more, but having a single thing they talk to, that means they probably have to talk to some sort of routing-like things -
AL (00:30:34):
Yeah, yeah.
EW (00:30:34):
- like a smartphone.
AL (00:30:37):
Sounds like the the blockchain discussion. No, I think you're right. Yeah, when I was at Intel, I was in the wearables division there, and yeah, there were a lot of very similar challenges.
AL (00:30:59):
A colleague of mine made...a pretty nifty ring actually that did a lot of things, but couldn't get the battery size small enough for a woman-sized hand. Definitely batteries were a huge kind of source of development at the time, especially for yeah, rings and bracelets and things like that.
EW (00:31:29):
Yeah, they've had some, like Ringly, those were cute, but it was clear they were really pushing. It didn't quite do as much as I wanted it to, and technology-wise I could see why they were trying to make it so small. It can't do everything.
EW (00:31:45):
That didn't have a display on the device. So you always had to interact with it -
AL (00:31:54):
Through the app, yeah.
EW (00:31:54):
- through your smartphone. Yeah.
AL (00:31:56):
Yeah. I had a Ringly too. But it looked really nice.
EW (00:32:01):
It's a balance. It looks really nice, but it's not as functional. And some of the functional ones, they don't look as nice, but they last for a month, which is pretty cool. It's yeah, it's all just trade-offs. I mean, that's all the engineering part.
AL (00:32:19):
Yeah. Well, also user interaction design. We did a few studies where we were trying to teach new gesture interactions to users for some of these prototype devices that we had. And it was almost like teaching a new language.
AL (00:32:38):
And the question is how many new languages do people want to learn? And I tried to create a sort of ring-based input for a head-mounted display, and kind of had to sort of even just looking at the usual interactions of scrolling up and scrolling down as a specific example, people got really confused.
AL (00:33:08):
Because some people...were used to scrolling their mouse a certain way, or they would relate it to the way they scrolled their phone. And that took a while to kind of get people used to. And I was kind of surprised at how that was one of the things...that I had to spend a lot of time with between different users.
EW (00:33:37):
It reminds me of the scene in Independence Day, where they have a post-it note that says up, down, left, and right. And then as soon as they get in and they try it, it's backwards. And so they just turn the post-it note upside down.
EW (00:33:51):
And I was like, "There's so many times I want to do that on a UI." And when I switch from using Linux to using Windows, there's something different about scrolling, and I just try all the options until whatever I'm in works. But it is confusing. And if you tried to add another set of choices for me, I would fail.
CW (00:34:17):
Well, that's the thing is, you talk about creating new languages in terms of gestures and things, but people are already being trained surreptitiously. And so -
EW (00:34:27):
Yeah.
CW (00:34:27):
- you have to have to kind of think about, "Okay, I'm going to do this, but have they been using something else that's similar enough that they're going to have muscle memory that's backwards or wrong? And does that limit you from making kind of more radical choices?
AL (00:34:44):
Yeah, definitely. Right. And the dominant technology tends to drive these user interactions that we're used to. And...even switching between an Android phone and a iOS device, the different interactions on this one type of screen, and people have a hard time switching between.
AL (00:35:10):
But it's these two companies that have kind of now dictated the gestural language of using a smartphone. I don't think it's been that sort of way yet with wearables. I don't think there's kind of a dominant tech there yet.
EW (00:35:28):
No.
CW (00:35:30):
No.
EW (00:35:30):
I haven't seen any. And I've always thought that if I was going to try to do a gestural thing, I would look really hard at ASL.
AL (00:35:41):
Yes.
EW (00:35:41):
Because it's already a sign language. And if you could get some crossover, it would be very nice, just because that's already done. And it's got a grammar, and it's a language language.
AL (00:35:56):
Yeah. I think there are quite a few projects in machine learning trying to study ASL languages, which I thought were quite fascinating.
EW (00:36:11):
Yeah. There's some that do the translation, that you can have an avatar go from English to ASL instead of having a human interpreter. I'm fascinated by that. I think that's just the coolest.
AL (00:36:25):
Yeah. Yeah. I think there's some definitely very interesting technology coming out for accessibility and especially with all the conferences that we've had virtually.
AL (00:36:39):
Recently I've noticed there's been a lot of good closed captioning. Auto captioning in sign language has still, I think, been a human there, but yeah, definitely huge potential for machines to help out with that.
EW (00:36:55):
When we talk about virtual reality and virtual worlds, do you consider things like Animal Crossing to be part of that, or is it separate?
AL (00:37:06):
Yeah, I kind of have a looser definition in terms of virtual worlds. And so I think Animal Crossing definitely for me falls into that, I guess, especially from the sort of social perspective where you can have multiple people in the world itself.
AL (00:37:28):
I mean it itself does have a 3D kind of element to it. And for all intents and purposes does sort of fulfill that virtualness from a environment perspective, from a story perspective. Yeah. And I think it's become a very socially relevant platform too.
EW (00:37:50):
You've done actual things in Animal Crossing. I mean, beyond dressing up your villagers as they are trying to fish.
AL (00:37:59):
Yeah. Yeah, last year I created a virtual protest in Animal Crossing sort of during the start of the Black Lives Matter movement. And...the Black Lives Matter movement started, I think in 2013, but the resurgence of it last year, and we were all kind of stuck at home, and I was looking for ways to in a way contribute.
AL (00:38:29):
And I was spending so much time in Animal Crossing. Initially I had just wanted to build a memorial and a tribute to all the black lives lost. And so I started creating this memorial and in thinking about what else I could do with that, I wanted to create a fundraiser.
AL (00:38:51):
And then that sort of turned into a bit of a protest. And yeah, I had help from some friends, in terms of creating the artwork and also kind of thinking about how actual protests would work. And we ended up creating a path that people could walk through when they came onto my island.
AL (00:39:14):
And we did this sort of processional march, and we sat down at the memorial and kind of chanted. And it was very neat in the way that it felt very life-like.
AL (00:39:27):
I've been to a few protests myself, and I was kind of surprised at how without, I guess, the loudness and the violence sometimes involved with it, but the spirit of it and the community, it really felt quite realistic. And I thought people would be coming in and leaving really fast.
AL (00:39:46):
But we ended up sitting for 45 minutes, an hour, to a time in a group, and kind of expressing how we were feeling, and all the frustration. And it was a super cool that people from all across different parts of America had come. I had a couple of people from Australia come. It was a great virtual gathering in that sense.
AL (00:40:11):
And I know...in Hong Kong they've been using Animal Crossing for protesting as well too. People have actually gotten arrested over it so anything in some places is actually quite an effective tool.
EW (00:40:27):
You've also done art. Big art. You worked to light the London bridge with Leo Villareal?
AL (00:40:38):
Villareal. Leo Villareal studio. Yeah. I worked with Leo on the software side of things. Yeah, he's an artist that worked on The Bay Lights.
CW (00:40:51):
Oh, okay.
EW (00:40:51):
The Oakland Bay Bridge. Is that what we're talking about?
AL (00:40:54):
Yeah. The Oakland Bay Bridge. Yeah, that was, I think, one of the first kind of large, urban scale works that he did. I wasn't with their studio at the time. But yeah, I found it really mesmerizing.
AL (00:41:10):
And so when an opportunity came up to do some work with them, I was very excited. And he got this commission to work with a nonprofit in London and working with the city and the different boroughs, because each bridge there is owned by two boroughs.
AL (00:41:31):
So for each bridge that we worked on, there had to be negotiations and planning approvals from all these different owners. So it was a much bigger project from that sort of bureaucratic perspective.
AL (00:41:44):
But yeah, I worked on building custom software that he uses to program and light up the bridges, and we have something kind of basically working hopefully for the next 10 to 20 years. So that was definitely a really fun project to be a part of.
CW (00:42:06):
Those big installations must be, I mean, people make LED light up things all the time with Arduinos, and WS2812s, and things like that. What's the challenge in moving from, "Okay. I can make some lights blink," to, "I can make giant lights on a bridge blink?" Does the technology not really transfer or is it kind of just scaled up?
AL (00:42:30):
I guess kind of the programming paradigm is very similar, but the scale of it definitely from the hardware perspective, so you can't really use these -
CW (00:42:48):
Yeah.
AL (00:42:48):
- you have to use kind of architectural grade, outdoor sort of fixtures that can kind of withstand all kinds of weather and stuff like that. And installing those requires a construction team...And it even needed to be more specialized. The electricians needed to be abseilers as well.
AL (00:43:10):
So they needed two types of licensing to be able to install the lights. Which was kind of insane, kind of watching these abseilers climb over bridges that were, some of them, hundreds of years old, installing these fixtures onto it. And so then it's about being able to speak the protocol that light fixture is using.
AL (00:43:35):
So Leo's studio has different sort of different protocols that we can use the software to then connect to these different fixtures. And every time there's a new project, it's a different protocol. So, it's always about kind of figuring that out, but from the software perspective, we try to build tools that are more modular.
AL (00:44:03):
And so that can be kind of used to connect to these sort of different systems. And it needs to be a little bit more robust in that we have to be able to monitor them. So we have kind of a bit of a monitoring software that we build and beefier computers.
CW (00:44:23):
Yeah.
EW (00:44:25):
It sounds like going from wearables that you do yourself to production, very much the same sort of problems.
AL (00:44:33):
Yeah. Yeah. So a lot of what I've done in a way had kind of touched on production and...yeah, production work is a whole different beast and skill set. And I kind of learned a lot of that actually at Intel where we were doing kind of...stage demos and had some production people come work with us.
AL (00:45:03):
And I learned a lot from them about...doing everything live, kind of live data is in a way, a very different beast to live streaming data, which is kind of what I work with as my current job, to kind of more static queries, and the sort of big data problems.
AL (00:45:28):
But yeah, they're both really interesting. And I think that has actually helped with the way I think about wearables. For example, I'm working on a project with Sophi Kravitz right now. We're making our own radio boards and thinking about using that at a scale of a hundred people.
AL (00:45:50):
And again, kind of back to the "What size battery can fit on a necklace," and that sort of thing. But then now kind of thinking...from a production perspective, what does that kind of need? How does that work with the scene and the environment?
AL (00:46:07):
And what is the audience participation, and the affect of that? And how can the people be part of the lighting, part of the environmental lighting, for example, are kind of things that I bring back to it now from working on kind of larger scale productions. So I think it's good to be able to work in both realms.
EW (00:46:36):
What is A MAZE. Fest? Changing subject entirely.
AL (00:46:42):
Oh, yeah, I'm glad you brought that up actually. A MAZE. Festival, it's a games festival, or, yeah, I guess...it is a festival, in Berlin that I got involved with a few years ago actually it's through my friend, Phoenix Perry. We collaborated on, I guess, an interactive musical artwork piece, and we actually built it in Berlin for the festival.
AL (00:47:18):
But it is an independent games festival. But it's a really cool one, because...it works really hard to kind of push the idea of what a game is forward and opening it up to playful media, performances, software experiments, anything sort of creative that has a bit of... Either uses like game mechanisms or maybe it's built with game engines.
AL (00:47:51):
And so I got invited back this year to be a judge on the panel and basically have been playing games nonstop for the last two weeks.
EW (00:48:03):
[Laughter] Oh, no, that sounds awful.
AL (00:48:05):
It's awful. Well, not nonstop, but whenever I have free time and that also involves kind of watching performances...
AL (00:48:14):
One of the pieces, this person built their own game so that they could do a hip hop performance in...Talk about building your own environment, and kind of the virtuality, and combining performance with it. And it isn't something that anybody else can play.
AL (00:48:39):
And we interact with it by watching, but that was in one of the games categories. And yeah, so it's a really interesting festival that's virtual this year as well.
AL (00:48:55):
And I definitely encourage you to check that out. It's coming up next on the 20, I forgot the exact dates, but 22nd to 25th of July, something somewhere around there. I can also send you a link to that.
EW (00:49:11):
We'll put it in the show notes.
AL (00:49:13):
Yeah.
EW (00:49:14):
And are these mostly electronic games? That's not the word I'm looking for. Or computer games?
AL (00:49:20):
Digital?
EW (00:49:21):
Digital, that's the word.
AL (00:49:23):
Yeah. I would say maybe 70% of them are fully digital, but a lot of them also have some sort of analog form. So games also include card games, board games, and also kind of in-person immersive games, so kind of think escape room type of thing, right?
AL (00:49:52):
So all of those kind of categories come into this, but because, especially last year specifically, all of the physical stuff wasn't really available, so a lot of these sometimes in-person things kind of took on a digital form.
AL (00:50:11):
And so some of the card games that were in their had apps, that also used the apps to kind of help do the storytelling mechanism. There was one which was kind of a Zoom, multi-person game where they kind of had actors.
AL (00:50:33):
It was kind of like a role-playing sort of thing where they had actors kind of talk you through different puzzles. It was around the question of urban development, brought people together, and kind of gave them different roles.
AL (00:50:47):
Maybe you're the urban developer and you're the resident, and had kind of interesting discussions around it with the sort of game kind of mechanism to move the story along. And that would last for about an hour.
AL (00:51:01):
So yeah, one of the games kind of had a blinking mechanism. So it kind of...got you to calibrate with your camera, and got you to calibrate your blinking, and used a kind of blinking, or closing your eyes, or opening your eyes kind of mechanism to propel the story forward.
AL (00:51:22):
And it was done in a very natural sort of way. So there are a lot of different kind of analog physical inputs to the sort of digital, kind of virtual space that we know now.
EW (00:51:38):
It's funny how games have gone from keyboard input to joysticks, to the incredible revolution of accelerometers and gyroscopes with the Wii U. We don't even have those games, why don't we have golf anymore?
CW (00:51:56):
There's a golf game.
EW (00:51:57):
Yeah. But not like -
CW (00:52:00):
It moved on to VR. You just have to do it in VR.
EW (00:52:03):
Yeah. Yeah. And I mean, the mats, and the VR, and the cameras have been really interesting, that you can get a lot of information from someone based on how they look or based on what they're looking at.
AL (00:52:21):
Yeah. The eye tracking, the head movement tracking, Facebook is doing a lot of development. A lot of their machine learning is using their Oculus headsets and tracking everything.
EW (00:52:35):
Do you have any thoughts for what you would want to have for the future of game inputs? What about you Christopher?
CW (00:52:48):
I think I'm becoming too old. I mean -
EW (00:52:52):
That's a silly answer.
CW (00:52:53):
I enjoy controller-based games these days. I mean, I do enjoy some of the VR stuff. And I think that's less about creating a new interface, and getting rid of interfaces entirely, and just making it your natural movements, which I guess is a type of interface. But, I don't know.
EW (00:53:10):
I mean, that's the accelerometers, and the gyros, and the Wii U.
CW (00:53:14):
Well, yeah.
EW (00:53:14):
It went from something totally artificial to something vaguely similar.
CW (00:53:20):
Yeah.
EW (00:53:20):
And the blinking sounds like something, I mean, that's something I do anyway. I'm not sure how it would make a story go, but okay. Yeah, I could imagine.
AL (00:53:35):
Yeah. Well, I guess...that's sort of replaced clicking. So I guess anything that can replace clicking, you could replace with the different kind of sensor input or different kind of gestural input. Gosh, it's really hard to think about what hasn't been done.
EW (00:53:56):
Well, yes. Yes. Let me put you on the spot. Could you please invent a whole new technology right now?
AL (00:54:04):
I mean, I've seen licking games, where you lick something.
EW (00:54:13):
That is not going to be the show title.
AL (00:54:17):
There's a game designer, [Kaho Abe], she did a lot for this medium and kind of bringing physical computing to games, and had some really interesting game mechanics around that. I think I've seen ones done with phones. I've seen a toaster? Am I making it up? No. I've seen a game made with a couch.
EW (00:54:51):
That's going to be hard to productize.
AL (00:54:54):
Yeah. I mean, I think, well,...any objects or interfaces that you're passionate about,...it's more about thinking about different playful ways that you could approach something.
AL (00:55:09):
And so I think there's definitely a lot of potential for people to create new mechanisms, and they may be more towards the artistic side of things, and maybe less of a productized kind of game. I think it's definitely a rich landscape to be engaged in.
EW (00:55:30):
You've said playful. What does that mean? I mean, is it one of those words that everybody knows what it means, but nobody can define it?
AL (00:55:42):
Ooh, I like that question. Yeah. It has kind of become this sort of catch-all in a way of, if you don't have a pure win-lose mechanic, but it still uses some kind of, I was going to say playful mechanism.
EW (00:56:08):
No fair using the word to define the word.
AL (00:56:12):
Yeah. You should look up the dictionary definition of play. But, yeah, I think for me, I use playful when there isn't necessarily a, yeah, win-lose dynamic, I would kind of say, but still has maybe rules or rewards, different ways of kind of interacting that are parameterized, let's say, but don't necessarily mean you could be a winner or loser.
EW (00:56:51):
I'm really glad I asked, because that isn't what it means to me at all, and so now I really want to understand more...And I thought about asking what is a game, but I guess I sort of did with the electronic part.
EW (00:57:07):
So it's something you do that doesn't have a win or a loss. Is it only when you're intending to play? Or can it be playful without rules? Where am I going with this, Christopher?
CW (00:57:25):
I don't know. I'm pretty confused now.
AL (00:57:30):
Yeah, I guess, I mean, well, how do you define, what feels -
CW (00:57:35):
Well,...my brain keeps going to whimsical, which is not the right thing.
EW (00:57:39):
Fun.
CW (00:57:39):
But sometimes -
EW (00:57:41):
Joy.
CW (00:57:41):
- playful implies whimsical to me, where there's something either humorous, or yeah, something that makes you smile.
EW (00:57:52):
And that can be concentrated. I mean, it could be hard, and still be play.
CW (00:57:56):
Yeah.
EW (00:57:56):
And playful.
CW (00:57:56):
Yeah.
EW (00:57:57):
But it requires that happiness for playful to work.
CW (00:58:05):
Also, maybe discovery and curiosity. So -
EW (00:58:07):
Yeah.
CW (00:58:07):
Yeah, it's nebulous.
AL (00:58:11):
Would you consider a puzzle playful?
EW (00:58:12):
The one we did, the Nautilus one from Nervous Systems. That was so fun.
AL (00:58:17):
Oh, what's that one?
EW (00:58:19):
It's cut into really hard shapes. And so it doesn't act like a normal puzzle.
AL (00:58:29):
Right, right. Yeah. I know they've been making puzzles. I should play that one. But yeah, so a puzzle could be potentially playful. Sometimes I think you some of the early stuff that Phoenix and I were working on, kind of turning different environments or different objects into musical instruments, I would call that playful.
AL (00:58:59):
But there isn't necessarily an outcome that you're trying to get to. If you have three or four people playing musical instruments or playing interactive objects that make music, those people decide what is the music or the sound that they create together.
AL (00:59:21):
And so those may not have rules, but the rules are kind of built into the input of the object itself. And there's only maybe so many ways that you could use that, for example. Another, I guess I would say playful game that I've built...three or four years ago now, it's called Star Catcher, where we had VR trackers on nets.
AL (00:59:52):
And there was a canopy of programmed LED lights, and they kind of looked like falling stars, and you could go in and catch them. And so when you put your wand underneath one of the strips, if the light hit the bottom of the strip, you would effectively catch the star.
AL (01:00:13):
And it sort of did have a little bit of a level element to it where every star you caught filled in a constellation, and you could kind of advance levels, but...you could totally ignore that mechanism, for example. And you didn't need that to kind of advance in the game.
AL (01:00:37):
So I would still call them more playful than some of the kind of more shooting games where you have to shoot 10 people to finish a level. And you either win, or you die.
EW (01:00:51):
Does playful mean without a goal?
CW (01:00:57):
Not necessarily to me.
AL (01:00:57):
I think -
EW (01:00:59):
No, I don't think so, because I can imagine getting to a goal.
AL (01:01:02):
Yeah. I definitely think you could have goals.
EW (01:01:05):
But it shouldn't be a goal that makes money or does useful things.
CW (01:01:11):
Well, I don't know. I think over defining it is probably not really -
EW (01:01:16):
Alright. Christopher's bored of my -
CW (01:01:18):
No, I just don't know that we're going to get to anything satisfying.
EW (01:01:21):
Etymology of play is, okay, I'm not going to look it up.
AL (01:01:25):
You definitely want to bring in a games professor, I think, for that conversation. Actually -
CW (01:01:34):
Is defining playfulness playful?
EW (01:01:37):
Well, okay.
AL (01:01:37):
Well, as long as we're having fun, actually, I think that's the rule here. So we could keep going. The listeners may not enjoy that.
EW (01:01:49):
And actually we probably should close up. We're about out of time. I'll consider play anyway. You guys can do other things. Adelle, do you have any thoughts you'd like to leave us with?
AL (01:02:02):
I guess, maybe reach out to me if you want to have further discussion about any of the things we talked about. I love talking about wearables, games, play, machine learning, cameras, whatever.
EW (01:02:22):
What kind of machine learning do you like to do?
AL (01:02:26):
Most of the stuff I'm doing right now is computer vision. And I did work with some musical machine learning projects. Those were really fun. I'd love to get more of that. But that doesn't really pay the bills.
EW (01:02:43):
Our guest has been Adelle Lin, Creative Engineer and international puppy herder.
CW (01:02:50):
Thanks, Adelle.
AL (01:02:52):
Thank you so much.
EW (01:02:53):
Thank you to Christopher for producing and co-hosting, and thank you to Sophi and Phoenix for suggesting Adelle. And thank you for listening. Oh, and Patreons, thank you for Adelle's mic. You can always contact us at show at embedded.fm, or hit the contact link on embedded.fm.
EW (01:03:13):
We will make sure to have Adelle's contact information, probably via Twitter and her website. So now a quote, well, of course not a quote to leave you with.
EW (01:03:26):
So from Middle English, "play" comes from Old English, which still is kind of play, and it means "quick motion, movement, exercise, sport, game, festivity, drama, battle, gear for games, implementation of a game, clapping with the hands, and applause." So I kind of understand why we can't define it. Nobody ever has.