463: Layers of Band-Aids
Transcript from 463: Layers of Band-Aids with Kevin Lannen, Christopher White, and Elecia White.
EW (00:06):
Welcome to Embedded. I am Elecia White, alongside Christopher White. Our guest this week is Kevin Lannen, embedded systems engineer at LUCI Mobility. We are going to talk about wheelchairs, control and people.
CW (00:21):
Hello, Kevin. Good afternoon. Morning. Evening.
KL (00:25):
Hey.
CW (00:25):
I do not know where you are, so I am just going to cover all the bases.
KL (00:30):
Yeah, thanks for having me on the show.
EW (00:32):
Could you tell us about yourself as if we met at, I do not know, Supercon? Hackaday Supercon?
KL (00:40):
Yeah. I am Kevin Lannen. I am an embedded systems engineer at LUCI, where I work on really everything from designing the electronics, hardware, embedded Linux. We do some bare metal C programming. All the way up to application software that controls our whole system.
(00:55):
I always try to push my understanding of the systems I work with, to really include as much of the stack as possible. And I really love doing system optimization, so squeezing as much performance out of the hardware and power budget that we have. Outside of my engineering work, I spend my time skiing and hiking a lot, here in Colorado.
EW (01:17):
What exactly is LUCI?
KL (01:19):
Yeah, so LUCI is a safety system that installs onto existing power wheelchairs. It provides sensors that provide a 360 degree view of obstacles around the chair, as well as the ground. So we actually can look and detect steps and drop-offs. Then we interface with the control system of the wheelchair, to prevent the user from being able to run into things, drive off curbs.
(01:44):
We also provide a little bit of cloud connectivity, that provides some alerts to the user's caregiver team. So you can keep tabs on your user. So if somebody tips over their wheelchair, we can send an alert and say, "Hey, you tipped over your wheelchair at this location, and they might need help."
EW (02:02):
All right. So I have questions about so many of those things, including controls. But first lightning round. Are you ready?
KL (02:09):
Yes!
CW (02:10):
Complete one project, or start two dozen?
KL (02:14):
Oh, three dozen. Is that an option?
EW (02:18):
Black diamond, double black diamond or green triangle?
KL (02:22):
Oh, double black diamond.
CW (02:25):
Wow! I have been ignoring Rust. Is it ready yet?
KL (02:28):
It depends on what you are doing, but for a lot of things, yes.
EW (02:32):
Favorite type of motor?
KL (02:36):
Ooh. Brushless, DC motors.
CW (02:37):
Have you started any motor fires?
KL (02:40):
I have not caught motors on fire. I have gotten some FETs awfully hot, to where I think one of them fell off of a board once, because it unsoldered itself.
EW (02:49):
Least favorite analog component?
KL (02:52):
[Least ooh.] Analog in general is kind of ugly and nasty.
CW (03:01):
<laugh> Favorite fictional robot?
KL (03:03):
R2-D2.
EW (03:05):
Good one. Favorite microcontroller?
KL (03:08):
Not necessarily my favorite. The one I know most about. I really like the MSP430 family, because they are super simple to use.
EW (03:15):
Mhm.
CW (03:18):
And finally, do you have a tip everyone should know?
KL (03:20):
Yes, it is debug to the root cause. So when you are debugging a problem, try and get all the way to what the actual problem is. And not just try and cover it up with layers and layers of band-aids on top of it. You really learn something when you actually dive in and solve a problem all the way to the bottom.
EW (03:43):
<music> We are happy to be sponsored this week by Memfault. Memfault provides a device reliability platform for IoT monitoring, debugging, and updates. Device operation no longer needs to be a scramble, as issues with fielded units pile up. Instead, Memfault gives developers a more scalable and sustainable process, to accelerate time to market, de-risk product launches, cut development costs, and deliver higher quality products.
(04:09):
So if you are wondering how you are going to monitor your units once they are shipped, or whether your firmware update plan is secure enough, it is time to take a look at Memfault. Or you can read their Interrupt blog, for all of its fantastic goodies on how to debug hard faults, monitor units, or generally write good embedded code.
(04:26):
Embedded.fm listeners will get 25% off their first year with Memfault, if you request a demo through go.memfault.com/demo-request-embeddedfm. It is a link you can find in our show notes. Thank you to Memfault for sponsoring this week's show. <music>
(04:44):
Okay, so LUCI. A few years ago we were at the beach and a woman in a wheelchair was being pushed over the bridge. The bridge had a little bit of unevenness to it. The pusher did not see the lip, and the user was dumped out of the chair. Do you remember this?
CW (05:05):
No, but that is probably for the best.
EW (05:08):
The chair landed on the person who was sitting in it. It was awful, and the person seemed okay. The pusher was very distraught. The feeling of powerlessness. LUCI is supposed to prevent that sort of thing?
KL (05:28):
Yeah, to an extent. We help prevent that with power wheelchairs, where we can detect places where it is unsafe for the wheelchair to go. Like dropping off the edge of a bridge, or something like that. It is a little harder for a manual chair where somebody is pushing.
(05:42):
But there are definitely a lot of off-road wheelchairs, and there are other things out there, but it is hard for people to get access to them, get funding for them, that sort of thing. So there are a lot of things that are both technology that can help prevent that, as well as there are some questions about how folks in wheelchairs get access to technology, and to the equipment they need to really live a fulfilling life.
EW (06:08):
I mean, it is always a matter of cost, right?
KL (06:13):
At the end of the day, yeah, it all comes down to cost. And some inertia. I mean, there are a lot of things that we at LUCI think we could reduce costs in a lot of areas, for people in wheelchairs. As far as, if you make something safer, fewer injuries that insurance might have to pay out, that sort of stuff.
(06:32):
But there is a lot of inertia behind it, of the insurance companies do not like to go after new billing codes and new technology, as much as I think the folks that are relying on this technology would like, for sure.
EW (06:46):
Is LUCI a certain type of wheelchair?
KL (06:49):
LUCI is not a wheelchair itself. We work with about four or five wheelchair manufacturers right now, and we integrate as an aftermarket accessory. We have a frame that installs on the wheelchair that has all the sensors. You wire it in, plug it in, and works with these existing wheelchairs on the market.
EW (07:10):
Existing powered wheelchairs, though?
KL (07:11):
Yes, existing power wheelchairs. Specifically what are called "Group 3 wheelchairs." So in the power wheelchair space, there are kind of three main categories of wheelchairs. You have got the mobility scooters, which folks see around a lot for helping people get around. And then there are the Group 2 chairs, which are a more advanced power base, with just a pretty simple seating system on top of it.
(07:38):
And then the chairs that most of our users are in, are in this Group 3, where they are very complex wheelchairs. But the main thing that gets more complex with them is the seating system. So these seating systems can go from tilt and recline, to being able to elevate so you are up at eye level with folks, to even putting you in a standing position in the wheelchair. That adds a lot of cost and complexity under the wheelchairs.
(08:06):
A lot of the innovation recently from the manufacturers, has really been on the seating side of these complex seating assemblies. But there has not been as much attention paid to the drive side, which is where we are focused on, with our safety system.
EW (08:21):
Group 1 wheelchairs are probably- You said mobility scooters, so I am thinking sometimes they even have those as the grocery store, for people who do not get around quickly or well.
KL (08:31):
Yes. Yeah, that is a great example. The grocery store ones are kind of the bottom of the totem pole there. Then you get the scooters that everyone drives around at Disney, is a pretty good example. Everyone has got a lot of folks who use the scooters to get around, so they are not having to walk as far. So those are for somebody who normally can walk, but cannot walk a super long distance, so the scooter helps them get around to complete their everyday activities.
EW (08:55):
And Group 2 sounds like it is similar. That it is for people who maybe cannot walk any distance, but are not in the chair a hundred percent of the time.
KL (09:08):
Yes, they are not in the chair a hundred percent of the time, so they do not have the complex seating and positioning needs, that somebody who is full-time in a wheelchair would need.
EW (09:18):
And that is where Group 3 comes in.
KL (09:20):
Yep. That is where the more complicated and much more expensive wheelchairs come in. We can talk about LUCI. People say, "Oh, LUCI is kind of expensive." We are just under $10,000. But some of these wheelchairs, the sticker price of them can be easily as much as a luxury car, in a lot of cases. So it is a really expensive product with all the complex needs that these users have.
EW (09:50):
I mean, you are spending your life in it.
KL (09:52):
Exactly.
EW (09:52):
This is not your desk chair. This is not your couch.
CW (09:57):
Even people spend a lot on desk chairs too, but not that much. You spend a lot of time with it. It makes sense to invest in it.
EW (10:04):
And it affects your health, it affects your ability to interact with other people. It is a home.
KL (10:14):
It is a home and it is your legs. So being able to get out in the community, get outside, go to the mall with friends, these are all things people in your wheelchair want to do. And so it is all about making it easier for them to get out and enjoy life.
(10:31):
We have seen with some people we have talked to and interviewed, huge mental health benefits of being able to get out and be a teenager, for instance, for some of these kids. That is a really empowering thing of working on this, is seeing how we have really changed some people's lives, of letting them get out and do stuff that they were not able to do before.
EW (10:52):
One of the things I saw in your videos that I had not considered at all, was crowds. I do not like crowds because I do not like crowds. But I know going to a soccer game or a concert or wandering the mall, you are with three or five people and they are shifting around and moving and walking. I am used to going wherever I want to go, and standing next to whoever I want to stand next to.
(11:18):
But it did not occur to me that as a wheelchair user, a powered wheelchair user, is keeping up with the crowd is a constant thing you have to control. It would be really easy to hurt somebody in front of you, or to stop suddenly and be hurt, because somebody walked into you.
KL (11:40):
Yeah, that is a huge source of anxiety for a lot of wheelchair users. Is they really do not want to run over somebody's toe or bump into somebody. These wheelchairs are three, 400 pounds, and so you can do a lot of damage pretty quickly.
(11:53):
Our CEO loves talking about Disney after the fireworks, because they brought their daughter Katherine to Disney. You run into a dilemma, as someone in a wheelchair. You can either leave before the fireworks, so that you are not in this big crowd. Or you have to sit there and wait for everybody to leave, so you have got a clear path to drive out. It really limits and changes how you think about these experiences out in public.
(12:20):
That one is a big thing that we have done a lot of work on. Our system kind of works like the adaptive cruise control in a car, when you are in a crowd. So you can just hold forward on the joystick, and it will keep up with the person in front of you, and move you along. That is a huge deal for a lot of people, being able to have confidence in these crowds, that they are not going to hurt somebody.
EW (12:45):
How does it work?
KL (12:48):
We use a variety of sensors. We have a saying- This came from @whitequark a long time ago on Twitter, that "All sensors are wrong and some are useful." So we use a variety of sensors to make up for the inherent things that they are good at and not good at.
(13:05):
We use some stereo vision cameras from Intel RealSense. We have some millimeter-wave radar that we have built on top of some of Texas Instrument's radar chips. We use some ultrasonics. And then we have got an IMU on board.
(13:17):
So we are constantly looking out and detecting objects, and then we scale down the joystick of the user, proportionally to how close they are to objects. So that as you approach something, if you are in a tight area, we slow the speed of the chair down, until you get right up against it and then we stop. And if you are in a wide open area, we let you drive around as you would normally, as if the system was not there.
(13:42):
We are starting to add a little bit of some semi-autonomous features to help people in common scenarios, like getting up ramps on wheelchair vans for instance. In these tight areas where you want to do the same thing every time and not think about it, we are working on how can we help automate those.
(13:57):
But we are not really an autonomous system in general. We do not want to take away the input of the user. That is really important. You do not want to have to- Like how your legs take you somewhere, without you telling them where to go sort of thing. So we are focused on the how do we engage the human, keep them in the loop of control, but provide them safety and confidence in what they are doing.
EW (14:27):
You mentioned your CEO. Dean?
KL (14:31):
Yes. Barry Dean.
EW (14:34):
He is a country music star?
KL (14:36):
Yes. He has written several country music hits. He is a singer-songwriter out in Nashville. He has written for Little Big Town. "Pontoon," along with quite a few other songs people have probably heard. So yeah, kind of an interesting story if we go from him being a singer-songwriter, to the CEO of a tech company working on power wheelchairs.
EW (15:01):
And it was because of his daughter?
KL (15:04):
Yes. His daughter Katherine has been in power wheelchair for pretty much her whole life. She has cerebral palsy. The very first time she got in the wheelchair at the hospital, it was not configured right, and she drove it down the hallway and into a wall. So from the very first time, the first experience they had with a wheelchair, they have known that these things are dangerous.
(15:29):
She also has some vision limitations, and reaction time limitations. It made everybody nervous. They wanted her to be safe as she drives around. And then as she got older, they also wanted her to be able to be more independent, to go do things on her own. She is super social, loves to go talk to people and hang out and stuff, so she wanted to be able to get around and do stuff on her own. LUCI was the answer we created for that.
EW (16:00):
Does having a prototypical or archetypical user help you define the system?
KL (16:08):
A hundred percent. We really focus on all sorts of use cases. We talk about users as far as- We have the safe drivers, that they are the people that drive slow naturally, even without LUCI, want to be safe, they are really nervous. Up to, we call our race car drivers, so the people that want to go fast everywhere all the time. Thinking about how those different people would use the system, really helps us tune the behavior and how we have developed the product for them.
(16:39):
The other thing we think about in terms of users, that is interesting with a medical product like this, is that the person in the wheelchair is not our only customer. We really talked that we have multiple customers. We have the person in the wheelchair that is actually using it. We also have the clinician that is prescribing the system. We have the care team or the parents or the spouse or the paid caregiver, that is looking after the person that is with them every day. They have to interact with it.
(17:10):
And then there is also who is paying for the system. So you may have the insurance, or the VA in some cases. Or in a lot of cases it is the family of somebody. They want this now and do not care about getting insurance to pay for it. We have had a lot of people just buy one on their own.
(17:25):
Thinking through all those use cases too, of it is not just about the person in the wheelchair. For these folks, life really is a team sport with this whole care team. So we really have to focus on meeting the needs of every single person in that chain.
EW (17:43):
When I spoke with somebody else who has worked on powered wheelchairs in the past, he said one of the main questions that came up often was, "Where is the cup holder?" Do you have to deal with that sort of thing?
KL (17:56):
Yeah, people want all sorts of different customizations. We have actually 3D printed some cup holders that we had on some chairs at the office, so we could have our cup of coffee while we were driving around the office in the wheelchairs. So these sorts of customizations and the stuff that people come up with for their chairs are really cool to see.
(18:16):
Both the folks themselves, as well as some of the ATPs, that are the wrench turners that help set up these wheelchairs and maintain them for the users. People come up with some really creative solutions to all sorts of different customizations and needs of the users.
EW (18:36):
What is an ATP?
KL (18:38):
They are the Assistive Technology Professional. It is a certification that you can get and it allows you to work on these wheelchairs. So if you get a wheelchair delivered to you, the ATP comes along and does the install and the setup, makes all the adjustments. They might help tune the control system on the wheelchair for how you want to drive it. They are kind of the- I guess a wheelchair mechanic is not a bad way to look at. But then they also do a whole lot more, as far as helping address the medical needs of the-
CW (19:11):
How open are these wheelchairs? You said you integrate with other manufacturers' wheelchairs, that is your thing, you are an add-on. How much time do you have to spend reverse engineering multiple manufacturers, or are there only a few out there? I know nothing about this industry.
KL (19:28):
Yeah, it is an interesting industry in that regard. So the answer to how open they are, is basically they are not. There are only a couple control systems on the market, or two or three as far as the electronics. We work with one of those control systems, that we have reverse-engineered pretty much everything about that. We do not really work with the manufacturer of that a whole lot. Most of the communications and stuff we had reverse-engineered.
(19:56):
Then we have worked with some of the wheelchair manufacturers themselves, that buy that control system, for the mechanical integration. We have these sensor pods that we have designed, that we can place all over the wheelchair. For each model of wheelchair, we do a unique sheet metal frame, that allows it to mount to whatever points are available to mount on the wheelchair, as our point of customization there.
(20:19):
That is where we have been trying to collaborate and get stuff a little more open. But it is a tough space, it is pretty closed. Even modern software development practices are not so much of a thing in the wheelchair industry right now.
CW (20:37):
Do those companies not see the need for this kind of enhancement? They do not know how to do it? Why have not they done this?
KL (20:47):
It is a bit of both. I think a lot of it is they do not know how. The wheelchair manufacturers are actually really good on some of the mechanical engineering. Some of the suspension systems on these, and the seating assemblies and systems that they have come up with, are pretty incredible.
(21:04):
The software integration point, and how they can use software to help out the users, is pretty new to them. We were definitely the first of the industry to do over-the-air updates. We have seen a couple manufacturers that seem to be following us in that regard.
(21:22):
We have also done some stuff that some folks are copying, as far as setup. So previously to configure a wheelchair, you had to plug in a little programmer, and pull up on your Windows laptop with the specific special software. Then the ATP could go tune settings on the wheelchair.
(21:37):
Now we are seeing a little more of that being driven through mobile apps and web interfaces to these. And we are seeing the software updates start to become a thing in that area. So folks are definitely looking at what we are doing and taking note. But it is an industry that has not historically spent a ton of money on R & D. Getting that ship turned to where they are looking at the world in a modern way, has not been a very fast process.
EW (22:07):
It reminds me of LeapFrog, where one of the best things they did before they went defunct, was they forced Mattel and other big name toy manufacturers, to put the whole alphabet on toys, instead of stopping at the letter F because that is as big as the toy was.
CW (22:23):
<laugh>
EW (22:23):
Do you think LUCI is putting pressure on other wheelchair manufacturers, through customers or through regulations?
KL (22:40):
It is a little bit of both. Definitely we are seeing customers asking for more. Having the users be able to actually customize their chair a little bit without needing the ATP. Some new chairs coming out, we are seeing some of those apps and stuff, that let people do some of that.
(22:58):
The other place we are looking is from the regulation side. We are involved with a couple standards committees, to try and work on defining actual standards for this smart technology, of how do you test a wheelchair that has these sorts of safety systems. There was no existing standard for that.
(23:18):
We made up a process internally, that we are pretty happy with right now. We are working with the standards organizations to try and figure out how we can actually turn that into a standard, so people can compare safety systems. There are some other options out on the market, that do not as much talk to the wheelchair, but have beepers and stuff. So kind of setting up that. How would you look at, or how would a company go about testing and comparing the capabilities of these safety systems?
EW (23:49):
Some of the LUCI documentation compares it to a car, including talking about the Level 2 auto drive. I can see how you would get to standards from there, because we are doing better with talking about cars. But cars do not go into houses, they do not go on sidewalks, they do not go in the grass. Is it a reasonable comparison?
KL (24:15):
It is a great place to start, and it was where we started. It definitely breaks down once you get more into the details. We looked at the cars and said, "Okay, they are doing these lane keep assist features. They are doing the automatic emergency braking. There is the adaptive cruise control, that you can get." You start putting those together.
(24:34):
That was some of our initial inspiration for really looking at this technology, was the wheelchairs had a seatbelt, was the only safety feature. So we looked at, "Okay, well what does the auto industry have? And how do we bring that technology to the wheelchairs?"
(24:47):
But then as you get deeper in, it is a very different set of testing. One of the big challenges with a wheelchair, is they can zero point turn 360 degrees. You can spin around. Most cars on the road cannot do that. So it requires a different level of testing, level of putting sensors around it. The sensor coverage has to be a lot more complete around the wheelchair, because it can move in really different ways, and it is operating in tight environments.
(25:17):
One of the things we struggled with early on, where we had to develop our own radar and ultrasonics, was we tried looking at automotive sensors. Most of them cannot detect something that is only four centimeters away from the sensor, or being really close.
(25:33):
We had to go take a step back and say, "Okay, we need to actually detect stuff in really close. Because as a wheelchair user, you have to be able to get right up next to a toilet to transfer, or get right through tight doorways." Some of the apartments people have for these, that they live in with a wheelchair, are incredibly small, really tight quarters.
(25:51):
Being able to navigate those, it really is a different control problem and way of looking at the world, from a wheelchair compared to a car. But the high level of how we talk about the autonomy and some of the testing that needs to be done, does end up being really similar to the automotive industry.
EW (26:11):
Your sensors. You mentioned vision and radar and ultrasonics. Do you- I will tell you what I think you do with them, and you can tell me how wrong I am.
CW (26:22):
<laugh>
EW (26:22):
I would want to create a cloud bubble of everything that you could identify that was near you. So that you could say, "Okay, there is someone walking in front of me from the left." Or, "There is a curb coming up, and it is three feet away, it is two feet away, it is one foot away." Sort of a point cloud of everything out there, which is kind of what radar does on its own. But it could be augmented with other things, through Kalman filters or other heuristic models. How close am I?
KL (27:00):
That is actually really close. Yeah, we combine all of the feeds from the different sensors into this one point cloud view of the world. If folks go look at- We have a video of a- There is a visualizer that is available in our app, that actually shows all the points around the chair.
(27:17):
We stream this over Bluetooth from the system onto your phone, so you can use it as a back up camera view. Almost looks like some cars have, that top-down view of the zone goes from green to yellow to red, as you get closer backing up to something.
(27:31):
We use that view of the world, and then we have zones around the chair. So we say, "Okay, if something is in the zone that is around your right front caster, then maybe you cannot go turn to the right." Or, "If something is right in front of the footplate, you cannot drive forward, but you can go every other direction."
(27:48):
We combine all these sensors and create that view of the world, and then got to scale the joystick in different directions, depending on where the user is trying to go.
EW (27:59):
Can you talk about the algorithm of combination?
KL (28:03):
Yeah, most of it right now we just add all the sensors up. We have some algorithms that take out specific data we know to be false with the radars. If you have certain stuff, it will show a false point right in front of it. But for the most part, we just combine them all.
(28:17):
The main separation we do, is we treat the world pretty differently between what we call collision, which is things you can run into, versus our ground detection. So the driving off steps. Steps are a little more tricky, because we only have really one sensor that can detect them, which is the camera. We have a couple different approaches and processing that we do on that data, to make sure we do not miss steps around.
(28:44):
The other thing we have to use for- Some of the ways we combine sensors to make up for shortfalls, is like the camera is not very good at seeing glass. So where we have a camera on the system, we have an ultrasonic right next to it, that is looking the same direction, so you would not back through a glass window for instance.
CW (29:01):
Mhm.
KL (29:01):
So that is where we have that, "All sensors are wrong, but some are useful," is some of them are going to give wrong data, mostly missing objects, but some of the other sensors might be able to pick up and kind of fill in that gap.
(29:13):
Ultrasonics have a hard time seeing a soft, fluffy thing sometimes, for instance, so then the radars that are looking in the same direction of those, can pick up the object that the ultrasonic would have missed.
EW (29:29):
Do you have trouble driving over black rugs?
CW (29:30):
<laugh>
EW (29:30):
I know our vacuum does. It is very afraid of black rugs.
KL (29:35):
We do not have too much problem with dark floors. Shiny floors can be a problem for stereo vision, because it will actually see the reflection rather than-
EW (29:44):
Mhm.
CW (29:44):
Yeah, I was going to ask about reflections. Yeah.
KL (29:45):
Yeah, reflections are nasty. We are constantly working to improve how we handle those. We have gotten a lot better than we used to be, but they are still pretty hard.
(29:55):
The cameras we use have a infrared projector on them, that sends out basically noise. It is a little infrared laser. That helps it resolve areas where there is not good lighting, or if it is a really flat surface, because stereo vision does not work really well. You need something that contrasts. So it adds contrasting noise into the scene. That helps in a lot of situations. But the reflections are really the difficult one to deal with.
CW (30:25):
Cannot just fix that with polarized filters?
KL (30:29):
We actually do use polarized filters, so that is one of our tricks. It does work pretty good, but-
CW (30:35):
As long as you are aligned right. <laugh>
KL (30:37):
Yes, and we do align them right. So yeah, we have different ones. The front cameras are vertical, and the back cameras horizontal.
CW (30:42):
Back
KL (30:43):
So we actually have different- The flipped polarizers between the two.
EW (30:48):
Going back to sensor fusion. I am afraid right now to say that word "machine learning" here, because there has been some crankiness about ChatGPT lately.
CW (30:59):
I am in favor of machine learning. I am just not in favor of large language models.
EW (31:05):
Have you tried out machine learning for these problems? Because it is a lot of data.
KL (31:09):
It is a lot of data, and machine learning is something we have- We have played with it a little bit, and I think there are some applications for it in our product. But right now we do not have any machine learning on our product. Everything is deterministic, which is really nice for-
EW (31:27):
Testing.
KL (31:28):
Documenting how it works.
CW (31:30):
Yeah.
KL (31:32):
And how the- We can make a case of like, "Okay, this is exactly how the algorithm works," and can explain it. That really helps with some of our safety case.
(31:40):
We are looking into it a little more, for some of the individual sensor processing. Like looking at the data we get back from an ultrasonic, and determining where an object is. That could potentially be a really good application of it. We have kind of explored it in a couple of those places, but as far as a whole world look at machine learning of all of our data, it is not really a good fit.
(32:03):
We are also pretty compute limited, is the other aspect. We do not have a GPU that we can throw all of our data through, and run a big ML model on, because we are limited by the power. We use the battery in the wheelchair, so we do not want to suck up too much of the range of the wheelchair through the battery.
EW (32:27):
There are a lot of different things you need to balance. I mean, there is ease of use-
CW (32:34):
There is the chair itself, that you need to balance.
EW (32:36):
<sigh> Yes, but cost and safety and agility and ease of use-
CW (32:41):
Power budget.
(32:42):
And power budget. Are there other criteria that go into designing a powered wheelchair like this? Or a control system like this?
KL (32:52):
Yeah, there is a lot that goes into- Like where we place the sensors is a big balancing act, of keeping the sensors where they can see everything. But keeping them out of the way where they are not going to get damaged. Or where they are not in the way of the user transferring out of the wheelchair.
(33:09):
One thing that was interesting when we first started looking at this, is there has been a lot of university research projects and stuff that have, "Oh, we built an autonomous wheelchair that can drive around." Which is really cool, until you look at the picture and realize that they put a several thousand dollar 3D lidar in the seat.
CW (33:29):
<laugh>
KL (33:29):
So yes, the wheelchair drives around, but where is the user going to sit? And how are they going to afford it, because you used a really expensive 3D lidar. There is a huge balancing act between the yes, everybody is like, "Oh, an autonomous wheelchair would be really cool." But when you actually look at what that would be, and what that would take, it is not actually what the users end up needing at the end of the day.
EW (33:55):
There are people for whom an autonomous wheelchair would be good. But taking away the autonomy probably is one of the things you have to balance. You want it to react safely and well, but you also do not want it to do- I still do not know what I am saying. Maybe the question is, how do you prevent the tech from doing stupid things, when the human really, really wants to do something stupid?
KL (34:20):
Yeah. We have a rule that the user is always in control. So there is a button on LUCI that lets you override it. If you need to go drive off a curb that we think is a little too high, you can hit that button and drive right off that curb. Or if you need to get- Anytime you disagree with LUCI and say, "Nope, I need to do this thing."
(34:44):
A great example of that is pushing open a door. So if the door does not have an auto door opener, you might need to actually run into the door to get it open. So you can hit the override button and do that.
(34:56):
There are several different modes. A lot of our users kind of customize that, for how they use override. Some people drive with LUCI in override a majority of the time, but then just turn it on when they are in tight situations.
(35:09):
So that is another thing of keeping the human in control, if we want them to be able to do what they need to do, and not be limiting of what they can do. We do not want to tell them, "No, you cannot do that." It is just, "Hey, we do not think that is a good idea right now. But if you need to go do that to get across the street, then go for it."
EW (35:31):
How much of the system is open source?
KL (35:35):
Not a ton. We have some open-source work that we are doing with a few research partners. There have been some grants we have gotten with- Northwestern has been a big one, where we are working on some open interfaces to use LUCI to drive wheelchairs. As well as provide an interface for some of their researchers, that are working more on the autonomy piece.
(35:58):
We just saw a demo the other day, where they have got a system where as you hold forward on the joystick, it tries to center you up to get through doorways and stuff like that.
(36:08):
So we have opened source that side for them, to help do research and look into techniques. That is actually a set of ROS packages that expose our data into ROS, so that they can use that for control. But the system itself, the code, is currently not open source.
(36:27):
We are opening up pieces as we have seen opportunities to collaborate, and give some of our- We have got some users that are very tech savvy, that are working on different ways of driving. So we have given them some hooks to plug their systems in, to help control how they drive the wheelchair.
(36:47):
It does not make a lot of sense to open source all of it, because with the hardware we have, it really is very specific to the hardware. You would not be able to use it stand-alone. But where we have found good opportunities for collaborating with other folks, we are definitely open to open sourcing it. So if other people have things they would like to do with it, be sure to let us know and have a conversation about that.
EW (37:13):
How would they reach you?
KL (37:16):
They can reach out- Probably the best way is my email, for LUCI stuff is [email address], and I can direct that to our group that is working on that interface.
EW (37:27):
We will leave that in the audio bits, but we are going to take that out of the transcript or you will get lots of spam.
KL (37:35):
<laugh> Sounds good.
EW (37:35):
Anybody misses that and wants to contact Kevin, just email the show, or hit the contact link on the show webpage.
(37:43):
This sounds like a fantastic job. How did you land here?
KL (37:56):
Yeah, so the very first prototype of this system was built by my boss who is Barry's brother, Jered. It was built while Jered was a professor at the Colorado School of Mines, while I was there. He was in charge of the senior design program. He was working on this on the side, just for Katherine.
(38:16):
We had what we called our SparkFun special prototype, that was a bunch of Raspberry Pis and Arduinos and off-the-shelf little ultrasonics hooked together. Jered had asked me to help on that, when he was first working on it. So I did some work on it while I was at Mines.
(38:36):
Their original plan was to go talk to the wheelchair manufacturers and say, "Hey, we have got this proof of concept. You guys should build this thing." And they went and talked to the wheelchair manufacturers and they all said, "Mm. No. Not interested."
(38:51):
I had by that point graduated and was working at Keysight for a year. Got a call from Jered and Barry and they said, "Hey, we are starting a company. We are going to build this thing." So I moved back to Colorado and joined on.
(39:05):
So I have been with it since it was a very early prototype, and then all the way through turn it into a full product. It has been a lot of fun. It is a great job. We have been able to work on a whole bunch of different hardware, see the impact of what we have done for a lot of people, and honestly, driving wheelchairs around the office a bunch is actually quite a bit of fun. So it has been a great job.
EW (39:30):
Do you have anything you do specifically to test, that is particularly fun? I am imagining wheelchair basketball.
KL (39:36):
We do not do wheelchair basketball. No. Our main testing is we do kind of a "butt in seat" for every release and every sprint, where we all hop in the chairs. We have a course around the office we drive them through. We take full speed runs at our target we call Gandalf, so "Thou shall not pass, or run him over." It is a mobile target, so when we hit it, it moves out the way and we do not smash stuff.
(40:03):
In our previous office, we had put numerous holes in drywall walls all over the place, so we have since learned that testing on actual walls is a pretty bad idea. We had a sign up that said "We crash wheelchairs, so you do not have to" above one of the holes. And then we go drive at the curb outside. We have got that.
(40:26):
We also have done stuff where we took a whole bunch of chairs, the whole team. Everybody grabbed a wheelchair and we rode a mile up the street to go get lunch one day. So we go try and take them out in the world and actually do stuff with the wheelchairs.
(40:41):
One of my funniest testing stories, unfortunately I was not a part of this. My brother also worked for us and Jered were out testing at a library. They went into the elevator with Ross in the chair, went upstairs, did some driving around, swapped who was in the chair, came back down. The person that was sitting by the elevator did the funniest double take as they came out with who was in the chair swapped.
(41:08):
We have had all sorts of funny interactions with the public, of having a bunch of folks that are swapping around who is in the wheelchair, and some general goofing around. But it is actually, it is quite a bit of fun to go take these out.
(41:19):
It really helps us learn what our users are up against, navigating the world. From bumpy sidewalks to getting across the street in time. The world not built for wheelchairs. It really changes how you view the world, when you try and drive one around for a bit.
EW (41:40):
Do people treat you differently?
KL (41:45):
Somewhat. A lot of people give you a really wide berth, which is interesting. Or they kind of let you go. We go test in the mall a lot, because it is a big open public space that we can drive around. And have a lot of different flooring, and different other stuff. Just as people give you a really wide berth around, you definitely get treated a little differently in the wheelchair, than you would just walking around normally.
EW (42:10):
Sure, you are in something that weighs several hundred pounds, that could run over toes.
KL (42:18):
Yes.
EW (42:18):
Yeah.
CW (42:19):
Do people realize that? I do not think able-bodied people realize how heavy and powerful those chairs are.
EW (42:28):
The powered ones especially. Yeah.
KL (42:31):
Yeah, the large chairs. One of our tests we have to do is tip them over, to make sure that our tip detection works. Actually it usually takes two people really leaning on a chair, to even get the thing tipped over.
CW (42:44):
Which is probably good.
KL (42:45):
They weigh a lot. Yes, it is a feature that they weigh 300 pounds, but then it also makes certain things harder with them.
EW (42:52):
Moving them into vehicles, I imagine.
KL (42:55):
Yes, getting them into vehicles. You need a ramp, you are not going to be able to, even with multiple people, you really cannot pick up a chair and set it in a car.
EW (43:04):
What is the longest time you have been in a chair without getting out?
KL (43:12):
Probably just for a few hours. I mean, we will drive around for maybe half a day. We have definitely done some testing, where we have spent a good amount of time in the chair. So half a day driving around the office, kind of trying to live in it.
(43:23):
Usually we end up doing that when there is some particular bug that we are trying to suss out, that only happens once every few hours or whatever, and so we will be in a chair for a good chunk of the day.
EW (43:35):
I was just thinking how hard it would be, if you did not get to get up and stretch after that. But that is-
(43:42):
Let us see. How much training does it usually take to get wheelchair skills? I have sat in one before, and I think I might have run over my own feet.
KL (43:58):
Yeah, it really varies. So for folks that have- For users that have normal motor control and normal cognitive skills, it can be pretty quick to figure out how to drive. But then the subtleties of how to approach and get through a door, what the right line to take so that you do not hit it, is something that a clinician may work with someone training them on.
(44:21):
All the way up to, it can be a really hard learning process. There are folks that drive- So if you do not have motor control, you might be driving with a head array, where you are moving your head to hit switches. Or you might be driving with a sip and puff, where you control the wheelchair literally by sipping and blowing through a little straw that is hooked up to a pressure sensor. And there are some solutions out there as well, that let you drive with your eyes, that do eye tracking.
(44:52):
Learning to use those systems, that is not a day or two, that is weeks or months of being in the chair and getting good at driving it. So that is really- We have been- There is a clinic here in Colorado, called Craig Hospital, that is a spinal cord injury rehab center.
(45:09):
They will get folks that went from being fully functional, maybe they had a sporting accident and ended up paralyzed. A big part of their visit there is learning how to live their life, and part of that is in the wheelchair. So they are learning how to drive, how to get around, how to transfer in and out of the wheelchair, all of that.
(45:32):
It is a huge variance depending on the needs of the user, but with some of these drive methods, it can really be a long process. That is one of the things that LUCI can really help with, is that training process. You can put somebody in it, they can drive, but you do not have to worry about them running into something, if they get it wrong. It is actually a really helpful tool just for the driver training, especially with kids and these alternative drives, where you do not want- The cost of an accident during that training can be really high.
EW (46:10):
So does LUCI work with the alternative drives?
KL (46:13):
Yes, we do. So those plug in the same as a joystick. They show up as you hit the button and it goes forward, or goes backwards or turns. We plug in and work with most of those.
(46:27):
There is a mode of driving that we do not quite support right now, that is called "latch driving," where- This is pretty typical for a sip and puff user, where you actually say, "Okay, go forward," and then you do not have to continue providing that forward input. It just goes forward until you give it a stop command. So that one is a little bit more difficult to work with.
(46:44):
But all of the switch, like the head array and stuff, we work with and we test with some of those pretty regularly. A lot of our chairs have variants of these alternative drives on them, so we can test with those and experience driving.
(46:59):
For a while, I had a chair that had a joystick that you had a little thing that mounted it, so you could drive with your chin. You would literally use your chin on a joystick. So think you have picked up your Xbox controller. Stick your chin on the joystick and try and navigate around your video game. It takes a little bit of getting used to, for sure.
EW (47:21):
Do you see many of those that are fully custom? I mean, there is more of a movement to use maker technology, adaptive design. There is a group called ATMakers. I think you sent me one about Adaptive Sports Connection, that helps people on a one-on-one basis to develop adaptive technologies. How much of that do you see, as you are working on LUCI?
KL (47:53):
We definitely see some of it. Yeah, those adaptive sports people have a program called "Go Baby Go." It is a fantastic program where they customize those power- Big wheels little power toys for young kids to use to get around that are disabled. So they may customize the steering wheel so they can use it more easily. Or put some bumpers on it so they do not get their fingers pinched, that sort of stuff. So that these kids can get around, that cannot otherwise walk.
(48:24):
There is quite a bit of research that shows that for these kids that cannot walk, giving them some method of getting around, really can help improve how they learn about the world and everything.
(48:36):
The other side of that that we are working on, is other ways to drive the wheelchair. So one of the things that is on our GitHub and is kind of open source, is we are working on a digital interface for the wheelchairs. That if you have LUCI, you can use off the shelf gaming controllers. Like you can use an Xbox controller or PlayStation controller to drive your wheelchair.
(49:00):
The one we are really excited about with that, is Xbox has an adaptive controller, that you can plug in all sorts of custom buttons and stuff into it. It has got just a bunch of like headphone jacks on it, that you can hook up switches to, to help people game. We are working on how you can hook that up to LUCI, and let people drive with that.
(49:20):
So instead of having to wait and go through the whole whatever is available from the wheelchair manufacturer for switches and stuff, you can use some of this tech that has been developed for the gaming world, to help control your wheelchair.
CW (49:33):
You mentioned earlier documentation and safety. Is this a medical device? Is the FDA the people you have to get approval from, or is this not regulated?
KL (49:44):
This is a medical device. So we are a FDA Class I device, so that is where we have to kind of control the manufacturing. So the wheelchairs themselves are Class II, so they have to go through the more rigorous approval process. And then we are an accessory to the wheelchair, so we are a Class I device.
(50:01):
So we do operate as though we are designing a medical device. We have a lot of quality control and design processes that have to be followed, to check all the boxes that the FDA needs checked for that.
(50:15):
As well as a lot of those processes, like having requirements and doing testing, are just a good practice. That after having done it a bunch for a medical device, I think I would have a hard time not doing that, on consumer devices. It is the stuff you should be doing anyways.
CW (50:31):
Yeah, I was going to ask how it worked, since you are an accessory to a medical- I am trying to think of other examples where- It is a little odd. Does the FDA have a process for that? Or do they just, "This is a new medical device. And you are in a different class, because you are attaching to something that is in a different class"? Or?
KL (50:51):
There is a bunch of guidance, and there was a little bit of prior- So the alt drives for instance, I was talking about, the head arrays and stuff, those are usually not. In some cases they are made by the wheelchair manufacturers, but they are often made by third parties. Those are an example, that there was that prior art of it is an accessory to a wheelchair. Those are all Class I devices as well. So we fit under that umbrella right now.
CW (51:16):
How much real estate does this take up on the- I imagine part of the design challenge is adapting a suite of sensors to something that is already large, and has its own sensors and its own- I mean, they do not have accessory bays for you to just put an electronic system in, and rails to attach new sensors, right?
KL (51:42):
Yeah, it is a big effort to get it to fit right. So that is where we have had this pod level approach, and we do the custom sheet metal for each model. But even with that, depending on how somebody has ordered a chair, the seating assembly may move differently.
(51:56):
So we have to keep it out of the way of the seat that can recline all the way back to, I do not know, it feels like you are upside down. You recline some of these seats all the way back, that folks use for pressure injury mitigation. So you actually roll back and take the load off your skin, to get blood flow back in.
(52:14):
There are a lot of spaces on the wheelchair, that we have to be very careful about what we put there. Making everything small and compact so it fits in. Keeping cables out of the way, so they do not get pinched by the mechanisms and everything.
(52:29):
So there is a lot of effort for every chair that we have designed for, to get all the sensors in a place where they are out of the way, and not getting crunched or run into. And that we can still see the world from them. You can put a sensor out of the way, but it might not be able to see what we need to see.
CW (52:50):
Mm-hmm. Yeah. Right.
EW (52:52):
How often do you have to talk about cost-reducing the system?
KL (52:56):
It is not a everyday conversation, but it is definitely something. It is less of a cost reduction, more of a complexity reduction. It is trying to take complexity out of the system, so that it is easier to service, or it is easier to install. Or maybe allows for less complex software. You get rid of a microcontroller, that is a great win of, now I do not have to write code for that microcontroller.
(53:21):
We think about more of system optimization. Cost is one axes on that, but there are a lot of other pieces that go into that. How long does it take for somebody to install one? These get installed often in the field, by somebody that has- Our goal is to keep the number of tools down that you need to install it with, that sort of stuff. There is a lot of that optimization that goes in, but it is not necessarily just along the cost vector.
EW (53:51):
That makes sense. Kevin, thank you so much for speaking with us. Do you have any thoughts you would like to leave us with?
KL (54:00):
Yeah, I have a thought. This came from the podcast you guys had recently, where they were talking about learning stuff from different programming languages. And they get some advice to everybody, is to go learn something in a field adjacent to what you are working on.
(54:12):
So if you typically write embedded code, go try and design a PCB, or maybe go learn a web framework. Do some web programming, or go learn Rust even. I know that is a sore topic here. But every programming language I have learned, has taught me a new way to approach problem solving, even when I am not programming in that language.
(54:33):
And doing hardware and embedded software design. There are a lot of things I do different on hardware because of that. Because I have been the software developer that the hardware was thrown over to, and realized I did not have the test points I needed. Or we did not match the reference design, so then all the code I wrote on the dev board, now I have to make a whole bunch of changes, to get it to work on the actual hardware.
(54:57):
So I think there is a lot of- Even if that is not something you are going to do regularly, is go learn something adjacent to what you are working on. Go talk to the hardware engineer. Go talk to the web programmers. There is a whole bunch we can learn outside your immediate skillset, and it will apply in ways that you would not have thought possible when you started it.
EW (55:20):
Our guest has been Kevin Lannen, embedded software engineer at LUCI Mobility.
CW (55:27):
Thanks, Kevin.
KL (55:27):
Thank you.
EW (55:29):
Thank you to Christopher for producing and co-hosting. Thank you to our Patreon group for their support. Thank you to Memfault for sponsoring this show. And of course, thank you for listening. You can always contact us at show@embedded.fm or hit the contact link on embedded.fm.
(55:45):
Now I actually have two quotes to leave you with, one for you and one for me. The first one, Helen Keller, "It has been said that life has treated me harshly, and sometimes I have complained in my heart, because many pleasures of human experience have been withheld from me. If much has been denied me, much, very much has been given me."
(56:07):
And the second from Charles Schultz, "Happiness is a warm puppy."