269: Ultra-Precise Death Ray
Transcript from 269: Ultra-Precise Death Ray with Alan Cohen, Elecia White, and Christopher White.
EW (00:00:06):
Hello, and welcome to Embedded. I am Elecia White, here with Christopher White. Have you ever wished for a book that could help you understand the path from idea to product? It turns out there is such a book, and it is a pretty good one. Its author is our guest, Alan Cohen. "Prototype to Product: A Practical Guide for Getting to Market."
CW (00:00:30):
Hi, Alan. Thanks for joining us.
AC (00:00:32):
Hi, thanks for having me.
EW (00:00:35):
Could you tell us about your background?
AC (00:00:38):
Sure. I am a systems engineer. My background, prior to doing systems engineering, is in electronics and software. My career has been in product development, primarily in the development of medical devices.
AC (00:00:51):
So probably about 80% of what I've done is medical devices, but I've done a few other things as well. And I'm also an author of the O'Reilly book, "Prototype to Product: A Practical Guide to Getting to Market."
AC (00:01:03):
Currently I'm at Massachusetts General Hospital in the Department of Radiation Oncology, where I'm helping to do some renovations on a very, very large medical device, something called a proton therapy system, which is basically an ultra-precise death ray that does a 3D scan of a tumor.
EW (00:01:23):
The listeners are going to be really angry when I don't ask anything about the ultra-precise death ray.
AC (00:01:30):
Yeah. [Laughter].
EW (00:01:31):
But maybe we'll get to that at the end or maybe not.
AC (00:01:34):
Okay.
EW (00:01:36):
Okay. So lightning round is on break due to recent extensive lightning-related questions. So you -
CW (00:01:44):
You've been spared.
EW (00:01:44):
You've been spared.
AC (00:01:46):
Okay.
EW (00:01:46):
I hope that's okay.
AC (00:01:49):
Okay.
EW (00:01:49):
So let's go straight to your book. I have heard that if everyone in Silicon Valley and other tech centers read and followed the advice in your book, products would be cheaper, less engineering time would be wasted, and general happiness would descend over the world.
CW (00:02:08):
And puppies.
EW (00:02:10):
Happiness and puppies? Is that all correct? I mean, is that a good summary of your book?
AC (00:02:16):
Well, I think that that's certainly the intent, and I think it would go a long way. It's not the only thing that would cause those things to happen. But yeah, I think that the intent is to give people a realistic overview of what the product development process is like.
AC (00:02:35):
A lot of people just sort of stumble into it, and try different things, and find out that it's a little different than what they thought it was going to be.
AC (00:02:43):
So the idea is to give people an idea of what it will be like, and also to give them some recipes for things that will help them to minimize the path between prototype and product, and getting to market.
EW (00:02:57):
So who is your intended audience? Is this a MBA sort of book or an engineering sort of book?
AC (00:03:04):
Well, the intent is...for it to be an everything and everybody kind of a book. So...one thing I should maybe describe a little bit is what a systems engineer is. I think a lot of your audience probably knows what that is, but some may not.
AC (00:03:18):
So a systems engineer is an engineer that's responsible for the system as a whole. So in any large engineering effort, particularly when there's hardware involved, there'll be different types of engineers. There'll be the software developers, there'll be hardware, electrical engineers, mechanical engineers, designers, various disciplines.
AC (00:03:41):
So a systems engineer traditionally is the person, or set of people, that try to bring everybody together as a group to make sure that you end up with a product that adheres to what it's sort of goal is, the vision of the product. As opposed to being different disparate sets of things that happen to be...pasted together.
AC (00:04:08):
And so as a systems engineer, I'm working with various people in various engineering disciplines, but also working with various management and sales types, and different people throughout the organization.
AC (00:04:23):
And I tend to spend a lot of time describing to different people...I guess, first of all, what the overall product development process is, but then also each group kind of wants to know what the other groups are doing and why things are taking so long.
AC (00:04:38):
Or...the CEO might say, "Well, I've got a $200 cell phone, and I've got USB on it." So I was going to take $20,000 in four weeks to design USB on our product. And so I have to describe those types of things.
AC (00:04:55):
So the hope was to write a book that would cover sort of a little bit of everything at a high level and how everything hangs together so that people participating in the project would understand those things.
CW (00:05:08):
And it's that job of integration that gets skipped over a lot.
CW (00:05:13):
And having somebody have that actual job of getting all of the different people in the same room, sometimes it's just getting people in the same room to argue or to hash things out, that's a really important thing that a lot of companies just kind of skip over until the end of the project when things don't work together.
AC (00:05:29):
Yes. I agree with you a hundred percent. It's critical. And the best time to do that is the beginning, as early as possible. Because as I touch on in my book, sort of the fundamental law of product development is that the earlier that you find issues, the things you want to change, the cheaper it is.
AC (00:05:50):
As time goes on, things become exponentially more expensive to change. So getting people together at the start and arguing in some sort of a process, that's sort of where the systems engineering comes in, it puts a process around those debates and arguments.
AC (00:06:06):
So you come out of those with everybody hopefully being on the same page, and requirements and specifications for a great product. And then people can go off and build it.
EW (00:06:18):
Requirements and specifications. Gosh, that sounds Waterfall-ish. That sounds so nice. That sounds so great.
AC (00:06:28):
Well it doesn't -
EW (00:06:28):
Can we please go back to a system that makes sense? Oh, sorry.
AC (00:06:34):
Yeah. Well, I shouldn't -
CW (00:06:37):
Well, coming from medical, that's unavoidable.
EW (00:06:40):
It's true.
AC (00:06:41):
Well, it is. Although I don't want to leave anybody with the impression that you start off with perfect -
CW (00:06:49):
Right.
AC (00:06:50):
Requirements and specifications. You can do a reasonable job at the beginning, better than probably most people do, from what I've seen in product development. But things will evolve as the system moves forward, as the development moves forward.
EW (00:07:08):
It's funny. In your book, as I discovered it, I started highlighting things, and I'm not much of a highlighter as far as books go, but stuff just made sense, and it made sense in ways that I could explain to other people.
EW (00:07:24):
One of the things you had was a quote from General Dwight Eisenhower, "Plans are worthless, but planning is everything."
EW (00:07:35):
And then you went on to say, "While specific plans are inevitably broken by the end of the first week of work, spending substantial time and effort in the planning process is indispensable." Could you talk a little bit more about that?
AC (00:07:50):
Sure. Yeah, so I think there's a few reasons why you put together a project plan. One very important reason is for estimation. Product development projects...tend to be over budget, sometimes by a lot, and they tend to be delayed sometimes by a lot.
AC (00:08:14):
A lot of the reason for that is just lack of understanding what the project actually involved. So by putting a lot of time into building a detailed project plan, even though a lot of your details might be wrong, you're going to catch a lot of things that you wouldn't sort of take into account if you're doing things off the top of your head.
AC (00:08:36):
So let's say you're talking about...a board spin, electronics, doing a new PC board...When you're sort of just guesstimating how long it's going take to do a new board spin, you might think about, "Well, the layout should take this long, and then we build it, and give it a week to test."
AC (00:08:58):
But you're not thinking about maybe, "Aw, gee, we've got to make sure we have a couple rounds of reviews of the board before we send it off to the vendor." And those things need to be scheduled with people.
AC (00:09:12):
People can't just pop out of their cubes and do a review when you want them to do a review, not if you want them to do a good review. You have to take into account the initial board bring-up, and what happens if it turns out that there's some problem on the board, and you need to do some cuts, and jumpers, and things like that.
AC (00:09:33):
So as you walk through the process in detail, you'll pick up a lot of issues. And then suddenly you'll realize that, "Gee, it's going to take twice as long as my gut told me it was going to take," when you put all these things together. So that's important. The other thing that's important is to understand dependencies.
AC (00:09:50):
I guess I touched on that a little bit, but there are different parts of the project that could be dependent on other parts.
AC (00:09:57):
And if you're not thinking about what those dependencies are, then you could get very surprised when some part of the project's ready to go forward...Let's say the electronics are done, but the software's not caught up yet, and it doesn't have certain milestones met yet.
AC (00:10:15):
So you need to understand what those things are that you're dependent on, that the different parts of the project are dependent on, in order for things to proceed at the fastest possible rate. Another thing that you can do during planning is to understand, "What are the signs that I'm succeeding or failing?
AC (00:10:35):
So you can think about, "How do I show myself and everybody on the project that things are progressing the way we want them to progress?" Or conversely, "If they're not, we can flag things when there's a problem and we have to attend to it." So that's kind of what I mean by planning.
AC (00:10:56):
The best laid plans will break quickly, but you'll at least have a good shot at a reasonable plan. And as time goes on, you keep your project plan up to date, and it'll give you a good idea of what's going to happen moving forward.
EW (00:11:18):
But these do take time. I mean, ideally they save time in the end, but so many times I hear from people, "Oh, writing a requirement spec is just too much effort and then we'll have to change it every week. We won't get anything done. We'll spend all our time dorking with the documents."
EW (00:11:39):
How can I tell them, "No, maybe if we write a good document and just update it once a month, we'll all be making the same project?" How do I get people to pay me to do the right sort of engineering instead of just slamming it all together?
AC (00:11:57):
...It's a very valid point. I think that sort of both sides have a good point, right? I think that engineers, doers, the people who are building the system, don't want to spend time dealing with filling out forms, and planning, and things like that, particularly not once they're in the thick of it.
AC (00:12:18):
And it's also difficult. It can be difficult for organizations to accept that kind of planning, spending money on that...The people who are non-technical,...let's say non-technical people, they want to see engineers doing engineering very often and not planning.
AC (00:12:44):
And I don't actually have a great answer to that other than to pick good companies to work for. So in my experience, company culture is very difficult to change. And...so the very good companies that I've worked for or with, there's a very good understanding of how important this process is.
AC (00:13:07):
And in fact, I'll tell you a little story. I remember one time I was working for a company...We had about 150 designers and engineers, and we were doing contract development. About half of our work was contract medical device development.
AC (00:13:26):
And one of the large medical device makers, a very good well-known maker, they needed some work done. They needed a new device developed, next generation of some device. And they sort of did a bake-off, and they started with about ten companies. And they got initial quotes from those companies and ours was one of them.
AC (00:13:45):
And I was, I co-led the effort to quote for them. So we made it through that first round. They picked the three ones they liked the best. And then they asked for very detailed quotes.
AC (00:13:57):
So we gave them a detailed quote, and I got a call. And the person who was leading the effort on their side said, "We'd like to come over and talk with you guys about your quote." So I said, "Sure, of course." So they came over, and they started by saying, "You guys were the most expensive by far of the bids we got."
AC (00:14:17):
So of course my heart sank, because usually cost is the driver. And they said, "But you guys were the only people who obviously understood what needed to be done and did the proper planning to understand what this project was going to look like. So therefore you guys get the job."
AC (00:14:32):
So in that case, even though we were more expensive, because that was an organization that understood how important the planning was, they would pay premium for that.
EW (00:14:42):
It's good to hear a success story when someone actually realizes that you've put thought into it and is willing to pay the extra money. But I know that when I've done a schedule, or I've seen when Chris does a schedule, and we think about the whole system, even though our part is the firmware, we tend to be the end.
EW (00:15:03):
So you kind of want to think about the whole system so that you don't just end up being the end of the dog, of the tail, of the wagging.
AC (00:15:12):
Yeah.
EW (00:15:13):
And yet I've been holding the result is far too long. We should be able to do better than that. We should be able to innovate.
CW (00:15:19):
"You engineers, you always sandbag things."
EW (00:15:22):
Exactly. And then at the end of the project, it pretty much went according to our schedule, because we had the mechanical electrical retry. Because of course it's going to happen. How do you make them listen?
EW (00:15:39):
How do you make them realize that, "No, really, this is what it's going to be. You can't say we're going to ship whatever we have in a year. It's a minimum viable product for a product that takes 18 months to do."
AC (00:15:53):
Yeah. So...I know some sort of tactics that have been helpful, but I don't have a magic bullet for it, certainly.
AC (00:16:04):
So one thing is, one of the nice things about a detailed project plan is that you can put that in front of people and say, "...Here, I've got 200 lines of tasks on this project plan on this Gantt chart, and they're documented. Let's walk through this together and maybe you can help me."
AC (00:16:25):
"Maybe I'm not understanding the project properly. And maybe, as we walk through this, you can tell me that there are things here that we don't need, or that I'm overestimating time on." And that sometimes can be very good, because then people, as they walk through it, they realize that you've really thought things through.
AC (00:16:41):
And also maybe you'll find out that you're over-engineering something. The other thing you could do is set things up as sort of contingencies based on assumptions not being correct.
AC (00:16:54):
So,...and actually, this is very important whenever you do a project plan, is to put together a set of assumptions...that are reasonable assumptions. Don't make kind of crazy assumptions.
AC (00:17:09):
And then say, "Hey, in the event that X happens, then this project changes, and here's sort of how it might change," rather than building everything right into the project plan initially.
EW (00:17:22):
Yeah...Contingency or plan Bs, paths, is nice. And I agree that bombarding them with actual information is sometimes helpful.
EW (00:17:36):
And yet I have still been in that meeting where they say, "Well, we just can't afford for it to go on this long, so it'll have to be shorter." And yeah, I guess I've walked from at least one of those companies, but I wish more people would realize.
CW (00:17:51):
Well, I think that...part of it is back scheduling, right? They decide we need the product by June of 2019.
EW (00:17:58):
Yeah.
CW (00:17:59):
And so everything has to be done by April of 2019, and then everything has to line up to that. And the resources you have are the same. And it's just kind of a fiction, right? And then -
EW (00:18:12):
Oh, yeah. Recently I had a client that was like, "Oh, we want it by March 1st." And...I gave them the schedule for my work, and it led out to June. And I said, "You're not going to have it by March 1st, no matter what I do. So yeah, it's not that I'm taking too long. It's just that you have no idea that it's going to take this long."
CW (00:18:30):
Yeah, I guess my point is that's contrary to both Agile and Waterfall, and any other method where Agile is supposed to be, well, there is no set date really. You're working towards a minimum viable product at all times, and you work in small increments.
CW (00:18:45):
And Waterfall as well, we spec everything out, and then we plan based on how long we think it's going to take, and then we get the date. So back scheduling is, I think, the root of a lot of the problems. And it's contrary to almost all planning.
EW (00:18:59):
But it's important for businesses -
CW (00:19:01):
Well, it's important, right. Right.
EW (00:19:01):
- because they have this much money.
CW (00:19:03):
There's this conflict. There's this conflict -
EW (00:19:03):
They have to make it at this market.
CW (00:19:04):
"We have to ship something for CES," or "for some conference." So yeah, there's always that tension.
AC (00:19:13):
Yeah. And I think certainly you can have a conversation around that. So...if a company has something they want to hit, like CES, or one of the medical trade shows, or something, I think it's good to sit down and have a conversation about that, and say, "Okay, well, let's talk about what could be done by then."
AC (00:19:35):
And then, try to prioritize and understand what's most important here. Is it having something, because you've promised your shareholders that you'd file with FDA by a certain date, or is it more important to get the final product out?
AC (00:19:52):
...It can be challenging. It takes awhile. I mean, planning..., it takes time and people often feel like that's taking away from "doing" time.
EW (00:20:03):
Yeah.
AC (00:20:04):
But it really is "doing" time...Your product will be completed faster and cheaper if you put that legwork in. And it's interesting if you look by industry.
AC (00:20:17):
The industries that do sort of the largest, scariest engineering projects, I think, so, aerospace, automotive, healthcare, medical devices, military,...they have very strong cultures of having a process where you do a good job, a thorough job, of planning beforehand. You don't just sit down. You start working.
EW (00:20:42):
Yeah. And when I went from a medical company to a military company, or FAA sort of company, it wasn't hard. I already knew all the processes.
EW (00:20:54):
But when I went from an FAA product to consumer toys, I was severely over-engineering things. And I just needed to understand, "No, it's okay if it fails in the field...It's fine."
AC (00:21:10):
Right.
EW (00:21:10):
Nobody dies. It's fine.
AC (00:21:12):
Right. Exactly. Well, you know -
EW (00:21:14):
Maybe a kid cries, but that's okay.
AC (00:21:16):
Yeah. Right. Yeah, absolutely.
EW (00:21:20):
One of the things in your book that I highlighted, I can't believe highlighted this much, "The goal of detailed product definition is to minimize bad surprises. I sometimes think we should change the phrase detailed planning to surprise management as it sounds less stuffy."
AC (00:21:39):
Yes.
EW (00:21:40):
Have you gotten any takers on that? Because...yeah, if I told a CFO or a CEO, "No, no, no. We're not going to go into detailed planning right now. We're just going to do a little bit of surprise management," they would be into that.
EW (00:21:56):
I mean, I do sometimes say risk management, and they're more willing to let me do a plan.
AC (00:22:03):
[Affirmative].
EW (00:22:03):
Have you had much success?
AC (00:22:06):
Well, yeah, I do almost always use that phrase when I'm discussing detailed requirements, and specifications, and planning, to continually remind people that that's what we're really trying to do here. In the medical area, risk management tends to mean something, a little different.
EW (00:22:25):
Yeah.
AC (00:22:26):
But it is managing risk, right? It's managing the risk of an unpredictable product or an unpredictable development cycle, which is ruinous.
AC (00:22:36):
If you can't predict how much something's going to cost and how long it's going to take, you can get into pretty big trouble as a company. So yes, I do use that phrase, and it does resonate with people.
EW (00:22:49):
Okay. So I have been getting into detailed parts of your book, but I probably should go back to the general question of how do you suggest getting from a prototype to a product? Yeah. Can you summarize your book in 25 to 30 words instead of 400 pages?
AC (00:23:13):
Let's see. Very thoughtful iteration...So there's a tension in product development, and it's kind of the tension between Agile and Waterfall in a way. When you're building a product, it's one thing to have a prototype, right? So nowadays it's so easy to put together a prototype...It's unbelievable -
EW (00:23:35):
It's so wonderful that we can now. It used to be hard, and now..., "I'll just go to SparkFun and Digi-Key. Boom, boom, boom."
AC (00:23:42):
Yeah, exactly. It's...incredible. My wife's tired of me sitting there and babbling about how amazingly easy it is to put together something that does something unbelievably cool. But building a product, I think, is now more difficult than it's ever been, because there's so many features in products.
AC (00:24:03):
There's processors, and everything, and software, and sometimes different layers of software, and all kinds of fancy electronics and stuff. Displays. So let's go back to Waterfall and Agile for a second. So Waterfall, it doesn't have a very good reputation these days, a very good name these days.
AC (00:24:27):
Because I think that what had happened in the past is people who maybe weren't engineers would come up with some sort of a schedule for building things and say,"...We know what we're going to do, so therefore we know what the product's going to be, and we know the engineer should know how to build it."
AC (00:24:45):
"So therefore, this is the schedule we're going to follow, and this is the path we're going to follow." And that doesn't work so well, I think, because there's a lot of things you don't know when you're building a product. You really usually don't know exactly what the customer wants.
AC (00:24:59):
One of the things I discuss in the book is that, in the first chapter, "The 11 Deadly Sins of Product Development," you should never assume that you know what the customer wants. You should also never assume that the customer knows what the customer wants.
EW (00:25:14):
Yes. Yes, yes, yes.
AC (00:25:14):
...Yeah. Right. You've been there, [huh]? The customers, I think, don't know what they want until they have it in front of them. So that means that you can't just sit down -
EW (00:25:28):
And then they just know they don't want this.
AC (00:25:30):
Well, you go through a lot of that, right?
EW (00:25:33):
Yeah.
AC (00:25:33):
And that's absolutely true. It's sort of a funny thing that when you build something that people like, maybe they're excited for five minutes, and then...they're bored.
AC (00:25:43):
Because it's like, "Okay. Yeah, this just does what I want. Okay." So,...I want a parade in bells, but I get, "Oh. Yeah, this is great. This is what I want. Thanks."
EW (00:25:58):
Yeah.
AC (00:26:00):
And then I feel like I've done a really good job.
CW (00:26:02):
"Oh, you did what I asked. Great."
AC (00:26:04):
Yeah. Right. Yeah. Or, "What I wanted." Well, sometimes they're good. The really good customers will say, "Oh, well, it's doing what I asked, but it turns out that that's actually not going to work well." It's like, "Yes, somebody who is sentient." That's great.
CW (00:26:19):
I don't know what I want, but that's not it.
AC (00:26:22):
Yeah. I think you have to start off assuming that that's generally true, particularly if it's a product that's something very new.
AC (00:26:30):
If it's the ninth generation of some product that's been around for eight generations, probably marketing has a pretty good idea of what the product should do, but if they're a good marketing group, they'll want to put prototypes in front of people, functional prototypes...
AC (00:26:46):
I mean, it could just be UIs that mock the UI that they're intending to build to make sure that people can use it properly.
AC (00:26:54):
But I think in general, and particularly for products that are really new things, until you put things in front of people, and you iterate a number of times until you get to the point where users can look at some sort of user experience, let's say, and say, "Yeah, that's the user experience I want, and that's what I need to get my job done," you really don't know what you want to build.
AC (00:27:18):
And that's kind of more Agile-ish right. So...when I think of Agile, I think iteration, and when I think of Waterfall, I think, prescribed. and I think somehow you need to strike the balance between those two, both in terms of the product's functionality but also in terms of the nuts and bolts engineering.
AC (00:27:43):
So, "Gee, I need to build a circuit that does such and such," or, "I need to write code that does this thing," you shouldn't assume that you're just going to know how to do that first time out. You need to have a process that allows for some iterations until you get things right.
AC (00:28:00):
But to do that in some sort of a framework where...it's not 100 monkeys typing on typewriters trying to get Shakespeare, that there's some guidance in there, some sort of a process.
EW (00:28:10):
I have seen iterative Waterfall as one of the things that goes in medical documents as the way the process works, where you do a fast Waterfall that's a prototype, and then you do a couple of slow ones, and then you do a fast one that's the final product after everything's been cleaned up.
EW (00:28:29):
Do you use that terminology or do you have another?
AC (00:28:33):
I have used that terminology. I think that that could be used in different ways. So I tend to be pragmatic in that I will use whatever terminology will resonate with the people who I'm working with and/or working for.
AC (00:28:51):
So, I would actually describe what I end up doing as being iterative Waterfall, although more iterations within a Waterfall. So there'll be specific processes, there'll be gates, that sort of gate moving from one phase of a project to another phase of a project.
AC (00:29:12):
And there are some, I think, very good reasons to do that. But within those phases, it'll be more Agile-ish..and there'll be a lot of iterating.
EW (00:29:26):
Yeah. But it's also important when you think about iterative Waterfalls and Agile that you do the whole process. You don't just start with, "Oh, okay. I'm in the middle of development phase, so I must just be doing development."
EW (00:29:42):
No, in the beginning of this development phase, you probably want to maybe respect this, maybe do some testing. You don't want to do it linearly. That's where Waterfall got us in trouble is, you were supposed to design, implement, and then test with no spins. It was just one way, and that's no good.
AC (00:30:04):
Oh, yeah. And you need to do all those things in parallel...I don't know if I cover this in the book. I probably do. I don't know if I have any illustrations.
AC (00:30:14):
But if you were to look at the different activities that need to be done, let's say, marketing research specifications, requirements specifications, building, testing, those things really happen in parallel to a large degree.
AC (00:30:36):
Now, there are some things that happen a lot at the beginning and very little at the end. So requirements and specifications, for example, and testing usually doesn't start during the planning phase, but you should be testing continuously while the product is being developed.
AC (00:30:54):
I think if you're not developing your tests while you're developing your product, you're going to run into some bad surprises.
EW (00:31:03):
Can you talk about this on a system scale? Because you're a systems engineer, but this same mentality works...for anybody.
EW (00:31:15):
I mean, for sitting down today and programming what I need to get done today, if I start out with the idea of requirements, which is, "What do I want to get done? What is this code supposed to do?"
EW (00:31:28):
And, "Testability. How am I going to make sure it does this," it's a lot different than sitting down and going, "Oh, well, I kind of want something like this and...yeah, I'm just going to type it. I don't need to think about it. I'm just going to type it."
EW (00:31:45):
Do you try to convince people to use this sort of process on a smaller scale within their own disciplines?
AC (00:31:57):
Oh, absolutely. Yeah. I think that the development process really needs to be set up to work that way as you're describing. The other thing that's very important, which you touched on, is people need to buy into what's being done.
AC (00:32:18):
So you can't just prescribe to, say, electrical engineers, "Here's how you're going to do this." I mean you can, but that's not fun for anybody.
EW (00:32:27):
No.
AC (00:32:27):
I think it's super-duper important that anything you do in the way of process, or really anything else, that it's important to do I guess quasi-organically, I'm not quite sure how to say it.
AC (00:32:42):
But basically work with people to put processes into place that they feel are going to be helpful to them, and aren't just those people with the suits telling them what to do.
EW (00:32:54):
It's not make-work -
AC (00:32:56):
Right.
EW (00:32:56):
- if you're doing it right.
AC (00:32:58):
Yeah...There should be a really good reason for everything. And if people don't understand why they're doing something, or they think it's a bad idea,...if you can't get to the point where you can convince them, then maybe you shouldn't be doing that, or you need to do it differently. I think that's critical.
CW (00:33:17):
Yeah. And I think people underestimate, well, it's going to sound funny, but, how good it feels when you execute to a plan.
EW (00:33:25):
Oh, yeah.
CW (00:33:25):
I know The A-Team, "I love it when a plan comes together," but we've sort of lost that in a lot of development lately where we're just all kind of fighting fires on a bi-weekly basis, it seems like, and making little bits of progress. And then at the end of it, somehow something comes out of it, and you're like, "Whoa. That was tough."
EW (00:33:44):
Look, a spaghetti monster.
CW (00:33:46):
But the times I've been involved in a development process where we had a good plan, yeah, that we iterated on, and then we actually managed to execute on that, that feels amazing. Because it kind of validates what you were thinking, right?
EW (00:33:58):
And you end up with something that you wanted.
CW (00:34:01):
Yeah.
EW (00:34:01):
As opposed to a bunch of little things that kind of work together.
CW (00:34:04):
Kind of a Darwinian approach to development, of, "Everything that dies, we abandoned, and this is the product that came out at the end."
EW (00:34:13):
I think Chris and I are totally on board, Al, but have you read "The Mythical Man-Month?"
AC (00:34:22):
Yes, I have. Actually,...because you had mentioned it recently in our correspondence, I just went back. I had rented about 20 years ago or something. And I went back and read it,...and I enjoyed reading it....I had, throughout my career, found that a lot of what Mr. Brooks said was absolutely true.
EW (00:34:41):
But so few people do it, and it would make engineering lives easier. It would make business lives easier. And yet people aren't willing to. Why?
CW (00:34:55):
I still have conversations where people say, "Oh, we need to give a copy of that to so-and-so." Or nine copies.
EW (00:35:02):
Nine copies.
CW (00:35:02):
Yeah.
EW (00:35:02):
So they can read it faster.
CW (00:35:04):
Yeah, yeah.
AC (00:35:08):
Right. They can make that baby in one month if they've got nine women, right? So here's something I think we don't talk about a lot, particularly as engineers, but psychology is an amazing thing. And actually, I come from a family of psychologists.
AC (00:35:28):
My sister's a practicing psychologist,...she has a PhD in psychology. My father has a PhD in psychology. And the ability for the human mind to sort of hold two contradictory things at the same time and not have them affect each other is pretty amazing to me.
AC (00:35:47):
So I find it's often the case that people will totally agree that...it's super important to find out all the potential pitfalls early in a project, but then an hour later to say, "We don't have time to figure this stuff out. We just have to get going."
EW (00:36:05):
Yes.
AC (00:36:05):
And it's just an amazing thing to watch. And you can remind them that, you can't make a baby in one month if you have nine women. You can remind them of various things that they know to be true, but sometimes they will just move in a certain direction. And I don't know how to change that.
AC (00:36:32):
I think it tends to be an organizational thing. There are some organizations that believe in mind over matter. And they usually do not do very well.
AC (00:36:44):
But I find that the really good companies, so for example, where I'm at now, Massachusetts General Hospital, and some of the other people I've worked for, people are very aware that reality is reality.
AC (00:36:56):
And you've got to accept it and do the best you can with it. And no amount of magical thinking is going to make things work out better.
EW (00:37:07):
But I've been at places where they say, "Oh, well, we'll just try."
AC (00:37:11):
Yeah.
EW (00:37:11):
"We'll just work hard."
CW (00:37:12):
Literally, "Give the old college try." I've been told that.
EW (00:37:16):
Do you ever feel like Cassandra doomed to tell prophecies no one believes?
AC (00:37:21):
Yes. In fact, I often tell people that I'm thinking of changing my name to Cassandra. So...yeah, that's my lot in life...And it's not because I'm particularly brilliant or anything. It's just, I've seen enough of this to know. When projects fail, they tend to fail in certain predictable ways.
AC (00:37:42):
That's actually where my first chapter came from. I think it's called "The 11" -
EW (00:37:47):
"11 Deadly Sins?"
AC (00:37:48):
- "Deadly Sins." Yeah. Right.
EW (00:37:49):
Yeah.
AC (00:37:49):
It's basically the things that tend to crop up all the time when projects fail, when product developments fail. And you can look at a project, and they're not doing some of those things, or they're committing some of those sins. And you just know there's going to be a problem, and all you can really do is step back and watch.
EW (00:38:11):
Yeah. I liked that chapter. Was it one of your favorites?
AC (00:38:14):
Yeah, it was my favorite.
EW (00:38:16):
I mean, you talk about the vice of laziness, about putting off serious testing, the vice of assumption, knowing what people want...It was so funny to have them be called vices, because that's what they are.
EW (00:38:32):
And it's laziness, or the vice of cluelessness, and the deadly sin there is number seven, not addressing regulations.
AC (00:38:45):
Right.
EW (00:38:45):
Oh, please honk your horn if you've been there.
AC (00:38:49):
Yeah. All the time. Yeah. It happens. Although...I can't take credit for couching those as deadly sins and vices. That chapter was originally titled "11 Ways to Screw Up a Project."
AC (00:39:05):
And my editor at O'Reilly, actually two editors there sort of put their heads together to figure out what would be a better way to describe that. So we can thank them for that cleverness.
EW (00:39:19):
It's a good framing. And I like the way that you have vices, and you have sins, and the sins are related to the vices, but they aren't one for one.
AC (00:39:29):
Right.
EW (00:39:30):
The vice of hubris, which is deadly sin number ten, not planning to fail.
EW (00:39:36):
I am so amazed at how much information you can get in a meeting where you say, "Okay, let's pretend it's six months after the ship date. And we have failed, failed in every way we can possibly fail. How do you think we did fail?" And going through, and realizing that people know what's going wrong, but you didn't ask the question right.
EW (00:40:02):
And that question leads to people saying, "Oh, well, we didn't do our marketing early enough," or "It's uncomfortable," or "People are allergic." And sure, you get a bunch of stuff you don't need. You get a lot of things that may not happen, that are easily preventable, but at least you get a picture.
AC (00:40:21):
Yeah. I love the way you phrased that. If it's six months after the project, I think you said six months, and we've totally failed. "Why have we failed?" I haven't heard that phrasing before, but I like that a lot.
AC (00:40:33):
But yeah, it's very important to try to get those things out on the table as early as possible, because people, like you said, they know. They know a lot of this.
AC (00:40:41):
They may not speak up about it though. Because nobody likes to talk about things that can go wrong, but that's how you make things go right, is by recognizing the things that can go wrong.
EW (00:40:52):
How was writing a book for you? You mentioned your O'Reilly editors.
AC (00:40:56):
Yes. Yeah. [Meghan Blanchette]. She was very, very helpful in making this a better book, as well as the copy editor, [Gillian McGarvey], who was phenomenal. She's the only copy editor I've ever had who was okay with my making up new words that don't exist yet in the English language. She actually thought that was fun.
AC (00:41:16):
Editors tend to really not like it when you make up new words though. She's just fantastic. So yeah, they were both very, very helpful in making the book what it is, for better. They're responsible for the good parts. I'll take credit for the bad parts.
AC (00:41:32):
So in terms of doing the book overall, yeah, I kind of have a love-hate relationship with writing. I enjoy it to some degree, and I don't enjoy it to some degree. I guess that's why it's love-hate. It's very challenging.
AC (00:41:50):
The thing I find I spend the most time on is thinking about how to explain something in a way that makes sense to people who aren't skilled in the art of whatever it is I'm explaining. And that can be challenging to do. So I end up taking a lot of walks and thinking about how to explain something.
AC (00:42:09):
And the writing part itself takes time, certainly, and the editing and all that. So that takes time out from other things, but ultimately it's a very rewarding experience. I guess anything really rewarding is rarely all positive all the time.
EW (00:42:27):
What you said there about thinking about how to explain it to someone who doesn't know it, I have trouble explaining that concept to people who haven't written a book or haven't thought about it a lot.
EW (00:42:42):
When you read a datasheet or when you read a technical book that maybe isn't that good, and then six months later after you've finally learned all the material, because you had to do it in order to get your job done, and you go back to the book, and it suddenly is a pretty good book, why didn't you like it before?
EW (00:43:00):
That's the sign of a book that is written for people who understand the material.
AC (00:43:08):
Right.
EW (00:43:08):
It's not teaching you. It might be a good reference book, but that split is so hard to explain to people who haven't tried it. I don't where I where going with that.
AC (00:43:24):
Yeah, I think the way you said that, well, I think you said it very well...
AC (00:43:28):
I think to write a good book, I don't know that I've ever written a good book, but for people who do write good books, there's a lot of thought that's put not just into what facts you're going to put out there, but how are you going to present those in a way that tells a story.
AC (00:43:48):
And puts something in the reader's head..., some sort of structure that helps them to understand what's happening, and how things go together, and that kind of thing. So a book, it's not a bunch of disparate facts. It's really a story. It should be a story, I think.
EW (00:44:05):
Yeah. And it has to build up. It can't just be, "Here, I'm going to throw these facts at you." It has to build a framework and then put things in the framework where they're appropriate.
AC (00:44:17):
Yeah. By the way, I've read your book as well. And I think your book did a very good job of that as well...The next time I have sort of a budding firmware developer, I will give them that book to read.
EW (00:44:35):
...Yeah, so one of the things I did was I had people in mind, particular people that I was trying to write for. And they had their background and their knowledge, and I tried not to assume anything else.
EW (00:44:50):
Did you have a particular person you were writing for, or how did you figure out what your reader was going to know and what they weren't going to know?
AC (00:45:00):
I think it was based on real experience. So I think everything in that book is really came from, "What are the things that I end up explaining to most people most of the time?" So there are some things, it's a little bit of a hodgepodge, and sometimes I feel like it's maybe too much of a hodgepodge.
AC (00:45:18):
But I figured I wasn't going to teach electrical engineers how to do electrical engineering or software developers had to do software development, other than there are a few issues in both of those disciplines, and in other things that I cover.
AC (00:45:30):
...Software developers, they know how to develop software, but they may not understand much about how to think about different operating systems, for example, because typically they're given an operating system to use and they may complain about it.
AC (00:45:49):
So let's say you're using an RTOS, right? Like Green Hills or something. And they're saying, "Oh,...why are we using this? This is just so freaking hard to use. It doesn't do anything for me." Well, there's reasons for that.
AC (00:46:01):
And I'm trying to put this in context, like, "Yes, you could use something that's more full-featured, but it's not going to be an RTOS, and it's not going to be as tested, and safety's not going to be as assured," and all those kinds of things, just to give people an idea of sort of why they're doing what they're doing.
AC (00:46:20):
I guess that could be applied to the book in general. It's trying to help different people who are involved in product development, trying to help them understand sort of what's going on holistically, and how do they fit in, and how can they work with other people so everybody will have a good time, and get a good product out the door.
EW (00:46:40):
I liked the systems aspect of it, talking about how boards are actually manufactured and how they're tested. I don't think I'd ever seen, I mean, I've used the bed of nails testers, but usually they're in the other room. I didn't have a good picture of it before your book.
EW (00:46:54):
And you talk about how you power things and go through the regulations for standards or regulations for safety and radios, and how you do all these things, including choose an operating system and processors. And yet you don't give enough information that someone could make that their career.
EW (00:47:18):
But you give enough information that a systems engineer, or a manager, or a CEO of a tiny company would know the right questions to ask, and wouldn't make assumptions that just because somebody else used Linux, that it is appropriate in their product. And I liked that about it.
AC (00:47:36):
Yeah. Linux. Let's just run Linux.
EW (00:47:40):
And yet the boards..., everything's getting cheaper. So why bother to not run Linux as long as you don't mind crashing occasionally?
AC (00:47:49):
Right. Or backporting fixes and all those types of things -
EW (00:47:55):
Security.
AC (00:47:55):
- which are all, yeah, well, security and reliability, and...I wouldn't say anything bad about Linux or any other operating, well, that's not true. There are probably some operating systems I could say bad things about, but they are what they are, right?
AC (00:48:12):
And it's really just understanding what's out there and what are the pluses and minuses in picking best set of compromises for what you need to do.
EW (00:48:21):
So would you write another book? Are you currently writing another book?
AC (00:48:26):
I am not yet writing another book....So I would write another book, and I'm sort of thinking about potential ideas for what such a book might look like. So I'm thinking about sort of more along the lines of systems engineering in a more detailed systems engineering book or software.
AC (00:48:50):
So I've become very interested in model-based systems engineering and software engineering, which I think is a very important topic. And it's a very powerful way to do things, but I don't think it's often used outside of a few industries.
AC (00:49:04):
So I'm kind of thinking about maybe doing something along those lines, but yeah, it's a lot of work.
EW (00:49:12):
Yeah.
AC (00:49:12):
And I'm lazy. So I don't know. We'll see.
EW (00:49:15):
You sent me a slide that included model-based, and I want to hear more about it. So let me describe this slide a little bit. It was about defects and software methodologies, and it talked about different software methods, Waterfall, iterative, object-oriented, Agile, stuff I don't know, stuff I don't know, model-based.
EW (00:49:36):
And then it talks about what the defect potential was with a number I didn't understand, and the number of defects you can expect to be delivered. And Waterfall was by far the worst and model-based was pretty good. Agile was between them. Can't we just type? But yeah. Okay. So what does model-based mean?
AC (00:50:02):
Well, the idea is that, so...let's move away from software for a second. Let's...say you're going to build some sort of a mold or something, right? You have a plastic part you want to build...If you were going to 3D print it, right, ultimately you're going to end up with G-code instructions that go to the 3D printer.
AC (00:50:26):
I think it's called G-code. But you're not going to sit there and write G-code that tells the print heads how to move and all that kind of stuff. You're going to draw some stuff on a screen that describes the part that you want. And then it's going to turn into an STL model and then turn it into the G-code and whatnot.
AC (00:50:42):
So, sort of the way that model-based engineering is similar,...in the case of software, right, ultimately you're going to end up with, with code, right? C code, or C++ code, or Python, or whatever it is.
AC (00:50:59):
But maybe there's a way to, sort of at a high level, draw diagrams that show what the system should do, and then have it...turn that into code. So, I guess, well, why would you do that? If you do things as diagrams, it makes it a lot easier for people to analyze what's going on.
AC (00:51:21):
And let's say if you've got a piece of code, let's say a state machine, for example. I don't know about you guys. You guys do the kind of work where I suspect you've been exposed to state machines, and have probably dealt with them to some degree or maybe to a larger degree.
AC (00:51:38):
But...I've never found a great way to do state machines and code where everything's clear, and easy, and people can do a good review of them, that kind of a thing. They're always just sort of clunky and nasty.
AC (00:51:52):
But if you were to draw a state machine as a diagram, you can do a great job of that, that everybody can understand, and all the stakeholders can look at it and say, "Yeah, that makes sense."
AC (00:52:00):
"These states make sense, and these are the right transitions," or, "Oh, we forgot about this event occurring that'll cause such and such transition and this other error state," and things like that. So by doing these things as diagrams, we can, I think, understand them a lot better at a high level and then reduce things to code.
AC (00:52:27):
So it basically lets you fail faster. You can put together diagrams rather than writing a ton of code and work with the diagrams and in some cases execute the diagram.
AC (00:52:38):
So the tool that I use for model-based software engineering is something called enterprise architect, but it's actually not a very expensive product. It's a very good product, and it will actually generate code in a lot of instances from the diagrams that you draw.
AC (00:52:53):
So for example, state machines, you could draw a reasonably simple state machine and it will kick out a thousand lines of code that's documented, and good code, and sturdy, and all that kind of stuff. So it's quite nice.
EW (00:53:09):
I'm having my normal "new things are bad engineering" reaction because, I mean, I remember UML, and I remember UML to C generators. Ow. Or C++ I guess. And I'm remembering every time I tried to learn LabVIEW and then ran away screaming. I'm not sold on the diagrams. Can you sell me a bit more?
AC (00:53:37):
Yeah,...well, LabVIEW I think is great, this is kind of an aside, but LabVIEW's great for certain types of things. I think it's great for test engineering. But for doing products, it's not great.
AC (00:53:54):
And hopefully that doesn't bias you too much against model-based software engineering. UML, yeah, that was kind of a big thing in the, I don't know, turn of the century, shall we say? And then I think people didn't see a lot of benefit for what they were doing.
AC (00:54:14):
And also UML is very picky. To do it in a way where you can generate code from that is pretty challenging. And actually the tool that I'm using, and actually the typical tools for doing model-based engineering, do use UML and an extension of that called SysML for system engineering.
AC (00:54:35):
And it is, I would say two things. One is that the learning curve is actually very steep for getting to the point where you can use this to do something useful. And also,...there are certain things that it's good for and there are certain things that it's not good for.
AC (00:54:58):
So my suspicion, I'm not an expert on this by any means. I should be being tutored on this by somebody who's an expert at it. There are certain things that are good, I think, that work well in terms of modeling and then generating code.
AC (00:55:15):
So for example, doing classes. If you're going to do classes and state machines, things like that, there's a lot of boilerplate code that it's easy to mess up if you're doing it by hand. But if you're drawing a nice diagram and then pressing a button, it'll generate nice code, that if you did it by hand, you'd probably screw up.
AC (00:55:35):
But if you're trying to do detailed algorithms and things like that, no, I don't think you're going to do that with a model. I don't think that's going to work too well. So, that was a lot of talking, and I'm not sure what I said, other than I think there's some very, very good uses for it. And...there's definitely ways to abuse it.
AC (00:55:55):
And it does require a steep learning curve, which is actually what makes it attractive to me as something to write about, which is that I think a lot of the steep learning curve has to do with there not being books available...that teach the user in a good way how to use these technologies.
EW (00:56:16):
One of the good things that comes from technologies like that, and I don't know about enterprise architect itself. I've never seen it. But the idea of this model-based system is, like you said about state machines, you put one together, and it's an understood state, and the code generates it, or understood pattern.
AC (00:56:38):
[Affirmative].
EW (00:56:38):
It's an understood pattern. And then the code is generated according to fairly standard methods. And then there are other patterns, many of these newer UML to C++ systems, like Java and Swift are very pattern-based. You want to do this, and so you use a subscriber-publisher pattern. You want to do that so you use this.
EW (00:57:10):
And so we end up with code that isn't reinvented. And going back to your 3D printer and CAD modeling, basically, if something is an off-the-shelf bolt, don't redesign it. Just use the bolt, give it the size you need, and the thread count you need, and go...Stop redesigning the bolts.
AC (00:57:31):
Oh, absolutely. Yeah. One of the things I tell engineers who I work with or who worked for me, I'll tell them, "Your job is to finish a product, not to design stuff," which is a little disappointing in some ways, but there's still plenty of room for creativity.
AC (00:57:50):
But the bottom line is that the job is to get it done as quickly and efficiently as possible. And if that means just getting something off-the-shelf, or using something that's already been designed, that's all the better.
EW (00:58:03):
Has writing a book changed the trajectory of your career?
AC (00:58:08):
I'd like to say the heavens opened up and all kinds of great things happened, but I would say not really. No, I mean, I'm actually pretty happy where I am in my career. But I mean, it's nice. People recognize it.
AC (00:58:27):
It is kind of a shorthand. I can give that to people and they will kind of quickly know that I kind of know a little bit of what I'm talking about, that kind of thing. And I am able to use it to on projects where...people will start talking about some topic, and I'll say, "Hey, you should read pages."
AC (00:58:46):
You're usually talking about three or four or five pages, because I tend to be pretty concise in the book about that topic. And that might be of interest to you. But no, I don't think it's changed my career trajectory in any sort of a huge way.
EW (00:58:58):
Have you ever taken it to an interview with potential clients and pulled it out at the right time?
AC (00:59:03):
I haven't had that opportunity. So, I mean, since I wrote it, I've been fortunate in that -
EW (00:59:17):
Of course, neither have I.
AC (00:59:17):
Yeah. At some point, maybe that'll happen, but no I've been, I mean, it has only been out for two, three years, I think, which isn't that long in book years.
AC (00:59:28):
And I've been just sort of working on some fun stuff and people have asked me to work on things. So I haven't had to really do that, do the going around looking for work kind of thing. So not yet, but I'm sure it'll help a lot if and when that happens.
CW (00:59:46):
I just wanted to briefly ask about Mass General Hospital, because when I worked at a medical place, we had a relationship with them, and I was surprised at the time that, "Oh, this hospital does all this research and stuff."
CW (00:59:59):
And I think people would be interested to just hear about what's done there. It's kind of a cool organization.
AC (01:00:06):
Yeah. Mass General is an amazing, amazing place. And I feel really, really fortunate to work there. So it's one of Harvard teaching hospitals, and it's the largest of their teaching hospitals...They have the largest research budget of any hospital in the United States.
AC (01:00:26):
I'm trying to remember what it is. It's a billion dollars plus or minus a year that they spend on research.
EW (01:00:34):
Wow.
AC (01:00:34):
Yeah, it's huge. They do a tremendous amount of research. They also do excellent patient care. And those two things very often don't go together.
AC (01:00:45):
But in the case of Mass General and the Brigham and Women's here and some other hospitals, they do go together. And...I can't say enough good things about it. There are so many smart people there. I mean, it's Harvard Medical School, right?
AC (01:01:01):
So...the people I work with are physicists...All physicists are just very smart people. And these are sort of cream of the crop or among that top, high-end group. There is great research going on. I mean, in the area I'm in, radiation oncology, there's a lot of people doing great research on better ways to do different kinds of therapy.
AC (01:01:26):
The system that I work on now, the proton therapy system, proton therapy for clinical use was really pioneered at Mass General/Harvard. Probably about 40 years ago, they started doing it. They were the first people to do it clinically on a large scale.
AC (01:01:45):
So yeah, any area of medicine, they're doing something really great, and...it's fun to work there. And people are really nice too, so it's great.
CW (01:01:56):
Awesome.
EW (01:01:56):
Cool. Thank you. I think we'll let you get back to your weekend. Al, do you have any thoughts you'd like to leave us with?
AC (01:02:04):
So, one of the things in writing a book, the book I wrote sort of talked a lot about, say tactics and strategies and how to do stuff. But one of the things that I didn't really touch on, which I think is super-duper important, is the people aspect of all this.
AC (01:02:22):
I think one of the things that you realize as a systems engineer, because you're working across so many different groups of people, is how important people and people's brains and their psychology are to a project.
AC (01:02:38):
And it's something that you guys sort of touched on earlier, about how do you get people to think about things in certain ways that make for good outcomes. The flip side of that is teams of people. One of the things...that's really, really super important when you're building a product is having a good team of people.
AC (01:03:00):
And there's really nothing more important, I think, than having a good team of people who are smart, motivated, egos aren't too big. Self-confidence is good but egos aren't so great for getting things done of any sort.
AC (01:03:16):
So I would say that I hope people read the book and take it to heart, but also be very thoughtful of the people you're working with in making sure that they're motivated, and that everybody's happy, and then they'll do a great job.
EW (01:03:35):
Cool. That is very good advice. Our guest has been Alan Cohen, author of "Prototype to Product: A Practical Guide for Getting to Market." It's published by O'Reilly and available at all the usual bookstores. Thank you for being with us.
AC (01:03:51):
Thank you, guys. It's been a pleasure.
EW (01:03:54):
Thank you to Christopher for producing and co-hosting. And of course, thank you for listening. You can always contact us at show@embedded.fm or hit the contact link on the website, embedded.fm, which has a blog if you didn't know.
EW (01:04:07):
And now a quote to leave you with. This was kind of odd, because I read Camille Fournier's "The Manager's Path" right before I started Al's book, and the two of them together, I have never highlighted so much in my life. And what Al was saying about teams, Camille has some things to say too.
EW (01:04:29):
"Do remember to be kind. It's natural and perfectly human to want to be liked by other people. Many of us believe that the way to be liked is to be seen as nice, that niceness itself is the goal. Your goal as a manager, however, should not be nice. It should be to be kind."
EW (01:04:47):
Embedded is an independently produced radio show that focuses on the many aspects of engineering. It is a production of Logical Elegance, an embedded software consulting company in California.
EW (01:04:59):
If there are advertisements in the show, we did not put them there and do not receive money from them. At this time, our sponsors are Logical Elegance and listeners like you.