Is 99.9% safe good enough for robotaxis?

0:00 Introduction: the promise of autonomous vehicles

Today in The Garage, we’re going to look at autonomous driving in a new way. Everyone is excited to see how much progress autonomous, vehicles have made in recent years with Waymo and Zoox and many other companies now commercially available. Many people in in cities like here in San Jose, the Bay Area where we live, can hail a robotaxi on their phone and it shows up with no driver and takes them on a ride. And it’s been incredibly exciting and successful to see. I, personally, in my career was working on robotaxis about five years ago when it wasn’t quite ready and it was a very frustrating experience for me. So to see how much progress they’ve made is really inspiring to see how that technology is going and an incredible use of AI.

Having said that, as we’re getting to higher numbers of robotaxis, the complexity of making robotaxis that work well to make them at the level of working well that is viable to scale to thousands and thousands of vehicles is a bigger challenge. And who best to talk about that than a real expert who’s been studying this for thirty years? Our guest today is professor, now newly retired, Phil Koopman from Carnegie Mellon University who spent thirty years studying robotaxis and other aspects of safety engineering and will tell us all about the pluses and the challenges to make robotaxis very viable in scale.

Let’s go.

I’m John Heinlein, Chief Marketing Officer with Sonatus. We’re so pleased to have Phil Koopman here with us today. Phil, welcome to The Garage.

Thanks for having me on. Happy to be here.

So we we both have a strong connection to Carnegie Mellon. I went to undergrad there. You were a professor for a long time. And we we I think we crossed paths when back at when I was a student and you were a grad student. Start by telling us your background.

Sure. My background is I started with an undergrad in computer engineering at RPI.

And after that, I served in the Navy because they were kind enough to pay for my undergrad. That that’s how it worked back then. I then worked well, then I went to Carnegie Mellon University, got a PhD, which is where we ran into each other. I was a teaching assistant in a hardware design course, bit-slice CPUs for people who know even what that is. And but it was a fun course. And after I graduated there, I went to work for Harris Semiconductor as a CPU designer. I spent time at United Technologies at the Central Research Center, got a lot of embedded domain experience, and then finally went back to Carnegie Mellon as a professor, taught there for almost thirty years, and just recently retired.

Well, congratulations on an incredible career, and you have a real distinguished background. But we’re gonna get to that. But before you, you gotta tell us a fun fact about you.

Sure. The the fun fact is I used to drive submarines in the navy during the cold war. I have a combat medal, but I can’t tell you how I got it.

That’s an incredible story. I love submarine movies, Das Boot and The Hunt for Red October. Such such great stories, so I’m jealous of your experience.

Well, Hunt for Red October is an exceptionally good one. It’s very accurate for the types of things I was doing back when I was doing it.

3:15 Fundamentals Of Safety Engineering

Very fun. So tell us about your career at CMU and especially about safety engineering. That was a key specialty you had. I wonder if you can introduce for our audience, what is the concept of safety engineering and how does it differ in some ways from sort of, if you will, classic engineering of of just building something? How does it complement that?

Sure. I’ve been doing self- driving car safety specifically for thirty years, and I’ve been doing a lot of other types of safety engineering for I lost count at 200 design reviews of embedded systems, and I lost count like 15 ago. And and some of them were just doesn’t work, but a lot of them were safety in all sorts of different domains. So I’ve seen lots of safety standards, lots of safety.

And if you have to boil safety down, it really comes down to if you’re building a system and you’re testing it, engineers are taught to make sure it works. And they’re really good at the happy case, but maybe not the unhappy case. And safety engineering isn’t about how to make it work. It’s about what happens when something goes wrong.

And specifically, what you’re taught to do is create a list of hazards, things that can cause problems. A lot of times, it’s personal injury, but it could be severe equipment damage. It could be a lot of things. It’s something bad will happen.

You make the list of here are all the things that can that can go wrong and here’s our plan to make sure that either it won’t happen or if it does happen, that the harm will be mitigated somehow. And so safety engineering is listing the hazards, figuring out how to mitigate the risk from the hazards, and making sure those safety requirements go into the engineering requirements, which means now you have to design the safety behavior intentionally and you have to test that the safety stuff really works. Just a really simple one: On a train, there’s air brakes.

If everything goes wrong, there’s a cylinder of compressed air and you hit the button and it slams on the brakes. And and what’s that for? Well, there’s lots of things that you go on a train, but most of them get cured by slamming on the brakes. So that’s the plan.

It’s interesting to to describe it that way and I think that, you know, as you’re right. As an engineer, you want to make something work, but, know, you have to really think about all the failure scenarios. And I think there’s a concept called fail safe and fail dangerous, which I think is is often often talked about. I think probably your goal, I’m assuming, is first avoid the failure happening at all in the first place. But if it does happen, have it fail in a graceful way that safety is not put at risk. Is that a fair way to describe it?

5:37 Fail Safe And Fail Operational

Yeah. That’s well, that’s the way it used to be until recently. And so classical systems, if you de-energize the system, it’s safe. You you have a chemical process and if you can get that thing wound down so it’s at room temperature instead of superheated steam, it’s a lot safer.

So so you wanna avoid the problem. You wanna have a way that if you have an unsafe situation, you can sort of wind it down to to become safe, and usually that means by dissipating the energy. You have a car, you bring it to a stop. But the catch is there’s some situations where winding it down to no energy isn’t good enough.

So you have an airplane and you have an engine fire at 35,000 feet. Turning off all the engines is the wrong move. Right? And so there’s fail safe, which usually means fail stop.

Just wind everything de energize everything and you’re safe. And that works in a lot of places. But in some places, if there there’s a high velocity motion involved, you need to actually keep working long enough to wind it down. And there’s a reason why passenger airliners have two engines, and there’s a reason why they have two or more sets of flight controls because when one fails, you have to fail not just fail stop, but fail operational long enough to get yourself into a safe position.

And that applies to cars too. If you’re 60 miles an hour on a highway and your steering is, well, the steering is a problem. It’ll shut down.

That doesn’t work.

You need the steering to work long enough to get the car shut down.

It’s true. And and you can’t even just come to a stop either because then another car behind you is gonna plow right into you.

So you you need to somehow gracefully…

You have to get off the road. You have to get out the travel lane. I mean, there may be a very rare case where there’s nothing you can do. If the if the if the engine falls out of the car, there’s only so much you can do. But in almost all cases, you want enough oomph to get yourself out of the travel lane. So you need you switch from the normal operation mode to a recovery operation mode with less capability and a very short mission time, but that backup has to be there. So you have to have backups or it doesn’t work.

That’s fantastic. We’re getting a degree in safety engineering here in in the podcast today.

I think you’ve been such a you mentioned you’ve been doing autonomous driving for for thirty years, and obviously, there’s been massive massive progress over that time from a glimmer in someone’s eye to you can hail a robotaxi right outside my office here. So I think let’s start talking first about some of the the rosy things that we’ve seen and then we’ll talk about some of the pragmatism that’s come up as well. We’re seeing now practical robotaxis, Waymo’s, Zoox, a few others that you can actually hire. Certainly there’s some solutions in China as well that maybe people know where they’re actually in practical deployment in certain cities, in certain limited operating domains. I’m sure we’ll talk about that.

So there must be incredible progress over the time you’ve been doing this.

Well, it’s been thirty years, but there has been great progress. So it’s important to say, yes. Absolutely. You can go hail a robotaxi. I’ve had rides in robotaxis. It’s amazing technology.

And the thing about safety is 99.9% is a good start. It’s not the finish line. Right?

And we’re it feels like we’re 99% there because my goodness, you can go hella robotaxi, but it did take thirty years. So thirty years ago, thirty one years ago, Carnegie Mellon had a car that went from Washington DC to San Diego 98% hands off the wheel, and it’s taken all this time to get that last 1.999%, whatever it is. Alright?

But it’s been great progress because that’s an amazing achievement. You can actually there’s nobody in there driving it. There’s, you know, there’s no man behind the curtain when most of the time when these things are driving. That’s great.

And it feels like we’re almost there, but I think what we’re going to talk about today is once you have a 100 robotaxis on the road, that’s amazing.

9:26 Challenges Of Scaling Robotaxi Fleets

But from a 100 robotaxis to a 100,000 robotaxis, it’s a whole different thing. And what I like to say is 99% done, 99% more to go.

I mean it’s it’s another way to say that because as you know as you mentioned when you think about you know it’s famous in business to talk about five nines which is 99.999 or we know how many ever nines you want. The thing is really what it depends on is what’s the numerator. Right? If if there’s a 100 robotaxis on the road, 99.9%, probably pretty good. When there’s a thousand, when there’s 10,000, it becomes a situation where you’re going to hit some of those those smaller nines.

Simple division.

Go ahead. Please.

Sure. So simple division. Just to illustrate the point. Something that happens once a year to a 100 robotaxis happens twice a week to 10,000 robotaxis.

And that’s not saying there’s anything with a robotaxi, but once a year, you probably may not even notice it happen, but when it’s happening twice a week, it matters. And that’s not that the robotaxi got worse. It’s that it’s a numbers game. The more you have, the more you have to pay attention to the things that were so rare they didn’t matter.

And so the the technology to run a 100 robotaxis is truly impressive, and it’s it’s here. It’s now. The next challenge is what do you do with the rare events that used to be so rare they weren’t a big deal because now they’re going to become a big deal at scale. That’s the challenge.

Well, this is an exciting topic. We’ve had guests over many years talk about here on The Garage talk about this this problem from a different lens.

A senior person from Mobileye earlier this season and others talking about what they’re doing. But you bring a really valuable and unique perspective. So let’s talk about some of the challenges you see in mass production and mass deployment of robotaxis.

11:12 Taxonomy of Failure Types

Well, I I have four bins, and I guess we’ll probably walk down the four bins, but let me give a preview. The the first one is mass failures. When there’s one robotaxi, if it stops in the middle of the street, it’s one robotaxi. If you have a 100 robotaxis all stopping in the middle of the same street, now you have a different problem.

So and they and software tends to fail under similar circumstances. So if all the robotaxis have the same software, that that’s that’s what computers do. They tend to fail at once. So you you have to have a plan for that.

The next one is there’s more to safety than crashes.

And so the industry has been really talking about reducing crash rates and then making great progress at that. But what we’ve learned as the number of robotaxis has gone from 10 to a 100 to a thousand, that there are other things that matter to safety that are sort of bubbling up. That that in the first days, I I wouldn’t have said, I wouldn’t have worried about things like this, but but it turns out they matter, so we need to learn from that. The third one is that edge cases, rare events, turned out to be more more well, they turned out to be a big challenge. I would say more than anyone thought, but I knew they were going to be a challenge. And the industry has a plan for that, but that brings new challenges. And the last one is accountability.

As you get scaled up, you have to have a better accountability plan than where we are now. And that’s not the robotaxi industry’s fault. That’s that our legal system isn’t really ready to handle what’s going on.

Thank you for that taxonomy and a good preview of where we’re going. And I think that’s that covers so many different it really covers it from different perspectives.

12:41 #1 Correlated Failures

So let’s start with the first one, which is you mentioned mass failures or when one fails that tends to correlate with other failures. Talk about that and and what’s the challenge and what should we do about it?

Well, when you talk about human drivers, human drivers make mistakes all the time. Cars break down. All sorts of things happen. But it’s rare for a 100 or a thousand cars to all have the same problem at the same time because every human driver is different different cars.

Sure. There are things that stop or or clog up traffic. If you have a blackout, it’s gonna cause some pandemonium for all drivers. But if you have a the same computer driver in every single robotaxi and that’s a significant fraction of the traffic in a city, then if something goes wrong, you’re gonna have a citywide problem.

And and don’t tell me that computers, network computers at large scale never go wrong. You know, CrowdStrike is just the latest poster child. Things happen.

Now there was a issue in in China, I think it was in Wuhan, where all the taxi robotaxis from a single company all went dead at the same time because there was some some sort of communication problem. And now these these people were stranded on high on elevated roadways, and and we’ve seen we’ve all seen the the the videos of US robotaxis caught in just mass events like a big party event or something like that where there’s a lot of cloggage or during a power blackout. You know, this is the kind of thing that happens. Now that doesn’t mean you can’t use the technology. It means you have to realize it’s inevitable. You know, let’s let’s put all our software as a service stuff on the same server in the same city and serve the world from that really bad idea. Right?

There well, guess what? Robotaxis is gonna have the same sort of vulnerabilities. You have to have a plan.

You mentioned when a significant fraction of the traffic is from a single company. So obviously, that’s not something we have today. But there are many people, maybe the bulls, the optimists about robotaxis who feel like they could at some point in the future obsolete or certainly reduce significantly the number of cars. So it sounds like that’s a real practical problem to think when there’s consolidation from a certain company. Now we’re not talking about cyber security. That’s probably a whole another thread we could do at another time. Whole whole thing.

Yep. You talk about mass hacking of a fleet of robotaxis, you can imagine bringing a city to a grinding halt. So that’s certainly a theoretical problem as well. But your point was even that could happen without a malevolent actor just due to a correlated failure.

That’s right. And there are some consequences beyond just the bunch of robotaxis stranded. If you heaven forbid, there’s an earthquake and if the robotaxis all get stuck in the earthquake now I know human cars have trouble in earthquake, but human drivers are pretty good at getting their car out of the way of emergency vehicles. And we’ve seen a lot of trouble with robotaxis struggling with that.

And, if it’s the ones, twos, they they figure it out. But if there are hundreds of robotaxis stuck and they’re blocking all the emergency vehicles in a in a disaster situation like that, it can be a problem. Now is it a big problem compared to human drivers? Interesting question.

But those are the kind of questions we need to ask. If you’re gonna scale this up to be a substantial fraction of the cars, you need to think about disaster response. You need to think about, well, okay. So there’s and I guess we should do this one next about remote assistance.

But if you have 50 remote assistants and 200 cars all get stuck at the same time, you know, what’s your plan? You need a plan.

Right. Yeah. In a similar way, I mean, we’ll talk later in in in item three about kind of corner cases. But let’s imagine, you know, you said an earthquake. Let’s imagine a terrible earthquake where a lot of the roads are cracked. All of a sudden now, you could experience a situation where many of the corner cases literally happen instantaneously where the roads are great and now the roads are cracked and robotaxis see obstacles.

And you don’t even need an earthquake. You just need a power blackout to take out all the traffic lights.

And you can say even if 90% of the cars get the traffic lights right, that 10% adds up pretty quick, and that this is actually something that’s happened. Or hurricane evacuation or or who knows? If there’s some situation they haven’t been trained for, they’re all gonna fail at the same time if it’s an environmental thing. Now this can be improved, so the the question is not not to say, well, there’s nothing you can do. The question is, are we looking ahead? Are we waiting for these things to happen before we respond, or are we getting ahead of it? And and what we need to do as we scale is we need to get ahead of it.

Okay. Alright. So that’s topic one. That’s that’s really super interesting and and scary. Topic two, you were talking about human drivers and and failover approaches.

17:23 #2 High-severity edge cases and human remote assistants

Right. So the this is the technical challenge for robotaxis since the first day has been edge cases. So rare events well, theoretically rare events, it gets it gets sort of tricky there. But if you think about something like a power blackout, power blackouts are rare. They don’t happen that often. Depends where you live.

But just because it’s rare, if it’s high consequence, it doesn’t mean you can ignore it if you care about safety. And there are things like pedestrians walking out in the middle of a block. It doesn’t happen all the time. It depends what city you live in.

I know some listeners are going, wait a minute. He doesn’t live in my city. But but people going across or or school kids coming out of their parents’ car and crossing the block to get to the school. Well, depending where you are, you may never have seen that or you may see it every day.

So rare rare is kind of relative. So this is why it gets tricky. You say, well, that doesn’t happen often. Well, it doesn’t matter if it doesn’t happen often.

If it’s high severity and it happens to you, you have to deal with it. Right? And and the catch with machine learning is machine learning is good at what it’s trained at, and it tends to be very brittle and and lack, what for better lack of a better word, I’ll call it common sense. Machine learning has no common sense.

So if it gets into a situation that it hasn’t been trained for, it’s prone to overconfidence. It doesn’t even realize it doesn’t know. It does something we would say is just stupid. And if that’s a high severity situation, that’s gonna be a problem.

So the edge cases are how do you deal with things the machine learning hasn’t been trained on? And whether you think it’s rare doesn’t matter. If it doesn’t know, it doesn’t know. It doesn’t it doesn’t help to say, well, I’ve seen that.

Well, if it hasn’t been trained, it hasn’t been trained. And so safety severe events are very rare. People, even if you conclude the drunks , it’s about one fatality per a 100,000,000 miles. A 100,000,000 miles is a lot of miles.

People are impressively good drivers, especially if they’re unimpaired. It’s way way better than that.

And so you can say, really, only the rare things matter. People will say, well, that’s rare, so it didn’t matter. Like, no. Fatalities are extraordinarily rare.

So and, in fact, the rare things are what control safety. So normal engineering is, I pressed the button. Did it work? Yeah.

Okay. Cool. Safety engineering is, I know that’s rare, but it’s high risk. And if you add up all the events over time, it’s too high a risk.

We have to mitigate it so even the rare stuff doesn’t cause a big loss event.

Right. And so you talked about, you know, strategies to to attack these rare events.

What are what are some strategies to to tackle them?

Well, it used to be it looked kinda hopeless because there’s an it it’s like you’re hunting for Pokemon except there’s an infinite number of Pokemon. You can’t collect them all because the world is full of craziness. It’s just absolutely full. And the the saying used to be what the industry used to say is, well, we’ll catch enough that we’re willing to suck up the occasional weirdness.

That if we’re a 100 times safer than a human driver and something bad happens, people, for a factor of a 100 people, are just gonna well, you know, that bad things happen. There’s nothing you do about it. But they’re not anywhere near a 100 times safer at its fatality rate. We we don’t know for sure, but there’s no way it’s a 100 times safer.

And they never actually got there. So the the industry basically declared defeat on edge cases.

We don’t have the ability to make AIs handle all the weird high severity stuff. We just don’t. So what’s their plan? But they’re on the road. How did they do that? The way they did it was they said, you know what? When the robot taxi gets into trouble, it’s gonna phone home and a real life human’s gonna help it out.

And that’s an ingenious way to handle edge cases. So now you don’t have to be trained on how to deal with everything. You just have to be trained well enough to know, I have no idea what’s going on. I’m gonna go go phone phone home.

So a lot of the incidents you see are there’s still room to improve in knowing when to phone home.

But when they phone home, then they have a human operator that says, oh, yeah.

I know you can’t see the traffic light because of road glare or the wind spun, trust me, bro, it’s green. Okay?

And safety is it had better be green when they say it’s green because the robotaxis just you know, the robotaxis has all these sensor inputs, but that remote humans is sort of casting the deciding the vote and many times determines what happens next.

Very interesting. So I think that obviously there was a bit of a dirty little secret of the industry for a while that a lot of these companies weren’t acknowledging they’re human backup drivers. But I think that most of them are acknowledging that they do that in some situations. Now indeed, it’s probably the rare case, but that they are there for those exceptional cases.

It it actually happens all the time. But but that’s not bad because getting hung up on autonomy purity is is beside the point. It I don’t care if it’s fully autonomous. In fact, I’m glad they have people helping it out when it doesn’t know because the alternative is scary.

Right? It’s a it’s a question of how many people do you have versus how many vehicles. And so if you have a 100 vehicles per person, it’s cost effective. Actually, if you have ten ten vehicles per person, beyond that, it’s kind of diminishing returns.

Right? The numerator is small. The divisor is high.

Right.

You know, at 10 to one, at 20 to 1, it’s it’s not that big a difference. So as long as you have a bunch of cars per person, the economics, which is what matters here, works out. So the next question is, are the remote are the remote assistance really safe? And the industry’s finally come clean that they have them, and I think that’s fine.

I don’t think that’s a negative. I think that’s realistic. It’s no problem. What the industry hasn’t has to come clean with still is that those people are making safety critical decisions, and we need to be attentive to the fact that they’re in a safety critical role.

The industry is still working on on sort of addressing that. But when when you have 10 or 20 remote assistants, you can hire the very best, the cream of the crop, and maybe you won’t have a lot of problems. Maybe it’s no big deal. But if you have a million robotaxis, you probably have a lot of call centers with remote assistance and you have to manage the safety there.

So that’s that’s a remaining scale up challenge.

Alright.

23:31 #3 Combinatorial Complexity In Edge Cases

There’s number two zero four and I think that’s a really interesting topic. I think the third is the corner cases and we touched on that a little bit in the previous section. So we had a guest from Mobileye on a few months ago and we were chatting about the one of the benefits that they have, they’re one of the largest driving databases in the world, is they can accumulate the corner cases and inject more corner cases into the training model for their customers and so on like that.

And definitely beneficial. But as we were chatting earlier, think the challenge with that approach is that, please you take the ball and run with it from here, is that that only talks about the problems you know about. And the trick is that there are many problems you don’t know about and then the unknown unknowns is I think is the biggest challenge. Am I right about that?

Yeah. That’s the infinite Pokemon problem that I talked about before. But the there’s a nuance here that matters. Traditionally, people think of edge cases as, oh, look. There’s a zebra in the middle of the road, and I don’t know what a zebra is. They think about about crazy objects or crazy events, and for sure, those are edge cases.

But intuitively, maybe you can catch almost all of them. The part that’s more subtle and more tricky is that there are combinations of known things that also form an edge case if you haven’t been trained on it. In particular, you could have a circumstance that combines known things in a way that has a different meaning than you’re trained to. I’ll give you a super simple example. If you have a stop sign this is a trick question, I’ll warn you. If you have a stop sign on a pole, what do you do?

You stop.

Right? You stop. Okay.

Then you go. Right?

Right.

Except trick question. If it’s a stop sign in a pole being held by a construction worker, you stop and you say stopped.

Well done. Well done.

More trick question, but I’m not done. More trick question. Stop sign in a pole and it’s in somebody’s hand and it’s not in front of their face, but it’s down by their leg, you ignore it because that’s a school crossing guard who’s not serious about having you stop.

And every you intuitively know this. Everyone intuitively knows this. But stop signs are really complicated. And if it’s a stop sign bolted onto a bus, whether you stop depends if it’s out or folded back.

Because one, you stop and say stop, and the other, you ignore it. So so my point is that edge cases are more than stop sign. When you’re trained on stop signs, you say you are, but why is your robotaxi running bicycle buses? Well, turns out stop signs aren’t so simple, and I took the I took the example everyone knows. It’s there’s more to it than that.

Right.

It’s such an interesting example. And, you know, I was I was driving last year, and a car decided to completely turn left into the oncoming traffic, turn left into my oncoming… and so you’re just not even used to seeing that because he’s just where he’s not supposed to be. So what do you do? Well, I got the hell out of the way is what I did. But you know, the question is you’re not even used to other people breaking the rules. And so not only do you have to follow the rules for yourself, but you have to imagine other drivers that are doing something even extremely crazy.

And in real time, have to use common sense. So I’ll I’ll do an example that I’d never occurred to me until recently. I’m driving along and a fire truck stops in front of me, and the firefighters are getting out. They open the door and there’s a stop sign painted on the inside of the door.

So and I’m like, I know what they want. They want to not get run over as they get out of the truck, so I believe I’ll stop and say stop till the door closes is what I’m gonna do. Right? So that and but people are great.

I mean, bet you would have done the same thing. Would have figured out from context that must be more Right. Be what this means. It’s on the driver manual.

It I was never trained on it. We’ve and and AI is spectacularly bad. It’s good at interpolating between known points. That’s what it does.

When they say journalize, it means we have training points and we do something in between. But if that something in between has a different meaning than it’s all its neighbors, it has a problem. And if it’s way we we have no idea, it has a problem.

It’s so interesting. And we we could probably have an entire conversation about that alone because obviously there’s a lot of talk about AI right now in the news, you know, and are we or aren’t we in AGI and so on. And the reality is these corner cases, there’s there’s miles to go before we sleep on fully general purpose AI.

Absolutely. But but rather than get sucked down into that tar pit, why don’t we talk about there’s more to safety than crashes? How’s that?

I think it’s great. Let’s do that and that’ll be our that’ll be our wrap up. This has been an incredible conversation.

28:05 #4 There’s more to safety than crashes

Okay. So the next one is there’s more to safety than crashes. Now this was a surprise to me. It wasn’t really top of mind for me. When we started, the narrative was it’s gonna save lives. It’s gonna have fewer crashes, and and that matters.

But once you get on par, let’s say that robotaxis today are on par for for human drivers, maybe a little better, maybe the same. The the truth is for fatalities, we don’t have the data to really know how that’s gonna turn out. Let’s just say that that’s where we’re going to end up. There’s no reason to believe that won’t happen with with due attention.

There’s more to safety than that.

And the more to safety than that is, for example, if you stop and block a fire truck or you block you block an ambulance with somebody having a heart attack and that ambulance can’t get through, that wasn’t a crash, but you can’t say that has nothing to do with safety.

Or you’re running by school buses. The school bus has a stop sign out and it’s extended so you know that you have to stop and stay stopped and your car blows by. Your robotaxi blows by. You didn’t hit a kid that time, but there’s a reason you’re supposed to stop because it’s a chaotic environment that no driver is good enough they can go at full speed and stop if the kid appears.

You don’t you don’t have the reaction time. Right? So you’re stopping for the safety of the kid. You can’t say that, well, I didn’t hit a kid this time, so it’s okay.

I mean, that doesn’t work. Right? If you’re not stopping, it’s a safety problem. So there are a bunch of things we found with safety that ordinary folks get very upset about safety, and it has nothing to do with crashes per mile, which is the metric the industry likes.

So as we scale up, we’re gonna see more of these kind of incidents that matter to stakeholders, especially municipal stakeholders, parents, things like this, that have to do with safety that really have nothing to do with crashes per mile. So as the robotaxi industry scales up, they’re gonna have to deal with that. And there was one more on my list I wanna make sure we get in, which is accountability for harm.

29:59 #5 Legal Accountability and Liability Gaps

So right now, if you’re a human driver and you break the law, you get a ticket, you could lose your license.

If you hurt someone while breaking the law, you can be sued wrongful death, those kind of tort, civil tort, negligent suits.

In many states, robotaxis well, in no state can a robotaxi be sued for tort negligence. It’s product liability, completely different. But in a lot of states, the person on the hook for traffic tickets is not the manufacturer. It’s the vehicle owner or the vehicle operator.

So if you go buy a fleet of 10 robotaxis instead of buying a condo to rent it out and you rent out your robotaxis and they get traffic tickets, as near as I can tell, that goes on your personal license, and you had no ability to control the driver. So there’s a lot of really funky stuff going on, and it’s the robotaxi industry past lobbying for state laws that blame everyone but them.

Yeah. And and with only a few robotaxis and they’re operating their own fleet and they can just pay out of petty cash the tickets in states some states they can’t even get tickets, but it’s really no big deal. But if you want to scale up to a million robotaxis, you’re going have to have a plan for what happens. They’re not going to be perfect. There are to be crashes. That doesn’t make them bad, but you need accountability and a way to resolve the crashes that make sense and that incentivizes safety in the future instead of what we have now, which is just a mess.

You know, really merging those two points. I think those are such interesting points. Mean, the first the first point you made about was how crashes is not the only failure. And indeed, you know, the industry says, you know, they’re much safer than human crash rates. And as you say, we don’t exactly know, but it could be true that that is the case. And yet, there are still unsafe things that are happening that that just so happens they get they get lucky.

That’s kind of where we are now.

That is kind of where we’re now. The industry says they’re safer on crashes. We can have a discussion about that, but even if we stipulate that’s true, what about not stopping for school buses?

I know. Exactly. And I’m I’m reminded back in CMU, I took a class called engineering and public policy, which was one of the most eye opening classes I’ve ever taken. And and they were talking about the space shuttle Challenger for those of you who remember that.

And the the point that they made during the class was that really that the the NASA at that time was playing a game of Russian roulette. And the saying from from the from the report on the space shuttle Challenger disaster was “when playing Russian roulette, the fact that the first shot got off safely is little consolation for the next.” Yeah. And so the fact is these unsafe things happening and yet not causing your problems means eventually there’s going to be a bullet.

I like to say that “getting lucky doesn’t scale.”

There you go. And then the second point you made was about accountability. And and there’s no doubt that this is a an active conversation that I’ve heard many times that there is a well understood and yet not addressed gap in this liability issue because if these things are going to scale, I think there’s going to need to be some reform of how this is done. Because the model right now is not scalable. It’s going need to be addressed in some way or other that’s going to make this viable because the current approach is clearly not viable. And it’s going to take political will and probably, unfortunate to say, some bad situations happening for people to wake up to what you’ve what you’ve what you’ve pointed out. And I think we should solve it because I think robotaxis are are a public good and that they they can and will be beneficial, but that we’re not all the way there yet to that problem.

I’d like to get ahead of the problem so that we don’t have a lot of folks who get unlucky early on having to bear the brunt of the burden of reforming the system. I’d rather get it performed without having to go through that process.

I fully agree and and I hope that that’s the way that the the politics goes. Look, this has been an incredible conversation, Phil. I’ve learned so much. I hope our guests have enjoyed this mini course in safety engineering and about robotaxis. Look, there’s still plenty to to be happy about with the incredible progress and for you to have been doing this thirty years and see how much this has come must be very exciting. But you’ve showed us that there’s still more to do.

I I think that the future can be very bright. There are some challenges for scaling up, and the way to succeed is for the industry to acknowledge and and face the challenges. That that’s how everyone wins. So thanks for having me. Really appreciate it.

Thank you so much for joining us. If you like what you’re seeing, please like and subscribe to see more episodes like this from The Garage, and we look forward to seeing you in another episode again very soon.