A Cruise self-driving car doing a test route in San Francisco in January 2019.Getty Images
It’s tough to know for sure which companies really have the best self-driving car technology, though Alphabet Inc.’s Waymo and General Motors Cruise can point to the only public data available in the U.S. that compares dozens of programs testing robotic fleets in California if they want to make that claim. They probably shouldn’t.
The successor to Google’s self-driving car project said its autonomous system only disengaged, meaning a human safety driver had to take the wheel, at a rate of once every 11,017 miles. That’s a “50% reduction in the rate and a 96% increase in the average miles traveled between disengagements” from last year, Mountain View, California-based Waymo said. And that happened as it boosted testing in the state to 1.2 million miles from 352,000 miles in 2017. GM Cruise reported the second-lowest disengagement rate, with a safety driver taking over once every 5,205 miles, 321% better than a year ago, according to data released by California’s Department of Motor Vehicles on Wednesday.
By comparison, Apple’s enigmatic robot car project appeared to have far more frequent disengagements, and Tesla, which Elon Musk touts as a self-driving leader, said it “did not test any vehicles on public roads in California in autonomous mode or operate any autonomous vehicles, as defined by California law.”
The value of the data is limited, however, as the figures don’t factor in the complexity of environments in which vehicles are tested–dense urban environments, versus low-speed suburbs or less complex highway driving–nor do they show conditions including weather, light or speed. And while California is the center of autonomous tech R&D, Arizona has become a bigger stage for testing and early deployment.
“There really is no way to verify” claims companies make based on the DMV data, said Mike Ramsey, a senior research director for Gartner. “Absent a standard process or ability to verify their recordkeeping, the reports are akin to restaurant ratings on Yelp. They are useful, but can’t really be relied on to reward or condemn companies.”
The 2018 tally included reports from 48 of the 62 companies that have permits to test autonomous vehicles in California, the DMV said. Total test mileage reported to the state jumped to 2.05 million miles, about 500,000 more than a year ago. Waymo, with a 110-vehicle fleet, accounted for more than half of those test miles.
Waymo
“A lower rate of disengagements shows that our cars are getting better at recognizing and handling a wide variety of driving situations, including ‘edge cases’ across the cities we’ve been testing in: those unusual situations that a human driver might see only once (or never) in a lifetime of driving,” said Waymo. It launched a commercial ride service using hundreds of self-driving minivans in suburban Phoenix in December.
(For more on the Alphabet program, see At Waymo, It’s Launch Time For Google’s Biggest Moonshot)
“We believe the key to self-driving technology safely improving and scaling is through a robust breadth of experience and scenario testing, represented by a wider array of data points beyond disengagements alone,” Waymo said.
Its California test fleet, which used to mainly test in the Silicon Valley suburbs, expanded into San Francisco last year to build up its ability to handle denser, more complex driving situations. By comparison, Cruise does the bulk of its robot-car driving in San Francisco, which the company says is more challenging.
Or as Kyle Vogt, Cruise’s cofounder and chief technology officer put it in a 2017 blog post: “Every minute of testing in San Francisco is about as valuable as an hour of testing in the suburbs.”
Apple’s data, filed for both 2017 and 2018, shows its 62-vehicle fleet logged almost 80,000 test miles in California last year – and that its self-driving system disengaged about once every mile. That appears to be the highest frequency among the latest reports.
San Francisco-based self-driving truck startup Embark filed a voluntary report indicating that its robot semis logged 124,062 test miles on California roads last year, although it wasn’t required to do so. (The state doesn’t yet have the same rules for driverless trucks that uses for cars.) On average, its system disengaged once every 1,392 miles, trailing Waymo and Cruise rates but appearing to outperform the bigger, better-funded programs at Apple or boutique startup Aurora, which just completed a $530 million fundraising round. Embark has raised $47 million to date.
“The mileage and disengagement rates released today demonstrate that Embark’s approach has been remarkably capital efficient in working toward the goal of human-equivalent safety,” Embark CEO Alex Rodrigues said. “By focusing on trucks on highways we’ve been able to safely progress towards commercialization much more quickly and cost-effectively.”
Embark, which hauls revenue-generating commercial loads on test trucks between Southern California and Arizona on a regular basis, said it’s also only the third company to log more than 100,000 miles in a year, behind Waymo and Cruise.
Three of the self-driving vehicles in Waymo’s test fleet parked at Google’s I/O Developers Conference in Mountain View, California in 2018. 2018 Getty Images