Bringing Forward Collision Warnings to our open source self-driving car

At comma.ai we are building openpilot, open-source software that can drive your car. Right now support includes Toyotas with TSS-P, the Honda Civic and CR-V (with Honda Sensing) and Acura ILX. In this post, I will describe how we built the Forward Collision Warning (FCW) system in openpilot. (note: the human was driving in all footage below)

Forward Collision Warning indicator on the dashboard

Our openpilot community also ported openpilot to a bunch of other cars, including a variety of Toyotas, Hondas, pre-autopilot Teslas and the Chevy Volt. This makes openpilot compatible with 5 out of the 10 most sold cars in the US in 2017. Currently, openpilot is a level 2 system and provides better lane keep assist and adaptive cruise control than the system that ships with the car. Also openpilot keeps improving with every update.

This makes openpilot compatible with 5 out of the 10 most sold cars in the US in 2017.

Each week, our fleet of openpilot users sends back hundreds of hours of video paired with sensor data from their car and EON. We use this data to train our neural networks, create accurate maps and reverse engineer more CAN bus signals. But this data can be used for much more than learning how to drive.

Having access to all these miles of driving data enables us to quickly develop new algorithms. Using our infrastructure, we can simply run our algorithms on all historical data. We don’t need to deploy our software in ‘shadow mode’ and wait weeks for a result. This gives us very good estimates on how well our system behaves, how many warnings were generated and how many of those were false positives.

What is an FCW?

In short, you want to get a warning when you have to decelerate quickly in the near future. This can happen because the car in front of you starts braking suddenly, or when the driver is not paying attention and gets very close the other car. The FCW system will then sound loud beeps and show a sign on the dashboard. Warning the user early can greatly increase the time the user has to respond and avoid a collision. According to [Lee et al., 2002] an FCW system can reduce rear-end collision by up to 80.7%.

An FCW system can reduce rear-end collision by up to 80.7%.

When searching for events where an FCW should trigger, we look for a maneuver where the car brakes with deceleration of 3 m/s² for at least 1 second. With this level of braking you go from 65 mph to a full stop in about 10 seconds. On the other hand, every warning generated by the system which is not followed by at least 2 m/s² deceleration is considered a false positive.

Building FCW

During our search, we are interested in three types of events: an FCW followed by braking (true positive), FCW not followed by braking (false positive) and braking not preceded by an FCW (false negative).

We’d like to minimize the false negatives, while still being careful not to generate too many false positives. After too many false positives the user will not trust the system anymore, or worse: turn the system off. We decided that a ratio of 1 false positive per 2 true positives is acceptable.

To trigger an FCW at the right moment we need to predict if any hard braking is required in the near future. Luckily, for our Adaptive Cruise Control, we used model predictive control to build an advanced planner which calculates the desired speed trajectory for the next 10 seconds. This uses an optimization algorithm, based on radar and vision information of the lead car (e.g. relative position and speed) and speed and acceleration of your own car.

This optimization algorithm tries to keep the car out of the ‘danger zone’ of the lead car, while minimizing passenger discomfort by acceleration and jerk. This ‘danger zone’ is defined as the distance in which the driver has less than 1.8 seconds to avoid a collision if the lead car would suddenly start braking very hard. According to [Zhang, Antonsson & Grote 2006],1.8 seconds should give most drivers enough time to react

This optimization runs 20 times per second and generates a speed and acceleration trajectory. If this trajectory contains a deceleration that is significantly lower than our current deceleration, we know hard braking is required in the near future and it is time to warn the driver.

This optimization runs 20 times per second and generates a speed and acceleration trajectory.

However, just using this trajectory would generate a lot of false positives. There are a lot of situations where a human drivers enter the danger zone while this is still pretty safe. For example, this happens when the driver is about to change lanes. By running the algorithm on a lot of data we were able to come up with conditions to filter these false positives, while still keeping most of the useful warnings.

Example of a false positive

Results

We ran our algorithm on a subset of our driving data since June 2017, about 6400 hours of driving. By changing parameters we found a tuning that resulted in 625 successful warnings and 246 false positives. There were 7531 false negatives, but most of those didn’t warrant an FCW (e.g. hard braking for a red light). This means our target of 1 false positive per 2 real warnings is reached and the system can be shipped.

Qualitative plot of false positives vs. events detected when increasing sensitivity

Conclusion

Of course, the development of the FCW system is not finished, because there are still situations where the system did not trigger a warning when it should have. By collecting more data and spending more time on this problem we can improve the performance over time.

Additionally, we could train a neural network based on raw camera images and feed this as an extra input to the FCW system. Alternatively we could train a net to decide whether to trigger an FCW based on the available metrics, instead of manually selecting the features for which we trigger.

However, data already shows the current system has the potential to improve the safety of our users. And unlike most of the systems built by car manufacturers, we have the ability to roll out over-the-air updates with system improvements.

Want to work on interesting problems like this? Check out our open positions. Also follow us on Twitter.