Extending Conventional LiDAR Metrics to Better Evaluate Advanced Sensor Systems
Executive Summary
As the autonomous vehicle market matures, sensor and perception engineers have become increasingly sophisticated in how they evaluate system efficiency, reliability and performance. Many industry leaders have recognized that conventional metrics for LiDAR data collection (such as frame rate, full frame resolution, and detection range) currently used for evaluating performance no longer adequately measure the effectiveness of sensors to solve real world use cases that underlie autonomous driving.
First generation LiDAR sensors passively search a scene and detect objects using background patterns that are fixed in both time (no ability to enhance with a faster revisit) and in space (no ability to apply extra resolution to high interest areas like the road surface or intersections).
A new class of advanced solid-state LiDAR sensors enable intelligent information capture that expands their capabilities and moves from passive “search” or detection of objects, to active search and, in many cases, to the actual acquisition and classification attributes of objects in real time.
Because early generation LiDARs used fixed raster scans, the industry was forced to adopt overly simplistic performance metrics that did not capture all the nuances of the sensor requirements needed to enable AVs. In response, AEye, the developer of iDAR technology (which includes agile LiDAR) is proposing the consideration of three new corresponding metrics for extending LiDAR evaluation. Specifically: extending the metric of frame rate to include intra-frame object revisit rate; expanding resolution to capture instantaneous enhanced resolution; and enhancing detection range to reflect the more critically important object classification range.
We are proposing that these new metrics be used in conjunction with existing measurements of basic camera and passive LiDAR performance as they measure a sensor’s ability to intelligently enhance perception and create a more complete evaluation of a sensor system’s efficacy in improving safety and performance in real-world scenarios.
Download “Rethinking the Three “Rs” of LiDAR: Rate, Resolution and Range” [pdf]
Introduction
We have often found it useful to leverage proven frameworks from advanced robotic vision research and apply them to LiDAR-specific product architecture. One that has proven to be both versatile and instructive has been work around object identification that connects search, acquisition (or classification) and action.
Search is the ability to detect any and all objects without the risk of missing anything.
Acquire is defined as the ability to take a search detection and enhance the understanding of an object’s attributes to accelerate classification and determine possible intent (this could be by calculating velocity or by classifying object type).
Act defines an appropriate sensor response as trained or as recommended by the vehicle’s perception system or domain controller. Responses can largely fall into four categories:
Continue scan for new objects (no enhanced information needed)
Continue scan but also interrogate the object further and gather more information on an acquired object’s attributes to enable classification
Continue scan but also continue to track an object classified as currently non-threatening
Continue scan but the control system is going to take evasive action.
Within this framework, performance specifications and system effectiveness need to be assessed with an “eye” firmly on the ultimate objective: completely safe operation of the vehicle. However, as most LiDAR systems today are passive, they are only capable of basic search. Therefore, conventional metrics used for evaluating these systems’ performance relate to basic object detection capabilities – frame rate, resolution, and detection range. If safety is the ultimate goal, then search needs to be more intelligent and acquisition (and classification) done more quickly and accurately so that the sensor or the vehicle can determine how to act immediately.
Rethinking the Metrics
Makers of automotive LiDAR systems are frequently asked about their frame rate, and whether or not their technology has the ability to detect objects with 10 percent reflectivity at some range (often 230 meters). We believe these benchmarks are required, but insufficient as they don’t capture critical details such as the size of the target, speed at which it needs to be detected and recognized, or the cost of collecting that information. We believe it would be productive for the industry to adopt a more holistic approach when it comes to assessing LiDAR systems for automotive use. Additionally, we make the argument that we must look at metrics as they relate to a perception system in general – rather than as an individual point sensor and ask ourselves: “What information would enable a perception system to make better, faster decisions?” Below, we have outlined the three conventional LiDAR metrics and a recommendation on how to extend these metrics.
Conventional Metric #1: Frame rate of 10Hz – 20Hz
New Metric: Object Revisit Rate
(The time between two shots at the same point or set of points)
Defining single point detection range alone is insufficient for sensor detection because a single interrogation point (shot) rarely delivers sufficient confidence – it is only suggestive. Therefore, passive LiDAR systems need multiple interrogation/detects at the same point or multiple interrogations/detects on the same object to validate an object or scene. The time it takes to detect an object is dependent on many variables, such as distance, interrogation pattern and resolution, reflectivity, or the shape of the objects to interrogate, and can “traditionally” take several full frames to achieve.
A key factor that is missing from the conventional metric is a finer definition of time. Thus, we propose that Object Revisit Rate becomes a new, more refined metric for automotive LiDAR because an agile LiDAR, such as AEye’s iDAR, can revisit an object within the same frame. The time between the first measurement of an object and the second is critical, as shorter object revisit times can help keep processing times low for advanced algorithms that need to correlate between multiple moving objects in a scene. The best algorithms used to associate/correlate multiple moving objects can be confused when many objects are in the scene and time elapsed between samples is high. This lengthy combined processing time is a primary issue for the industry.
The agile AEye iDAR platform accelerates revisit rate by allowing for intelligent shot scheduling within a frame. Not only can iDAR interrogate a position or object multiple times within a conventional frame, it can maintain a background search pattern while overlaying additional intelligent shots with the same frame. For example, an iDAR sensor can schedule two repeated shots on a point of interest in quick succession (30ms). These multiple interrogations can then be contextually integrated with the needs of the user (either human or computer) to increase confidence, reduce latency, or extend ranging performance.
These interrogations can also be data dependent. For example, an object can be revisited if a (low confidence) detection occurs, and it is desirable to quickly validate, or reject, enabled with secondary data and measurement, as seen in Figure 1. A typical completive full frame rate (traditional classic) for conventional sensors is approximately 10Hz, or 100 msec. This is also, for said conventional sensors, equivalent to the “object revisit rate.” With AEye’s flexible iDAR technology, the object revisit rate is now different from the frame rate and it can be as low as 10s of microseconds between revisits to key points/objects as the user/host requires – easily 100x to 1000x faster than alternative fixed scan sensors.
Figure 1. Advanced Agile LiDAR Sensors enable intelligent scan patterns such as the “Foveation in Time” Intra-Frame Revisit Interval and random scan pattern of iDAR (B) compared to Revisit Interval on a typical fixed pattern LiDAR (A)
What this means is that a perception engineering team using dynamic object revisit capabilities can create a perception system that is at least an order of magnitude faster than what can be delivered by conventional LiDAR without disrupting the background scan patterns. We believe this capability is invaluable in delivering level 4/5 autonomy as the vehicle will need to handle significantly complex corner cases, such as identifying a pedestrian next to oncoming headlights or a semi-trailer laterally crossing the path of the vehicle.
Within the “Search, Acquire, and Act” framework, an accelerated object revisit rate, therefore, allows for faster acquisition because it can identify and automatically revisit an object, painting a more complete picture of it within the context of the scene. Ultimately, this allows for collection of object classification attributes in the sensor, as well as efficient and effective interrogation and tracking of a potential threat.
Real-World Applications
Use Case: Head-On Detection
When you’re driving, the world can change dramatically in a tenth of a second. In fact, two cars traveling towards each other at 100 kph are 5.5 meters closer to each other after 0.1 seconds. By having an accelerated revisit rate, we increase the likelihood of hitting the same target with a subsequent shot due to the decreased likelihood that the target has moved significantly in the time between shots. This helps the user solve the “Correspondence Problem” (determining which parts of one “snapshot” of a dynamic scene correspond to which parts of another snapshot of the same scene), while simultaneously enabling the user to quickly build statistical measures of confidence and generate aggregate information that downstream processors might ..
