Executive Summary
Conventional metrics (such as frame rate and resolution) used for evaluating LiDAR systems don’t adequately or explicitly address real-world problems facing autonomous driving. Therefore, AEye, the developer of iDAR™ technology, proposes two new corresponding metrics for evaluation: object revisit rate and instantaneous resolution. These additional metrics are necessary to better describe the safety and performance of more advanced LiDAR sensors in real-world scenarios.
Download “Deconstructing Two Conventional LiDAR Metrics” [pdf]
Introduction
How is the effectiveness of an intelligent detection system measured? Conventional metrics used for evaluating LiDAR systems rely on frame rate and resolution (as well as range which we will discuss at a later time) as the touchstones of success. However, AEye believes that these measurements are inadequate for evaluating the effectiveness of more advanced LiDAR systems for autonomous vehicles. In this white paper, we discuss why object revisit rate and instantaneous resolution are more meaningful metrics to assess the capabilities of our iDAR system, and why these metrics are ultimately more advantageous for autonomous vehicle development.
Deconstructing the Metrics
Makers of automotive LiDAR systems are frequently asked about their frame rate, and whether or not their technology has the ability to detect objects with 10 percent reflectivity at some range and at some frame rate with some arbitrary resolution. While most manufacturers can readily answer these questions, we believe that this description is insufficient and that the industry must adopt a more holistic approach when it comes to assessing LiDAR systems for automotive use. Additionally, we must think of them as they relate to a perception system in general—rather than as an individual point sensor. Below, we have outlined two conventional LiDAR metrics and AEye’s additional metrics.
Conventional Metric #1: Frame rate of xx Hz
AEye’s Metric
Objective revisit rate (the time between two shots at the same point or set of points)
Defining single point detection range alone is insufficient because a single interrogation point (shot) rarely delivers enough confidence—it is only suggestive. Therefore, we need multiple interrogation/detects at the same point or multiple interrogation/detects on the same object to validate or comprehend an object or scene. The time it takes to detect an object is dependent on many variables, such as distance, interrogation pattern and resolution, reflectivity, or the shape of the objects to interrogate, and can “traditionally” take several full frames to achieve. What is missing from the conventional metric, therefore, is a finer definition of time. Thus, AEye proposes that object revisit rate becomes a new, more critical metric for automotive LiDAR because an agile LiDAR such as AEye’s iDAR can have object revisit rate that is vastly superior to its traditional classic frame rate.
The time between the first measurement of an object and the second is critical, as shorter object revisit times can help keep processing times low for advanced algorithms that need to correlate between multiple moving objects in a scene. Additionally, too long of an object revisit time at fast velocities could be the difference between detecting an object before it’s too late and loss of life, since even the best algorithms used to associate/correlate multiple moving objects can be confused when many objects are in the scene and time elapsed between samples is high.
The agile AEye platform accelerates revisit rate by allowing for intelligent shot scheduling within a frame, including the capability to interrogate a target position or object multiple times before the traditional classic frame is completed. For example, an iDAR sensor can schedule two repeated shots on a point or points of interest in quick succession. These multiple interrogations can then be used according to the scene context and the needs of the user (either human or another computer) to increase confidence (or even extend ranging performance).
These interrogations can also be data dependent. For example, an object can be revisited if a (low confidence) detection occurs, and it is desirable to quickly validate, or reject, said detect with a secondary measurement, as seen in Figure 1. A typical completive full frame rate (traditional classic) for conventional sensors is approximately 10Hz, or 100 msec. This is also, for said conventional sensors, equivalent to the “object revisit rate.” With AEye’s flexible iDAR technology, the object revisit rate is now different from the frame rate and it can be as low as 10s of microseconds between revisits to key points/objects as the user/host requires—easily 3 to 4 orders of magnitude faster than alternative fixed scan sensors.
Figure 1. “Foveation in Time” Intra-Frame Revisit Interval and random scan pattern of iDAR (B) compared to Revisit Interval on a typical fixed pattern LiDAR (A)
What this means is that an effective perception engineering team using dynamic object revisit capabilities can create a perception system that is at least an order of magnitude faster than what can be delivered by conventional LiDAR. We believe this capability is invaluable in delivering level 4/5 autonomy as the vehicle will need to handle significantly complex corner cases.
Real-World Application: When you’re driving, the world can change dramatically in a tenth of a second. In fact, two cars closing at a mutual speed of 200 km/hour are 18 feet closer after 0.1 seconds. By having an accelerated revisit rate, we increase the likelihood of hitting the same target with a subsequent shot due to the decreased likelihood that the target has moved significantly in the time between shots. This helps the user solve the “Correspondence Problem” (determining which parts of one “snapshot” of a dynamic scene correspond to which parts of another snapshot of the same scene), while simultaneously enabling the user to quickly build statistical measures of confidence and generate aggregate information that downstream processors might require (such as object velocity and acceleration). While the “Correspondence Problem” will always be a challenge for autonomous systems, the ability to selectively increase revisit rate on points of interest can significantly aid higher level inferencing algorithms, allowing them to more quickly determine correct solutions.
Furthermore, only allocating shots to extract velocity and acceleration when detections have occurred (part of the acquisition chain) rather than allocating repeat shots everywhere in the frame vastly reduces the required number of shots per frame. For example, even in dense traffic, only 1% of the occupancy grid may contain detections. Adding a second detection, via iDAR, to build a velocity estimate on each detection increases the overall number of shots by only 1%, whereas obtaining velocity everywhere, as mandated by fixed scan systems, doubles the required shots (100%, i.e., 2x increase). This speed and shot saliency ultimately makes autonomous driving much safer because it eliminates ambiguity and allows for more efficient use of downstream processing resources. Solving other “Correspondence Problems” (think: camera/LiDAR) with iDAR is the subject of a future paper.
The AEye Advantage: Whereas other LiDAR systems are limited by the physics of fixed laser pulse energy, fixed dwell time, and fixed scan patterns, AEye’s iDAR technology is a software definable system that allows downstream processors to tailor their data collection strategy to best suit their information processing needs at design time and/or run time. Physics, of course, remains the ultimate arbiter, with the primary physics constraints being the photon budget (laser average power), and the speed of light induced round trip flight time, but the AEye software agility allows us to achieve the limit of physics in a tailored (as opposed to global) fashion. The achievable object revisit rate of AEye’s iDAR system for points of interest (not just the exact point just visited) is microseconds to a few milliseconds, compared to conventional LiDAR systems that require many tens or hundreds of milliseconds between revisits, and therefore, a high degree of object correspondence ambiguity. This gives the unprecedented ability to calculate things like object velocity in any direction faster than any other system.
The ability to define the new metric, Object Revisit Rate, which is decoupled from the traditional “frame rate,” is important also for the next metric we introduce. This second metric helps to segregate the basic idea of “search” algorithms from “acquisition” algorithms: two algorithm types that should never be confused. Separation of these two basic types of algorithms provides insight into the heart of iDAR, which is the Principle of Information Quality as opposed to Data Quantity. Or, in other words: “more information, less data.”
Conventional Metric #2: Fixed (angular) resolution over a fixed Field-of-View
AEye’s Metric
Instantaneous (angular) resolution
The assumption behind the use of resolution as a conventional metric is that it is assumed the Field-of-View will be scanned with a constant pattern. This makes perfect sense for less intelligent traditional sensors that have limited or no ability to adapt their collection capabilities. Additionally, the conventional metric assumes that salient information resident within the scene is uniform in space and time, which we know is not true. Because of these assumptions, conventional LiDAR systems indiscriminately collect gigabytes of data from a vehicle’s surroundings, sending those inputs to the CPU for decimation and interpretation (wherein an estimated 70 to 90 percent of this data is found to be useless or redundant, and thrown out). It’s an incredibly inefficient process. Note this is doubly inefficient: the active collection of..
