Deconstructing Two Conventional LiDAR Metrics

Executive Summary

Conventional metrics (such as frame rate and resolution) used for evaluating LiDAR systems don’t adequately or explicitly address real-world problems facing autonomous driving. Therefore, AEye, the developer of iDAR™ technology, proposes two new corresponding metrics for evaluation: object revisit rate and instantaneous resolution. These additional metrics are necessary to better describe the safety and performance of more advanced LiDAR sensors in real-world scenarios.

Download “Deconstructing Two Conventional LiDAR Metrics” [pdf]

Introduction

How is the effectiveness of an intelligent detection system measured? Conventional metrics used for evaluating LiDAR systems rely on frame rate and resolution (as well as range which we will discuss at a later time) as the touchstones of success. However, AEye believes that these measurements are inadequate for evaluating the effectiveness of more advanced LiDAR systems for autonomous vehicles. In this white paper, we discuss why object revisit rate and instantaneous resolution are more meaningful metrics to assess the capabilities of our iDAR system, and why these metrics are ultimately more advantageous for autonomous vehicle development.

Deconstructing the Metrics

Makers of automotive LiDAR systems are frequently asked about their frame rate, and whether or not their technology has the ability to detect objects with 10 percent reflectivity at some range and at some frame rate with some arbitrary resolution. While most manufacturers can readily answer these questions, we believe that this description is insufficient and that the industry must adopt a more holistic approach when it comes to assessing LiDAR systems for automotive use. Additionally, we must think of them as they relate to a perception system in general—rather than as an individual point sensor. Below, we have outlined two conventional LiDAR metrics and AEye’s additional metrics.

Conventional Metric #1: Frame rate of xx Hz

AEye’s Metric
Objective revisit rate (the time between two shots at the same point or set of points)

Defining single point detection range alone is insufficient because a single interrogation point (shot) rarely delivers enough confidence—it is only suggestive. Therefore, we need multiple interrogation/detects at the same point or multiple interrogation/detects on the same object to validate or comprehend an object or scene. The time it takes to detect an object is dependent on many variables, such as distance, interrogation pattern and resolution, reflectivity, or the shape of the objects to interrogate, and can “traditionally” take several full frames to achieve. What is missing from the conventional metric, therefore, is a finer definition of time. Thus, AEye proposes that object revisit rate becomes a new, more critical metric for automotive LiDAR because an agile LiDAR such as AEye’s iDAR can have object revisit rate that is vastly superior to its traditional classic frame rate.

The time between the first measurement of an object and the second is critical, as shorter object revisit times can help keep processing times low for advanced algorithms that need to correlate between multiple moving objects in a scene. Additionally, too long of an object revisit time at fast velocities could be the difference between detecting an object before it’s too late and loss of life, since even the best algorithms used to associate/correlate multiple moving objects can be confused when many objects are in the scene and time elapsed between samples is high.

The agile AEye platform accelerates revisit rate by allowing for intelligent shot scheduling within a frame, including the capability to interrogate a target position or object multiple times before the traditional classic frame is completed. For example, an iDAR sensor can schedule two repeated shots on a point or points of interest in quick succession. These multiple interrogations can then be used according to the scene context and the needs of the user (either human or another computer) to increase confidence (or even extend ranging performance).

These interrogations can also be data dependent. For example, an object can be revisited if a (low confidence) detection occurs, and it is desirable to quickly validate, or reject, said detect with a secondary measurement, as seen in Figure 1. A typical completive full frame rate (traditional classic) for conventional sensors is approximately 10Hz, or 100 msec. This is also, for said conventional sensors, equivalent to the “object revisit rate.” With AEye’s flexible iDAR technology, the object revisit rate is now different from the frame rate and it can be as low as 10s of microseconds between revisits to key points/objects as the user/host requires—easily 3 to 4 orders of magnitude faster than alternative fixed scan sensors.

Figure 1. “Foveation in Time” Intra-Frame Revisit Interval and random scan pattern of iDAR (B) compared to Revisit Interval on a typical fixed pattern LiDAR (A)

What this means is that an effective perception engineering team using dynamic object revisit capabilities can create a perception system that is at least an order of magnitude faster than what can be delivered by conventional LiDAR. We believe this capability is invaluable in delivering level 4/5 autonomy as the vehicle will need to handle significantly complex corner cases.

Real-World Application: When you’re driving, the world can change dramatically in a tenth of a second. In fact, two cars closing at a mutual speed of 200 km/hour are 18 feet closer after 0.1 seconds. By having an accelerated revisit rate, we increase the likelihood of hitting the same target with a subsequent shot due to the decreased likelihood that the target has moved significantly in the time between shots. This helps the user solve the “Correspondence Problem” (determining which parts of one “snapshot” of a dynamic scene correspond to which parts of another snapshot of the same scene), while simultaneously enabling the user to quickly build statistical measures of confidence and generate aggregate information that downstream processors might require (such as object velocity and acceleration). While the “Correspondence Problem” will always be a challenge for autonomous systems, the ability to selectively increase revisit rate on points of interest can significantly aid higher level inferencing algorithms, allowing them to more quickly determine correct solutions.

Furthermore, only allocating shots to extract velocity and acceleration when detections have occurred (part of the acquisition chain) rather than allocating repeat shots everywhere in the frame vastly reduces the required number of shots per frame. For example, even in dense traffic, only 1% of the occupancy grid may contain detections. Adding a second detection, via iDAR, to build a velocity estimate on each detection increases the overall number of shots by only 1%, whereas obtaining velocity everywhere, as mandated by fixed scan systems, doubles the required shots (100%, i.e., 2x increase). This speed and shot saliency ultimately makes autonomous driving much safer because it eliminates ambiguity and allows for more efficient use of downstream processing resources. Solving other “Correspondence Problems” (think: camera/LiDAR) with iDAR is the subject of a future paper.

The AEye Advantage: Whereas other LiDAR systems are limited by the physics of fixed laser pulse energy, fixed dwell time, and fixed scan patterns, AEye’s iDAR technology is a software definable system that allows downstream processors to tailor their data collection strategy to best suit their information processing needs at design time and/or run time. Physics, of course, remains the ultimate arbiter, with the primary physics constraints being the photon budget (laser average power), and the speed of light induced round trip flight time, but the AEye software agility allows us to achieve the limit of physics in a tailored (as opposed to global) fashion. The achievable object revisit rate of AEye’s iDAR system for points of interest (not just the exact point just visited) is microseconds to a few milliseconds, compared to conventional LiDAR systems that require many tens or hundreds of milliseconds between revisits, and therefore, a high degree of object correspondence ambiguity. This gives the unprecedented ability to calculate things like object velocity in any direction faster than any other system.

The ability to define the new metric, Object Revisit Rate, which is decoupled from the traditional “frame rate,” is important also for the next metric we introduce. This second metric helps to segregate the basic idea of “search” algorithms from “acquisition” algorithms: two algorithm types that should never be confused. Separation of these two basic types of algorithms provides insight into the heart of iDAR, which is the Principle of Information Quality as opposed to Data Quantity. Or, in other words: “more information, less data.”

Conventional Metric #2: Fixed (angular) resolution over a fixed Field-of-View

AEye’s Metric
Instantaneous (angular) resolution

The assumption behind the use of resolution as a conventional metric is that it is assumed the Field-of-View will be scanned with a constant pattern. This makes perfect sense for less intelligent traditional sensors that have limited or no ability to adapt their collection capabilities. Additionally, the conventional metric assumes that salient information resident within the scene is uniform in space and time, which we know is not true. Because of these assumptions, conventional LiDAR systems indiscriminately collect gigabytes of data from a vehicle’s surroundings, sending those inputs to the CPU for decimation and interpretation (wherein an estimated 70 to 90 percent of this data is found to be useless or redundant, and thrown out). It’s an incredibly inefficient process. Note this is doubly inefficient: the active collection of the data requires energy (from the laser source) and time (from the LiDAR scanner), whilst the subsequent passive interpretation of the data also requires energy (computer processing power) and time (data transfer, search and retrieval).

But AEye’s iDAR technology was deliberately developed to break these assumptions. The AEye team believes that intelligent and agile scanning is more efficient and provides greater safety through faster response times and higher quality (timely) information. Agile LiDAR, which enables faster Object Revisit Rates, also enables dynamic foveation—which can change the instantaneous resolution throughout the FOV. This idea is not new. As humans, we don’t “take in” everything around us equally. Rather, the visual cortex filters out irrelevant information, such as an airplane flying overhead, while simultaneously (not serially) focusing our eyes on a particular point of interest. Focusing on a point of interest allows other, less important objects to be pushed to the periphery. This is called foveation, where the target of our gaze is allotted a higher concentration of retinal cones, thus, allowing it to be seen more vividly. However, iDAR can do this more completely. Whereas humans typically only foveate on one area, iDAR can do this on multiple areas simultaneously and in multiple ways. Furthermore, since humans rely entirely on light from the sun, moon, or extant artificial lighting, human foveation is on receive only, i.e., passive. iDAR, in contrast, foveates on both transmit (regions that the laser light chooses to “paint”) and receive (where/when the processing chooses to focus).

So, in a conventional LiDAR system, there is (i) a fixed Field-of-View and (ii) a fixed uniform or patterned sampling density, choreographed to (iii) a fixed laser shot schedule. AEye’s technology allows for these three parameters to vary almost independently. This leads to an almost endless stream of potential innovations and will be the topic of a later paper.

Although there will always be a max Optical I/O capacity, which we will discuss at a later date, our agile iDAR system allows the user to dynamically change the angular density over the entire Field-of-View, necessarily allowing for new constructs or tools as needed within the max Optical I/O capacity limits, enabling the robust collection of useful, actionable information.

Instantaneous resolution is required to convey that resolution is not, in iDAR, something dictated by physical constraints alone, such as beam divergence, or number of shots per second. Rather, it is determined by optimizing resources in a dynamic fashion.

The Point

These two new metrics of Object Revisit Rate and Instantaneous Resolution are not only more relevant and indicative of what a LiDAR can and should do, but they are also synergistic in combination, allowing us to define new constructs such as Special Regions of Interest (ROI). An example of this follows.

Figure 2 below shows two squares, Square A and Square B. Both squares have a similar number of shot points within them. Square A represents a uniform scan pattern, typical of conventional LiDAR sensors. These fixed scan patterns produce a fixed frame rate with no concept of an ROI. Square B shows an adjusted, unfixed scan pattern. As we can see, the shots in Square B are gathered more densely within and around the ROI (the small box) with the square.

Looking at the graphs associated with Squares A and B, we see that, additionally, the unfixed scan pattern of Square B is able to produce revisits to an ROI within a much shorter interval than Square A. Square B can not only complete one ROI revisit interval, but multiple ROIs within a single frame, whereas, Square A can not complete even one revisit. iDAR does what conventional LiDAR cannot: it enables dynamic perception (just like in humans), allowing the system to focus in on, and gather more comprehensive data about, a particular Region of Interest at unprecedented speed.

Figure 2. Region of Interest (ROI) Revisit Rate and foveation of iDAR (B) compared to conventional scan patterns (A)

Real-World Application: So once again, when points or objects of interest have been identified, we can “foveate” our system in space and/or time to gather more useful information about them. For example, let’s say the system encounters a fast pedestrian that is jaywalking across the street suddenly and directly in the path of the vehicle. Because the path is lateral, even current radars and coherent LiDARs will have trouble recognizing the threat: e.g., lateral velocity vs radial velocity vector. However, because iDAR enables a dynamic change in temporal sampling density and spatial sampling density within a Special Region of Interest, it can focus more of its attention on this jaywalker, and less on irrelevant information, such as parked vehicles along the side of the road (which, presumably, iDAR has already identified long ago and is simply tracking them). Ultimately, this allows our system to more quickly, efficiently, and accurately identify critical information about the jaywalking pedestrian (like their lateral velocity, as well as any abrupt changes). Substitute this jaywalker with a fast moving car and you immediately see the lasting benefits. Therefore, the iDAR system swiftly provides the most useful, actionable data to the domain controller to help determine the best timely course of action. No other system on the market even comes close to this.

The AEye Advantage: A major advantage of iDAR is that it is agile in nature, meaning that the main parameters do not have to be fixed, and therefore, it can take advantage of concepts like time multiplexing. It can actually trade off temporal sampling resolution, spatial sampling resolution, and even range simultaneously at multiple points in the “frame” for any of the other two. This allows the system to have tremendous value in perception and do some amazing things that no other system can.

Conclusion

In this white paper, we have discussed how object revisit rate is a more meaningful metric than frame revisit rate alone because the time between object detections is extremely valuable, and cannot be ignored at the cost of vehicle reaction time to threats. This is tantamount to ignoring safety. Because multiple detects at the same point/object are required to fully comprehend an object or scene, measuring object revisit rate is a more useful and critical metric for automotive LiDAR than (static) frame rate. Additionally, we have argued that quantifying (angular) resolution is insufficient. It is important to further quantify instantaneous resolution (or dynamic resolution) because intelligent and agile resolution in scanning is more efficient and provides greater safety through faster response times, especially when pairing ROIs with convolutional neural networks (a future paper). The agile iDAR system enables this type of toolkit that reduces latency and bandwidth in a dramatic way. It allows for dynamic search, acquisition, and tracking of important objects in a scene at the sensor level, mimicking the process of human perception.

As perception technology improves and more real-world challenges are faced, we must be able (and willing) to adopt new metrics to assess its performance. Because AEye’s iDAR system has advanced the state of artificial perception technology in the market, current, conventional metrics can not accurately assess its capabilities. AEye has products that are completely software definable, meaning, that they fulfill current use cases and will support any additional use cases in the years to come.

In this paper, we have offered two more meaningful metrics for evaluating the safety and performance for our system for real-world automotive applications and more will follow.