Driving road safety forward: a structured approach for automating ADAS/AD function validation

Improving road safety and driving comfort is the driving force behind the development of Advanced Driver Assistance Systems (ADAS) and Autonomous Driving (AD) technologies. ADAS plays a critical role in reducing human error, but validating these systems efficiently is increasingly challenging due to tighter timelines and cost pressures. The emergence of Software-Defined Vehicles (SDVs), which allow post-production updates and evolving algorithms, intensifies the need for robust, repeatable hardware validation. To keep pace, the industry must rethink its validation strategies—prioritizing early fault detection through advanced simulation and modeling to ensure reliability and accelerate time to market. Traditionally, the testing journey for these systems has faced a significant challenge: a disconnect between software simulations and real-world on-road testing. While software simulations provide valuable insights and play a crucial role in the early stages of development, they fall short when it comes to accurately replicating sensor behaviors and their inherent imperfections. On the other hand, on-road testing is indispensable for validating system performance in real traffic conditions, but it is expensive, inherently risky, and lacks repeatability, especially in the early phases of development.

To truly guarantee the safety of both people and infrastructure, the verification and validation (V&V) process of ADAS/AD systems requires:

• Software simulations
• In-lab scene generation
• Road testing
• Compliance with standards such as safety of the intended functionality (SOTIF, ISO 21448) and functional safety (FuSa, ISO 26262) which focus on understanding and mitigating safety risks.

In today’s ADAS/AD development, OEMs and Tier 1 suppliers have the capability to generate synthetic simulation data for specific scenarios and run tests to assess how well their individual vehicle software functions perform in those tested / simulated scenarios. However, fully understanding how the vehicle behaves at system level with respect to safety and comfort, requires a well-structured approach.

Together with our partners (Keysight, IPG Automotive, and TNO), the ambition is to showcase that the process described above can be automated not only in execution. But also, in the identification of the most relevant scenarios and the post-processing of the simulation and test results, therefore, objectively quantifying the potential functional insufficiencies a vehicle system can endure, and how these could potentially compromise safety.

This allows OEMs and Tier 1 suppliers to very efficiently reduce their development cycles and objectively measure performance improvement (if any). This approach also emphasizes that it is possible to have this process executed across multiple contributing parties and vendors. To achieve this, the steps to be taken are:

• First, it is essential to identify the relevant scenarios that should be tested based on the vehicle's operational design domain (ODD) — the specific conditions and environments the vehicle is intended to operate in.
• Next, it involves defining the parameters of interest for the tuning of the vehicle function (such as interdistance, overlap, velocity, relative velocity, acceleration) linked to those scenarios. Those parameters will also require to be allocated to a certain statistical distribution which is based on real driving experience.
• The challenge continues with the need to analyze the data lake generated from the testing and simulations and produce safety and performance metrics.

The proposed automated ADAS/AD functions validation methodology streamlines the real-world scenario selection, simulates the chosen scenarios using a radar sensor emulator that feeds sensor data into the simulation, and emphasize functional insufficiencies — such as “misdetection” — to enhance overall system safety.

Automate vehicle simulation scenarios with radar sensors in modeFRONTIER

CarMaker from IPG Automotive is a simulation software that can be used for testing vehicles by recreating real-world test scenarios in a virtual environment, simulating every type of road and traffic, and performing realistic execution through the event and maneuver-based testing method. It includes a complete vehicle model, with the possibility to change many elements of vehicle dynamics, and an intelligent driver model, in which it is possible to modify the driver’s behavior.

To demonstrate the automated ADAS/AD validation approach, a simplified scenario with multiple vehicles is considered. It consists of an ego vehicle (white) equipped with a radar sensor, and two target vehicles (red and blue) cutting in and cutting through in front of the ego vehicle. The perception quality attributes for both target vehicles are:

• Object presence
• Object type
• Lateral and longitudinal position
• Lateral and longitudinal velocity

By coupling modeFRONTIER with CarMaker, it is possible to systematically modify a set of input parameters (such as vehicle interdistance, overlap, velocity, relative velocity, acceleration, distance, velocities, lane dimension, guard rail presence and position) that are critical for the risk calculation.

modeFRONTIER then automatically triggers the generation of scenario instances, each representing a unique combination of these parameter values, which are subsequently simulated in CarMaker. This approach enables the efficient exploration of a vast scenario space.

Radar simulation models cannot perfectly replicate the real-world characteristics of a physical radar sensor, in particular when considering the radar supplier’s proprietary filtering algorithms that eliminate irrelevant signals or noise to produce a “clean” radar image to be exploited then by the perception algorithm here as well in many cases a property of the radar supplier. These algorithms are often intellectual property of the radar manufacturer, making accurate simulation challenging. To overcome this issue, and to get closer to the setup that will exist in the car on the road, the Keysight Radar Scene Emulator (RSE) is included in the ADAS/AD functions validation process automation.

Bring validation of ADAS into the lab from the road with Keysight’s Radar Scene Emulator (RSE)

How do we make sure the radar sensors behave as in a real vehicle? A solution that allows for repeatable tests under real-world conditions, with the real sensors, in the lab, is needed. Rixels are RF transceivers small enough to fit into a chip-sized unit. Each one is like a pixel on a TV screen. By putting eight of them on one board and stacking multiple boards next to each other, the matrix of rixels creates a high-resolution wall. This is analogous to a high-definition screen with pixels that display different colors and brightness. Similarly, rixels “display” distance, velocity, and echo strength.

The Keysight RSE leverages echo emulation to generate realistic radar echoes, simulating complex environments with the radar's field of view. Using hundreds of RF transceivers known as ‘rixels’, arranged in a high-resolution matrix, the RSE creates a dynamic scene that models distance, velocity, and object size and classification with precision. Unlike traditional methods that simulate only a few point objects, Scene Emulation enables full-scene modeling. The radar sensor, housed in a sealed chamber, is shielded from external interference, ensuring clean and repeatable testing conditions. This setup allows the actual radar hardware to be integrated directly into the test system, which significantly increases the integration test's fidelity level. It also ensures that the emulated scenario perceived by the radar is synchronized with the simulated environment: a brake actuation, for example, will cause a vehicle to pitch forward, resulting in a more downward field of view of the radar sensor.

The RSE can emulate complex and detailed roadway scenarios in your lab. The RSE allows you to create scenarios containing up to 1024-point targets with independent attributes: speed, direction, distance from the ego vehicle, angle, and more. The Keysight RSE can cover mandatory and proprietary test cases for radar-based ADAS/AD functions.

Watch this video to understand more about what Keysight RSE does.

The collaboration with IPG Automotive provides simulation models for all sensor technologies and the simulation models of the object as they react to the sensors. Keysight and IPG Automotive have integrated the Radar Scene Emulator with CarMaker's scenario simulation ability, enabling the generation of realistic radar reflection data for simulated traffic situations. This enables hardware-in-the-loop (HIL) testing where radar signals emulate actual driving environments while the vehicle and ADAS logic run in simulation. As a result, a co-simulation framework is established between IPG Automotive's CarMaker and the Keysight RSE, which is subsequently automated using modeFRONTIER.

To take advantage of such technology, a co-simulation framework is established between IPG Automotive's CarMaker and the Keysight RSE:

• CarMaker feeds the environmental information into the Keysight RSE and provides to the Keysight RSE the objects that are present within the radar Field of View (FoV).
• The Keysight RSE generates the radar reflection signal towards the real radar installed in the RSE using the radar cross section information provided in CarMaker.

The output from the co-simulation contains:

• The CarMaker outputs such as vehicle positions, velocities and radar synthetic data.
• The real raw radar output and its perception (object level information).

When looking at the results, a radar point cloud is generated both by the Keysight RSE and CarMaker. When overlapping the two point clouds it is possible to make an initial assessment on the radar perception capabilities (identification of all involved objects, ghost objects, etc). To summarize, the CarMaker simulation software from IPG Automotive produces a synthetic version of the scenario while the Keysight radar scene emulator generates real radar data for the physical radar installed in the RSE allowing a realistic response from the radar itself.

ADAS/AD function performance estimation with TNO

One of the bigger challenges at the end of such a simulation pipeline is to assess the performance of the stack-under-test, and determine whether its failure rates and accuracy are sufficient to maintain safety of the vehicle, given the ODD and safety requirements defined upfront. Such performance metrics are not only important to predict the operational performance of the vehicle, but also to prove the safety of the vehicle to legislators in the form of a safety case.
TNO processes the produced data, i.e., a large variety of scenarios in the ODD from CarMaker (ground truth data), the Keysight RSE (measurement data) and the input parameters from modeFRONTIER (scenario setup). Through physics-based relationships, based on the known vulnerabilities and/or functional insufficiencies of, in this case, a radar-based system, a probabilistic model is created. This probabilistic model is subsequently calibrated based on the scenario data, the ground-truth data, and the measurement data. As a result, the model which is calibrated on the stack-under-test can reason about the reliability of the system, given its environment. First of all, this allows forward reasoning, i.e., given a large set of scenarios in which performance needs to be assessed, the trustworthiness of the system (e.g., detection performance, accuracy in positioning and sizing) can be determined, which is a major factor towards safety assurance. In addition, the method also allows backward reasoning, i.e., given the assumption that the stack-under-test could fail, what are the contributing factors to such a failure? This backward reasoning therefore allows for exploration of potentially undiscovered edge-cases, allowing an iterative process to ensure completeness in the verification and validation procedure of the system.

Taking these results, the model can be taken a step further, by incorporating risk-calculations based on the uncertainty in detection, positioning and sizing. This allows for a comprehensive evaluation of vehicle safety on a system level, enabling a large step forward in the challenges of verification and validation of ADAS and ADs. In the scope of this demonstrator, a case-study has been chosen where the presence of a guard-rail, under specific circumstances, could obfuscate the presence of a vehicle assuming the use of a radar sensor. Such a case is well-known in the radar-sensing community and could potentially result in false negatives, hence posing a safety hazard for ADAS and/or AD systems. Here, modeFRONTIER orchestrates the pipeline of CarMaker and the Keysight RSE to generate relevant and realistic data given the ODD. Subsequently, the TNO probabilistic model is calibrated on this data, allowing it to, given environmental conditions (e.g., guard rail locations, potential vehicle locations) reason about the probability of misdetection of a vehicle.

The results show the ability of the TNO probabilistic model to predict obfuscation of the vehicle by the guard rail, i.e., the selected functional insufficiency in the scope of this study. Follow-up steps in this study would be to select additional insufficiencies (e.g., reflections from the road surface, or object fusing during cut-in/cut-out maneuvers) and extend the probabilistic model with further data from the pipeline created through modeFRONTIER, CarMaker, and the Keysight RSE.

Why automating ADAS/AD functions validation matters

The main advantage of using ESTECO technology together with IPG Automotive, Keysight, and TNO lies in the enhanced ability to deliver efficient, accurate, and comprehensive virtual testing and validation solutions for ADAS/AD development. This synergy allows automotive OEMs and Tier 1 suppliers to:

• Shorten development cycles through front-loading testing and validation in virtual scenarios, minimizing physical prototype costs and delay.
• Objectively quantify the level of risk of the vehicle and consider the function proposed.
• Optimize the resource usage (such as experts, track and open road testing).
• Estimate the value on how good the function is and identify which are the critical test cases.
• Explain how the results have been achieved in a transparent manner towards the certification authority.
• Meet critical safety standards by ensuring all steps are transparent and compliant with the rigorous requirements of SOTIF and ISO 26262.