How to optimize ADAS and AD systems for real-world performance

A major advancement in Autonomous Driving (AD) capabilities is on the horizon. However, most vehicles today are equipped with only basic advanced driver assistance systems (ADAS) features while few have AD capabilities. We refer to the society of automotive engineers (SAE) level 0, level 1 and level 2 driving automation systems. At these levels, the driver retains full responsibility for any vehicle interventions drivers deem inappropriate.

Today, only a limited number of vehicles have reached Level 3 automation, where the vehicle can act independently. However, even at this level, the driver must be prepared to take over if the vehicle encounters a situation it can’t manage. According to McKinsey, consumers want access to AD features and are willing to pay for them. This trend is projected to generate additional value for the auto industry. Based on consumer interest in AD features and commercial solutions available on the market today, ADAS and AD could potentially generate between $300 billion and $400 billion in the passenger car market by 2035.

To successfully advance ADAS and AV systems to SAE levels 3 and 4, OEMs need to embrace a new approach that focuses on simulation-driven development processes. The increasing number and complexity of ADAS functions necessitate extensive testing - hundreds of millions of kilometers - to validate performance. This makes physical testing financially non-viable. So, how can we deliver reliable, compliant AV systems that function in a variety of scenarios?

At ESTECO, we propose a white-box verification system that leverages statistical tools to post-process road testing data and identify critical conditions for safety case definitions. We recently showcased our approach for ADAS/AD validation at Operational Safe Systems (OSS.5) Europe, a premier event for professionals working in the field of operational, functional and system safety for highly automated vehicles. Through our software solutions ESTECO VOLTA and modeFRONTIER, we aim to streamline the investigation of ADAS/AD performance across different sensors, algorithms, actuation and scenarios. This provides a robust solution to drive automation in the automotive industry forward by pinpointing critical test cases and assessing function risks.

Insights from OSS.5 Europe

This year over 150 top industry experts in safety, security, validation, and verification came together to exchange knowledge and insights. ESTECO hosted a thematic workshop (World Café) to discuss the necessity of simulation technology in AD system development. Representatives from Continental, Fraunhofer Institute, Scania, Stellantis, TNO and Zeeker Automotive Europe shared their challenges, ideas and solutions to accelerate ADAS/AD testing.

Defining safety cases through simulation and AI

Simulations can generate an infinity of test scenarios, but not all are relevant or realistic. The challenge lies in ensuring that generated scenarios are viable test cases, even if their occurrence is improbable. With a comprehensive scenario database, relevant AI/ML algorithms can intelligently sample test cases and identify edge cases that lack sufficient data. Certifying AI is an important, complex topic as it involves using ML based models to provide a result for a specific query. The challenge lies in building an acceptable level of confidence on the function and requires an explanation as to how this level of confidence was achieved. Simulations should serve as a supporting tool for the verification and validation (V&V) of AD vehicles, requiring collaboration among stakeholders at both company level and across the industry.

Integrating V-cycle development with CI/CD

While Continuous Integration and Continuous Delivery (CI/CD) processes are prevalent in the software industry, the traditional V-cycle method is commonly used in manufacturing. The development of AD vehicles requires a hybrid approach, combining both methodologies to address the extensive software functions involved.

Business processes allow people to work in a structured manner and to track the project process at any point in time. By having a business process to capture the traditional V-cycle development process, we can also add another business process to consider the software development process required (CI/CD) which intervenes during the various stages of development. Depending on the teams involved, some will see the V-cycle as the way forward while others (software development teams) may believe CI/CD is the only way to manage vehicle development. Realistically, the truth probably lies somewhere in between.

Edge cases and risk acceptance

A major challenge is calculating and integrating risk into the development process. Risk combines exposure (the probability of an event) with severity (the level of impact / injury). By integrating risk as an objective in the simulation process, we can run an optimization where the risk is maximized to identify critical conditions. This assessment helps determine whether the risk of critical scenarios are acceptable (the risk is lower than what a human driver would tolerate ) or if adjustments to algorithms and/or sensors are necessary. But, is this enough? Collaborating with regulatory authorities is crucial to define acceptable confidence levels for certification.

Simulation models for safety cases

Many software vendors promote physics based sensors in simulations, but are they really necessary? The necessity for high detail is debatable. A simulation can never replicate real-world scenarios but it can identify corner cases and critical conditions such as blooming or ghost object generation. Simulation models require tuning and calibration to represent a test. Representing reality in a simulation environment is complex as it involves simulating vehicle behavior and sensor reactions. Different simulation models are built for different purposes and it follows that different levels of precision are required. What remains to be seen is, how detailed do these levels need to be? A possible path could be to leverage the approach used by the Safety Of The Intended Functionality (SOTIF). Within this context, engineers can define where the simulation will yield wrong results. The goal is to determine the minimum requirements for a simulation model to be meaningful while maintaining an appropriate level of fidelity.

How ESTECO Technology addresses the key ADAS/AD verification and validation challenges

Validating an ADvehicle involves an infinite array of potential load cases due to the diverse environments in which these vehicles operate - from busy urban streets to open highways and differing weather conditions. It’s virtually impossible to identify all these factors in physical road testing. AD simulations that consider vehicle dynamics, sensors and controls can facilitate the generation of realistic driving scenarios. But, these need to be realistic and somehow related to the real driving conditions. At ESTECO, we don’t believe that the ADAS/AD verification and validation challenge can be addressed alone.

Simulation and modeling plays an important role in the validation of ADAS/AD algorithms. By using sophisticated virtual environments and mathematical models, simulation allows for the rigorous testing and validation of ADAS/AD algorithms in various edge cases and scenarios that may be challenging to replicate in physical testing. We are currently collaborating with TNO on data mining activities to create a scenario database. This is only the beginning. We know that proving a system’s capabilities in all possible scenarios in a defined Operation Design Domain (ODD) is unfeasible. It is important to have a way to calculate the confidence level the function meets. For calculating the confidence level (risk), we sampled the scenario database within defined ODDs and automated the simulation process involving all the needed simulation software packages for the AD scenarios to be accounted for.

With our digital engineering software solutions ESTECO VOLTA and modeFRONTIER, we aim to help ADAS/AD engineers to streamline the validation process through a scenario-driven design validation and optimization approach. Our white-box verification system isn’t tied to any specific software package or manufacturer. It leverages statistical tools to post-process road testing data and identify critical conditions for the safety case definition.

Here is how our scenario-based verification plan formalizes the validation process:

1. Data collection: gather vehicle data, perform data mining and create a scenario database.
2. Automation: automate ADAS/AD functions with vehicle dynamics sensors and algorithms within the modeFRONTIER process automation and design optimization simulation workflow.
3. Sampling: explore the design space by sampling from initial scenario database and explore the design space to understand most relevant parameters and their limitations.
4. Scenario generation: generate many scenarios with AI/ML algorithms to identify most critical test cases
5. Data sharing: store and share generated data across teams using the ESTECO VOLTA digital engineering platform for SPDM and MDO.
6. Safety case development: build the safety case and define the limits of ADAS/AND functions.

Automated driving is the future, but validating these complex systems remains a significant challenge. ESTECO’s scenario-based verification plan is a portable white-box methodology that streamlines the ADAS/AD validation process, regardless of the manufacturer or system involved.