Testing Autonomous Driving Systems (ADS) is essential to ensure their safety and reliability in dynamic and unpredictable real-world environments. Model-Based Testing (MBT) has proven effective for complex software systems, including Cyber-Physical Systems and the Internet of Things. However, for intelligent systems like ADS, much of their behavior can only be observed during operation. This calls for adaptive MBT approaches, where the test model—the foundation for generating test cases—evolves through continuous interaction with the system under test (SUT) during test execution. To address this need, we propose LiveTCM, a framework that adaptively generates test cases by interacting with ADS within simulation environments.
Back to HomeLiveTCM builds upon our previous work, RTCM[1], a methodology for statically generating test cases from test case specifications (TCS).
RTCM is supported by a metamodel that formalizes key testing concepts such as test setup, test sequence, and test oracle, and enables the transformation of TCS in natural language into structured instances of the metamodel, which can then be converted into executable test cases.
To streamline this transformation process, RTCM includes a tabular template primarily for documenting event flows, along with a set of well-defined keywords such as VALIDATES THAT and INVOKE API to guide the accurate generation of test cases.
However, RTCM and similar MBT approaches fall short in testing ADS, as they rely on static test generation and do not sufficiently account for the evolving nature of these systems and their operational environments.
This highlights the need for a new methodology capable of generating and executing TCS dynamically, through continuous interaction with the system and its environment. Furthermore, the resulting TCS should be treated as valuable artifacts that contribute to a deeper understanding of both the system and its operational context.
The LiveTCM Editor serves as the primary interface for users to interact with the LiveTCM framework. It provides a dedicated pane for specifying and visualizing key components of a TCS, including the precondition (test data specification), test setup, and test steps, which are organized as structured flows of events.
Test Case Specification Generator of LiveTCM can automatically and dynamically generate, execute, and verify TCS together with various test scenario generation strategies such as DeepCollision[2].
Explore our foundational research and related publications
Y. Shi, C. Lu, M. Zhang, H. Zhang, T. Yue and S. Ali, "Restricted Natural Language and Model-based Adaptive Test Generation for Autonomous Driving," 2021 ACM/IEEE 24th International Conference on Model Driven Engineering Languages and Systems (MODELS), 2021, pp. 101-111, doi: 10.1109/MODELS50736.2021.00019.
View Publication