TrafficComposer - Project Page

Abstract

Autonomous driving systems (ADS) require extensive testing and validation before deployment. However, it is tedious and time-consuming to construct traffic scenarios for ADS testing. In this paper, we propose TrafficComposer, a multi-modal traffic scenario construction approach for ADS testing. TrafficComposer takes as input a natural language (NL) description of a desired traffic scenario and a complementary traffic scene image. Then, it generates the corresponding traffic scenario in a simulator, such as CARLA and LGSVL. Specifically, TrafficComposer integrates high-level dynamic information about the traffic scenario from the NL description and intricate details about the surrounding vehicles, pedestrians, and the road network from the image. The information from the two modalities is complementary to each other and helps generate high-quality traffic scenarios for ADS testing. On a benchmark of 120 traffic scenarios, TrafficComposer achieves 97.0% accuracy, outperforming the best-performing baseline by 7.3%. Both direct testing and fuzz testing experiments on six ADSs prove the bug detection capabilities of the traffic scenarios generated by TrafficComposer. These scenarios can directly discover 37 bugs and help two fuzzing methods find 33%–124% more bugs serving as initial seeds.

Framework

The workflow of TrafficComposer. TrafficComposer leverages an LLM-based textual description parser and a novel visual information extractor to distill information from the two input modalities. Then, TrafficComposer aligns the information from both modalities and encapsulates it into a comprehensive traffic IR. Subsequently, TrafficComposer automatically converts this integrated IR into an executable traffic scenario in a driving simulator.

Video Demonstrations

The ego vehicle (Autoware-operated) follows a highway route. When attempting to exit, it fails to detect a slowing vehicle in the right lane, triggering hard braking, leading to a skid on the wet road surface.

The ego vehicle (Autoware-operated) follows a leading car that suddenly brakes. To prevent a collision, the ADS executes a sharp left turn, resulting in colliding into the wall.

The ego vehicle (Autoware-operated) attempts to overtake stationary traffic. During this maneuver, it unlawfully crosses a solid lane divider and fails to activate its turn signals appropriately.

The ego vehicle (MMFN-operated) hits a pedestrian in a heavily foggy weather.

The ego vehicle (Autoware-operated) attempts to exit the emergency lane but fails to detect a decelerating bus on the left. In response, the ADS executes a corrective maneuver, overturning to the right and colliding with the wall.

The ego vehicle (TransFuzer-operated) fails to detect lane boundaries, causing it to enter the oncoming traffic lane in heavy fog.

Citation

          @article{tu2025trafficcomposer,
            title={Multi-modal Traffic Scenario Generation for Autonomous Driving System Testing},
            author={Tu, Zhi and Niu, Liangkun and Fan, Wei and Zhang, Tianyi},
            journal={Proceedings of the ACM on Software Engineering},
            year={2025},
            number={FSE},
            publisher={ACM New York, NY, USA},
          }