Vision–language models trained only on synthetic traffic videos often generalize poorly to real-world scenes because changes in appearance can affect both multiple-choice reasoning and free-form caption generation. We propose a semantic state bridge for synthetic-to-real traffic-safety understanding, where phase-level VQA predictions are mapped into factorized, domain-stable semantic states such as orientation, relative position, action, attention, speed, and environmental context. The framework combines question-aware visual observations with lightweight task-specific adapters, routes predictions according to question type, and converts answer-option scores into canonical state probabilities. For state types with reliable temporal structure, transition priors estimated from synthetic annotations are used to decode consistent state trajectories across the five event phases. The resulting states directly determine selected VQA answers and provide confidence-gated factual constraints for correcting inconsistent caption attributes and actions produced by an independently trained caption model. This design preserves the different optimization objectives of VQA and captioning while coupling them through a common semantic interface at inference time. Under synthetic-only task training and real-world evaluation, the proposed system improves VQA accuracy from 81.85% to 84.77% and increases the caption score from 29.15 to 30.17 without additional caption-model training. These results suggest that explicit semantic-state reasoning provides an effective interface for transferring traffic understanding from synthetic observations to real-world vision–language tasks.
The 2026 AI City Challenge Track 2 studies synthetic-to-real traffic-safety understanding: task-specific models are trained only on the synthetic SynWTS digital twin and evaluated on real WTS traffic videos. This creates a substantial domain shift in texture, illumination, sensor characteristics, object appearance, and motion realism, affecting both visual question answering and caption generation.
Instead of relying primarily on appearance alignment, this work introduces a semantic state bridge built around traffic concepts that remain meaningful across synthetic and real domains. Phase-level VQA predictions are converted into canonical semantic states such as orientation, relative position, action, attention, speed, and distance. Question-aware frame routing and task-specific adapters provide the observations, while transition priors learned from synthetic annotations enable temporally consistent state decoding across the five event phases.
The VQA and captioning branches remain independently trained but interact at inference time through the semantic state representation. High-confidence VQA states are reused as validated facts to correct inconsistent attributes and actions in generated captions. This allows structured reasoning to improve both tasks without additional caption-model training or task-specific training on real WTS data.
On the official real-world public split, the semantic state bridge improves VQA accuracy from 81.85% to 84.77%, a gain of 2.92 percentage points. Structured spatial world-state decoding provides the largest individual improvement, contributing +2.44 points, while action re-decoding and temporal state conditioning further improve accuracy.
For captioning, state-grounded Fact-Stitch refinement increases the official caption score from 29.15 to 30.17, a total gain of +1.02 without retraining the caption model. The complete system achieves a full-test S2 score of 57.3307 and ranks 2nd in the 2026 AI City Challenge Track 2.
A key finding is that appearance transfer contributes only +0.04 points on the real VQA test set, while semantic-state reasoning accounts for most of the improvement. This suggests that domain-stable traffic semantics transfer more effectively than photometric appearance adaptation in this synthetic-to-real setting.