In tri-modal visual question answering (VQA), auxiliary text — captions, OCR outputs, retrieved descriptions — is commonly added to complement image and question, yet its reliability is often uncontrolled. We show that the most harmful auxiliary text is not the most factually incorrect, but the text that aligns with the question while contradicting the image and favoring a specific distractor, systematically redirecting predictions. To isolate this, we introduce the Textual Reliability Ladder, a controlled diagnostic that decomposes auxiliary text along three axes — image consistency, question relevance and option support. Across ScienceQA, VCR, A-OKVQA and Causal-VidQA and several recent VLMs, such distractor-supporting text induces the largest accuracy drops (up to 53.1%) and concentrates errors on specific incorrect options. We further propose a training-free, inference-time intervention that counteracts this redirection via noise-stability steering and dynamic grounding, reducing redirected errors while largely preserving performance under faithful text. The results argue that auxiliary-text reliability must be judged at the decision level, not by factual correctness alone.
Modern VQA increasingly runs in a tri-modal setting where auxiliary text (captions, OCR outputs, retrieved descriptions) is fed alongside the image and question. When accurate this text helps, but its reliability is rarely controlled — and this work shows that not all unreliable text is equally harmful. The most damaging case is "option-redirecting" text: text that is aligned with the question yet inconsistent with the image and quietly supportive of a specific wrong option. Rather than causing diffuse confusion, it systematically pushes the model toward one distractor — a decision-level failure that factual-correctness metrics miss.
To study this precisely, the paper introduces the Textual Reliability Ladder, a controlled protocol that fixes the image, question and options and varies only the auxiliary text along three axes — image consistency, question relevance and option support — yielding five regimes (Faithful, Off-Question, Contradictory, Option-Redirecting, Irrelevant) plus a No-Aux baseline. It then proposes a training-free, inference-time intervention with two stages: noise-stability steering (NSS), which estimates global steering directions from contrastive residuals, and dynamic grounding (DG), which aligns textual representations with question-relevant visual evidence per instance.
Across four datasets (ScienceQA, VCR, A-OKVQA, Causal-VidQA) and six recent VLMs spanning the Qwen2.5-VL, Qwen3-VL and InternVL3.5 families at multiple scales, option-redirecting auxiliary text consistently causes the largest accuracy degradation — up to 53.1% — while other unreliable-text regimes cause only mild changes, and it concentrates errors on specific options rather than spreading uncertainty. The proposed training-free NSS+DG intervention recovers the most accuracy exactly under option-redirecting text (double-digit gains, up to +15.1 points) while leaving performance under faithful text largely intact. The paper is accepted at ACM MM 2026.