← All publications
CVPR 2025 Workshops (CVPRW)

VRAG: Retrieval-Augmented Video Question Answering for Long-Form Videos

Bao Tran Gia, Khiem Le, Tien Do, Tien-Dung Mai, Thanh Duc Ngo, Duy-Dinh Le, Shin'ichi Satoh

Abstract

The rapid expansion of video data across various domains has heightened the demand for efficient retrieval and question-answering systems, particularly for long-form videos. Existing Video Question Answering (VQA) approaches struggle with extended video sequences due to high computational costs, loss of contextual coherence, and challenges in retrieving relevant information. We introduce VRAG: Retrieval-Augmented Video Question Answering for Long-Form Videos, a framework that brings a retrieval-augmented generation (RAG) architecture to the video domain. VRAG first retrieves the most relevant video segments, then applies chunking and refinement to identify key sub-segments, enabling precise and focused answer generation. This maximizes the effectiveness of the Multimodal Large Language Model (MLLM) by ensuring only the most relevant content is processed.

Overview

VRAG brings retrieval-augmented generation (RAG) to long-form video question answering. A query — Known-Item Search or VQA — is encoded and run through a multimodal retrieval system that combines semantic-embedding search, on-screen text search, audio-based search, object filtering and temporal search. The top candidates are then re-scored by an MLLM-based re-ranking module before a dedicated VQA module produces the final answer.

The re-ranking module merges each candidate shot with its temporal neighbours and prompts a multimodal LLM to assess relevance, sharpening the ranking. The VQA module then chunks the selected long video, uses the MLLM to keep only the segments relevant to the question, and reasons over that focused short segment — so the model spends its context budget only on content that matters.

Figures
The VRAG framework. A query is routed through multimodal retrieval (semantic, on-screen text, audio, object and temporal search); an MLLM re-ranking module re-scores the top-N candidates; and a VQA module chunks and refines the chosen video to generate the answer.
Fig. 1. The VRAG framework. A query is routed through multimodal retrieval (semantic, on-screen text, audio, object and temporal search); an MLLM re-ranking module re-scores the top-N candidates; and a VQA module chunks and refines the chosen video to generate the answer.
Re-ranking module: the retrieved shot T is merged with its temporal neighbours (T−3 … T+3) and scored by a multimodal LLM, sharpening relevance before answer generation.
Fig. 2. Re-ranking module: the retrieved shot T is merged with its temporal neighbours (T−3 … T+3) and scored by a multimodal LLM, sharpening relevance before answer generation.
VQA module: a long video is split into segments; the MLLM selects the segments relevant to the question, then reasons over the focused short segment to produce the answer.
Fig. 3. VQA module: a long video is split into segments; the MLLM selects the segments relevant to the question, then reasons over the focused short segment to produce the answer.
Results

Evaluated on a benchmark of KIS and VQA tasks derived from the Video Browser Showdown (VBS) 2019–2025, VRAG improves both retrieval precision and answer quality over a no-re-rank baseline and naive VQA prompting, across several multimodal LLMs. The work was presented in the CVPR 2025 Workshops (CVPRW).