Towards Scalable and Context-Aware Multimodal Interactive Video Retrieval

Abstract

Interactive video retrieval (IVR) is challenging due to its multimodal nature, ambiguous user intent and strict real-time constraints. We introduce a unified, multimodal, context-aware framework for large-scale IVR. A fast, scalable design leverages GPU-accelerated indexing, parallel inference and lightweight fusion for sub-second responsiveness at million-scale; a multimodal architecture integrates vision-language embeddings, scene text, speech transcripts and object-level cues through adaptive score fusion to mitigate modality bias; and a multi-stage sequential retrieval paradigm handles ambiguous or partially ordered queries via coarse-to-fine expansion and context-aware alignment.

Overview

The framework follows a unified two-phase architecture. Offline, videos are segmented into shots; keyframes are extracted and indexed with multimodal representations — semantic embeddings, scene text, speech transcripts and object detections — using Milvus for vector search and Elasticsearch for text. Online, user queries are encoded with the same models, candidates are retrieved via hybrid multimodal search, then refined through reliability-weighted score fusion, optional object/audio alignment and sequential event retrieval.

Three contributions: (1) a GPU-accelerated, scalable backend that sustains real-time interaction at million-scale; (2) reliability-weighted multimodal fusion across an ensemble of vision-language models (CLIP, BEiT-3, InternVL-G) plus text-in-image and spoken-content retrieval, mitigating modality bias under noisy evidence; (3) a two-stage context-aware sequential event retrieval that expands candidates within a temporal neighborhood for robust reasoning over loosely ordered events.

Figures

Fig. 1. System architecture. The offline phase segments videos into shots, extracts keyframes and indexes multimodal features — semantic embeddings, scene text, speech transcripts and object detections (Milvus for vector search, Elasticsearch for text). The online phase runs hybrid multimodal search with adaptive score fusion and sequential event retrieval.

Fig. 2. Video Browser Showdown (VBS) 2025 — final team scores. Our team (NII-UIT) placed first with 7,566 points, ahead of all established interactive systems.

Fig. 3. Ho Chi Minh AI Challenge — top-40 team scores. The framework secured three consecutive first-place finishes (2022–2024) on multilingual broadcast media.

Fig. 4. The interactive retrieval interface answering a Known-Item Search (KIS) query, ranking candidate keyframes from the large-scale V3C collection in real time.

Fig. 5. Browsing interactions: from any result, users expand to temporally nearby keyframes, visually similar keyframes, or jump straight to video playback.

Results

First place at the Video Browser Showdown (VBS) 2025, surpassing established state-of-the-art interactive systems under heterogeneous, time-critical conditions. Three consecutive first-place finishes at the Ho Chi Minh AI Challenge (AIC) 2022–2024 on multilingual broadcast media. The framework was further validated on four datasets spanning open-domain web video (V3C, ~28,450 videos / 3,800 hours), underwater footage (MVK), laparoscopic surgical recordings (LapGynLHE) and Vietnamese news (AIC, 1,471 videos / 328 hours), using a benchmark of 72 VBS and 50 AVS queries — confirming scalability, robustness and cross-domain generalization.