Publications

Evidence Placement Matters: Pixel-Grounded Re-Ranking for Training-Free Composed Video Retrieval

Long Le, Khang Nguyen, Bao Tran, Thanh Duc Ngo

Training-free composed video retrieval (CoVR) often converts a reference video into language, rewrites the generated caption according to an edit instruction, and retrieves with the resulting target description. This creates an information bottleneck because visual evidence omitted during generation cannot be recovered by the final matcher. We instead ask what evidence should occupy the final compositional decision. PiL-RANK is a training-free framework that keeps direct visual evidence in the loop until that decision. A frozen retriever first builds a top-k shortlist, then a frozen MLLM listwise re-ranks it from raw query and candidate frames, while generated descriptions serve only as verifiable priors. On WebVid-CoVR-Test, pixel-grounded re-ranking alone improves R@1 from 54.73 to 62.52 with BLIP and from 54.81 to 64.55 with CLIP-L. The prior adds a smaller increment, reaching 64.20 and 65.81. Its effect depends on the pixels available for verification. Replacing query and candidate visual evidence with generated captions flips the identical prior's sign and drops performance below the first-stage baseline. Shortlist expansion to the largest candidate set supported by the MLLM image context (k = 23), combined with rank fusion, raises R@10 from 86.15 to 90.41. The same pipeline also transfers to CIRR, lifting a simple frozen CLIP-L baseline from 13.44 to 42.02 R@1. Generated language is therefore best treated as a prior to be checked against pixels rather than as evidence to retrieve with.

HairCraft: Controllable Hairstyle Transfer via Instance-Specific Adaptation

Quoc-Anh Bui-Huynh, Thanh Duc Ngo

Hairstyle transfer aims to realistically integrate a hairstyle from a reference image onto a source image while preserving facial identity and visual coherence. Current methods struggle with complex hairstyles and subtle details due to reliance on generic encoders, leading to visual artifacts and unrealistic blending. To overcome this, we propose HairCraft, an instance-specific adaptation method explicitly designed for precise hairstyle transfer. HairCraft personalizes a generative diffusion model to each hairstyle instance using only a single reference image, allowing detailed and faithful reproduction of complex hair features. We introduce a novel soft inpainting mechanism, carefully refining hair boundaries during denoising to ensure seamless and natural integration. HairCraft also supports optional user interaction through natural-language prompts and spatial masks, enabling flexible control over hairstyle attributes and placement. Quantitative evaluations and perceptual user studies show that HairCraft outperforms existing state-of-the-art methods in realism, detail preservation, and user control.

Protocol-Aware Global Assignment for Text-Based Person Anomaly Search

Hien Pham Duy, Huy Nguyen Pham Gia, Bao Tran, Thua Nguyen, Tien Do, Thanh Duc Ngo, Duy-Dinh Le, Shin’ichi Satoh

Text-based person anomaly search is commonly solved by ranking the gallery independently for each query. This query-wise inference ignores a key property of the verified PAB2025 evaluation mapping: each query has one positive image, and no positive image is shared across queries. Consequently, duplicated top-1 predictions are mutually incompatible, and at least one competing decision must be incorrect. We address these conflicts through protocol-aware set-level inference. Starting from retrieval and cross-encoder scores, we construct a bipartite graph from each query’s leading candidates and identify connected components containing duplicated top-ranked images. Within each component we solve a maximum-weight one-to-one assignment in which every query also holds a slack option priced at a fixed score budget; a rank prior shapes the candidate graph and its collisions, while the slack price suppresses unreliable reassignments inside the objective. The decision layer is model-agnostic in formulation and requires no retraining, although its score calibration is reranker-dependent. On the public PAB2025 annotations, component assignment alone improves the rank-prior blended reranker by 1.9 mAP@10 and 3.8 Recall@1 points. Pairwise Similarity-Coverage Analysis (SCA) followed by component assignment reaches 98.64 mAP@10 and 97.47 Recall@1. The associated challenge system obtains 98.35 mAP@10 on the hidden-label PAB2026 test set. These results indicate that explicitly modeling cross-query incompatibilities can recover a substantial portion of residual top-rank errors once candidate recall is saturated.

A Semantic State Bridge for Sim2Real Traffic-Safety VQA and Captioning

Nhat Do Minh, Minh Nguyen Huu Tuan, Lam Hoang Le Kim, Long Vo Nguyen Minh, Huan Phan Dinh Minh, Bao Tran, Tien Do, Thanh Duc Ngo, Duy-Dinh Le, Shin’ichi Satoh

Vision–language models trained only on synthetic traffic videos often generalize poorly to real-world scenes because changes in appearance can affect both multiple-choice reasoning and free-form caption generation. We propose a semantic state bridge for synthetic-to-real traffic-safety understanding, where phase-level VQA predictions are mapped into factorized, domain-stable semantic states such as orientation, relative position, action, attention, speed, and environmental context. The framework combines question-aware visual observations with lightweight task-specific adapters, routes predictions according to question type, and converts answer-option scores into canonical state probabilities. For state types with reliable temporal structure, transition priors estimated from synthetic annotations are used to decode consistent state trajectories across the five event phases. The resulting states directly determine selected VQA answers and provide confidence-gated factual constraints for correcting inconsistent caption attributes and actions produced by an independently trained caption model. This design preserves the different optimization objectives of VQA and captioning while coupling them through a common semantic interface at inference time. Under synthetic-only task training and real-world evaluation, the proposed system improves VQA accuracy from 81.85% to 84.77% and increases the caption score from 29.15 to 30.17 without additional caption-model training. These results suggest that explicit semantic-state reasoning provides an effective interface for transferring traffic understanding from synthetic observations to real-world vision–language tasks.

When Helpful Text Hurts: Option-Redirecting Bias in Vision–Language Models

Tam Le Thi Thanh, Hoang Tran Van, Hanh Nguyen, Thanh Duc Ngo

In tri-modal visual question answering (VQA), auxiliary text — captions, OCR outputs, retrieved descriptions — is commonly added to complement image and question, yet its reliability is often uncontrolled. We show that the most harmful auxiliary text is not the most factually incorrect, but the text that aligns with the question while contradicting the image and favoring a specific distractor, systematically redirecting predictions. To isolate this, we introduce the Textual Reliability Ladder, a controlled diagnostic that decomposes auxiliary text along three axes — image consistency, question relevance and option support. Across ScienceQA, VCR, A-OKVQA and Causal-VidQA and several recent VLMs, such distractor-supporting text induces the largest accuracy drops (up to 53.1%) and concentrates errors on specific incorrect options. We further propose a training-free, inference-time intervention that counteracts this redirection via noise-stability steering and dynamic grounding, reducing redirected errors while largely preserving performance under faithful text. The results argue that auxiliary-text reliability must be judged at the decision level, not by factual correctness alone.

From Dialogue to Evidence: Retrieval-State-Conditioned Interaction for Text-Based Person Retrieval

Bao Tran, Thanh Duc Ngo

Interactive text-based person retrieval resolves ambiguity in an underspecified initial query through iterative clarification. Existing methods rely on free-form dialogue, so acquired evidence is only implicitly encoded in dialogue history, remaining uncertainty is not explicitly represented, and generated questions are often weakly grounded in the visual differences that determine ranking — a problem amplified by the long-tailed distribution of visual attributes. We reformulate interactive retrieval as a retrieval-state-conditioned evidence-acquisition problem over a grounded discrete concept space of reusable concept atoms, a shared representation for the initial query, candidate-side evidence and user feedback. At each round the system selects the next concept atom conditioned on the current candidate set using a redundancy-aware utility that jointly favors ambiguity reduction, candidate discrimination and high-quality partitioning of the candidate pool. Natural language serves only as the interaction interface, while decision-making is performed over structured concept-level evidence — replacing unconstrained dialogue generation with structured, utility-driven clarification and requiring no additional question-generation or interaction-policy training. On Interactive-PEDES under a unified multi-round protocol, the method consistently outperforms language-centric baselines, with R@1 improving by 31.01% over five rounds.

Towards Scalable and Context-Aware Multimodal Interactive Video Retrieval

Bao Tran, Khiem Le, Thanh Duc Ngo

Interactive video retrieval (IVR) is challenging due to its multimodal nature, ambiguous user intent and strict real-time constraints. We introduce a unified, multimodal, context-aware framework for large-scale IVR. A fast, scalable design leverages GPU-accelerated indexing, parallel inference and lightweight fusion for sub-second responsiveness at million-scale; a multimodal architecture integrates vision-language embeddings, scene text, speech transcripts and object-level cues through adaptive score fusion to mitigate modality bias; and a multi-stage sequential retrieval paradigm handles ambiguous or partially ordered queries via coarse-to-fine expansion and context-aware alignment.

ITSELF: Attention Guided Fine-Grained Alignment for Vision-Language Retrieval

Tien-Huy Nguyen, Huu-Loc Tran, Thanh Duc Ngo

Vision Language Models (VLMs) have rapidly advanced and show strong promise for text-based person search (TBPS), a task that requires capturing fine-grained relationships between images and text to distinguish individuals. Previous methods address these challenges through local alignment, yet they are often prone to shortcut learning and spurious correlations, yielding misalignment. Motivated by the finding that encoder attention surfaces spatially precise evidence from the earliest training epochs, we introduce ITSELF, an attention-guided framework for implicit local alignment. At its core, Guided Representation with Attentive Bank (GRAB) converts the model's own attention into an Attentive Bank of high-saliency tokens and applies local objectives on this bank, learning fine-grained correspondences without extra supervision. Multi-Layer Attention for Robust Selection (MARS) aggregates attention across layers with diversity-aware top-k selection, and an Adaptive Token Scheduler (ATS) schedules the retention budget from coarse to fine over training.

NII-UIT at VBS2026: Towards Effective Visual Question Answering for Interactive and Multimodal Video Retrieval

Bao Tran, Tien Do, Thanh Duc Ngo, Duy-Dinh Le, Shin'ichi Satoh

Interactive video retrieval increasingly requires answering targeted questions about fleeting moments in long videos, under ambiguous queries and tight time limits. This paper presents the NII-UIT system for the Video Browser Showdown (VBS) 2026, addressing three limitations of current systems: localizing brief answer-bearing segments, the high verification effort demanded of users under time pressure, and limited integration of diverse modalities such as visual content and speech. The system introduces an Answer Span Prediction module to highlight candidate temporal regions, a Candidate Answer Suggestion mechanism that aggregates multimodal cues into verifiable answer options, and a dedicated In-Video Retrieval component.

MVDGC: Joint 3D and 2D Multi-view Pedestrian Detection via Dual Geometric Constraints

Thinh Phan, Hao Vo, Khoa Vo, Cuong Pham, Thanh Duc Ngo, Ngan Le