← All publications
WACV 2026 · IEEE/CVF Winter Conference on Applications of Computer Vision

ITSELF: Attention Guided Fine-Grained Alignment for Vision-Language Retrieval

Tien-Huy Nguyen, Huu-Loc Tran, Thanh Duc Ngo

Abstract

Vision Language Models (VLMs) have rapidly advanced and show strong promise for text-based person search (TBPS), a task that requires capturing fine-grained relationships between images and text to distinguish individuals. Previous methods address these challenges through local alignment, yet they are often prone to shortcut learning and spurious correlations, yielding misalignment. Motivated by the finding that encoder attention surfaces spatially precise evidence from the earliest training epochs, we introduce ITSELF, an attention-guided framework for implicit local alignment. At its core, Guided Representation with Attentive Bank (GRAB) converts the model's own attention into an Attentive Bank of high-saliency tokens and applies local objectives on this bank, learning fine-grained correspondences without extra supervision. Multi-Layer Attention for Robust Selection (MARS) aggregates attention across layers with diversity-aware top-k selection, and an Adaptive Token Scheduler (ATS) schedules the retention budget from coarse to fine over training.

Overview

ITSELF targets text-based person search (TBPS), where a system must retrieve the right individual from fine-grained image–text relationships. Rather than relying on explicit local alignment — which is prone to shortcut learning and spurious correlations — ITSELF performs implicit local alignment guided by the model's own attention.

At its core, Guided Representation with Attentive Bank (GRAB) converts encoder attention into an Attentive Bank of high-saliency tokens and applies local objectives on that bank, learning fine-grained correspondences without extra supervision. Multi-Layer Attention for Robust Selection (MARS) aggregates attention across layers and performs diversity-aware top-k selection, while the Adaptive Token Scheduler (ATS) anneals the token-retention budget from coarse to fine over training — preserving context early and focusing on discriminative detail later.

Figures
ITSELF architecture. GRAB turns the encoder's own attention into a bank of high-saliency tokens; MARS performs multi-layer, diversity-aware selection and the Adaptive Token Scheduler drives implicit local alignment under combined global and local objectives.
Fig. 1. ITSELF architecture. GRAB turns the encoder's own attention into a bank of high-saliency tokens; MARS performs multi-layer, diversity-aware selection and the Adaptive Token Scheduler drives implicit local alignment under combined global and local objectives.
Analysis and ablations: per-layer attention entropy across training, and the effect of layer-setting and token discard-ratio choices on retrieval performance.
Fig. 2. Analysis and ablations: per-layer attention entropy across training, and the effect of layer-setting and token discard-ratio choices on retrieval performance.
Qualitative text-based person search. For each query, ITSELF ("Ours") retrieves more accurate matches than the RDE baseline, with attention concentrating on discriminative regions.
Fig. 3. Qualitative text-based person search. For each query, ITSELF ("Ours") retrieves more accurate matches than the RDE baseline, with attention concentrating on discriminative regions.
Results

On three widely used TBPS benchmarks, ITSELF reaches state-of-the-art performance and strong cross-dataset generalization, without any additional prior supervision. The work appears at WACV 2026 (IEEE/CVF Winter Conference on Applications of Computer Vision).