Interactive text-based person retrieval resolves ambiguity in an underspecified initial query through iterative clarification. Existing methods rely on free-form dialogue, so acquired evidence is only implicitly encoded in dialogue history, remaining uncertainty is not explicitly represented, and generated questions are often weakly grounded in the visual differences that determine ranking — a problem amplified by the long-tailed distribution of visual attributes. We reformulate interactive retrieval as a retrieval-state-conditioned evidence-acquisition problem over a grounded discrete concept space of reusable concept atoms, a shared representation for the initial query, candidate-side evidence and user feedback. At each round the system selects the next concept atom conditioned on the current candidate set using a redundancy-aware utility that jointly favors ambiguity reduction, candidate discrimination and high-quality partitioning of the candidate pool. Natural language serves only as the interaction interface, while decision-making is performed over structured concept-level evidence — replacing unconstrained dialogue generation with structured, utility-driven clarification and requiring no additional question-generation or interaction-policy training. On Interactive-PEDES under a unified multi-round protocol, the method consistently outperforms language-centric baselines, with R@1 improving by 31.01% over five rounds.
Text-based person retrieval usually assumes the initial description uniquely identifies the target, but real queries are underspecified and errors come from unresolved ambiguity among visually similar candidates that differ only in subtle, identity-defining details. Interactive retrieval can acquire the missing evidence, yet a language-driven paradigm couples interaction to unstructured generation: acquired evidence stays implicit in the history, remaining uncertainty is not represented, and questions are weakly grounded in what actually separates candidates.
This work reformulates interaction as retrieval-state-conditioned evidence acquisition over a grounded discrete concept space. Queries, candidate-side evidence and user feedback are unified as reusable concept atoms, and a retrieval-conditioned planner selects the next atom to clarify using a redundancy-aware utility that balances ambiguity reduction, candidate discrimination and clean partitioning of the current candidate set. Language is only the interface; decisions are made over structured concept-level evidence, so no extra question-generation or interaction-policy module is trained. A key insight is the long-tailed distribution of visual attributes — frequent attributes are easy to ask about but quickly become weakly discriminative, whereas long-tail attributes are more informative, so the planner deliberately shifts its question budget toward those informative mid- and tail concepts.
On Interactive-PEDES under a unified multi-round protocol, the method delivers faster early-stage refinement and larger cumulative gains than language-centric baselines across interaction rounds, with R@1 improving by 31.01% over five rounds under a shared IRRA backbone. Analysis shows the baseline over-queries a few high-frequency (head) concepts, while the proposed planner reallocates its questions toward more informative mid- and tail concepts, which yields the largest per-tier improvements from round 0 to round 5. The paper is accepted at ACM MM 2026.