Text-based person anomaly search is commonly solved by ranking the gallery independently for each query. This query-wise inference ignores a key property of the verified PAB2025 evaluation mapping: each query has one positive image, and no positive image is shared across queries. Consequently, duplicated top-1 predictions are mutually incompatible, and at least one competing decision must be incorrect. We address these conflicts through protocol-aware set-level inference. Starting from retrieval and cross-encoder scores, we construct a bipartite graph from each query’s leading candidates and identify connected components containing duplicated top-ranked images. Within each component we solve a maximum-weight one-to-one assignment in which every query also holds a slack option priced at a fixed score budget; a rank prior shapes the candidate graph and its collisions, while the slack price suppresses unreliable reassignments inside the objective. The decision layer is model-agnostic in formulation and requires no retraining, although its score calibration is reranker-dependent. On the public PAB2025 annotations, component assignment alone improves the rank-prior blended reranker by 1.9 mAP@10 and 3.8 Recall@1 points. Pairwise Similarity-Coverage Analysis (SCA) followed by component assignment reaches 98.64 mAP@10 and 97.47 Recall@1. The associated challenge system obtains 98.35 mAP@10 on the hidden-label PAB2026 test set. These results indicate that explicitly modeling cross-query incompatibilities can recover a substantial portion of residual top-rank errors once candidate recall is saturated.
Text-based person anomaly search requires matching a natural-language description not only to a person’s appearance, but also to fine-grained actions, events, object interactions, and temporal states. Modern systems typically retrieve and rerank candidates independently for each query, even when the evaluation protocol imposes additional structure across the complete query set.
This work observes that the verified PAB2025 mapping is injective: every query has exactly one positive image, and no positive image is shared by two queries. Therefore, when multiple queries independently select the same image at rank 1, their predictions are jointly inconsistent. Moreover, resolving one collision may create another because alternative candidates can connect several queries into larger components.
We introduce protocol-aware set-level inference to resolve these conflicts jointly. Retrieval rankings from multiple late-interaction vision-language models are fused with Reciprocal Rank Fusion and rescored by Qwen3-VL-Reranker. Rank-prior blending then combines retrieval agreement with cross-encoder confidence. Finally, duplicated predictions and their alternatives form a bipartite graph, and each connected collision component is solved using exact minimum-cost flow with per-query slack options.
The assignment layer requires no model retraining and modifies only the final decision rule. It is applied selectively to collision components, leaving the majority of queries unchanged while jointly coordinating ambiguous top-ranked predictions.
On PAB2025, the three-model late-interaction RRF fusion achieves 95.28 mAP@10 and 91.20 Recall@1 while reaching 100% Recall@50, showing that the remaining errors are primarily caused by candidate ordering rather than candidate omission.
After cross-encoder reranking and rank-prior blending, independent query-wise inference reaches 96.61 mAP@10 and 93.48 Recall@1. Component-wise protocol-aware assignment alone raises performance to 98.53 mAP@10 and 97.32 Recall@1, outperforming pairwise Similarity-Coverage Analysis.
The complete pipeline, combining pairwise SCA with component assignment, achieves 98.64 mAP@10 and 97.47 Recall@1 on PAB2025. Assignment improves over SCA across every analyzed collision-component size and changes only queries involved in structured conflicts, while 80.8% of queries outside collision components remain unaffected.
Using the same configuration calibrated on PAB2025, the challenge system achieves 98.35 mAP@10 and 97.22 Recall@1 on the hidden-label PAB2026 test set, ranking 3rd in AI City Challenge 2026 Track 4.