CoEvolve: Construct‑to‑Edit Visual Grounding with Bidirectional State Refinement

Interactive demo

GT

IoU —

▶ Play ‹ Prev Next ›

Real trajectory from the paper (Figure 10): the query "broccoli on top right", RER's five committed regions, and both BDR passes, with coordinates in the paper's [0, 1000] frame.

<refine> reasoning text (fixed) x₁=815 y₁=20 x₂=1000 y₂=420

IoU —

▶ Play ‹ Prev Next ›

How BDR is trained: coordinates are deliberately corrupted, and the refiner learns to reconstruct them in place with global context (the causal mask is removed). Over 30,969 controlled anchors, this lifts mIoU from 58.80 to 86.19.

Abstract

Visual grounding localizes an object described by language with a bounding box. Most multimodal grounding models compress target identification, spatial reasoning, and boundary estimation into one terminal prediction. Free-form rationales make reasoning linguistically explicit but do not necessarily expose measurable, editable spatial states. This makes intermediate localization errors difficult to diagnose and correct, allowing incorrect region choices and imprecise boundaries to persist in the final box. Autoregressive re-generation further lacks an explicit mechanism for synchronous state repair.

We introduce CoEvolve, a construct-to-edit framework that separates grounding into explicit state construction and state editing. Region-Evolution Reinforcement (RER) organizes grounding analysis into a coarse-to-fine semantic–spatial trajectory, with each reasoning step committing to an explicit candidate region. Bidirectional Denoising Refiner (BDR) treats this trajectory as an editable state: bidirectional same-position reconstruction produces a synchronous localization candidate from a fixed observed response, while geometry- and behavior-level supervision teaches BDR to preserve accurate anchors, correct unreliable states, and consolidate reliable improvements.

Evaluations cover natural-image and remote-sensing grounding. CoEvolve achieves 85.72% mIoU across RefCOCO-family test splits and 76.23% on DIOR-RSVG after domain-specific retraining. Under controlled box corruption, BDR raises mIoU from 58.80% to 86.19%. State-source comparisons further support the complementarity of structured state construction and bidirectional state editing.

How CoEvolve works

A single terminal prediction does not show where an error came from. Because CoEvolve writes down the spatial state at every step, wrong region choices and loose boundaries are visible in the trajectory. BDR can then repair the existing state directly instead of re-generating the whole response; the reasoning text is never modified.

Qualitative examples

Results

Table 1 · Grounding on the RefCOCO family (Acc@0.5) CoEvolve trains only on RefCOCOg train, without target-dataset fine-tuning. Overall averages all eight splits. ‡ backbone baseline locally reproduced.

Table 2 · Localization at stricter IoU thresholds on five test splits Each cell reports Acc@0.7 / Acc@0.9; aggregate columns are sample-weighted. ‡ locally reproduced; † aggregates reconstructed from reported size groups.

Table 3 · Remote-sensing grounding on DIOR-RSVG RER and BDR are retrained on the DIOR-RSVG training split. † our reproduced native backbone, evaluated without DIOR-RSVG adaptation.

Downstream segmentation: prompting a frozen SAM3 with CoEvolve boxes yields 78.19 Macro-8 cIoU across the eight RefCOCO-family splits, without mask fine-tuning.

BibTeX

@article{sun2026coevolve,
  title   = {CoEvolve: Construct-to-Edit Visual Grounding with
             Bidirectional State Refinement},
  author  = {Sun, Dongwei and Zhang, Yujie and Yao, Bowen and
             Liu, Pei and Yao, Jing and Cao, Xiangyong},
  journal = {arXiv preprint arXiv:2610.01710},
  year    = {2026}
}