想快速判断两张图片“看起来像不像”？本文带你用开源的 LocateAnything-3B 模型，从物体检测框的 IoU 角度，计算两张图的相似程度。即使你是第一次接触目标检测或 IoU，也能跟着一步步完成。

---

一、背景：为什么用“框的 IoU”而不是“像素级相似”？

很多人想做图片对比时，第一反应是：用 ResNet50 这样的经典分类网络提取图片特征，然后计算特征向量的余弦相似度。这确实可行，但它更偏向“整体风格”或“语义类别”相似，很难回答一个具体问题：“这两张图里出现了哪些物体？它们的位置和大小一致吗？”

例如，监控摄像头前后两帧，我们希望知道有没有车辆移动；或者同一场景拍摄的两张照片，想知道是否几乎完全重合。这时候，直接比较“图里的物体检测框”会更加直观。

IoU（Intersection over Union，交并比） 就是衡量两个框重叠程度的经典指标

IoU 值在 0（完全不重叠）到 1（完全重合）之间。如果把图片 A 和图片 B 中对应的物体框一一配对，计算所有配对的 IoU 平均值，就能得到一个很实用的“图片结构相似度”。

那怎么得到物体框呢？我们选择 LocateAnything-3B 模型。它是一个基于多模态大语言模型的零样本物体定位器——你不需要提前训练，只需要告诉它“找什么人、车、树”，它就能直接输出框的坐标。完美契合我们的需求。

---

二、操作步骤

下面我们将一步步完成环境搭建、模型推理、封装 API，最后写一个 Demo 来对比两张图片并计算 IoU 相似度。

1. 环境搭建

首先创建独立的 Conda 环境，安装必要的依赖。我们使用 PyTorch 2.11、CUDA 12.8 版本，并安装模型下载工具 modelscope。

# 创建并激活 Conda 环境
conda env remove -n "LocateAnything-3B" -y
conda create -n "LocateAnything-3B" -y "python=3.12" pip
conda activate LocateAnything-3B
# 或者用 source /opt/conda/bin/activate LocateAnything-3B

# 下载模型（国内可用 modelscope 加速）
pip install modelscope
export MODELSCOPE_DOMAIN=www.modelscope.ai
modelscope download --model nv-community/LocateAnything-3B --local_dir ./modles

# 安装 PyTorch 及相关库
python -m pip install --index-url https://download.pytorch.org/whl/cu128 \
  "torch==2.11.0" "torchvision==0.26.0" "torchaudio==2.11.0"
pip install opencv-python-headless==4.11.0.86 transformers==4.57.1 \
	Pillow==11.1.0 peft decord==0.6.0 lmdb==1.7.5

# 进入模型目录
cd modles	

💡 提示：如果你的 GPU 不支持 CUDA 12.8，可以到 PyTorch 官网 查找适配你 CUDA 版本的安装命令。

2. 测试模型是否正常

我们将编写一个 detect_all_objects.py 脚本，它调用 LocateAnything-3B 对图片进行目标检测，并输出结构化的结果。核心逻辑：

• 构建提示词（Prompt）：告诉模型“找出图片中所有属于以下类别的物体：人、车、树……”
• 解析模型的文本输出，从中提取 <box> 标签里的坐标
• 将归一化的坐标（0~1000）映射回原始图像尺寸
• 根据 IoU 阈值去除重叠框（去重）
• 在图片上绘制检测框并保存

cat > detect_all_objects.py << 'EOF'
#!/usr/bin/env python3
import argparse
import json
import os
import re
import time
from pathlib import Path

from PIL import Image, ImageDraw

DEFAULT_CATEGORIES = [
    "person",
    "car",
    "truck",
    "bus",
    "van",
    "bicycle",
    "motorcycle",
    "scooter",
    "chair",
    "bench",
    "tree",
    "building",
    "sign",
    "pole",
    "charging station",
]


def parse_categories(text: str) -> list[str]:
    if not text.strip():
        return list(DEFAULT_CATEGORIES)
    categories = []
    for part in re.split(r"</c>|,|\n", text):
        value = part.strip()
        if value:
            categories.append(value)
    return categories


def build_phrase_prompt(query: str) -> str:
    return f"Locate all the instances that match the following description: {query}."


def build_category_prompt(categories: list[str]) -> str:
    return f"Locate all the instances that matches the following description: {'</c>'.join(categories)}."


def parse_labeled_boxes(answer: str, image_width: int, image_height: int, default_label: str) -> list[dict]:
    boxes = []
    current_label = default_label
    token_re = re.compile(
        r"<ref>(.*?)</ref>|<box><(\d+)><(\d+)><(\d+)><(\d+)></box>",
        flags=re.DOTALL,
    )
    for match in token_re.finditer(answer):
        ref_text = match.group(1)
        if ref_text is not None:
            current_label = ref_text.strip().rstrip(".") or default_label
            continue
        x1, y1, x2, y2 = [int(value) for value in match.groups()[1:]]
        boxes.append(
            {
                "label": current_label,
                "x1": x1 / 1000 * image_width,
                "y1": y1 / 1000 * image_height,
                "x2": x2 / 1000 * image_width,
                "y2": y2 / 1000 * image_height,
            }
        )
    return boxes


def resize_image_long_edge(image: Image.Image, long_edge: int) -> tuple[Image.Image, float]:
    if long_edge <= 0:
        return image, 1.0
    width, height = image.size
    current_long_edge = max(width, height)
    if current_long_edge <= long_edge:
        return image, 1.0
    scale = long_edge / current_long_edge
    resized = image.resize(
        (max(1, int(round(width * scale))), max(1, int(round(height * scale)))),
        Image.Resampling.BICUBIC,
    )
    return resized, scale


def box_area(box: dict) -> float:
    return max(0.0, box["x2"] - box["x1"]) * max(0.0, box["y2"] - box["y1"])


def box_iou(a: dict, b: dict) -> float:
    inter_x1 = max(a["x1"], b["x1"])
    inter_y1 = max(a["y1"], b["y1"])
    inter_x2 = min(a["x2"], b["x2"])
    inter_y2 = min(a["y2"], b["y2"])
    inter_w = max(0.0, inter_x2 - inter_x1)
    inter_h = max(0.0, inter_y2 - inter_y1)
    inter = inter_w * inter_h
    if inter == 0:
        return 0.0
    union = box_area(a) + box_area(b) - inter
    return inter / union if union > 0 else 0.0


def deduplicate_boxes(boxes: list[dict], iou_threshold: float) -> list[dict]:
    kept = []
    for box in sorted(boxes, key=box_area, reverse=True):
        if any(box["label"] == kept_box["label"] and box_iou(box, kept_box) >= iou_threshold for kept_box in kept):
            continue
        kept.append(box)
    return sorted(kept, key=lambda item: (item["label"], item["y1"], item["x1"]))


def draw_boxes(image: Image.Image, boxes: list[dict], line_width: int = 4) -> Image.Image:
    annotated = image.convert("RGB").copy()
    drawer = ImageDraw.Draw(annotated)
    for index, box in enumerate(boxes, start=1):
        xy = [(box["x1"], box["y1"]), (box["x2"], box["y2"])]
        label = f'{index}:{box["label"]}'
        drawer.rectangle(xy, outline="red", width=line_width)
        drawer.text((box["x1"] + 4, max(0, box["y1"] - 16)), label, fill="red")
    return annotated


def run_inference(image, prompts, args):
    from batch_utils import generate_batch_hybrid

    return generate_batch_hybrid(
        [(image, prompt) for prompt in prompts],
        temperature=args.temperature,
        top_p=None if args.top_p < 0 else args.top_p,
        top_k=None if args.top_k <= 0 else args.top_k,
        repetition_penalty=args.repetition_penalty,
        max_new_tokens=args.max_new_tokens,
        scheduler=args.scheduler,
        group_size=0,
    )


def apply_precision_mode(precision: str):
    if precision == "bf16":
        return

    from torchao.quantization import Float8DynamicActivationFloat8WeightConfig, quantize_
    from batch_utils import hybrid_runtime

    model = hybrid_runtime._model
    if model is None:
        raise RuntimeError("Model must be loaded before applying FP8 quantization.")

    config = Float8DynamicActivationFloat8WeightConfig()
    if precision == "fp8-text":
        quantize_(model.language_model, config)
        return
    if precision == "fp8-full":
        quantize_(model, config)
        return
    raise ValueError(f"Unsupported precision mode: {precision}")


def resolve_attention_backend(attn: str, query: str, categories: list[str] | None) -> tuple[str, str]:
    if attn != "auto":
        return attn, "explicitly requested"
    if query.strip():
        word_count = len(re.findall(r"\w+", query))
        if word_count <= 3:
            return "la_flash", "auto: short phrase grounding query"
        return "sdpa", "auto: longer free-form query"
    if categories is not None and len(categories) <= 1:
        return "la_flash", "auto: single category"
    return "sdpa", "auto: multi-category prompt"


def rescale_boxes(boxes: list[dict], scale_x: float, scale_y: float) -> list[dict]:
    if scale_x == 1.0 and scale_y == 1.0:
        return boxes
    scaled = []
    for box in boxes:
        scaled.append(
            {
                **box,
                "x1": box["x1"] * scale_x,
                "y1": box["y1"] * scale_y,
                "x2": box["x2"] * scale_x,
                "y2": box["y2"] * scale_y,
            }
        )
    return scaled


def main():
    parser = argparse.ArgumentParser(description="Detect objects in an image with LocateAnything-3B.")
    parser.add_argument("--model", default=".", help="Model directory.")
    parser.add_argument("--image", default="../20260708_103819_402267.jpg", help="Input image path.")
    parser.add_argument("--out", default="out.jpg", help="Annotated output image path.")
    parser.add_argument(
        "--query",
        default="",
        help="Free-form query mode. If set, script runs phrase grounding instead of category detection.",
    )
    parser.add_argument(
        "--categories",
        default="",
        help="Comma-separated or </c>-separated category list. Empty means built-in defaults.",
    )
    parser.add_argument("--iou-threshold", type=float, default=0.75, help="IoU threshold for deduplication.")
    parser.add_argument("--attn", default="auto", choices=["auto", "sdpa", "eager", "magi", "la_flash"])
    parser.add_argument(
        "--scheduler",
        default="pipeline",
        choices=["eager", "hold_ar", "ar_first", "pipeline", "adaptive"],
    )
    parser.add_argument("--max-new-tokens", type=int, default=8192)
    parser.add_argument("--temperature", type=float, default=0.7)
    parser.add_argument("--top-p", type=float, default=0.9)
    parser.add_argument("--top-k", type=int, default=0)
    parser.add_argument("--repetition-penalty", type=float, default=1.1)
    parser.add_argument("--iterations", type=int, default=1, help="Number of measured inference iterations.")
    parser.add_argument("--warmup-iterations", type=int, default=1, help="Number of warmup iterations to exclude.")
    parser.add_argument(
        "--resize-long-edge",
        type=int,
        default=0,
        help="If > 0, resize the input image so its long edge is at most this value before inference.",
    )
    parser.add_argument(
        "--precision",
        default="bf16",
        choices=["bf16", "fp8-text", "fp8-full"],
        help="Inference precision mode. FP8 modes are experimental torchao quantization paths.",
    )
    args = parser.parse_args()
    if args.iterations < 1:
        raise ValueError("--iterations must be >= 1")
    if args.warmup_iterations < 0:
        raise ValueError("--warmup-iterations must be >= 0")

    if args.query.strip():
        categories = None
        prompts = [build_phrase_prompt(args.query)]
        default_label = args.query.strip()
    else:
        categories = parse_categories(args.categories)
        prompts = [build_category_prompt(categories)]
        default_label = "object"

    selected_attn, selected_attn_reason = resolve_attention_backend(args.attn, args.query, categories)

    os.environ["LA_FLASH_MODEL"] = args.model
    os.environ["LA_FLASH_ATTN"] = selected_attn
    os.environ["LA_FLASH_HYBRID_SCHEDULER"] = args.scheduler

    from batch_utils import get_last_hybrid_stats, load
    from batch_utils.hybrid_runtime import load_pil

    load()
    apply_precision_mode(args.precision)

    original_image = Image.open(args.image).convert("RGB")
    if args.resize_long_edge > 0:
        image, resize_scale = resize_image_long_edge(original_image, args.resize_long_edge)
        image_load_mode = "explicit_resize_long_edge"
    else:
        image = load_pil(args.image)
        resize_scale = image.width / original_image.width if original_image.width > 0 else 1.0
        image_load_mode = "legacy_max_dim"
    raw_results = []
    iteration_times = []

    total_iterations = args.warmup_iterations + args.iterations
    for iteration in range(1, total_iterations + 1):
        start_time = time.perf_counter()
        answers = run_inference(image, prompts, args)
        elapsed = time.perf_counter() - start_time
        if iteration > args.warmup_iterations:
            iteration_times.append(elapsed)
        raw_results = [
            {
                "iteration": iteration,
                "is_warmup": iteration <= args.warmup_iterations,
                "prompt": prompt,
                "answer": answer,
            }
            for prompt, answer in zip(prompts, answers)
        ]

    boxes = parse_labeled_boxes(answers[0], image.width, image.height, default_label=default_label)
    boxes = rescale_boxes(
        boxes,
        scale_x=original_image.width / image.width,
        scale_y=original_image.height / image.height,
    )

    boxes = deduplicate_boxes(boxes, args.iou_threshold)

    annotated = draw_boxes(original_image, boxes)
    annotated.save(args.out, quality=95)

    result = {
        "image": str(Path(args.image)),
        "output_image": str(Path(args.out)),
        "query": args.query or None,
        "categories": categories,
        "requested_attn": args.attn,
        "selected_attn": selected_attn,
        "selected_attn_reason": selected_attn_reason,
        "image_load_mode": image_load_mode,
        "original_image_size": [original_image.width, original_image.height],
        "inference_image_size": [image.width, image.height],
        "resize_long_edge": args.resize_long_edge,
        "resize_scale": round(resize_scale, 4),
        "precision": args.precision,
        "iterations": args.iterations,
        "warmup_iterations": args.warmup_iterations,
        "inference_times_seconds": [round(value, 4) for value in iteration_times],
        "min_inference_time_seconds": round(min(iteration_times), 4),
        "max_inference_time_seconds": round(max(iteration_times), 4),
        "num_boxes": len(boxes),
        "boxes": [
            {
                "index": index,
                "label": box["label"],
                "x1": round(box["x1"], 2),
                "y1": round(box["y1"], 2),
                "x2": round(box["x2"], 2),
                "y2": round(box["y2"], 2),
            }
            for index, box in enumerate(boxes, start=1)
        ],
        "raw_results": raw_results,
        "stats": get_last_hybrid_stats(),
    }
    print(json.dumps(result, ensure_ascii=False, indent=2))

if __name__ == "__main__":
    main()
EOF

重点说明几个地方：

• 坐标归一化：模型输出的坐标范围是 0–1000，所以我们按图像实际宽高进行缩放。
• 去重：模型有时会对同一物体给出多个高度重叠的框，我们用 NMS（非极大值抑制）的思想，以 0.75 的 IoU 阈值保留置信度最高（面积最大）的框。
• 结果 JSON：最后脚本会将所有框、推理耗时等信息打印为 JSON，便于后续程序读取。

运行测试（假设当前目录有一张测试图片）：

python detect_all_objects.py  --warmup-iterations 3 \
	--iterations 5 \
	--image ../20260708_103819_402267.jpg \
	--out out.jpg > out.log 2>&1

输出的 out.log 中会包含检测到的 18 个框的信息，例如：

  "requested_attn": "auto",
  "selected_attn": "sdpa",
  "selected_attn_reason": "auto: multi-category prompt",
  "image_load_mode": "legacy_max_dim",
  "original_image_size": [
    1920,
    1536
  ],
  "inference_image_size": [
    1024,
    819
  ],
  "resize_long_edge": 0,
  "resize_scale": 0.5333,
  "precision": "bf16",
  "iterations": 5,
  "warmup_iterations": 3,
  "inference_times_seconds": [
    1.9965,
    1.9884,
    1.9891,
    2.0055,
    2.0017
  ],
  "min_inference_time_seconds": 1.9884,
  "max_inference_time_seconds": 2.0055,
  "num_boxes": 18,
  "boxes": [
    {
      "index": 1,
      "label": "building",
      "x1": 618.24,
      "y1": 254.98,
      "x2": 1814.4,
      "y2": 751.1
    },

同时会生成一张 out.jpg，上面画出了所有检测框。

3. 封装成 API

为了更方便地在其他项目里复用，我们把检测逻辑封装成一个 LocateAnythingDetector 类。这个类会：

• 在初始化时加载模型，自动切换注意力后端（sdpa、la_flash 等）
• 提供 detect() 方法，输入图片路径，返回框列表
• 额外提供静态方法 compute_iou() 和 compare_box_groups()，用于对比两组框的 IoU

其中 compare_box_groups() 是核心对比函数，它做了两件重要的事：

1. 构建 IoU 权重矩阵：将图片 A 的所有框作为行，图片 B 的所有框作为列，填充两两之间的 IoU。
2. 用匈牙利算法进行最优匹配：因为每张图的框数量可能不同，我们希望通过一对一匹配，让总体 IoU 最大。匈牙利算法能在多项式时间内找到这样的最大权重匹配。

这样，我们就得到了一个“对应框”的配对列表，可以计算出平均 IoU、最大/最小 IoU 以及未匹配上的框。

cat > locateanything_detector.py << 'EOF'
#!/usr/bin/env python3
from __future__ import annotations

import os
import time
from collections import defaultdict
from pathlib import Path
from typing import Any, Sequence

from PIL import Image

from detect_all_objects import (
    DEFAULT_CATEGORIES,
    apply_precision_mode,
    box_iou,
    build_category_prompt,
    build_phrase_prompt,
    deduplicate_boxes,
    draw_boxes,
    parse_categories,
    parse_labeled_boxes,
    rescale_boxes,
    resolve_attention_backend,
    resize_image_long_edge,
)


class LocateAnythingDetector:
    """Reusable LocateAnything-3B detector wrapper."""

    def __init__(
        self,
        model_path: str = ".",
        attn: str = "auto",
        scheduler: str = "pipeline",
        precision: str = "bf16",
        max_new_tokens: int = 8192,
        temperature: float = 0.0,
        top_p: float = 0.9,
        top_k: int = 0,
        repetition_penalty: float = 1.1,
    ) -> None:
        self.model_path = model_path
        self.attn = attn
        self.scheduler = scheduler
        self.precision = precision
        self.max_new_tokens = max_new_tokens
        self.temperature = temperature
        self.top_p = top_p
        self.top_k = top_k
        self.repetition_penalty = repetition_penalty
        self.last_result: dict[str, Any] | None = None

        load_attn = self.attn if self.attn != "auto" else "sdpa"
        os.environ["LA_FLASH_MODEL"] = self.model_path
        os.environ["LA_FLASH_ATTN"] = load_attn
        os.environ["LA_FLASH_HYBRID_SCHEDULER"] = self.scheduler

        from batch_utils import generate_batch_hybrid, get_last_hybrid_stats, load
        from batch_utils.hybrid_runtime import _set_llm_mode, load_pil

        self._generate_batch_hybrid = generate_batch_hybrid
        self._get_last_hybrid_stats = get_last_hybrid_stats
        self._load = load
        self._load_pil = load_pil
        self._set_llm_mode = _set_llm_mode

        self._load()
        apply_precision_mode(self.precision)
        _, _, self._model = self._load()
        self._current_attn = getattr(self._model, "_la_flash_requested_attn", load_attn)

    def _switch_attention_backend(
        self,
        query: str,
        categories: list[str] | None,
    ) -> tuple[str, str]:
        selected_attn, selected_reason = resolve_attention_backend(self.attn, query, categories)
        if selected_attn == self._current_attn:
            return selected_attn, selected_reason

        try:
            self._set_llm_mode(self._model, selected_attn)
            self._current_attn = selected_attn
            return selected_attn, selected_reason
        except Exception as exc:
            if selected_attn == "sdpa":
                raise
            self._set_llm_mode(self._model, "sdpa")
            self._current_attn = "sdpa"
            return "sdpa", f"{selected_reason}; fallback to sdpa: {exc}"

    def _prepare_categories(self, categories: str | Sequence[str] | None) -> list[str]:
        if categories is None:
            return list(DEFAULT_CATEGORIES)
        if isinstance(categories, str):
            return parse_categories(categories)
        values = [str(item).strip() for item in categories if str(item).strip()]
        return values or list(DEFAULT_CATEGORIES)

    def _format_boxes(self, boxes: list[dict[str, Any]]) -> list[dict[str, Any]]:
        return [
            {
                "index": index,
                "label": box["label"],
                "x1": round(box["x1"], 2),
                "y1": round(box["y1"], 2),
                "x2": round(box["x2"], 2),
                "y2": round(box["y2"], 2),
            }
            for index, box in enumerate(boxes, start=1)
        ]

    def detect(
        self,
        image_path: str,
        *,
        query: str = "",
        categories: str | Sequence[str] | None = None,
        resize_long_edge: int = 0,
        iou_threshold: float = 0.75,
        save_annotated_path: str | None = None,
        return_details: bool = False,
    ) -> list[dict[str, Any]] | dict[str, Any]:
        if query.strip() and categories is not None:
            raise ValueError("Use either query or categories, not both.")

        if query.strip():
            parsed_categories = None
            prompts = [build_phrase_prompt(query.strip())]
            default_label = query.strip()
        else:
            parsed_categories = self._prepare_categories(categories)
            prompts = [build_category_prompt(parsed_categories)]
            default_label = "object"

        selected_attn, selected_reason = self._switch_attention_backend(query, parsed_categories)

        original_image = Image.open(image_path).convert("RGB")
        if resize_long_edge > 0:
            image, resize_scale = resize_image_long_edge(original_image, resize_long_edge)
            image_load_mode = "explicit_resize_long_edge"
        else:
            image = self._load_pil(image_path)
            resize_scale = image.width / original_image.width if original_image.width > 0 else 1.0
            image_load_mode = "legacy_max_dim"

        start_time = time.perf_counter()
        answers = self._generate_batch_hybrid(
            [(image, prompt) for prompt in prompts],
            temperature=self.temperature,
            top_p=None if self.top_p < 0 else self.top_p,
            top_k=None if self.top_k <= 0 else self.top_k,
            repetition_penalty=self.repetition_penalty,
            max_new_tokens=self.max_new_tokens,
            scheduler=self.scheduler,
            group_size=0,
        )
        elapsed = time.perf_counter() - start_time

        boxes = parse_labeled_boxes(
            answers[0],
            image.width,
            image.height,
            default_label=default_label,
        )
        boxes = rescale_boxes(
            boxes,
            scale_x=original_image.width / image.width,
            scale_y=original_image.height / image.height,
        )
        boxes = deduplicate_boxes(boxes, iou_threshold)

        formatted_boxes = self._format_boxes(boxes)
        if save_annotated_path:
            annotated = draw_boxes(original_image, boxes)
            annotated.save(save_annotated_path, quality=95)

        self.last_result = {
            "image": str(Path(image_path)),
            "query": query or None,
            "categories": parsed_categories,
            "requested_attn": self.attn,
            "selected_attn": selected_attn,
            "selected_attn_reason": selected_reason,
            "image_load_mode": image_load_mode,
            "original_image_size": [original_image.width, original_image.height],
            "inference_image_size": [image.width, image.height],
            "resize_long_edge": resize_long_edge,
            "resize_scale": round(resize_scale, 4),
            "precision": self.precision,
            "inference_time_seconds": round(elapsed, 4),
            "num_boxes": len(formatted_boxes),
            "boxes": formatted_boxes,
            "raw_answer": answers[0],
            "stats": self._get_last_hybrid_stats(),
        }
        return self.last_result if return_details else formatted_boxes

    @staticmethod
    def compute_iou(box_a: dict[str, Any], box_b: dict[str, Any]) -> float:
        return round(box_iou(box_a, box_b), 4)

    @staticmethod
    def _hungarian_min_cost(cost_matrix: list[list[float]]) -> list[int]:
        size = len(cost_matrix)
        u = [0.0] * (size + 1)
        v = [0.0] * (size + 1)
        p = [0] * (size + 1)
        way = [0] * (size + 1)

        for row in range(1, size + 1):
            p[0] = row
            column = 0
            minv = [float("inf")] * (size + 1)
            used = [False] * (size + 1)
            while True:
                used[column] = True
                current_row = p[column]
                delta = float("inf")
                next_column = 0
                for candidate_column in range(1, size + 1):
                    if used[candidate_column]:
                        continue
                    cur = cost_matrix[current_row - 1][candidate_column - 1] - u[current_row] - v[candidate_column]
                    if cur < minv[candidate_column]:
                        minv[candidate_column] = cur
                        way[candidate_column] = column
                    if minv[candidate_column] < delta:
                        delta = minv[candidate_column]
                        next_column = candidate_column
                for candidate_column in range(size + 1):
                    if used[candidate_column]:
                        u[p[candidate_column]] += delta
                        v[candidate_column] -= delta
                    else:
                        minv[candidate_column] -= delta
                column = next_column
                if p[column] == 0:
                    break
            while True:
                next_way = way[column]
                p[column] = p[next_way]
                column = next_way
                if column == 0:
                    break

        assignment = [-1] * size
        for column in range(1, size + 1):
            if p[column] != 0:
                assignment[p[column] - 1] = column - 1
        return assignment

    @staticmethod
    def _best_one_to_one_matches(weight_matrix: list[list[float]]) -> list[tuple[int, int, float]]:
        num_rows = len(weight_matrix)
        num_cols = len(weight_matrix[0]) if weight_matrix else 0
        if num_rows == 0 or num_cols == 0:
            return []

        size = max(num_rows, num_cols)
        max_weight = max((max(row) for row in weight_matrix), default=0.0)
        padded_cost = [[max_weight for _ in range(size)] for _ in range(size)]
        for row in range(num_rows):
            for col in range(num_cols):
                padded_cost[row][col] = max_weight - weight_matrix[row][col]

        assignment = LocateAnythingDetector._hungarian_min_cost(padded_cost)
        matches: list[tuple[int, int, float]] = []
        for row, col in enumerate(assignment):
            if row < num_rows and 0 <= col < num_cols:
                weight = weight_matrix[row][col]
                if weight > 0:
                    matches.append((row, col, weight))
        return matches

    @staticmethod
    def compare_box_groups(
        boxes_a: Sequence[dict[str, Any]],
        boxes_b: Sequence[dict[str, Any]],
        *,
        match_by_label: bool = True,
    ) -> dict[str, Any]:
        iou_matrix: list[list[float]] = []
        for box_a in boxes_a:
            row: list[float] = []
            for box_b in boxes_b:
                if match_by_label and box_a.get("label") != box_b.get("label"):
                    row.append(0.0)
                else:
                    row.append(round(box_iou(box_a, box_b), 4))
            iou_matrix.append(row)

        grouped_a: dict[str, list[int]] = defaultdict(list)
        grouped_b: dict[str, list[int]] = defaultdict(list)
        if match_by_label:
            for index_a, box_a in enumerate(boxes_a):
                grouped_a[str(box_a.get("label"))].append(index_a)
            for index_b, box_b in enumerate(boxes_b):
                grouped_b[str(box_b.get("label"))].append(index_b)
            labels = sorted(set(grouped_a) | set(grouped_b))
        else:
            labels = ["__all__"]
            grouped_a["__all__"] = list(range(len(boxes_a)))
            grouped_b["__all__"] = list(range(len(boxes_b)))

        match_map: dict[int, tuple[int, float]] = {}
        matched_b_indices: set[int] = set()
        for label in labels:
            indices_a = grouped_a.get(label, [])
            indices_b = grouped_b.get(label, [])
            if not indices_a or not indices_b:
                continue
            label_matrix = [[iou_matrix[index_a][index_b] for index_b in indices_b] for index_a in indices_a]
            for local_a, local_b, iou in LocateAnythingDetector._best_one_to_one_matches(label_matrix):
                global_a = indices_a[local_a]
                global_b = indices_b[local_b]
                match_map[global_a] = (global_b, iou)
                matched_b_indices.add(global_b)

        best_matches: list[dict[str, Any]] = []
        best_ious: list[float] = []
        for index_a, box_a in enumerate(boxes_a):
            if index_a in match_map:
                box_b_index, best_iou = match_map[index_a]
                best_matches.append(
                    {
                        "box_a_index": index_a,
                        "box_b_index": box_b_index,
                        "label_a": box_a.get("label"),
                        "label_b": boxes_b[box_b_index].get("label"),
                        "iou": round(best_iou, 4),
                    }
                )
                best_ious.append(best_iou)
            else:
                best_matches.append(
                    {
                        "box_a_index": index_a,
                        "box_b_index": None,
                        "label_a": box_a.get("label"),
                        "label_b": None,
                        "iou": 0.0,
                    }
                )
                best_ious.append(0.0)

        return {
            "num_boxes_a": len(boxes_a),
            "num_boxes_b": len(boxes_b),
            "match_by_label": match_by_label,
            "iou_matrix": iou_matrix,
            "best_matches": best_matches,
            "matched_pairs": sum(1 for match in best_matches if match["box_b_index"] is not None),
            "unmatched_box_a_indices": [match["box_a_index"] for match in best_matches if match["box_b_index"] is None],
            "unmatched_box_b_indices": sorted(set(range(len(boxes_b))) - matched_b_indices),
            "mean_best_iou": round(sum(best_ious) / len(best_ious), 4) if best_ious else 0.0,
            "max_best_iou": round(max(best_ious), 4) if best_ious else 0.0,
            "min_best_iou": round(min(best_ious), 4) if best_ious else 0.0,
        }
EOF

📘 匈牙利算法小科普：假设你要给 N 个工人分配 N 个任务，每个工人完成每个任务有不同的成本，匈牙利算法可以找到总成本最小的指派方案。这里我们把“成本”换成“负 IoU”，就变成了让总 IoU 最大化。

---

4. 为 API 增加一个 Demo

现在用这个 API 写一个直观的对比 Demo。它会：

• 对两张图片分别检测
• 调用 compare_box_groups 得到对比结果
• 找到 IoU 最小的那个匹配对（即两张图中“移动最大”或“位置差异最大”的物体）
• 画框时，将这对框用黄色高亮，其余用红色
• 将两张标注图上下拼接，保存为一张对比图

cat > demo_locateanything_iou.py << 'EOF'
#!/usr/bin/env python3
import argparse
import json
from pathlib import Path

from PIL import Image, ImageDraw

from locateanything_detector import LocateAnythingDetector


NORMAL_COLOR = "red"
HIGHLIGHT_COLOR = "yellow"


def draw_boxes_with_highlight(
    image_path: str,
    boxes: list[dict],
    highlight_index: int | None,
    title: str,
) -> Image.Image:
    image = Image.open(image_path).convert("RGB")
    drawer = ImageDraw.Draw(image)
    drawer.text((12, 12), title, fill="white")
    for index, box in enumerate(boxes):
        color = HIGHLIGHT_COLOR if highlight_index == index else NORMAL_COLOR
        xy = [(box["x1"], box["y1"]), (box["x2"], box["y2"])]
        label = f'{box["index"]}:{box["label"]}'
        drawer.rectangle(xy, outline=color, width=4)
        drawer.text((box["x1"] + 4, max(0, box["y1"] - 16)), label, fill=color)
    return image


def stack_images_vertically(top: Image.Image, bottom: Image.Image) -> Image.Image:
    output_width = max(top.width, bottom.width)
    output_height = top.height + bottom.height
    canvas = Image.new("RGB", (output_width, output_height), color="black")
    canvas.paste(top, (0, 0))
    canvas.paste(bottom, (0, top.height))
    return canvas


def main() -> None:
    parser = argparse.ArgumentParser(description="Demo for comparing detection IoU across two images.")
    parser.add_argument("--image-a", default="../20260708_103819_507912.jpg", help="First image path.")
    parser.add_argument("--image-b", default="../20260708_103819_402267.jpg", help="Second image path.")
    parser.add_argument("--out", default="iou_comparison.jpg", help="Output path for vertically stacked visualization.")
    parser.add_argument(
        "--categories",
        default="",
        help="Comma-separated or </c>-separated categories. Empty uses detector defaults.",
    )
    parser.add_argument("--query", default="", help="Optional phrase grounding query. Cannot be used with categories.")
    parser.add_argument("--resize-long-edge", type=int, default=0, help="Optional long-edge resize before inference.")
    parser.add_argument("--attn", default="auto", choices=["auto", "sdpa", "eager", "magi", "la_flash"])
    parser.add_argument(
        "--scheduler",
        default="pipeline",
        choices=["eager", "hold_ar", "ar_first", "pipeline", "adaptive"],
    )
    parser.add_argument("--precision", default="bf16", choices=["bf16", "fp8-text", "fp8-full"])
    parser.add_argument("--match-by-label", action="store_true", default=True, help="Only compare boxes with same label.")
    args = parser.parse_args()

    detector = LocateAnythingDetector(
        attn=args.attn,
        scheduler=args.scheduler,
        precision=args.precision,
    )

    detect_kwargs = {
        "query": args.query,
        "categories": args.categories or None,
        "resize_long_edge": args.resize_long_edge,
        "return_details": True,
    }
    result_a = detector.detect(args.image_a, **detect_kwargs)
    result_b = detector.detect(args.image_b, **detect_kwargs)
    comparison = detector.compare_box_groups(
        result_a["boxes"],
        result_b["boxes"],
        match_by_label=args.match_by_label,
    )

    matched_pairs = [match for match in comparison["best_matches"] if match["box_b_index"] is not None]
    min_match = min(matched_pairs, key=lambda item: item["iou"]) if matched_pairs else None
    highlight_a_index = min_match["box_a_index"] if min_match is not None else None
    highlight_b_index = min_match["box_b_index"] if min_match is not None else None

    title_a = f"A: {Path(args.image_a).name}"
    title_b = f"B: {Path(args.image_b).name}"
    if min_match is not None:
        title_a += f" | min IoU box: {min_match['iou']}"
        title_b += f" | min IoU box: {min_match['iou']}"

    annotated_a = draw_boxes_with_highlight(args.image_a, result_a["boxes"], highlight_a_index, title_a)
    annotated_b = draw_boxes_with_highlight(args.image_b, result_b["boxes"], highlight_b_index, title_b)
    stacked = stack_images_vertically(annotated_a, annotated_b)
    stacked.save(args.out, quality=95)

    payload = {
        "image_a": {
            "path": args.image_a,
            "selected_attn": result_a["selected_attn"],
            "inference_time_seconds": result_a["inference_time_seconds"],
            "num_boxes": result_a["num_boxes"],
            "boxes": result_a["boxes"],
        },
        "image_b": {
            "path": args.image_b,
            "selected_attn": result_b["selected_attn"],
            "inference_time_seconds": result_b["inference_time_seconds"],
            "num_boxes": result_b["num_boxes"],
            "boxes": result_b["boxes"],
        },
        "comparison": comparison,
        "visualization": {
            "output_path": args.out,
            "normal_color": NORMAL_COLOR,
            "highlight_color": HIGHLIGHT_COLOR,
            "highlighted_match": min_match,
        },
    }
    print(json.dumps(payload, ensure_ascii=False, indent=2))


if __name__ == "__main__":
    main()

EOF

---

5. 运行 Demo，看看实际效果

我们用两张拍摄角度略有不同的街景照片做测试，只关注 person,car,truck,bus,bicycle,motorcycle 这几类。

python demo_locateanything_iou.py \
	--image-a ../20260708_103819_507912.jpg \
	--image-b ../20260708_103819_402267.jpg \
	--categories "person,car,truck,bus,bicycle,motorcycle" \
    --out person_iou_demo.jpg

程序输出的 JSON 包含详细的对比统计：

    "matched_pairs": 14,
    "unmatched_box_a_indices": [],
    "unmatched_box_b_indices": [],
    "mean_best_iou": 0.9696,
    "max_best_iou": 1.0,
    "min_best_iou": 0.8891
  },
  "visualization": {
    "output_path": "person_iou_demo.jpg",
    "normal_color": "red",
    "highlight_color": "yellow",
    "highlighted_match": {
      "box_a_index": 9,
      "box_b_index": 9,
      "label_a": "car",
      "label_b": "car",
      "iou": 0.8891
    }
  }

• 平均 IoU 高达 0.97，说明两张图里车辆的位置和大小非常一致。
• 最小 IoU 为 0.89，仍然是较高值，没有出现“一个框完全找不到对应”的情况（unmatched 列表为空）。
• 高亮的那对框是 car，虽然框没有完全重合，但依然非常接近，很可能是拍摄视角微小变化导致的。

同时生成的 person_iou_demo.jpg 会将两张图上下放置，并用黄色标出这对“最不重合”的汽车框，一目了然。

---

三、总结与下一步

通过这次实践，我们实现了一个基于视觉大模型的目标检测 IoU 对比管线：

1. 用 LocateAnything-3B 零样本获取物体框，无需训练，灵活性极高。
2. 利用 匈牙利算法 自动找到两张图之间最优的框匹配，计算出多种 IoU 统计量。
3. 通过简单的高亮可视化，快速定位变化区域。

这种方案非常适合以下场景：

• 监控变化检测：发现两帧间消失或新出现的物体。
• 无人机/卫星图像对比：评估同一地区不同时相的建筑物变化。
• 自动化测试：验证 UI 截图中元素布局是否发生变化。

当然，该方法也有局限：

• 语义理解依赖模型：如果模型漏检或误检，IoU 比较就会失真。可以考虑提升输入图片分辨率或微调模型。
• IoU 不感知物体朝向或外观：两辆车即使框 IoU=1，也可能颜色完全不同。需要结合特征相似度才能全面判断。
• 匹配算法的改进：目前是最优一对一匹配，但现实可能存在一对多、多对多的对应关系，可尝试更复杂的二分图匹配或 Soft IoU。