Table of Contents
- cs.CL [Total: 17]
- cs.CV [Total: 47]
- cs.AI [Total: 4]
- cs.CR [Total: 1]
- cs.LG [Total: 5]
- cs.RO [Total: 7]
- cs.SD [Total: 1]
- cs.SE [Total: 3]
- eess.IV [Total: 3]
- physics.flu-dyn [Total: 1]
- q-bio.QM [Total: 1]
cs.CL [Back]
[1] Li$^{2}$SAO: Lifting Light Singl-Rollout Asynchronous RL Optimization via Group-Relative Actor-Critic Bootstrapping cs.CL | cs.LGPDF
Zichen Bai, Yanli Wang, Yang Liu, Junhao Shi, Xueqin Liu
TL;DR: 论文研究异步强化学习用于LLM后训练时,SAO依赖闭源预训练value model的问题,并提出轻量方案Li^2SAO。作者先复现无预训练value model的LiSAO,发现GRPO在单轮推理任务更好,而LiSAO在多轮agentic任务更好,但LiSAO因critic冷启动在早期训练明显落后。Li^2SAO在critic冷启动阶段用group-relative advantage训练actor、用group-centered return训练critic,之后切换到single-rollout GAE,从而兼顾早期稳定性和后期性能上限。
Details
Motivation: 异步RL能提升LLM后训练效率,但SAO依赖闭源的、经scaling value pretraining得到的value model,轻量复现的LiSAO因value model/critic未训练好而在早期训练落后于GRPO。论文旨在消除该依赖并解决critic冷启动问题。
Result: 实验覆盖单轮推理和多轮agentic任务;摘要未给出具体数值或benchmark名称,仅定性表明Li^2SAO在早期训练中赶上或超过GRPO,并进一步提升后期性能上限。
Insight: 创新点是group-relative actor-critic bootstrapping:critic冷启动时用group-relative advantage训练actor、group-centered return训练critic,待critic可用后切换到single-rollout GAE。客观看,其价值在于结合GRPO的group相对优势与SAO的单rollout异步优化,缓解异步RL中value model冷启动和闭源依赖问题。
Abstract: Asynchronous reinforcement learning (RL) is becoming a mainstream paradigm for post-training large language models (LLMs), as it improves training efficiency by updating the model as rollouts arrive. Recent work argues that group-wise sampling in GRPO does not naturally fit asynchronous RL training because rollout lengths follow a long-tailed distribution, and proposes Single-rollout Asynchronous Optimization (SAO) to improve stability and off-policy behavior, especially on agentic tasks. However, SAO relies on a closed-source value model obtained from scaling value pretraining. In this paper, we analyze GRPO and LiSAO, our lightweight reproduction of SAO without the pretrained value model, on single-turn reasoning and multi-turn agentic tasks. We find that GRPO outperforms LiSAO on reasoning tasks, while the reverse holds on agentic tasks. Since the value model of LiSAO is not yet well trained , LiSAO lags behind GRPO substantially in early training steps. To address this issue, we propose Li$^{2}$SAO, which Lifts Light Single-rollout Asynchronous Optimization with group-relative actor-critic bootstrapping: while the critic is cold, a group-relative advantage trains the actor and a group-centered return trains the critic, after which training switches to single-rollout GAE. Experiments show that Li$^{2}$SAO catches up with or surpasses GRPO in early training and further raises the later performance ceiling. Code is available at https://github.com/jd-cc-coder/Li2SAO.
[2] Mage-VL-Cowork: A Full Training Recipe to Learn Hybrid Computer Use From Evolved Harnesses cs.CL | cs.LGPDF
Xiaoyi Zhang, Haoqing Wang, Xinjie Zhang, Xiang Ming, Xing Wang
TL;DR: 论文提出 Mage-VL-Cowork,一个 4B 混合计算机使用代理,能统一进行视觉 GUI 交互与代码/终端执行。该方法基于多模态对齐检查点,通过覆盖 midtraining、SFT 和 RL 的五阶段渐进式课程端到端训练,并配套 Mage-CUA-Forge 大规模合成桌面与网页任务数据。最终在 OSWorld-Verified 上达到 66.0%,在同参数规模下取得 SOTA。
Details
Motivation: 现有计算机使用代理往往难以在单一策略中无缝结合视觉 GUI 操作与代码、终端执行,且缺少可扩展的数据构造和完整训练配方。论文旨在降低任务构造成本,并提供从通用多模态模型到混合 CUA 的系统化训练流程。
Result: Mage-CUA-Forge 合成了 293,806 个桌面和网页任务,覆盖 6,915 项技能,并将重复性单任务开销降低 82.6%。Mage-VL-Cowork-4B 在 OSWorld-Verified 上达到 66.0%,在其参数规模下达到 SOTA,同时推理成本更高效。
Insight: 创新点在于用可复用的技能条件生成工作流和应用专用准备器规模化构造任务数据,并通过演化教师 harness、实例化专用 CUA harness 获取高质量轨迹与分阶段监督。整体上,五阶段渐进式训练课程将 midtraining、SFT 和 RL 串联,是可借鉴的混合代理训练配方。
Abstract: Effective computer-use agents (CUAs) should seamlessly combine visual GUI interaction with code and terminal execution within a single policy. We present Mage-VL-Cowork, a 4B hybrid CUA trained from a general multimodally aligned checkpoint through an end-to-end, progressive five-stage curriculum across midtraining, SFT, and RL. To power this curriculum at scale, Mage-CUA-Forge replaces per-task agent construction with reusable, skill-conditioned generation workflows and application-specific preparers, synthesizing 293,806 desktop and web tasks across 6,915 skills while reducing recurring per-task expenditure by 82.6%. Rather than directly distilling a default teacher, we evolve the teacher harness and instantiate specialized CUA harnesses to yield high-quality trajectories alongside stage-specific supervision. Mage-VL-Cowork-4B finally climbs to 66.0% on OSWorld-Verified through different training stages with more efficient reasoning cost, achieving state-of-the-art performance at its parameter size and pushing forward the frontier of on-device intelligence.
[3] CoFiBreak: A Coarse-to-Fine Diversity Reward for Effective Multi-Turn Jailbreak cs.CLPDF
Jialin Song, Xiaodong Liu, Weiwei Yang, Mingqian Feng, Chengquan Guo
TL;DR: 论文提出 CoFiBreak,一种用于 GRPO 训练的由粗到细多样性奖励机制,用于自动化多轮越狱红队测试。粗粒度奖励鼓励攻击者使用 GRPO 组内欠代表的攻击策略,细粒度奖励在移除有害意图信息后惩罚重复的人设、框架和输出格式,从而同时提升策略层和实现层的攻击多样性,并保持攻击成功率与推理效率。
Details
Motivation: 仅以攻击成功为唯一奖励的强化学习会让攻击者 LLM 收敛到狭窄的提示集合,导致攻击多样性崩塌,使安全评估反复测试近重复的攻击模式而非真实有害意图;现有方法往往需要以攻击成功率或推理成本为代价换取多样性。
Result: 以 Llama3.1-8B-Instruct 为受害者训练,在 HarmBench 上发现的独特成功细粒度策略数是最强基线的 1.6-2.0 倍;在 AdvBench 上达到 98.8% ASR@10,多样性分数为所有对比方法中最高,是最强攻击基线的 2.1 倍;迁移到未见过的 Claude-Opus-4.6 仍达 71.0% ASR@10。
Insight: 创新点在于将多样性奖励拆分为粗(策略选择)与细(策略实现)两级,并仅对已超过有效性阈值的攻击施加多样性奖励,避免以无效攻击换取多样性;同时奖励只在训练阶段计算,推理时无额外开销。
Abstract: Automated red teaming trains an attacker large language model (LLM) to elicit harmful responses from a victim LLM, but reinforcement learning can collapse attackers onto a narrow set of prompts when attack success is the only reward. Such a collapse of attack diversity can bias safety evaluation toward repeatedly testing near-duplicate attack patterns rather than the underlying harmful intents. Existing methods often trade off attack success or inference cost for greater content or strategy diversity. We propose CoFiBreak, a coarse-to-fine diversity reward for Group Relative Policy Optimization (GRPO) that improves attack diversity while preserving strong attack effectiveness and low inference cost. The coarse reward broadens which strategies the attacker uses by encouraging strategies underrepresented within each GRPO group, while the fine reward broadens how those strategies are realized by discouraging reuse of the same personas, framings, and output formats after harmful-intent information is removed from prompt embeddings. Diversity rewards apply only to attacks that already exceed an effectiveness threshold, so diversity cannot be gained through ineffective attacks. Since both rewards are computed only during training, CoFiBreak also retains inference efficiency. Trained with Llama3.1-8B-Instruct as the victim, CoFiBreak discovers 1.6-2.0x as many distinct successful fine-grained strategies as the strongest baseline across frontier victims on HarmBench. It reaches 98.8% ASR@10 (attack success rate with 10 generations per intent) on AdvBench with the highest diversity score among all compared methods, which is 2.1x that of the strongest-attacking baseline. The resulting attacks transfer to unseen frontier LLMs, reaching 71.0% ASR@10 on AdvBench with the unseen victim Claude-Opus-4.6. The code will be released.
[4] EDU-Jev Training System-One Decision Models for Education cs.CLPDF
Jiazheng Li, Yulan He
TL;DR: 论文提出EDU-Jev系列教育决策模型(0.8B/2B/4B),面向从固定选项中选择答案的教育场景,如评分、判定答案是否符合标准、定位解题首个错误步骤。模型不生成文本,而是单次前向直接输出预定义选项上的概率分布,并用指针头将预测限制在允许选项中。
Details
Motivation: 生成式语言模型用于教育自动决策时,输出需要解析、推理成本高,且用文字表达的置信度不一定反映答对概率;因此需要更直接、可校准且高效固定选项决策模型。
Result: 在12个教育基准上,EDU-Jev 4B平均准确率67.6%,比五个商业决策API中最强模型高9.4个百分点,比最佳开源4B决策模型高18.7个百分点;在4个跨域教育集上匹配最强API,但通用决策与推理任务仍落后。2B模型在单张H100上约每秒150次决策。
Insight: 创新点在于将教育决策建模为单次“系统一”式固定选项概率预测:用指针头约束输出空间,不生成文本,并用软交叉熵单阶段监督训练以支持校准。客观看,其可借鉴之处是面向受限选项的直接判别式建模,兼顾推理效率、输出可解析性和概率校准。
Abstract: Many automated decisions in education involve choosing from a fixed set of options: assigning a rubric score, checking whether an answer meets a marking criterion, or identifying the first wrong step in a solution. Generative language models are increasingly used to make these decisions, but their responses require parsing, longer outputs increase inference costs, and confidence expressed in words does not necessarily reflect the probability of being correct. We present EDU-Jev, a family of single-pass (“System One”) decision models at 0.8B, 2B and 4B parameters. Given an assessment context and a question with predefined answer options, the model returns a probability distribution over those options without generating text. A pointer head scores a marker for each option, ensuring the predictions are restricted to the allowed choices. Training uses a single supervised stage with soft cross-entropy against the available target label distributions; calibration is assessed empirically. Across twelve education benchmarks, EDU-Jev 4B achieves an average accuracy of 67.6%, exceeding the strongest of five commercial decision APIs by 9.4 percentage points and the best open 4B decision model by 18.7 points. On four cross-domain education sets, it matches the strongest API, which remains ahead on general decision-making and reasoning tasks. The 2B model processes about 150 decisions per second on a single H100 GPU.
[5] S2TDR: Structured Multimodal Reconstruction for Sticker-Aware Dialogue Understanding cs.CLPDF
Yuanchen Shi, Fang Kong
TL;DR: 论文提出S2TDR,一个面向贴纸对话理解的结构化多模态重构框架。它将含贴纸的对话轮次转换为上下文合适的文本话语,流程包括视觉线索提取、情感与意图推断、上下文感知对话重构,并将重构后的对话交给现有文本对话模型使用。实验表明该方法在情感分析、意图识别和对话摘要等任务上通常优于直接融合贴纸的方法。
Details
Motivation: 社交媒体对话中的贴纸常表达非字面、依赖语境的情感、立场和意图,通用视觉融合或独立图像描述难以准确处理其语义。论文希望把贴纸语义以结构化方式融入对话理解,并兼容现有文本对话模型。
Result: 在情感分析、意图识别和对话摘要任务上,S2TDR普遍提升了多个下游模型,优于直接贴纸集成方法;组件消融、OCR为空贴纸和重构质量分析进一步表明结构化重构有效且可解释。摘要未给出具体数值或SOTA声明。
Insight: 创新点在于不直接做视觉融合,而是将贴纸解释重构为上下文合适的文本话语,并分解为视觉线索提取、情感意图推断和上下文重构三步;OCR仅作辅助线索,使方法可解释、可接入现有文本对话模型。客观看,其可借鉴之处是“结构化语义重构”作为多模态对话融合的中间表示。
Abstract: Stickers are widely used in social media dialogues to express affect, stance, and intent beyond literal text. Their meanings are often non-literal and context-dependent, making them difficult to handle with generic visual fusion or standalone captioning. We propose Sticker-to-Text Dialogue Reconstruction (S2TDR), a structured multimodal reconstruction framework that converts sticker turns into contextually appropriate textual utterances. S2TDR decomposes sticker interpretation into visual cue extraction, affective-intent inference, and context-aware dialogue reconstruction, using OCR text only as an auxiliary cue when available. The reconstructed dialogue can then be consumed by existing text-based dialogue models. Experiments on sentiment analysis, intent recognition, and dialogue summarization show that S2TDR generally improves multiple downstream models over direct sticker integration methods. Further analyses on component ablations, OCR-empty stickers, and reconstruction quality indicate that structured reconstruction provides an interpretable and effective way to integrate sticker semantics into dialogue understanding.
[6] TraceSpec: Understanding and Improving Speculative Decoding for LLM Agents cs.CLPDF
Jihao Zhao, Shuaishuai Zu, Pengfei Li, Biao Qin, Guanghuan Fang
TL;DR: 论文研究LLM智能体多轮执行场景下的投机解码,指出聚合接受率指标会掩盖不同轨迹与阶段中的差异,难以指导草稿监督。作者提出TraceSpec框架,将轨迹评估与自适应草稿训练结合,构建TraceSpec-Traj数据集,并利用状态条件的前缀感知接受差距动态分配预测预算,以提升智能体场景下的投机解码效率。
Details
Motivation: 现有投机解码评估多依赖聚合指标,而LLM智能体包含推理、行动和工具反馈等异构决策,聚合接受率无法说明应在哪些轨迹状态下加强草稿监督。论文希望理解上下文长度、工作负载和执行阶段对接受率的影响,并据此改进草稿训练。
Result: 在三种目标模型规模和两种温度设置下,TraceSpec较DSpark基座将重放吞吐提升12.5%至22.6%;Qwen3-4B消融显示,自适应分配相较固定全轮监督将接受长度提升3.3%至10.0%。
Insight: 主要创新在于将智能体轨迹评估与自适应草稿训练连接,构建覆盖浏览和软件工程的训练/测试状态语料,并发现聚合接受率上升时匹配的后期浏览动作内接受率可能下降。方法上以状态条件估计前缀感知接受差距来分配固定预测预算,体现了从轨迹条件而非全局统计指导投机解码训练的思路。
Abstract: LLM agents execute tasks through repeated model calls over evolving histories of reasoning, actions, and tool feedback. Aggregate speculative-decoding metrics merge these distinct decisions, providing limited guidance on where draft supervision is needed. We present TraceSpec, a framework connecting trajectory evaluation with adaptive draft training. It combines dynamic execution with fixed-prefix replay and introduces TraceSpec-Traj, a corpus of 10,000 training states and 2,500 test states spanning browsing and software engineering. Analysis of five speculative baselines reveals that aggregate acceptance can rise with context length while declining within matched late browsing actions. Workload and phase further distinguish these trends, motivating supervision guided by trajectory conditions. TraceSpec trains on intermediate assistant turns and allocates a fixed prediction budget using state-conditioned estimates of prefix-aware acceptance gaps. Across three target sizes and two temperatures, it improves replay throughput by 12.5% to 22.6% over its DSpark base. Qwen3-4B ablations show that adaptive allocation increases accepted length by 3.3% to 10.0% over fixed full-turn supervision, with additional comparisons substantiating the roles of state conditioning and consecutive-token agreement. The resulting evaluation resources, trajectory findings, and training strategy connect the analysis of agent execution to practical draft improvement.
[7] Beyond Response Scoring: An AI-aware Cognitive Supervision Framework for Automated Mathematical Short Answer Grading cs.CLPDF
Shengran Guo, Min Chi, Feifei Qiao, Varad Patwardhan
TL;DR: 论文提出AICG,一个面向数学简答题自动评分的双分支多任务框架。第一分支用RoBERTa结合回归、序数回归和分类头进行单题评分;第二分支借助专家Q矩阵将回答关联到技能,并联合预测G-DINA衍生的技能掌握度和学习增益,通过共享编码器为评分提供认知监督。框架还引入Gen-AI检测器估计AI辅助概率并转化为可靠性权重,以减弱不可靠认知证据的影响。
Details
Motivation: 现有自动简答题评分方法多只输出分数,难以理解和推理含符号、公式的复杂数学答案,也难以区分同分但回答质量和技能状态不同的情况;同时忽略了生成式AI使用可能带来的评分上偏和对学生知识掌握评估的干扰。
Result: 在49,402份教师评分回答、覆盖19个学期的数据上,相较RoBERTa回归,AICG将MAE降低50.3%,QWK从0.798提升到0.927。消融表明Q矩阵引导的认知监督是主要提升来源;AI-aware加权在无加权认知模型基础上进一步降低整体MAE 4.2%,并在最近四个后Gen-AI学期持续提升MAE和QWK。
Insight: 摘要宣称的创新点包括:超越分数标签,用技能掌握和学习增益提供认知监督;引入Gen-AI检测与可靠性加权以缓解AI辅助带来的评估偏差。客观看,其创新主要在于将心理测量中的Q矩阵/G-DINA认知诊断与Transformer评分多任务联合建模,并显式建模生成式AI使用对自动评分可靠性的影响。
Abstract: Transformer-based language models have achieved remarkable success in automated short answer grading (ASAG). However, current methods in ASAG struggle to comprehend, reason about and evaluate complex mathematical answers involving symbols and formulas. These score-only methods fail to generalize across diverse responses and skill states associated with identical scores. Moreover, they ignore the potential upward bias introduced by generative Artificial Intelligence (AI) usage in assessing students’ performance and knowledge mastery. To address these limitations, we propose AICG (AI-aware Cognitive-supervised Grading), a two-branch multi-task framework. The first branch combines RoBERTa-based regression, ordinal, and classification heads for individual-answer grading. The second branch uses an expert-designed Q-matrix to associate responses with assessed skills and jointly predicts G-DINA-derived skill mastery and learning gain. Through the shared encoder, these skill-level targets provide cognitive supervision for answer grading beyond score labels. AICG further uses a Gen-AI detector, trained through supervised contrastive pretraining followed by linear classification, to estimate assistance probabilities and convert them into reliability weights, attenuating unreliable cognitive evidence during learning assessment. We conduct experiments on 49,402 instructor-graded responses spanning 19 academic semesters. Relative to RoBERTa regression, AICG reduces mean absolute error (MAE) by 50.3% and improves quadratic weighted kappa (QWK) from 0.798 to 0.927. Ablations identify Q-matrix-guided cognitive supervision as the primary source of improvement, while AI-aware weighting further reduces overall MAE by 4.2% over the unweighted cognitive model and consistently improves MAE and QWK across the four most recent post-Gen-AI semesters.
[8] BlindToChange: AI Agents Can Find Changes When Asked But Do They Look When Needed? cs.CL | cs.AIPDF
Anirudh Phukan, Tanya Goyal, Danish Pruthi
TL;DR: 论文提出 BlindToChange 基准,评估 AI agent 在用户请求未显式提及近期变化时,是否会主动搜索并发现影响答案的现实事件。基准由 Wikidata 自动构建且可定期刷新,通过“原本明显候选因近期事件不再适用”的推荐请求来测试。实验发现 agent 被直接询问时能找到变化,但在需要主动核实时存在显著性能差距。
Details
Motivation: 现有 BrowseComp 等 agentic search benchmark 明确告诉 agent 要找什么,无法暴露 agent 未意识到需验证自身假设的问题;真实用户请求常只给最终需求,需 agent 自行发现相关变化及其后果。
Result: 在 BlindToChange 上,近期事件未被显式提及时,系统性能相比直接询问相关事件最多下降 27.5 分,即使高推理努力仍存在大差距;超过一半失败源于 agent 未直接搜索验证相关事实是否变化。
Insight: 创新在于把“主动发现并纳入未明说的近期变化”作为新评测轴,并用 Wikidata 自动抽取事件实现可刷新 benchmark;客观看,其价值在于揭示 agent 搜索行为中的假设验证缺口,而不仅是答案正确率。
Abstract: Given that AI agents can search the web broadly and deeply in real time, it is tempting to conclude that their outputs account for recent world events. We show that this conclusion does not hold in practice. While agents can search and easily find recent changes, the correctness of their responses depends on whether they search and what they search for. Popular agentic search benchmarks like BrowseComp cannot expose failures wherein agents do not recognize their own assumptions that should be verified, since these benchmarks explicitly state what needs to be found. In contrast, many user requests state only the final information need (e.g., an itinerary for a trip), leaving the agent to discover relevant changes (e.g., the temporary closure of a museum) and account for their consequences. To highlight such gaps, we introduce BlindToChange, a benchmark that measures whether agents proactively discover recent events that affect their responses. Our benchmark comprises requests for recommendations where an obvious candidate is no longer applicable due to a recent real-world event. These events are drawn automatically from Wikidata, allowing BlindToChange to be periodically refreshed. Strikingly, we find a big gap – as high as 27.5 points, even at high reasoning effort – in systems’ performance, when the recent event is not explicitly mentioned compared to when we ask about it directly. Analyzing search trajectories, we learn that more than half of the failures can be attributed to the fact that agents do not directly search to verify whether a relevant fact has changed. Our benchmark highlights, and tracks, a new axis of progress, wherein systems account for recent information that is not explicitly mentioned in the user request.
[9] KEEPER: Learning Control-Sufficient Textual Memory for Long-Horizon LLM Agents cs.CL | cs.AIPDF
Jinkyeong Choi, Chaebin Jeong, Junsung Jeon, Donghyeon Park
TL;DR: 论文关注长视野LLM智能体递归文本记忆中的错误累积问题,提出“自一致性偏差”概念,即传入记忆中已有错误声明会持续保留。作者在共享递归接口下比较四种辅助目标在信息内容与更新敏感性维度上的表现,分析不同监督目标与智能体性能的关系。结果表明奖励相关预测在离线强化学习前后均取得最高任务性能,并优于联合监督。
Details
Motivation: 递归文本记忆可能把早期错误带入后续更新,而仅看任务表现无法证明记忆可靠;论文旨在识别并缓解错误声明在记忆中的持续保留问题。
Result: 在评估条件中,奖励相关预测在离线强化学习前后任务性能最高,并优于联合监督;对一条基线轨迹的声明级分析显示,即使有纠正性观察,错误仍会持续存在。摘要未提及具体benchmark名称。
Insight: 创新点在于提出自一致性偏差来刻画记忆错误持久性,并在统一递归接口下系统比较不同辅助监督目标;同时主张除任务结果外,还应评估错误持久性和有效信息保留。客观看,其价值在于将记忆可靠性拆解为可分析维度,并强调监督目标选择对长视野智能体记忆控制的影响。
Abstract: Recursive textual memory can carry earlier errors into later updates, yet task performance alone does not establish memory reliability. We define self-consistency bias as the persistence of incorrect claims already present in the incoming memory. We present KEEPER, a comparative analysis of four auxiliary objectives along information-content and update-sensitivity axes within a shared recursive interface, examining how supervision targets relate to agent performance. Among evaluated conditions, reward-related prediction achieve the highest task performance before and after offline reinforcement learning, outperforming joint supervision. A claim-level analysis of one baseline trajectory documents error persistence despite corrective observations. Together, these analyses motivate evaluating error persistence and valid-information retention alongside task outcomes.
[10] GraphNIAH: A Benchmark for Long-Context Reasoning over Real-World Graphs cs.CLPDF
Minh-Nam Tran, Abulhair Saparov
TL;DR: 论文提出 GraphNIAH,一个基于真实论文引用关系构建的动态长上下文图推理基准,用于评估 LLM 在大型图上下文中的多跳推理能力。该基准通过全自动流水线采样子图并生成带可验证答案的长上下文 QA。作者评测了 GPT-5.6、Gemini-3、Qwen3.5 家族的 8 个模型,发现随着推理路径变长,模型准确率显著下降,在较深路径上接近随机猜测。
Details
Motivation: 现有 NIAH 基准存在依赖合成数据、复用旧数据集带来污染风险、需要人工标注,以及将长上下文多跳推理过度简化等问题,无法反映真实场景中需要在多条证明路径上进行规划的长上下文推理。
Result: 在 8 个 LLM 上评测发现:问题深度为 16 和 32 时,GPT-5.6 和 Qwen3.5 在 YES/NO 问题上不优于随机猜测,且多倾向回答 NO;8 个模型中有 4 个在计数示例上完全错误,其余模型随子图规模增大而变差。
Insight: 创新点在于利用模型知识截止日期之后的真实引用关系构建动态、无污染基准,并自动化生成可验证的长上下文图多跳 QA。客观来看,其价值在于揭示 LLM 即使所需事实都在上下文中,也常提前停止搜索引用关系的失效模式,并提供可扩展的评测管线。
Abstract: The needle-in-a-haystack (NIAH) benchmark was introduced as a test for measuring a model’s ability to find small relevant facts within a very large body of input text. However, existing NIAH benchmarks each suffer from one or more of the following limitations: (1) reliance on synthetic data, so their evaluations may not generalize to real-world settings; (2) reuse of previously published datasets, leading to the risk of benchmark contamination; (3) dependence on manual annotation, which requires human effort to create and update data and is prone to future benchmark contamination; and (4) unrealistic simplification of long-context multi-hop reasoning where each next step is trivial given previous steps, while real-world reasoning often requires planning over multiple possible proof paths. To address these limitations, we introduce \Benchmark, a dynamic benchmark built from real citation relationships among scientific papers published after the evaluated models’ knowledge cutoff date, which utilizes real-world information while ensuring no benchmark contamination. The fully automated benchmark construction pipeline samples representative subgraphs of the citation graph formed by these papers and converts each subgraph into a long-context QA example with a verifiable answer. We use GraphNIAH to evaluate eight LLMs from the GPT-5.6, Gemini-3, and Qwen3.5 families. We find that accuracy falls as questions require longer reasoning paths: at question depths of 16 and 32, the GPT-5.6 and Qwen3.5 models answer YES/NO questions no better than random guessing, mostly by answering NO; four of the eight models answer no counting example correctly, and the others model get worse as the subgraph size increases. These findings suggest that current LLMs often stop searching the citation relationships in their context before they reach the answer, even though every fact they need is in the context.
[11] Stepwise-Querying: Distilling Reasoning without Access to Teacher Chain-of-Thought cs.CLPDF
Dicong Yu, Wenhua Wang, Tian Wang
TL;DR: 论文提出 Stepwise Querying,在黑盒教师模型仅提供最终答案或简短解法、无法获取其内部思维链的情况下,通过独立查询让教师对每个步骤给出中间推导、有效性检查和纠错,再将组装后的解法用于学生模型的标准监督微调。作者在多个数学与代码基准上验证方法有效性,并提出 SEIG 诊断指标分析逐步解释相对于全局解释带来的信息增益。
Details
Motivation: 推理蒸馏通常依赖教师生成的思维链监督,但黑盒场景下教师可能只暴露最终答案和简要解法,缺少中间计算与有效性条件,限制了可用于蒸馏的推理监督。论文旨在不访问教师内部 CoT 的情况下,仍能获取细粒度推理监督。
Result: 在两个教师、两个学生和四个数学/代码基准上,该方法在 16 次比较中有 14 次取得最高分,优于三个基线。GSM8K 分析表明,即使最终答案错误,只要保留有效中间推理,轨迹仍可能有用;进一步过度分解带来的额外收益有限。
Insight: 创新点在于用逐步独立查询替代对教师 CoT 的直接访问,主动 elicits 中间推导、有效性检查与纠错,并组装成可用于 SFT 的推理监督;同时提出 SEIG 诊断指标衡量逐步解释相对全局解释的信息增益。客观上看,其可取之处是把黑盒蒸馏从仅答案监督扩展到步骤级监督,并关注错误答案轨迹中有效中间推理的利用。
Abstract: Reasoning distillation is widely used to transfer the reasoning capabilities of large language models to smaller students through teacher-generated supervision. In the black-box setting, however, teachers may expose only final answers and brief solutions, leaving intermediate calculations and validity conditions unavailable and thus limiting the reasoning supervision available for distillation. We propose Stepwise Querying, which elicits step-specific explanations without requiring access to the teacher’s internal chain-of-thought (CoT) traces. Independent queries to the same teacher request intermediate derivations, validity checks, and corrections for individual steps, the assembled solutions are then used for standard supervised fine-tuning. Across two teachers, two students, and four mathematics and code benchmarks, our method achieves the highest score in 14 of 16 comparisons with three baselines. We further introduce Stepwise Explanation Information Gain (SEIG), a diagnostic based on the log-ratio of a frozen student’s probabilities of correct step completion under stepwise and global explanations. Analyses on GSM8K show that incorrect-answer trajectories can remain useful when they preserve valid intermediate reasoning, while further decomposition yields limited additional gains.
[12] Peeling the Matryoshka: Extracting Ontological Structure from Retrieval Embeddings cs.CL | cs.IR | cs.LGPDF
Eunchan Kim, Inwhee Joe
TL;DR: 论文发现 Matryoshka 检索嵌入的前缀维度与分类体系粗到细粒度对齐,提出 PEEL:在冻结嵌入上训练顺序嵌入探针,并用最小生成树/树状图解码分类 DAG;仅用概念词表即可高保真恢复分类结构并跨域迁移。
Details
Motivation: 分类体系构建成本高:聚类方法需逐语料调参且层级不稳定,LLM 流程依赖昂贵且不可跨域迁移的任务训练;论文探究预训练检索嵌入是否隐含可解码的层级结构。
Result: 在两个 OLLM 基准上以高结构保真度恢复分类 DAG,并在无目标边监督下迁移到新领域;端到端 OLLM 流程作为跨任务参考。
Insight: 创新点在于发现 MRL 前缀维度与分类粒度对齐,并用顺序嵌入目标加最小生成树解码,将冻结检索嵌入显式转化为父子 is-a 结构;客观看是一种低成本、少监督、可迁移的嵌入结构探测方法。
Abstract: Taxonomy construction is essential for knowledge-intensive NLP yet remains costly: clustering-based methods (i) need careful per-corpus tuning and yield unstable hierarchies, while LLM-based pipelines (ii) depend on expensive task-specific training that does not transfer across domains. We report an empirical finding that bears on both: in Matryoshka Representation Learning (MRL) embeddings, where each leading prefix is itself a usable, coarser vector, prefix length aligns with the coarse-to-fine granularity of gold taxonomies; shuffled- and reverse-dimension controls isolate the alignment to the learned early prefix dimensions. The pattern holds across three architecturally distinct MRL models. To test whether this regularity is usable, we introduce PEEL (Peeling Embeddings to Extract Latent ontologies): a minimal probe over these frozen embeddings, trained once with an order-embedding objective that represents hierarchy as vector containment; a minimum spanning arborescence then decodes taxonomic DAGs. Using only a concept vocabulary (no documents), PEEL recovers taxonomic DAGs with high structural fidelity on two OLLM benchmarks, transferring to a new domain with no target-edge supervision. The end-to-end OLLM pipeline, which jointly discovers concepts and induces structure from document corpora, is reported as a cross-task reference. Pre-trained retrieval embeddings encode a decodable granularity gradient; a minimal probe renders it as explicit parent-child (is-a) structure.
[13] Aligning MLLMs with the Latent Structure of Human Cognition via Behavior-Derived Semantic Dimensions cs.CL | cs.AIPDF
Ning E, Changde Du, Yizhuo Lu, Huiguang He
TL;DR: 论文提出一种基于行为衍生语义维度的框架,用于将多模态大语言模型与人类认知的潜在多维结构对齐。方法利用470万次人类Odd-One-Out判断构建的66维可解释语义空间,为每个三元组推断最显著的决策维度,并生成维度驱动的语言理由,使模型同时学习选择和理由。实验表明该方法提升了模型与人类相似性判断的一致性,并在标准多模态基准上保持竞争力,同时增强了模型与人脑fMRI表示的对齐。
Details
Motivation: 现有多模态大模型推理性能强,但其内部表示和决策标准是否与人类认知的潜在结构一致尚不明确。论文旨在解决MLLM与人类相似性判断背后的多维心理表示不对齐的问题。
Result: 实验显示该方法提高了模型与人类相似性判断的一致性,并在标准多模态基准上保持有竞争力的性能;在独立fMRI自然场景数据集NSD上,searchlight RSA分析表明模型与人类表示对齐增强。
Insight: 创新点在于利用大规模人类Odd-One-Out行为判断构建的可解释66维语义空间,将最显著决策维度转化为语言理由进行对齐训练,并结合行为相似性判断与fMRI表示相似性分析验证对齐效果。客观来看,其可借鉴之处是以可解释认知维度作为显式监督信号,而非仅拟合最终答案,并提供了行为与神经层面结合的对齐评估路径。
Abstract: Multimodal Large Language Models (MLLMs) have achieved impressive performance across a wide range of reasoning tasks, yet it remains unclear whether their internal representations and decision criteria align with the latent structure of human cognition. In this work, we propose a novel framework to align MLLMs with the multidimensional mental representations underlying human similarity judgements. We build on a previously established behavior-derived semantic embedding space, in which objects are represented along 66 interpretable dimensions derived from 4.70 million human Odd-One-Out (O1O) judgements. For each triplet, we infer the most salient decision dimension and convert it into a dimension-grounded linguistic rationale. This enables MLLMs to learn both the final choice and a justification consistent with the inferred behavioral criterion. Experimental results demonstrate that our approach improves model consistency with human similarity judgements while retaining competitive performance on standard multimodal benchmarks. Furthermore, using searchlight Representational Similarity Analysis (RSA) on an independent fMRI Natural Scenes Dataset (NSD), we observe increased representational alignment between model and human. Ultimately, our findings point to a data-driven route for incorporating human cognitive structure into MLLMs alignment. The code is available \href{https://github.com/ening-open/cogalign}{here}.
[14] CRoS: Causality Reward on Scrambled CoTs Improves Accuracy and Reasoning Faithfulness cs.CL | cs.AIPDF
Xiao Yu, Justin Chih-Yao Chen, Zaid Khan, Hyunji Lee, Mohit Bansal
TL;DR: 论文提出CRoS,一种通过打乱思维链(CoT)步骤顺序而不改变内容来构造的因果奖励指标,用于可验证奖励强化学习(RLVR)训练语言模型推理。该方法避免现有因果度量因删除步骤或注入错误而改变每步信息所带来的混淆,并在Reasoning Gym上使用Gemma-3-4B验证其能提升域外准确率、CoT质量和CoT忠实性。
Details
Motivation: RLVR通常只依赖最终答案正确性,无法保证CoT对答案有因果贡献;现有因果度量通过删除步骤或注入错误扰动CoT,会改变每步信息,使高因果分可能只是对信息变化敏感而非真正因果,并作为奖励时导致CoT信息量下降、探索和自我纠错减少。
Result: 在Reasoning Gym与Gemma-3-4B上,CRoS在五个未见逻辑和游戏数据集上达到46.8%准确率,超过最强基线43.8%;CoT-Pass@K在K=1到128上平均43.4%优于41.4%;CoT忠实性提示承认分数为34.0%,优于20.2%。
Insight: 论文宣称的创新点是用“打乱步骤顺序但保留内容”作为因果扰动,从而隔离信息变化带来的混淆,获得更干净的CoT因果奖励。客观来看,该设计将因果评估聚焦于推理步骤顺序的贡献,有助于同时提升答案准确率和推理忠实性。
Abstract: Reinforcement learning with verifiable rewards (RLVR) has become the standard method for training language models to reason. However, RLVR typically relies solely on outcome correctness, which does not guarantee that the CoT causally contributes to the final answer. Without this connection, outcome rewards may fail to reinforce high-quality reasoning. Existing causality metrics assess CoT causality by perturbing the CoT, such as deleting reasoning steps or injecting errors, and measuring changes in the final answer. However, these perturbations inevitably alter information in each step. Consequently, high causality scores may reflect sensitivity to information changes introduced by perturbation rather than true causality. Such confounding effects further induce undesirable reasoning behaviors when these metrics are used as training rewards, including generating less informative CoTs and reducing exploration and self-correction. To address this problem, we propose CRoS (Causality Reward on Scrambled CoTs), a novel CoT causality metric that perturbs the order of reasoning steps without altering content, thereby preserving information within each step. On Reasoning Gym with Gemma-3-4B, CRoS outperforms existing causality and outcome rewards in out-of-domain accuracy, CoT quality, and CoT faithfulness. Specifically, CRoS achieves 46.8% accuracy on five unseen logic and games datasets, outperforming the strongest baseline (43.8%). It also achieves better CoT quality, consistently outperforming baselines in CoT-Pass@K from K=1 to 128, averaging 43.4% vs. 41.4%. For CoT faithfulness, CRoS achieves a hint acknowledgment score of 34.0% vs. 20.2%.
[15] Subquadratic Attention with a Hierarchical Tree-Structured Indexer cs.CL | cs.AIPDF
Leon Lufkin, Cutter Dawes, Alexander Speicher, Kamesh Krishnamurthy, Tomás Figliolia
TL;DR: 论文提出一种基于分层树结构索引器的稀疏注意力方法 ZHA,用于缓解大语言模型长上下文中的二次注意力瓶颈。ZHA 在 KV cache 上构建递归压缩的 key 摘要树,查询自顶向下遍历以检索稀疏 key 子集,并通过迭代增加树深度使路由与最终注意力的总成本随序列长度次线性增长,从而实现整体次二次但超线性的注意力扩展。
Details
Motivation: 标准注意力随序列长度二次增长,限制 LLM 长上下文能力;现有滑动窗口/线性注意力表达力不足,稀疏注意力索引器本身又可能二次增长或仅带来常数级加速。论文希望从渐近复杂度上实现次二次稀疏注意力。
Result: 摘要未给出具体数值,但声称 ZHA 仅需少量 KL 蒸馏训练即可适配现有预训练模型,并在长上下文检索、推理、数学和 agentic coding 任务上保持性能;在超过 100 万 token 上下文时显著优于竞争方法,并有潜力扩展到数亿甚至十亿级有效上下文。
Insight: 创新点在于用分层树状递归压缩的 KV 摘要作为索引器,使查询通过自顶向下路由检索稀疏 key,并通过控制树深度实现整体次二次复杂度;同时强调可低成本迁移到预训练模型、对深树几何稳健。客观看,其价值主要在于将分层索引/稀疏检索与 KV 压缩结合,并从渐近复杂度和可部署适配两方面推进长上下文注意力。
Abstract: The quadratic scaling of attention is a fundamental bottleneck limiting the long-context capabilities of large language models (LLMs). A variety of approaches have attempted to circumvent this, but they either fail to match attention’s expressiveness (e.g., sliding-window or linear attention variants with fixed-size state) or only achieve constant-factor speedups (e.g., sparse attention methods whose indexers themselves scale quadratically). In this paper, we derive an asymptotically subquadratic sparse attention method based on a hierarchical, tree-structured indexer, which we call Zyphra Hierarchical Attention (ZHA). ZHA constructs a tree of recursively compressed key summaries over the KV cache, which future queries traverse from the top down to retrieve a sparse subset of keys. Crucially, by iteratively growing the tree’s depth, the total cost including routing and final attention grows sublinearly in sequence length, resulting in overall subquadratic (but superlinear) scaling for attention. We demonstrate that ZHA can be cheaply applied to existing pretrained models with only a short KL distillation training stage over a few billion tokens, and maintains performance on long-context retrieval and challenging reasoning, mathematical, and agentic coding evaluation tasks. Moreover, we show that our method is highly robust to tree geometry including very deep trees of up to 6 levels. Finally, we develop a detailed performance model and show that our approach scales subquadratically, dramatically outperforms competing methods beyond 1 million context, and ultimately has the potential to extend effective context windows to the hundreds of millions or billions of tokens.
[16] DecisionRank: On-Policy Distillation for Decision-Model Rerankers cs.CLPDF
Yanran Wu, Dian Yu, Chongyang Bai, Jieyi Zhang, Rui Liu
TL;DR: 论文提出 DecisionRank,用物品空间在线策略蒸馏(item-space OPD)将生成式 LLM 的排序知识迁移到直接输出分数的决策模型重排器中。方法把学生模型对文档的打分排序成排名列表,再让生成式教师基于该列表提供条件反馈,直接监督这些分数,并结合离线排序监督训练。
Details
Motivation: 决策模型可直接预测选择或分数,避免自回归解码延迟,但生成式教师通过文档 ID 序列的排名列表表达偏好,而学生模型依据首 token 位置的分数决定排序,二者存在表示与接口不匹配,导致排序知识迁移困难。
Result: DecisionRank 在 TREC DL、BEIR、BRIGHT 等通用与推理密集型检索 benchmark 上达到 SOTA;item-space OPD 在四个 benchmark 上稳定优于其他训练目标。1.7B 模型在 TREC DL19–23 上超过 27B 生成式教师;8B 模型超过生成式 listwise 对照,并在 TREC DL 上快 24.5 倍、BRIGHT 上快 16.7 倍。
Insight: 核心创新是用学生分数排序构造 rollout,让生成式教师以排名列表为输入给出条件反馈,从而直接监督学生分数,弥合教师序列式偏好与学生分数式排序之间的鸿沟;结合离线排序监督,在保持决策模型低延迟的同时迁移 LLM 排序知识。
Abstract: Decision models target structured tasks by directly predicting choices or scores, avoiding autoregressive decoding latency. However, transferring a generative LLM’s ranking knowledge to a decision model remains challenging: the teacher expresses preferences through a ranked list of document ID sequences, while the student determines the ranking using scores from the first-token position. To bridge this gap, we introduce DecisionRank, a decision model trained through item-space on-policy distillation (OPD). Item-space OPD constructs the student’s rollout by sorting its document scores into a ranked sequence. A generative teacher takes the resulting ranking list as inputs and provides conditional feedback to directly supervise those scores. Combined with offline ranking supervision, DecisionRank achieves state-of-the-art performance on both general and reasoning-intensive retrieval benchmarks such as TREC DL, BEIR, and BRIGHT. Across four benchmarks, item-space OPD consistently outperforms alternative training objectives. Even at 1.7B parameters, DecisionRank surpasses its 27B generative teacher on TREC DL19–23. At 8B, it outperforms its generative listwise counterpart while running $24.5\times$ faster on TREC DL and $16.7\times$ faster on BRIGHT.
[17] Flide: Accelerating Decision Inference with Exact State Forking cs.CLPDF
Bobo Li, Jiajia Song, Hao Fei, Wynne Hsu, Mong Li Lee
TL;DR: 论文提出 Flide,一种面向 LLM 智能体决策模型的高效精确推理引擎。它将决策请求的逻辑树结构与单个扁平化 token 流的物理执行解耦,通过 ForkAttention 在标准注意力和循环 Gated DeltaNet 层中利用 GPU 常驻索引实现精确状态分叉,从而避免前缀重计算和输出概率失真。
Details
Motivation: 现有生成导向推理引擎难以高效支持决策模型的分支/树状推理负载,要么产生严重的前缀重计算开销,要么因近似导致输出概率失真。论文旨在同时保证推理吞吐和数学精确性。
Result: 在公开混合模型和决策模型上,Flide 相比 vLLM 和 SGLang 最高快 9.5 倍,并在总变差距离上最接近 float32 参考结果。
Insight: 创新在于把请求分支拓扑表达为 GPU 常驻元数据,并用 ForkAttention 在标准注意力和循环层中实现与隔离前向传播等价的精确状态分叉;同时用紧凑预录执行图适配混合请求,并支持从早期检查点恢复扩展的智能体历史。
Abstract: Decision models such as Jev have emerged as a highly promising paradigm for LLM agent systems, providing lightweight and reliable categorical reasoning for rapid System One execution. However, existing generation-oriented inference engines fail to support these workloads effectively, suffering from severe prefix recomputation overheads or distorted output probabilities. To address this challenge, we propose Flide, an efficient and exact inference engine that decouples the logical tree structure of a decision request from its physical execution within a single flattened token stream. Through ForkAttention, Flide navigates request dependencies using GPU-resident indices across both standard attention and recurrent Gated DeltaNet layers, guaranteeing in exact arithmetic that every query branch evaluates identically to an isolated forward pass. By expressing branch topologies directly in GPU-resident metadata, Flide accommodates diverse request mixtures with a compact set of pre-recorded execution graphs while enabling expanding agent histories to resume from checkpoints saved in earlier passes. Extensive evaluations on public hybrid and decision models demonstrate that Flide answers queries up to 9.5 times faster than vLLM and SGLang while staying closest to a float32 reference in total variation distance. By unifying physical execution throughput with mathematical exactness, Flide establishes a principled foundation for scalable, latency-critical decision intelligence in agent systems.
cs.CV [Back]
[18] Drone-PUR: From Visual Clue Discovery to Understanding and Reasoning in UAV Scenes cs.CVPDF
Jiahe Wu, Bing Cao, Xingyue Guo, Ruipu Zhao, Beiming Yu
TL;DR: 论文提出 Drone-PUR,一个面向无人机场景的视觉线索发现、解释与整合的数据集和基准,覆盖感知、理解、推理三个能力维度,包含 11 个任务、175k 训练样本和 20k 人工审核测试样本,涵盖单图、图像序列、跨模态图像和多视角场景。作者评估了 22 个通用视觉语言模型,发现其在空间判断和跨模态/跨视角推理上存在持续局限,并进一步构建 UAV-PUR 领域适应模型。
Details
Motivation: 通用视觉语言模型在无人机场景中受小目标、细微属性和视角变化影响,视觉观察不完整或不准确会限制感知与推理能力;现有基准对视觉线索发现、解释和整合的系统评估不足。
Result: 评估 22 个通用 VLM 显示其在空间判断、跨模态和跨视角推理上仍有明显局限;基于 Qwen3-VL 分阶段微调的 UAV-PUR 4B 和 8B 模型,相比预训练版本总体多选准确率分别提升 18.33 和 17.52 个百分点。
Insight: 创新点在于围绕视觉线索发现、解释与整合构建无人机场景数据集和基准,并设计需要多线索组合的推理样本;同时通过多模型评估揭示感知与推理的关联及单任务微调下的选择性双向迁移,为领域适应提供系统分析。
Abstract: The reasoning capabilities of vision-language models (VLMs) can be limited by incomplete or inaccurate visual observations in the general domain. In unmanned aerial vehicle (UAV) scenes, small targets, subtle attributes, and changing viewpoints further complicate perception and understanding. Therefore, we introduce Drone-PUR, a dataset and benchmark organized around visual clue discovery, interpretation, and integration, corresponding to the three capability dimensions of perception, understanding, and reasoning. Reasoning samples are designed to require models to combine multiple visual clues to derive visually grounded conclusions. Its 11 tasks provide 175k training and 20k test samples spanning single images, image sequences, cross-modal images, and multiple viewpoints. All test samples undergo human review. Evaluating 22 general-purpose VLMs reveals persistent limitations in spatial judgment and cross-modal and cross-view reasoning. Across models, stronger perception is associated with better reasoning, while single-task fine-tuning further reveals selective bidirectional transfer between these capabilities. To examine domain adaptation, we develop UAV-PUR from Qwen3-VL through staged fine-tuning with auxiliary supervision. Its 4B and 8B variants improve overall multiple-choice accuracy by 18.33 and 17.52 percentage points over their pretrained counterparts, respectively. The dataset and project resources are available at https://drone-pur.github.io.
[19] EgoHSI3R: Online 4D Hand-Scene Interaction Reconstruction from Egocentric Video cs.CVPDF
Mingzhi Pei, Zhangchen Ye, Ruoqu Chen, Liu Cao, Mengdi Xu
TL;DR: 论文提出 EgoHSI3R,一个统一的单目第一人称视频在线 4D 手-场景交互重建前馈框架。它通过循环场景表示累积几何证据并提供共享世界空间与尺度,结合手部先验和局部场景上下文,实现相机运动、稠密场景几何、世界空间双侧 MANO 手及每顶点手部接触的联合估计。
Details
Motivation: 现有方法通常只恢复相机坐标系下的手或世界空间手部轨迹,缺少场景点图和交互信息的联合恢复,构建完整重建往往需要逐序列优化。论文希望在无需逐序列优化的情况下,从第一人称单目视频在线恢复一致的手、相机和场景。
Result: 尽管以因果在线方式运行,EgoHSI3R 在 ARCTIC 上取得所评估方法中最低的世界空间手部关节误差,推理速度快于现有世界空间手部重建流程,并在相机和场景重建精度上具有竞争力。
Insight: 创新点在于统一的在线前馈 4D 手-场景交互重建框架:用循环场景表示提供共享世界空间和尺度,用 hand-centric prompts 融合预训练手先验与局部场景上下文,并通过 MANO 顶点查询场景特征预测稠密每顶点接触,从而联合估计相机、场景、双手和接触。
Abstract: Egocentric videos offer an abundant source of hand-scene interactions for embodied learning, which requires recovering hands, camera, and scene consistently in a shared 3D world. Existing methods focus on either camera-frame hands or world-space hand trajectories, without jointly recovering the scene pointmaps or interaction information. Building a complete reconstruction from such specialized predictions therefore typically requires per-sequence optimization. We propose EgoHSI3R, a unified feed-forward framework for online 4D hand-scene interaction reconstruction from egocentric monocular video. To jointly recover these components, it maintains a recurrent scene representation that accumulates geometric evidence over time and provides a shared world space and scale. We introduce hand-centric prompts that fuse pretrained hand priors and local scene context, while MANO vertex queries attend to scene features to predict dense per-vertex contact. These designs enable a single network to jointly estimate camera motion, dense scene geometry, world-space bilateral MANO hands, and per-vertex hand contact. Despite operating causally, EgoHSI3R achieves the lowest world-space hand joint error on ARCTIC among the evaluated methods, while offering faster inference than existing world-space hand reconstruction pipelines and competitive camera and scene reconstruction accuracy.
[20] AquaProbe: Efficient Adaptation of Frozen Vision Models for Underwater Images cs.CV | cs.LG | cs.ROPDF
Md Shihabul Islam Shovo, Nishi Kanta Paul, Trung Dong, Xiaomin Lin
TL;DR: 论文提出 AquaProbe,用于分析水下图像相对陆地预训练视觉模型时,域差距在冻结编码器不同深度上的位置如何随水体类型和成像距离变化。方法利用配对干净图像与物理渲染图像,将新引入的域差异与编码器累计差异分离,并据此在固定 LoRA 参数预算下进行比例化秩分配,同时引入基于估计条件簇的三路路由器。
Details
Motivation: 水下环境的浑浊、后向散射和波长相关衰减会改变场景外观,使陆地图像预训练的视觉模型在水下不可靠;现有适配方法要么在编码器深度上均匀放置适配器,要么基于两个固定域之间的差异选择层,未考虑域差距位置随水型和成像范围变化的问题。
Result: 基于测量轮廓的比例 LoRA 秩分配在分割任务上达到 70.1 mIoU,在相同 0.59M 适配器参数预算下比均匀 LoRA 秩分配高 1.9 mIoU;以估计条件簇为条件的三路路由器在标准 SUIM 划分上达到 70.6 mIoU,并在留出退化条件下达到 63.4 AUDRC。
Insight: 创新点在于用 AquaProbe 度量域差距位置随水类型和成像距离的变化,并分离新引入差异与冻结编码器累计差异,从而按测量轮廓分配固定 LoRA 预算;客观看,其可借鉴之处是将适配器容量分配从均匀或固定域差异选择,推进到条件自适应、按深度域差距剖面分配,并结合条件簇路由提升水下退化场景适配效率。
Abstract: Reliable visual perception is essential for safe autonomous underwater exploration. Yet vision models pretrained on terrestrial images become less reliable underwater, where turbidity, backscatter, and wavelength-dependent attenuation alter scene appearance. Existing adaptation methods either spread adapters equally across the encoder depth or select layers using the divergence between two fixed domains. In this paper, we introduce AquaProbe for measuring how the location of the domain gap changes with water type and imaging range. AquaProbe separates newly introduced divergence from that accumulated through a frozen vision encoder using paired clean and physically rendered images and allocates a fixed LoRA budget according to the measured profile. The segmentation mean Intersection-over-Union (mIoU) for the proportional LoRA rank allocation, based on the measured profile, is 70.1; this is a better result than uniform LoRA rank allocation by 1.9 mIoU points while having the same parameter budget of 0.59 million adapter parameters. The three-way router, conditioned on estimated condition clusters, achieves 70.6 mIoU on the standard SUIM split and 63.4 AUDRC under held-out degradation.
[21] Coupled or Independent? Fault Coupling in Agentic Medical Vision Systems cs.CV | cs.CRPDF
Nishi Kanta Paul, Md Shihabul Islam Shovo, Md Mostafijur Rahman, Jamil Fayyad
TL;DR: 论文研究带工具的视觉语言智能体在传感器读数不可信时,工具输出与推理通道共享视觉故障的耦合问题。作者构建基于分割、对比检索和迭代视觉语言推理器的皮肤科智能体,定义故障耦合分类,并在PAD-UFES-20和HIBA上分析注入攻击、图像损坏、检索方式及通道比较对联合故障检测的影响。
Details
Motivation: 带工具智能体可暴露中间输出供检查,理论上比单次模型更安全;但若视觉输入本身故障,工具与推理可能共同出错,因此需要检验这一安全收益是否成立。
Result: 在PAD-UFES-20上,文字注入可产生可检测的方向性检索捕获,而物理损坏不能;先裁剪检索产生55%全样本联合错误一致(故障样本71%),全图检索为35%(43%)。二值决策下硬标签通道比较无法检测联合失败,分歧召回率仅0.29(先裁剪)对0.57(全图),且顺序效应方向在HIBA上复现。
Insight: 论文宣称的创新是提出故障耦合分类并系统评估医疗视觉智能体中间输出检查的局限;客观看,其价值在于揭示共享视觉故障会使工具与推理错误同向,且硬标签比较在二值决策下天然失效,提示安全评估需关注跨通道耦合而非仅单通道错误。
Abstract: Vision-language agents equipped with tools expose intermediate outputs for inspection and comparison with the model’s reasoning, which may offer safety benefits over single-shot models. This paper tests that potential benefit when the sensor reading itself is untrusted, so that the tool and reasoning channels share a common visual fault. We consider a dermatology agent based on segmentation, contrastive retrieval, and an iterative vision-language reasoner and define a coupling taxonomy that distinguishes single-channel errors from joint errors that agree or disagree. On PAD-UFES-20, we show that typographic injection produces detectable directional retrieval capture, whereas physical corruption does not. Crop-first retrieval preserves the payload at a readable scale and produces 55% joint wrong agreement across all cases (71% of faulty cases); full-image retrieval produces 35% (43%). Under a binary decision, if both channels are wrong, their predictions must agree, so hard-label channel comparison cannot detect joint failures, limiting disagreement recall to 0.29 (crop-first) versus 0.57 (full-image). The direction of the order effect is reproduced on the independent HIBA test set.
[22] Let LiDAR Decide Where to Look: Geometry-Measured Token Selection for Efficient Sparse Latent World Models cs.CVPDF
Zeping Zhang, Zhuoya Zhao, Robert Laganière
TL;DR: 论文发现视觉中心自动驾驶模型中大量图像token与规划无关且稀释注意力,随机丢弃75%token反而优于密集模型。为此提出几何感知token选择(GATS),利用LiDAR和雷达点投影在视觉骨干运行前选择少量关键图像token,并输入自监督潜在世界模型预测未来特征和3D位移,大幅降低计算量并提升规划性能。
Details
Motivation: 解决视觉中心自动驾驶模型每帧编码数千图像token导致计算冗余和规划注意力被稀释的问题,旨在仅保留驾驶关键内容以降低端到端计算成本。
Result: 在nuScenes和NAVSIM上,25%预算下GATS达到0.520m L2位移误差和88.9 EPDMS,超过密集基线(0.573m/85.9)且总FLOPs少6.6倍;10%预算下仍优于密集基线;在739个意图关键场景中覆盖95.8%危险车辆,碰撞率0.09%,比密集模型(0.34%)低73.5%。
Insight: 创新点在于利用实测3D几何(LiDAR/雷达投影)而非视觉特征在骨干前选择token,实现早期计算削减;结合自监督潜在世界模型预测未来特征和3D位移,无需框标注。客观分析:随机丢弃已揭示冗余,但几何引导选择更精准;早期选择机制有效降低端到端FLOPs,无标注世界模型增强规划安全性。
Abstract: Vision-centric autonomous driving models encode thousands of image tokens per frame, yet most depict sky, buildings, or empty road—content irrelevant to planning. Worse, these tokens dilute the planner’s attention: we show that randomly discarding 75% of tokens already outperforms the dense model, because the planner can concentrate on the remaining driving-critical content. Building on this finding, we present Geometry-Aware Token Selection (GATS), which uses LiDAR and radar point projections to select a small fraction of image tokens before the vision backbone runs. Because the selector operates on measured 3D geometry rather than visual features, the backbone encodes only the selected patches, cutting end-to-end FLOPs by up to $18\times$. Each kept token receives a 3D anchor from its LiDAR point and is fed to a self-supervised latent world model that predicts future features and 3D displacements without box annotations. At 25% budget, GATS achieves a 3-sec L2 displacement error of 0.520 m on nuScenes and an EPDMS of 88.9 on NAVSIM—surpassing the dense baseline (0.573 m / 85.9) at $6.6\times$ fewer total FLOPs. At an extreme 10% budget (24 GFLOPs, $18\times$ reduction), it still beats the dense baseline on both nuScenes (0.537 m vs. 0.573 m L2) and NAVSIM (87.9 vs. 85.9 EPDMS). On 739 intent-critical scenarios, GATS covers 95.8% of hazardous vehicles and achieves a 0.09% collision rate—73.5% lower than the dense model (0.34%).
[23] UFMF: Radar-Guided Kinematic Priors for Real-Time 4D Occupancy Forecasting cs.CVPDF
Zeping Zhang, Zhuoya Zhao, Robert Laganière
TL;DR: 论文提出UFMF,一种面向实时4D占用预测的超快多模态预测器。它采用无注意力的对角状态空间时序核心,并引入雷达引导的速度感知动态先验,将未来位移分解为线性外推与随预测时域变化的残差,通过逐单元流场在rollout中传输紧凑动态潜空间;同时用世界坐标系动态监督和类别条件垂直轮廓监督提升轨迹一致性与BEV到3D抬升。
Details
Motivation: 实时4D占用预测需在严格延迟预算下可靠外推自车与周围动态目标运动;现有占用世界模型精度高但计算昂贵,轻量BEV预测器虽满足实时性,却在较长时域对快速运动目标模糊或抑制。
Result: UFMF在单张RTX 4090上达到122 FPS,在nuScenes上取得0.45的快速位移mIoU,以远低于近期占用世界模型的计算成本超过其性能。
Insight: 创新点在于将雷达引导的运动学先验与轻量状态空间时序核心结合,用线性外推加学习残差和逐单元流场建模未来位移,并加入世界坐标系动态监督与类别条件垂直轮廓监督;客观看,其价值在于以结构化先验和辅助约束弥补轻量实时模型在快速运动与BEV到3D转换上的短板。
Abstract: Real-time 4D occupancy forecasting requires reliably extrapolating both ego motion and surrounding actor motion under strict latency budgets. Recent occupancy world models improve long-horizon fidelity through generative decoders and 3D volumetric modeling, yet their computational cost makes real-time deployment challenging. Light-weight BEV forecasters meet latency requirements but often blur or suppress fast-moving actors at longer horizons. We present UFMF (Ultra-Fast Multi-modal Forecaster), a real-time forecaster that keeps an attention-free diagonal state-space temporal core while injecting targeted kinematic and geometric structure where lightweight models typically fail. A radar-guided velocity-aware dynamic prior (VADP) decomposes future displacement into a linear extrapolation plus a learned horizon-dependent residual. The resulting per-cell flow fields transport a compact dynamic latent subspace during rollout. Two auxiliary training constraints complement this branch: world-frame dynamic supervision enforces inter-horizon trajectory coherence, and class-conditioned vertical-profile supervision regularizes BEV-to-3D lifting. UFMF operates at 122 FPS on a single RTX 4090 and reaches 0.45 fast-displacement mIoU on nuScenes, outperforming recent occupancy world models at a fraction of their computational cost.
[24] AVID: A Weakly Aligned RGB-Infrared Dataset and Perturbation Benchmark for Tiny Aerial Targets cs.CVPDF
Dharini Raghavan, Vishnu Rao, Sethu Selvi S
TL;DR: 论文提出 AVID,一个面向远距离小型无人机与鸟类区分任务的弱对齐 RGB-红外视频 benchmark,包含 32,040 对帧、115 个视频和 35,952 个标注,目标普遍极小。论文还提出对象相对对齐比例与对齐负担曲线来量化跨模态几何困难,并构建 AVID-C 模拟 RGB、红外或双模态的七类图像退化。基于固定特征的分析表明,RGB 与红外特征结合可提升鸟-无人机类间可分性。
Details
Motivation: 远距离小无人机检测易与鸟类混淆,且地基相机细节有限;RGB 与红外可互补,但其效果受目标尺寸、跨模态对齐和图像质量影响,缺少系统 benchmark。
Result: AVID 中 68.1% 目标几何平均框边长为 8–16 像素,99.0% 满足 COCO 小目标标准;300 对样本中 32.3% 全局平移超过 5 像素。固定特征分析中,12–16 像素目标的 Bhattacharyya 误差界为 RGB 0.148、红外 0.198、融合 0.046,但论文强调这不等同于检测器精度。
Insight: 创新点在于构建弱对齐 RGB-红外微小空中目标数据集与退化 benchmark,并提出对象相对对齐比例和对齐负担曲线,支持尺度感知、对齐感知和传感器特异的鲁棒性研究;融合模态在特征可分性上的收益也提供了客观参考。
Abstract: Long-range detection of small drones is complicated by their visual similarity to birds and by the limited detail available from ground-based cameras. Combining visible and infrared observations offers complementary information, but their usefulness depends on target size, cross-modal alignment, and image quality. We present AVID, a benchmark for studying these factors using real RGB-infrared bird-drone surveillance footage originating from the Drone Detection Dataset and curated for the IEEE ICIP 2025 Video and Image Processing Cup. AVID comprises 32,040 corresponding frame pairs from 115 videos, with 35,952 existing bird and drone annotations at $320\times256$ resolution. Targets occupy an unusually narrow size range: 68.1% have a geometric-mean bounding-box side of 8–16 pixels, and 99.0% meet the COCO small-object criterion. To characterize the geometric difficulty of combining the modalities, we introduce an object-relative alignment ratio and an alignment burden curve. In a sample of 300 frame pairs, 32.3% have estimated global translations greater than 5 pixels; among the 264 pairs with matching annotations, 16.3% exceed one median target side. We further introduce AVID-C, which simulates seven families of image degradation at five severity levels independently in RGB, infrared, or both. A fixed-feature analysis examines how scale, degradation, and added cross-modal shifts affect fitted bird-drone class overlap. At 12–16 pixels, the Bhattacharyya error bounds are 0.148 for RGB, 0.198 for infrared, and 0.046 for their combined features. These values describe separation in the specified feature and density models, not detector accuracy. Together, AVID and AVID-C support scale-aware, alignment-aware, and sensor-specific studies of tiny aerial targets.
[25] FieldGuideRAG: Image-Grounded GraphRAG for Fine-Grained Recognition with Vision-Language Models cs.CV | cs.IRPDF
Jiacheng Ding, Yanliang Qi, Cong Guo, Peihua Mai
TL;DR: 论文提出 FieldGuideRAG,一种免训练的图检索增强生成(GraphRAG)方法:在知识图谱的每个物种节点存一张真实照片,用冻结的视觉语言模型将查询与候选节点照片逐一验证,融合地理范围边进行细粒度物种识别,并利用事实边回答生态问题。方法还能用验证分数检测图中缺失的物种。
Details
Motivation: 现有多模态 GraphRAG 多用文本描述实体,难以传递区分相似物种所需的视觉细节;细粒度物种识别需要图像级证据,并希望免训练地结合图谱中的亲缘、地理和生态知识。
Result: 在 CUB-200、Wikidata 图和四个 iNaturalist 域上,节点照片比物种名提升验证效果,文本描述增益很小;候选池验证比 SigLIP 检索器最高高 16 分,iNaturalist 上范围边带来 7–19 分提升,事实边回答 28,820 个生态问题最高正确率 89.5%,并能检测图中缺失物种。
Insight: 创新在于把真实照片作为图谱节点的一等证据,用冻结 VLM 做节点级视觉验证,并将地理范围边和事实边融入识别与问答;同时以验证分数实现缺失物种检测。客观看,其价值在于避免文本描述丢失细粒度视觉信息,并以免训练方式把 GraphRAG 扩展到图像接地的细粒度识别。
Abstract: We present FieldGuideRAG, a training-free GraphRAG method in which a vision-language model, or VLM, recognizes fine-grained species by comparing the query with real photographs stored on the nodes of a knowledge graph. Multimodal GraphRAG systems describe each entity in text, and text conveys little of the visual detail that separates look-alike species. Our graph stores one photograph on every species node and links each species to its relatives, its geographic range and its ecological facts. A frozen VLM verifies the query against the photograph of each candidate node. FieldGuideRAG fuses the verification scores with the range edge and answers questions about the organism from the fact edges of the verified species. On CUB-200, a Wikidata graph and four iNaturalist domains, the node photograph improves verification over the species name on every graph, while text descriptions add little. Verification of the candidate pool exceeds a SigLIP retriever by up to 16 points, the range edge adds 7 to 19 points on iNaturalist, and answering from fact edges is correct on up to 89.5% of 28,820 ecological questions. The verification scores also detect queries whose species is missing from the graph. Graphs, prompts and code will be released.
[26] Prior-Guided Single-Exposure HDR Novel View Synthesis via Base-Gain Representation cs.CV | cs.MMPDF
Armin Shafiee Sarvestani, Sheyang Tang, Jialu Xu, Zhou Wang
TL;DR: 论文提出 PriorGS,一种基于 3D Gaussian Splatting 的单曝光 HDR 新视角合成框架,从单曝光 LDR 视图恢复 HDR 场景并估计 CRF 以支持重曝光。方法利用 2D 单曝光 HDR 重建方法生成伪 HDR 监督,并通过闭环重建锚定输入 LDR 视图,同时用可学习先验正则化 CRF 建模。为缓解伪 HDR 跨视图不一致,论文引入 3D base-gain HDR 表示,将 HDR 辐射分解为 LDR base 与 3D gain 场,并支持 3D HDR 外观编辑。
Details
Motivation: 单曝光 HDR 新视角合成缺乏 HDR 真值,仅用单曝光 LDR 视图直接监督会欠约束 HDR 重建,难以保证忠实恢复;同时 LDR 视图只能有限范围监督 CRF,需引入先验以可靠支持超出范围的重新曝光。
Result: 摘要称大量实验表明 PriorGS 显著优于基线,但未给出具体 benchmark、定量指标或明确的 SOTA 结论。
Insight: 创新点包括:用 2D 单曝光 HDR 重建先验提供伪 HDR 监督,并以闭环重建保持与输入 LDR 视图一致;用可学习先验正则 CRF,保证单调性并促进合理响应;提出 3D base-gain HDR 表示,将 HDR 辐射分解为 LDR base 与 3D gain 场,以提升对伪 HDR 跨视图不一致的鲁棒性并支持 HDR 外观编辑。客观看,核心创新在于将 2D HDR 先验、3DGS 闭环优化与 base-gain 分解结合,以缓解无 HDR 真值下的欠约束和跨视图不一致问题。
Abstract: Single-exposure high dynamic range (HDR) novel view synthesis (NVS) recovers HDR scenes from low dynamic range (LDR) views captured at a single exposure and estimates camera response function (CRF) for LDR re-exposure. Without HDR ground-truth, using single-exposure LDR views as the sole source of direct supervision underconstrains HDR reconstruction and cannot ensure its faithful recovery, motivating the need for learned LDR-to-HDR reconstruction priors. Moreover, single-exposure LDR views supervise CRF only over a limited range, necessitating a learnable prior for reliable re-exposure beyond it. We propose PriorGS, a 3D Gaussian Splatting framework that addresses both challenges by (1) guiding HDR recovery with pseudo-HDR supervision from 2D single-exposure HDR reconstruction methods while anchoring it to input LDR views through a closed reconstruction loop; and (2) regularizing CRF modeling with a learnable prior that guarantees monotonicity and promotes plausible responses. Yet cross-view inconsistencies in independently generated pseudo-HDR targets impair scene reconstruction when HDR is encoded directly as Gaussian’s color attribute. We therefore introduce a 3D base-gain HDR representation that factors HDR radiance into an LDR base and an overlaid 3D gain field, promoting robustness to such inconsistencies and enabling 3D HDR appearance editing. Extensive experiments demonstrate PriorGS’s substantial superiority over baselines.
[27] Dreams Are Cheaper Than Flights: Curated World-Model Data Improves Aerial Detection cs.CV | cs.AIPDF
Rugved Katole, Vedant Patil, Jenna Kline, Christopher Stewart
TL;DR: 论文研究用视频世界模型生成航拍合成数据,以补充或替代昂贵、缓慢的真实野外数据,并验证其对目标检测器现场性能的影响。作者用7B参数世界模型基于真实无人机视频扩展训练集,筛选合成帧后加入或替换真实数据均能提升检测性能,且总成本低于实地采集。
Details
Motivation: 机器人感知依赖数据,但野外数据采集慢、成本高且可能造成干扰;论文希望验证世界模型生成的合成数据能否达到真实现场数据的效用,并让检测器在实际场景中有效。
Result: 在同一无人机留出视频上,加入合成帧使mAP@50-95提升4.3%,替换真实数据提升10.3%,在城市基准上最高提升33.5%;尽管专家估计拒绝约87.5%的生成片段,GPU成本约1100美元,低于实地人工超3200美元且真实训练集效用更低。
Insight: 创新点在于以真实帧为种子,按部署传感器、高度和环境扩展世界模型生成视频,并通过筛选将数据成本从野外采集转向计算。客观看,其关键不是无条件合成,而是“真实种子+ curated frames”的数据扩展与筛选策略。
Abstract: Robot perception is increasingly data driven, yet field data is slow, costly, and sometimes disruptive to collect. Video world models offer a cheaper source: seeded with real frames, they extend a recorded scene into new footage with the deployment’s sensor, altitude, and environment. We ask whether such synthetic data matches the utility of real field data and whether detectors trained with it perform in the field. Seeding a 7B-parameter world model with wildlife drone video, we more than double the training set with curated frames. On held-out videos from the same drones, mAP@50–95 rises 4.3% when these frames are added to real data and 10.3% when they replace it, and up to 33.5% on an urban benchmark. Although an expert rejects an estimated 87.5% of generated clips, the data costs about $1,100 of GPU time, against more than $3,200 of field labor alone for a less useful real training set. Synthetic data thus shifts data costs from fieldwork to compute.
[28] WebOverlay: High-Throughput Vision-Only Distillation for Extreme Aspect-Ratio Web Overlay Detection cs.CV | cs.AI | cs.IRPDF
Jiannan Huang, Jerry Ding, Li Yu, Zhenting Li
TL;DR: 论文提出WebOverlay,一个仅视觉的高吞吐量网页覆盖层检测器,通过从视觉-语言教师模型蒸馏训练,采用动态分块策略处理极端宽高比的网页渲染,并结合宽高比感知的边界裁决机制,避免DOM解析,实现高吞吐量和低成本。
Details
Motivation: 网页覆盖层阻碍自动爬虫并降低下游文档表示质量,现有MLLM准确但延迟和成本高,轻量视觉模型难以处理极端垂直宽高比导致文本特征丢失和误报,需要高吞吐低成本的检测方案。
Result: WebOverlay显著优于基于DOM的启发式方法,与专有MLLM基线相比达到有竞争力的精确率和召回率,同时吞吐量高几个数量级,部署成本大幅降低。
Insight: 摘要宣称的创新点包括动态分块策略、宽高比感知的边界裁决机制以及仅视觉蒸馏绕过DOM解析;客观分析其创新在于针对极端宽高比网页的适应性分块与边界处理,以及通过蒸馏获得高效仅视觉模型实现高吞吐。
Abstract: Web overlays, such as cookie consent banners, newsletter prompts, and age gates, obstruct automated web crawlers and degrade downstream document representations. While proprietary multimodal Large Language Models (MLLMs) can accurately identify overlays, their latency and inference cost preclude web-scale ingestion. Conversely, lightweight vision models struggle with full-page renderings characterized by extreme vertical aspect ratios, where standard isotropic resizing collapses critical text features and causes false positives. In this work, we present WebOverlay, a high-throughput vision-only detector trained via distillation from a frontier vision-language teacher. We establish a dynamic tiling strategy that partitions extreme aspect-ratio web renders into aspect-preserving patches with spatial overlap, coupled with an aspect-ratio-aware boundary adjudication mechanism to eliminate boundary artifacts. To eliminate cross-modal latency overheads, our student model relies purely on visual cues, bypassing DOM tree parsing. WebOverlay significantly outperforms DOM-based heuristics and achieves competitive precision and recall against heavy proprietary MLLM baselines while delivering multiple orders of magnitude higher throughput and substantially reduced deployment costs.
[29] Grounding Image Composition with Label-Conditioned Visual Evidence cs.CVPDF
Armin Dadras, Tingyu Lin, Robert Sablatnig, Franziska Proksa, Markus Seidl
TL;DR: 论文提出了KU-PCPE,一个标签条件区域证据基准,用于摄影构图。给定图像和已判定适用的构图标签(如中心、对角线、三分法),任务分割出解释该标签为何适用的可见区域。数据集包含4,093个单标注训练掩码和1,327个专家测试任务(每个图像-标签任务三个掩码),并评估模型与人类专家之间的差距。
Details
Motivation: 现有构图标签数据集仅提供图像级标签和类特定几何元素,但未显示人们将哪些可见区域与每个构图类别关联,需要填补标签识别与视觉证据grounding之间的空白。
Result: 留出专家达到0.867 Dice,最强可部署模型0.664,proposal oracle 0.672,最佳VLM提案选择器0.395,表明当前模型与人类专家存在显著差距。
Insight: 创新点在于提出标签条件区域证据任务和基准,引入专家多掩码标注和segment-versus-line评级,将构图理解从分类推进到区域级解释,强调可解释性和人类对齐评估。
Abstract: Image composition labels such as center, diagonal, or rule of thirds describe visual structure rather than object categories. KU-PCP provides image-level labels and class-specific geometric elements, but does not show which visible regions people associate with each of its nine classes. We introduce KU-PCP Elements (KU-PCPE), a label-conditioned region-evidence benchmark for photographic composition. Given an image and a label already judged applicable, the task is to segment the visible regions that explain why the label applies. KU-PCPE contains 4,093 single-annotator training masks and a 1,327-task expert test split with three masks per image-label task. The annotations also include exploratory segment-versus-line ratings that describe whether region or semantic-guideline evidence better explains the label. A held-out expert reaches .867 Dice against the other experts, while the strongest deployable model reaches .664, a proposal oracle .672, and the best vision-language model (VLM) proposal selector .395. KU-PCPE exposes the gap between recognizing a composition label and grounding the human-aligned visual evidence that makes it plausible in a particular image.
[30] An Empirical Study on Learning Rooftop Solar Potential from Satellite Imagery Directly Without Explicit Geometry cs.CVPDF
Joshua Lee
TL;DR: 论文提出 Direct Rooftop Solar Prediction(DRSP),一种端到端方法,直接将卫星影像和粗略区域日照信息映射到应用级屋顶太阳能输出,而不依赖 LiDAR、DSM 或显式屋顶几何重建,也不需要几何中间监督。作者构建了从 Google Solar API 面板信息自动生成可用屋顶掩码与空间太阳能目标的数据流水线,得到美国中西部约 6 万和德州约 8.7 万屋顶的两个数据集,并系统分析地理训练分布、模型深度、图像分辨率、区域日照条件和热图回归目标的影响。
Details
Motivation: 传统屋顶太阳能潜力估计常依赖 LiDAR、数字表面模型或先显式重建屋顶几何的流程,成本高且流程复杂。论文希望验证能否直接从二维卫星影像学习屋顶级太阳能信息,从而简化大规模太阳能评估。
Result: 论文验证了 DRSP 的可行性,并系统刻画了地理训练分布、模型深度、图像分辨率、区域日照条件化和热图回归目标等因素的影响;结果表明可直接从二维影像学习有用的屋顶尺度太阳能信息,未给出明确 SOTA 数值结论。
Insight: 创新点在于端到端直接预测应用级太阳能输出,避免显式几何输入、几何中间预测和几何监督;同时利用 Google Solar API 构建程序化数据整理流水线,并围绕多个关键因素进行系统性实证分析。
Abstract: Rooftop solar potential estimation often relies on LiDAR, digital surface models (DSMs), or pipelines that explicitly reconstruct roof geometry before estimating solar potential. We propose Direct Rooftop Solar Prediction (DRSP), an end-to-end formulation that maps satellite imagery and coarse regional sunlight information directly to application-level solar outputs. DRSP neither requires explicit geometric representations as input nor predicts roof orientation, slope, height, DSMs, or other geometric quantities as intermediate outputs, and therefore requires no geometric intermediate supervision. To enable large-scale study of this formulation, we develop a programmatic data-curation pipeline that converts per-panel information from the Google Solar API into usable-rooftop masks and spatial solar-energy targets, yielding two geographically distinct datasets containing approximately 60K rooftops from the U.S. Midwest and 87K from Texas. We demonstrate the feasibility of DRSP and systematically characterize the effects of geographic training distribution, model depth, image resolution, regional sunlight conditioning, and heatmap regression objectives. Overall, our results suggest that useful rooftop-scale solar information can be learned directly from 2D imagery without explicit geometric intermediate representations, offering a simpler path toward large-scale solar assessment. Our code and dataset are available at https://github.com/joshualee124/direct_rooftop_solar_prediction.
[31] CompoSET: A Controlled Single-Edit Benchmark for Vision-Language Compositionality cs.CVPDF
Laure Delisle, Pietro Perona
TL;DR: 论文提出 CompoSET,一个用于评估视觉-语言组合理解能力的受控单编辑基准。每个基础图文对都有一个仅改变单一属性(如颜色、数量、空间关系或物体状态)的 foil 图像,从而在保持场景不变的情况下检验模型是否能正确绑定属性与实体。
Details
Motivation: 现有图文匹配基准中成对图像往往存在场景、布局和内容等多重差异,平均约97%像素变化,导致组合性评估被全局图像差异混淆,无法真正隔离组合理解能力。
Result: CompoSET 包含80个自然场景、1776个实例、16种编辑类型和3种描述详细程度;测试22个双编码器视觉语言模型后发现,所有模型表现均处于或低于随机水平,尽管它们在先前基准上得分较好。
Insight: 创新点在于用单编辑 foil 构造视觉上近乎相同(平均仅8%像素变化)的受控图文对,并为每个基础场景提供约22个 foil,以分离场景难度与编辑难度,从而更纯净地衡量组合性理解。
Abstract: Vision-language compositionality is the ability to identify which properties, relations, quantities, and states bind to which entities in both an image and a caption, and to match those bindings across modalities. Existing benchmarks evaluate this ability through image-text matching: a model ranks candidate image-caption match similarities and succeeds when the correct match comes on top. In such datasets, paired images differ by more than a single binding: scene, layout, and content all differ, with 97% of pixels changing on average. Such pairs do not isolate compositionality: results are confounded by global image differences. We introduce Compositional Single-EdiT (CompoSET), a benchmark dataset in which each base image-caption pair has an edited counterpart, or foil, that preserves the scene but differs by a single controlled edit, such as color, count, spatial relation, or object state. Base and foil images preserve naturalistic scene complexity yet are visually near-identical: only 8% of pixels change on average. Our dataset has many foils per base scene (~22) in order to separate scene difficulty from edit difficulty, a distinction one-foil-per-base benchmarks cannot make. CompoSET contains 1,776 instances across 80 naturalistic scenes, spanning 16 edit types in 4 semantic categories, with captions at 3 verbosity levels. We test 22 dual-encoder VLMs and find that all models perform at or below chance level on CompoSET, despite scoring well on prior benchmarks. This suggests that apparent progress under prior image-text matching protocols may be an artifact arising from confounds in benchmark design, and that compositional understanding in dual-encoder VLMs is far from solved. CompoSET addresses the confounds and offers a better controlled benchmark to measure future progress.
[32] MotorSense: A Video-EMG Dataset of Motoric Representations for Action Understanding cs.CV | cs.ROPDF
Eadom Dessalene, Michael Maynord, Amir-Hossein Shahidzadeh, Botao He, Yasamin Tabatabaee
TL;DR: 论文提出 MotorSense,这是首个大规模配对的自中心视频与表面肌电(EMG)数据集,包含 100 小时无脚本双手操作数据,并同步视频、双手 EMG、IMU 和音频。作者利用视频-EMG 跨模态预训练学习感觉运动对齐,使视觉与运动空间形成共享的动作组织,并将所得视觉模型迁移到动作识别、3D 手部重建、3D 接触重建等下游任务,还展示用 EMG 条件控制视频扩散模型生成细粒度手部操作视频。
Details
Motivation: 传统计算机视觉模型仅从视觉结果学习手部动作,缺乏 underlying embodied motor signals;而人工标注成本高、主观性强且在遮挡下不可靠。论文希望通过 EMG 提供密集、无遮挡、无标注者偏见的运动信号,弥补视觉模型缺少具身运动信息的问题。
Result: 在四类以动作为中心的计算机视觉任务和八个常用视频数据集上,基于感觉运动对齐的预训练方法在所有任务和数据集上均优于所有基线。摘要还宣称其视觉表示学习基础优于传统视频自监督学习方法。
Insight: 创新点在于构建首个大规模自中心视频-EMG 配对数据集,并利用 EMG 作为可扩展的密集运动监督信号学习视频-运动跨模态对齐。客观来看,其可借鉴之处包括将生理运动信号引入视觉预训练、以传感器信号替代昂贵人工标注,以及将学到的对齐表示用于判别式下游任务和 EMG 条件视频生成。
Abstract: Computer vision models learn hand action only from its visual consequences, without access to the underlying embodied motor signals. We address this absence with MotorSense, the first large-scale paired egocentric video and surface electromyography (EMG) dataset. MotorSense pairs bimanual EMG bands with egocentric video and yields synchronized video, bimanual EMG and IMU, and audio across 100 hours of unscripted manipulation. EMG offers a powerful scalable alternative to manual annotation: it captures finger movement, hand configuration, and the forces exerted between hand and environment as a dense signal free of occlusion and labeler bias, whereas manual annotation is costly, subjective, and unreliable under occlusion. We use MotorSense to learn sensorimotor alignment, producing an organization of action that is shared across the visual and motoric spaces, through cross-modal pre-training between video and EMG. The resultant vision models are then transferred to downstream tasks that benefit from sensorimotor alignment: in this case, action recognition, 3D hand reconstruction, and 3D contact reconstruction. We further demonstrate a generative application in which EMG motor signals are used to condition a video diffusion model, yielding videos of fine-grained hand manipulation. Across four action-centric computer vision tasks and eight popular video datasets, we demonstrate that sensorimotor alignment provides a stronger foundation for visual representation learning than conventional self-supervised learning methods over video, outperforming every baseline across all tasks and datasets evaluated.
[33] Beyond CLIP: Calibrated (Few-Shot) Defect Detection With System One Decision Models cs.CV | cs.AIPDF
Fadel M. Megahed, Ying-Ju Chen
TL;DR: 论文研究制造视觉检测中的极少样本缺陷检测,提出将Jev和Clef-Flash两种结构化决策模型用于每类仅2张参考图像的k-shot分类/检测,并提供自动精炼文本描述与缺陷概率校准方法。在五个制造案例中,该方法在相同参考图条件下准确率超过CLIP,并通过文本精炼和校准提升可靠性。
Details
Motivation: 高良率或小批量制造过程中缺陷图像极难收集,CLIP类方法虽将需求降至每类50-100张,但仍难以满足实际场景;论文旨在探索每类仅k=2张参考图像时的缺陷检测可行性。
Result: 在五个制造案例、每类2张参考图下,Jev测试准确率为0.75-1.00,Clef-Flash为0.71-0.99,均高于同条件CLIP的0.67-0.84;agentic文本精炼在8个模型-应用组合中的6个显著提升准确率,校准方法在15个配置中的13个降低Brier score。
Insight: 创新点包括面向Jev/Clef-Flash的k-shot缺陷检测框架、自动精炼指导分类的文本描述,以及校准缺陷概率以提升检测决策可靠性;客观看,其价值在于将结构化决策模型引入极低样本制造检测,并强调概率校准与可复现工具支持。
Abstract: Cameras and vision sensors are widely used for product and process inspection in manufacturing environments. Traditionally, this process required thousands of images of both nominal and defective products to train such models. Recent research shows that vision transformer (ViT) models, such as CLIP, can reduce this requirement to 50-100 images per nominal and defective class. However, collecting even 50 defective images can be difficult in high-yield or low-volume manufacturing processes. In this paper, we examine the utility of two recently proposed structured decision models (Jev and Clef-Flash) in applications using as few as k=2 reference images per class at inference. In the process, we provide three main contributions: (a) a framework for k-shot learning of these models for image classification/inspection applications; (b) an agentic approach for automatically refining the text descriptions that guide image classification in these decision models; and (c) an approach to calibrate the resulting defect probabilities to improve their reliability for inspection decisions. We test these contributions in five manufacturing case studies. Using two reference images per class, both Jev and Clef-Flash outperform CLIP’s test accuracies with the same references (0.75-1.00 [Jev] and 0.71-0.99 [Clef-Flash] vs 0.67-0.84 [CLIP]). Furthermore, we show that our agentic refinement of text descriptions significantly improves accuracy across 6 of 8 model-application combinations, and our calibration approach reduces the Brier score in 13 of the 15 model-application configurations. To support the reproducibility of our work, we provide our replication code and have created the R “decisions” package for evaluating additional models. We also provide a publicly available web app that allows practitioners to inspect their own images with no code.
[34] Hawkeye: Reading Illegible Text with Vision Language Models cs.CVPDF
Brian Tang, Kang G. Shin
TL;DR: 论文针对现代视觉语言模型在运动模糊、低分辨率等长尾退化文本识别上的不足,构建了 BSTRB 模糊场景文本识别基准,包含来自 1699 个真实视频序列的 8398 个噪声图像样本及高质量标注;并提出 Hawkeye,一个在 BSTRB 上微调、利用上下文先验和像素级对齐来阅读与推理难以辨认文本的视觉语言模型。
Details
Motivation: 现有视觉语言模型在通用场景文本识别上表现优异,但在运动模糊或低分辨率文本的长尾分布上持续失败,尤其在智能眼镜等第一视角应用中问题突出;论文旨在表征 VLM 的分辨率失效边界并提升不可读文本识别能力。
Result: Hawkeye 在 BSTRB 上整体匹配更大的视觉语言模型,并在文本高度不超过 20 像素时超过它们;方法支持单次推理,适合实时部署。
Insight: 创新点在于构建真实视频来源的模糊场景文本识别基准 BSTRB,并通过上下文先验与像素级对齐微调 VLM,使其能够阅读和推理低质量文本,同时刻画当前 VLM 开始失败的分辨率阈值。
Abstract: While modern Vision-Language Models (VLMs) have been remarkably successful in general scene text recognition, they consistently struggle with the long-tail distribution of text that is degraded by motion blur or low resolution. These limitations are particularly apparent in egocentric applications like smart glasses, where text is often captured in noisy environments. In this paper, we introduce the \textbf{BSTRB} (Blurry Scene Text Recognition Benchmark), a novel benchmark of 8,398 noisy image samples with high-fidelity annotations derived from 1,699 real-world video sequences. To support this, we present \textbf{Hawkeye}, a VLM fine-tuned on BSTRB to read and reason about illegible text by leveraging contextual priors and pixel-level alignment. Our results show that \name matches much larger VLMs on BSTRB overall and exceeds them on text at or below 20 pixels, in a single pass suited to real-time deployment. We characterize the resolution at which current VLMs begin to fail.
[35] ReVEAL: Eliciting Role-Aware Latent Visual Evidence from MLLMs for Reasoning Video Object Segmentation cs.CVPDF
Zhongyuan Bao, Cunjie He, Lejun Zhang, Zenglin Xu, Yanwei Fu
TL;DR: 论文提出 ReVEAL,一种解耦的“thinking-with-video”框架,用于推理视频目标分割(ReVOS)。它让多模态大模型在视频推理中生成角色感知的视觉证据令牌,并通过动态注意力层路由聚合为角色特定 grounding 证据,直接引导冻结的 SAM2 分割,而无需将 MLLM 与分割模型进行昂贵的跨模型对齐训练。
Details
Motivation: 现有基于 MLLM 的 ReVOS 方法通常依赖昂贵的跨模型对齐,并将 MLLM 训练绑定到下游分割模型,限制了解耦性和泛化能力;论文希望从 MLLM 推理中直接提取可靠的潜在视觉证据来指导分割。
Result: 在五个 benchmark 上,ReVEAL 优于无对齐基线,并与基于对齐的方法保持竞争力;摘要未给出具体数值。
Insight: 创新点在于引入 Visual Evidence Tokens 将视频区域/实体与目标、背景、干扰物角色关联,并用 Dynamic Attention Layer Router 跨层聚合角色特定注意力证据,辅以 Contrastive Attention Grounding Loss;客观看,其解耦式设计使 MLLM 推理证据可直接驱动冻结分割模型,避免跨模型对齐是其可借鉴之处。
Abstract: Reasoning video object segmentation (ReVOS) requires segmenting the target specified by an implicit query throughout a video. Existing MLLM-based methods typically require costly cross-model alignment, tying MLLM training to a downstream segmentation model. We present ReVEAL, a decoupled thinking-with-video framework for ReVOS that elicits role-aware latent visual evidence from MLLM reasoning to directly guide segmentation. ReVEAL introduces Visual Evidence Tokens (VETs) into video reasoning to associate necessary visual regions or entities with target, background, or distractor roles, while a Dynamic Attention Layer Router adaptively aggregates VET-to-video attention across layers into role-specific grounding evidence. We train the MLLM to perform VET-augmented reasoning on our constructed ReVEAL-47K dataset, with a Contrastive Attention Grounding Loss as auxiliary supervision to apply role-specific constraints to VET attention and optimize layer routing for reliable latent grounding. At inference, ReVEAL identifies the target through this thinking-with-video process and fuses the resulting role-specific evidence to guide a frozen SAM2 in predicting target masks. Experiments on five benchmarks show that ReVEAL outperforms alignment-free baselines and remains competitive with alignment-based methods.
[36] SketchDirector: Complex Scene Sketching via Agentic Layout Reasoning and Detail Refinement cs.CVPDF
Yiqi Liang, Xiaodong Cun
TL;DR: 论文提出 SketchDirector,一个面向文本引导多概念复杂场景草图的智能体框架。它先用 LLM 将场景提示分解为多个由 Bézier 曲线表示的可编辑草图资产,再通过带自进化记忆的智能体推理循环规划空间布局,最后利用扩散先验进行细节细化并保持几何结构。论文还构建了 MCSketch8K 数据集,实验表明其在语义和构图质量上优于现有基线。
Details
Motivation: 现有草图生成方法多依赖图像引导,且主要局限于单个对象或简化场景,文本引导的复杂场景草图生成尚未被充分探索。论文旨在自动生成多概念、构图合理的复杂场景草图。
Result: 在 MCSketch8K 相关实验中,SketchDirector 在语义质量和构图质量上优于 SOTA 基线,并在细节细化过程中有效保持整体轮廓组织;摘要未给出具体定量数值。
Insight: 论文宣称的创新点包括:用 LLM 将场景分解为 Bézier 曲线表示的草图资产、基于自进化记忆的智能体布局推理、结合扩散先验的细节细化,以及推出 MCSketch8K 数据集。客观上看,其创新在于将复杂场景草图生成拆解为“语义分解—布局规划—细节增强”的智能体式流程,并兼顾可编辑矢量表示与结构保持。
Abstract: Existing sketch generation methods often rely on image guidance and are largely limited to individual objects or simplified scenes, leaving text-guided complex scene sketching underexplored. We present SketchDirector, an agentic framework that mimics the human sketching workflow for automatic multi-concept scene sketching. Given a scene-level text prompt, SketchDirector first employs a large language model~(LLM) to decompose the scene into multiple draft assets, where each is represented by a set of Bézier curves. Then, an agentic reasoning loop iteratively plans and executes their spatial arrangement, guided by self-evolving memory. Finally, a detail refinement module leverages diffusion priors to enhance sketch quality while preserving geometric structure. To support future research and applications in multi-concept scene sketch generation, we further introduce the \textit{MCSketch8K} dataset, comprising editable vector assets, compositional layouts, and object-level manipulation annotations. Extensive experiments show that SketchDirector outperforms state-of-the-art baselines in semantic and compositional quality while effectively preserving overall contour organization during refinement. The project page is: \href{https://yiki77.github.io/SketchDirector/}{https://yiki77.github.io/SketchDirector/}.
[37] GenDSR: Transferring Representations from Video Generation Models for Dynamic Spatial Reasoning cs.CV | cs.AIPDF
Ke Yang, Zhenyu Zhang, Jun Li
TL;DR: 论文提出 GenDSR 框架,将视频生成模型(VGM)的中间时空表示迁移到视觉语言模型(VLM)中,用于动态空间推理。方法不把 VGM 特征当作单一先验,而是构建保留整体生成上下文的 raw 分支和强调时间变化的 delta 分支,再通过任务条件化的 tokenwise 门控融合并注入原生视觉表示。
Details
Motivation: 动态空间推理要求 VLM 在保持场景状态的同时追踪空间关系随时间的变化,这对现有 VLM 仍然困难;而视频生成模型在建模时间连贯的世界动态方面具有丰富的时空表示,可作为可迁移的先验知识。
Result: 在 DSR-Bench 上,GenDSR 达到 67.2%,超过此前最佳结果,取得新的 state of the art。
Insight: 创新点在于将 VGM 的中间表示迁移到 VLM,并通过 raw 分支与 delta 分支分别建模整体生成上下文和时间变化,再利用任务条件化的 tokenwise 门控进行融合注入,从而增强动态空间推理能力。
Abstract: Dynamic spatial reasoning is essential for connecting visual intelligence to thephysical world, yet remains challenging for vision-language models (VLMs), which must preserve scene state while tracking how spatial relations evolve over time. Meanwhile, video generation models (VGMs), trained to synthesize temporally coherent world dynamics, offer a rich source of spatiotemporal representations. We present GenDSR, a framework that transfers intermediate representations from VGM into VLM for dynamic spatial reasoning. Rather than treating a VGM feature as a monolithic prior, GenDSR constructs two complementary branches: a raw branch preserving holistic generative context and a delta branch emphasizing temporal variation. Task-conditioned tokenwise gates then fuse both signals and inject the result into the native visual representation. On DSR-Bench, GenDSR achieves 67.2%, surpassing the previous best result and establishing a new state of the art.
[38] Queryable Gaussian View Index: Text Retrieval and Selective Camera Localization cs.CVPDF
Tianyu Xiong, Rui Li, Jiaqi Yang
TL;DR: 论文提出 QGVI,用于查询已有 3D Gaussian 场景资产及其标定参考数据,包含文本-视角检索 QGVI-T 和选择性 RGB 相机定位 QGVI-I。QGVI-T 将 Gaussian 渲染视图构建为可语言检索的场景位置索引,绑定结构化描述与已知位姿/地点标识;QGVI-I 基于 LoGS 几何估计并用不同内点查询关键点过滤,输出资产坐标系下的位姿。实验在九个开发场景上验证检索与定位性能,并报告低延迟和低误接受/误拒绝率。
Details
Motivation: 动机是让已有 3D Gaussian 场景资产可被文本查询,并在不进行在线渲染、不调用 VLM、不训练场景特定位姿网络的情况下,实现高效文本-视角检索和选择性 RGB 相机定位。
Result: 在 9 个开发场景上,QGVI-T 的联合相关性与有效性 Top-1/Top-5 为 58.3%/78.3%,高于 CityLoC 重实现的 26.1%/34.6%;QGVI-I 在 1 m/2° 阈值下场景平均位姿成功率达 97.3%,合并误接受/误拒绝率为 0.707%/1.142%,文本查询中位延迟 0.020 s。
Insight: 创新点在于将 Gaussian 渲染视图离线组织为可语言检索的索引,并把结构化 caption、相机位姿和地点 ID 绑定,避免在线渲染/VLM/场景特定位姿网络;QGVI-I 用不同内点查询关键点过滤 LoGS 几何估计且省略光度优化,提升选择性定位的可靠性。
Abstract: We present Queryable Gaussian View Index (QGVI) for querying existing Gaussian scene assets and their associated calibrated reference data. Text-view retrieval and selective RGB camera localization return poses in the asset’s coordinate frame. QGVI-T turns Gaussian-rendered views into a language-searchable index of scene locations, binding structured captions to known camera poses and place identifiers. Semantic and lexical retrieval select views from distinct places without online rendering, VLM calls, or a scene-specific pose network. The image route, QGVI-I, filters LoGS-based geometric estimates using distinct inlier query keypoints. It omits photometric refinement and preserves accepted pose coordinates. On nine development scenes, QGVI-T achieves 58.3% Top-1 and 78.3% Top-5 joint relevance-and-validity success, compared with 26.1% and 34.6% for our CityLoC reimplementation with refinement. QGVI-I achieves 97.3% scene-mean pose success at 1 m/2 degrees, with pooled false-acceptance and false-rejection rates of 0.707% and 1.142%. A paired cached evaluation retains all 1,775 initializer poses correct at this threshold. Recorded median text query latency is 0.020 s. Matched controls show competitive BM25 retrieval and slightly lower false acceptance for thresholded full LoGS; grid augmentation suggests modest gains on novel-view-derived descriptions. We report model-judged semantic success separately from source-camera recovery, with source-verified or estimated scales for geometric evaluation.
[39] Parse Before You Execute: Efficient Spatial Reasoning with Executable Primitives cs.CVPDF
Xiwei Liu, Yulong Li, Xinling Zhuang, Xuhui Li, Yichen Li
TL;DR: 论文提出免训练的智能体式空间推理范式 PEPS,针对工具集成式视觉语言模型在空间推理中“查询编译承诺不足”的问题,先解析查询为紧凑的可执行需求,再通过固定几何 API 和确定性计算选择性执行,并由验证器检查执行充分性。方法在多个空间推理基准上提升准确率,同时减少迭代修复轮数、API 成本和端到端延迟。
Details
Motivation: 现有工具集成式空间推理常把查询编译为代码、约束、工具计划或证据等开放式产物,执行前需要解决的要求不明确,导致不必要的工具调用和迭代精炼,并可能从 incomplete spatial commitments 生成看似合理但不可靠的推理轨迹。
Result: 在多个挑战性空间推理基准上,PEPS 相比强工具集成基线持续提升推理准确率,并显著降低精炼轮数、API 成本和端到端延迟,表明显式需求承诺可同时提升可靠性和执行效率。
Insight: 核心创新是 parse-before-execute 原则与 FESM 类型化模式(Frame、Entity、State、Metric):解析器显式承诺可执行需求,执行器仅按需 grounding 几何原语,验证器检查执行充分性以支持早停或定向修复,从而减少开放式工具使用和冗余迭代。
Abstract: Spatial reasoning in Vision-Language Models (VLMs) has increasingly benefited from tool-integrated pipelines that ground computation in explicit geometry. However, existing systems often compile queries into relatively open-ended artifacts, such as code, task constraints, tool plans, or gathered evidence, leaving what must be resolved before execution under-specified. This can lead to unnecessary tool acquisition and iterative refinement, while still producing plausible reasoning traces from incomplete spatial commitments. We identify this failure mode as under-committed query compilation. To address it, we propose PEPS, a training-free agentic paradigm Parsing Executable Primitives for Spatial reasoning under a typed FESM schema over Frame, Entity, State, and Metric. PEPS follows a parse-before-execute principle: the Parser commits each query to a compact set of executable requirements; the Executor selectively grounds the required primitives through fixed geometric APIs and deterministic computation; and the Verifier checks execution sufficiency, enabling early termination or targeted repair. Extensive experiments across challenging spatial reasoning benchmarks show that PEPS consistently improves reasoning accuracy while substantially reducing refinement rounds, API cost, and end-to-end latency over strong tool-integrated baselines. These results demonstrate that explicit requirement commitment can make tool-integrated spatial reasoning not only more reliable, but also more execution-efficient.
[40] DUGO: Dual-Scale Gaussian Memory for Online Temporal Action Localization in Egocentric Procedural Videos cs.CV | cs.LGPDF
Mohammed Aman Bhuiyan, Md. Sazzad Hossain Adib, Samiul Basir Bhuiyan, Mir Faiyaz Hossain, Syed Mohammed Shamsul Islam
TL;DR: 论文提出DUGO,一个面向第一人称程序性视频的在线时序动作定位框架。它通过双尺度时序编码器同时建模局部运动动态和长程时序依赖,并引入高斯记忆库以随机、输入条件化的高斯原型存储历史动作信息。在EGTEA、EPIC-Kitchens-100等数据集上取得SOTA,并在Muses等非第一人称基准上也表现提升。
Details
Motivation: 在线时序动作定位无法访问未来帧且不能回改预测,现有方法多依赖短时锚点上下文或单尺度历史记忆,难以捕捉多时间尺度展开和长程推理的动作。
Result: 在EGTEA和EPIC-Kitchens-100等第一人称程序性视频数据集上达到SOTA,并在Muses等非第一人称基准上取得显著提升。
Insight: 创新点在于双尺度时序编码器结合深度局部表示与全局自注意力,以及用高斯记忆库建模随机输入条件化动作原型,并在编码器和锚点查询阶段通过门控交叉注意力利用记忆。
Abstract: Online Temporal Action Localization (OnTAL) aims to detect and localize action instances from untrimmed streaming videos in real time. Unlike offline approaches, OnTAL models have no access to future frames and cannot go back and refine earlier predictions, which makes the task considerably harder given the strict real-time constraints and narrow temporal window available. Most existing OnTAL methods lean on either brief anchor-based context or a single-scale memory of past frames, which limits how well they can capture actions unfolding across multiple time scales or requiring long-range reasoning in Procedural Videos. To address these gaps, we present DUGO (Dual-Scale Encoding and Gaussian Memory Online), a new OnTAL framework that jointly models local motion dynamics and long-range temporal dependencies. At its core, DUGO uses a Dual-Scale Temporal Encoder, pairing depth-wise local representations with global self-attention so it can pick up both fine-grained temporal patterns and broader contextual cues. We also introduce a Gaussian Memory Bank, which represents action memory as stochastic, input-conditioned Gaussian prototypes, allowing for richer temporal memory through gated cross-attention applied at both the encoder and anchor-query stages. Extensive experiments on procedural egocentric datasets (EGTEA, EPIC-Kitchens-100) demonstrate that DUGO achieves state-of-the-art performance while also providing substantial improvements on non-egocentric benchmarks such as Muses. The project page is available at: https://aman554-eq.github.io/DUGO
[41] SACQ: Structure-Aware Codebook Quantization for Ultra-Low-Bit Vision-Language-Action Models cs.CV | cs.LG | cs.ROPDF
Haesung Lee, Insu Han
TL;DR: 论文提出 SACQ,一种面向超低位宽视觉-语言-动作(VLA)模型的权重向量码本量化框架,旨在降低机器人 VLA 模型的显存占用。方法包含结构感知码本拟合、动作感知混合位分配以及两阶段后量化细化,并在 GR00T N1.7 和 SimplER 等设置上验证了超低比特量化的有效性。
Details
Motivation: VLA 模型能力强但内存占用大,难以部署在资源受限机器人平台;直接从大语言模型继承的训练后量化方法在 VLA 超低精度下会退化,且未显式考虑下游操作结构和量化对策略动作的影响。
Result: 在 GR00T N1.7 上,LIBERO 四个套件平均成功率达 93.85%,语言/动作权重仅 1.5 bit,内存降低 4.83 倍;结合 INT8 激活量化和低精度注意力仍保持 93.55% 成功率,并比 BF16 快 1.51 倍。SimplER 上 2 bit 时 Visual Matching 和 Variant Aggregation 成功率分别为 63.76% 和 52.27%,其中前者高于 BF16 的 58.90%,后者略低于 BF16 的 53.45%。
Insight: 创新点在于将码本量化与 VLA 下游动作误差和层级敏感性结合:用角色特定二次度量进行结构感知码本拟合,按最终动作误差增量分配混合位宽,并通过二阶误差补偿和面向解码动作的校准细化码本与行组尺度。客观来看,其核心价值是把量化目标从重构误差转向策略动作保持,更贴合 VLA 超低比特部署需求。
Abstract: Vision-Language-Action (VLA) models enable capable robot behavior, but their large memory footprint limits deployment on resource-constrained robotic platforms. Weight quantization is a natural remedy, yet post-training quantization (PTQ) recipes inherited from large language models degrade when applied to VLAs at ultra-low precision. Existing low-bit PTQ methods often fail to explicitly account for the structure of downstream operations and allocate precision without considering its effect on the policy actions. We introduce SACQ, an ultra- low-bit weight-only vector-codebook quantization framework for VLAs. SACQ combines (1) structure-aware codebook fitting with role-specific quadratic metrics that reflect input statistics and downstream sensitivity; (2) action-aware mixed-bit allocation based on the increase in final-action error as individual layers are compressed; and (3) two-stage post-quantization refinement that reassigns codeword indices with second-order error compensation and calibrates codebooks and row-group scales against the decoded action. Using GR00T N1.7, SACQ achieves 93.85% average success across four LIBERO suites at 1.5 bits per language/action weight while reducing model memory by 4.83x. With INT8 activation quantization and low-precision attention, it retains 93.55% success while running 1.51x faster than BF16. On SimplER, SACQ further achieves 63.76% and 52.27% success under Visual Matching and Variant Aggregation at 2 bits per language/action weight, compared to 58.90% and 53.45% for BF16, respectively.
[42] Beyond Trial-and-Error: A Geospatial-Guided Active Reasoning Framework for Robust Aerial Vision-and-Language Navigation cs.CVPDF
Jiawei Ge, Wenkang Gong, Jiuxin Cao, Bo Liu, Lei Cui
TL;DR: 论文提出GeoReasoner,用于高空视角下目标小且视觉歧义的Aerial VLN任务,将传统反应式试错搜索转变为地理空间引导的主动推理。方法先进行预搜索地理空间推理以获得区域级先验并约束目标假设空间,再通过假设驱动探索主动获取判别性证据,最后用一致性感知验证机制减少过早或错误的目标确认。
Details
Motivation: 现有Aerial VLN方法沿用室内VLN的反应式范式,反复基于局部观测更新状态,未充分利用导航中稳定的地理空间结构,导致视觉相似干扰物下容易强化局部错误目标假设,定位不可靠。
Result: 在真实世界CityNav benchmark上,GeoReasoner在零样本Aerial VLN方法中取得新SOTA,并与监督方法相比具有竞争力。
Insight: 创新点在于从反应式试错转向地理空间引导的主动推理:利用区域级地理先验约束假设空间,假设驱动地主动采集判别证据,并通过一致性感知验证对抗性评估互补证据。客观看,其价值在于把稳定地理结构作为先验、将被动累积证据改为主动证据获取,并引入停止前的多源一致性验证以降低误确认。
Abstract: Aerial Vision-and-Language Navigation (VLN) requires UAVs to locate language-specified goals in large-scale urban environments, where objects often appear small and visually ambiguous from high-altitude viewpoints. Existing methods largely follow a reactive paradigm common in indoor VLN, repeatedly updating navigation states from local observations while underutilizing the geospatial structure that remains stable during navigation. As a result, visually similar distractors can cause a reactive goal search to repeatedly reinforce locally plausible goal hypotheses based on passively accumulated evidence, leading to unreliable localization. To address this, we propose GeoReasoner, a robust framework that shifts the paradigm of Aerial VLN from reactive trial-and-error to geospatial-guided active reasoning. Specifically, GeoReasoner first performs pre-search geospatial reasoning to derive region-level priors and constrain the goal hypothesis space. Building on these, it further conducts hypothesis-driven exploration to actively acquire discriminative evidence and refine competing goal hypotheses. Finally, consistency-aware goal verification adversarially assesses complementary evidence sources before stopping, hence reducing premature or erroneous goal confirmations. Experiments on the real-world CityNav benchmark demonstrate that GeoReasoner sets a new SOTA among zero-shot Aerial VLN methods while competing favorably with supervised approaches.
[43] Parameter-Efficient Fine-Tuning of Transformer Encoders for Audio-Image-Symbolic Music Retrieval cs.CV | cs.SD | eess.ASPDF
Eliseo Fuentes-Martinez, Antonio Hidalgo-Centeno, Jorge Calvo-Zaragoza, Antonio Javier Gallego
TL;DR: 论文研究面向音频、乐谱图像和符号乐谱的多模态音乐检索,系统比较全量微调、LoRA 和 QLoRA 在九个预训练音乐编码器上的参数高效微调效果,并通过“编码器适配×共享投影”的 2x2 实验分析单模态判别力提升是否能转化为多模态表示改进。结果表明 PEFT 通常能以更少参数匹配或超过全量微调,但单模态收益并不稳定迁移到多模态任务。
Details
Motivation: 基于乐谱的多模态音乐检索需要各模态内部具有判别性、跨音频/图像/符号乐谱又彼此兼容的表示;论文关注 PEFT 是否适合适配预训练音乐编码器,以及单模态性能提升能否带来更好的多模态组织。
Result: LoRA 或 QLoRA 在 18 个模态-任务组合中的 14 个被选中,常匹配或优于全量微调且优化参数更少;TriScore 上最终适配并投影的标题表示用于曲目级检索,平均 Top-5 和 Top-10 准确率分别为 55.32 和 64.68。但单模态增益对多模态分类的迁移并不一致,适配与投影在不同任务上表现为互补、冗余或部分干扰。
Insight: 创新点在于跨九种音频、图像和符号乐谱编码器系统比较全量微调与 LoRA/QLoRA,并用受控 2x2 设计解耦编码器适配与共享投影的作用;客观来看,其价值主要是揭示了 PEFT 的高效性以及“单模态判别力提升不等于多模态表示更优”这一实证结论。
Abstract: Score-based multimodal music retrieval requires representations that are both discriminative within each modality and compatible across audio recordings, sheet music images, and symbolic scores. This paper investigates the effectiveness of parameter-efficient fine-tuning (PEFT) for adapting pretrained music encoders and examines whether modality-specific improvements translate into better multimodal representations. We compare full fine-tuning, LoRA, and QLoRA across nine pretrained audio, image, and symbolic score encoders on TriScore and the Musical Theme Dataset. Encoders are first evaluated and selected through supervised unimodal classification, using frozen-encoder linear probing as a reference. We then introduce a controlled 2x2 design crossing encoder adaptation with task-specific shared projection, comparing frozen and adapted representations before and after projection. LoRA or QLoRA is selected in 14 of 18 modality-target combinations, frequently matching or outperforming full fine-tuning while optimizing substantially fewer parameters. However, these unimodal gains do not transfer uniformly to multimodal classification. Adaptation and projection are complementary for composer classification on TriScore, largely redundant for TriScore instrument and the Musical Theme Dataset instrument and ensemble tasks, and partially interfering for composer classification on the Musical Theme Dataset. The final adapted-and-projected title representation on TriScore supports piece-level retrieval, reaching average Top-5 and Top-10 accuracies of 55.32 and 64.68, respectively. Cross-modal exclusion and neighborhood analyses further reveal strong modality-specific local structure in the projected spaces. Overall, PEFT provides an effective and computationally efficient alternative to full fine-tuning, but improved unimodal discrimination alone does not guarantee better multimodal organization.
[44] Attention Maps Know When to Think: Training-Free Difficulty-Aware Chain-of-Thought for Reasoning Segmentation cs.CV | cs.AIPDF
Chanseul Cho, Jaesung Jun, Dongheon Lee, Youngmin Ro
TL;DR: 论文指出多模态大模型在推理分割中无差别使用Chain-of-Thought会引发过度思考,导致查询意图与视觉定位错位。为此提出免训练的难度感知CoT框架DA-CoT,利用注意力图熵判断输入难度并选择性触发CoT,同时引入双向反思机制在初始输出失败时翻转推理决策。方法在推理分割基准上取得SOTA并显著提升推理效率。
Details
Motivation: 现有MLLM在推理分割等任务中常默认使用CoT,但不必要的推理会引入过度思考并损害视觉 grounding。论文希望根据输入内在难度自适应决定是否使用CoT,避免过度思考和思考不足。
Result: DA-CoT在Reasoning Segmentation基准上达到state-of-the-art性能,同时显著提升推理效率。摘要未给出具体数值,但宣称兼顾性能与效率。
Insight: 创新点在于发现注意力图熵可可靠反映任务难度,并以此作为免训练的CoT触发信号;双向反思机制可在初始输出失败时自动翻转推理决策,增强对过度思考和思考不足的鲁棒性。客观来看,将模型不确定性用于自适应推理控制,兼具可解释性和部署友好性。
Abstract: Chain-of-Thought (CoT) prompting has become a standard reasoning strategy in Multi-modal Large Language Models (MLLMs), showing strong performance in complex vision-language tasks such as reasoning segmentation. However, our analysis reveals that indiscriminate CoT usage can lead to degraded performance by inducing unnecessary reasoning, a phenomenon we term overthinking. In reasoning segmentation, overthinking often leads to misalignment between the query’s intent and visual grounding. To address this, we propose Difficulty-Aware Chain-of-Thought (DA-CoT), a training-free adaptive framework that selectively triggers CoT based on the input’s intrinsic difficulty. We find that the entropy of the model’s attention maps reliably reflects task difficulty: low entropy indicates confident, localized grounding, while high entropy suggests uncertainty and potential benefit from CoT. DA-CoT uses this entropy signal to determine whether to apply CoT prompting. Additionally, we introduce a Bidirectional Rethinking mechanism that enables the model to automatically flip its reasoning decision when the initial output fails, enhancing robustness by recovering from failures induced by overthinking and underthinking. DA-CoT demonstrates its effectiveness by achieving state-of-the-art performance on the Reasoning Segmentation benchmarks while significantly improving inference efficiency.
[45] GRIT: Gradient Masking for Visually Misgrounded Tokens in Text-Rich Reasoning cs.CV | cs.AI | cs.CLPDF
Chanseul Cho, Noori Bae, Dongheon Lee, Youngmin Ro
TL;DR: 论文关注视觉语言模型在文本丰富推理任务中RLVR效果受限的问题,指出模型常出现最终答案正确但中间步骤视觉信息误感知的CVMR现象。作者通过token级注意力分析发现CVMR token与良好视觉grounding token的模式存在系统性差异,提出将视觉grounding融入RLVR的GRIT训练框架,并构建GRIT-Bench基准。
Details
Motivation: 标准RLVR仅奖励最终答案正确性,无法区分推理过程是否真正基于视觉信息,因而会同等强化正确grounded与视觉误grounded的推理,限制了文本丰富推理任务的效果。
Result: 实验表明GRIT在推理准确率和视觉grounding忠实度上均显著优于现有RLVR方法;摘要未给出具体数值或SOTA表述。
Insight: 主要创新是定义并利用CVMR:借助token级注意力模式识别视觉误grounded token,并在RLVR训练中对这类token进行干预/梯度掩码,使优化信号兼顾答案正确与视觉grounding;同时提出可系统评估文本长度和推理深度的GRIT-Bench。
Abstract: Reinforcement learning with verifiable rewards (RLVR) has recently been extended to vision-language models (VLMs), yet its effectiveness remains limited for text-rich reasoning tasks. We highlight a key factor contributing to this limitation: models frequently arrive at correct final answers despite misperceiving visual information during intermediate steps. We define this phenomenon as Correct-but-Visually-Misgrounded Reasoning (CVMR). Because standard RLVR rewards only final-answer correctness, it fails to differentiate between properly grounded and misgrounded reasoning, reinforcing both equally. Through token-level analysis, we reveal that CVMR tokens exhibit distinct attention patterns that differ systematically from well-grounded ones. Based on this observation, we propose GRIT, a training framework that incorporates visual grounding into RLVR to mitigate the impact of CVMR. Furthermore, to address the lack of text-rich reasoning benchmarks that allow for systematic evaluation across text length and reasoning depth, we introduce GRIT-Bench. Experiments demonstrate that GRIT significantly improves both reasoning accuracy and grounding fidelity over existing RLVR methods.
[46] TADP: Dual-Branch Temporally Adaptive Differential Privacy for Expression-Aware Video Anonymization cs.CVPDF
Feng Xu, Xun Li, Lars Petersson, Yulei Sui, David Ahmedt-Aristizabal
TL;DR: 论文提出 TADP,一种用于表情感知视频匿名化的时变差分隐私框架。它采用双分支分别提取身份与表情表示,在固定长度片段替换邻接关系下,用时间自适应高斯机制对两分支进行隐私化,并通过自适应 RDP 组合联合释放的隐私预算,最后将片段级 RDP 保证转换为 (ε,δ)-DP。软量化和时序处理只在隐私化之后进行,作为 DP 后处理。
Details
Motivation: 解决推理阶段生成隐私保护视频时,如何在抑制身份泄露的同时保留面部表情识别效用和视频时序稳定性。
Result: 在人脸表情数据集上,相对评估的隐私保护基线,方法在评估攻击下降低了身份泄露,并保持可用的感知质量、时序稳定性和较高的表情识别效用;形式上满足片段级差分隐私,经验上取得较好的隐私-效用权衡。
Insight: 创新点在于身份/表情双分支的隐私化设计、仅依赖同一片段内先前隐私化释放的时变高斯噪声校准,以及面向联合释放的自适应 RDP 组合;将量化与时序处理置于隐私化之后作为 DP 后处理,有助于兼顾匿名化与表情效用。
Abstract: We present TADP, a temporally adaptive differential privacy (DP) framework for inference-time privacy-preserving video generation under fixed-length clip-replacement adjacency. TADP uses a dual-branch design to extract identity and expression representations, clips and privatizes both branches with temporally adaptive Gaussian mechanisms whose noise scales depend only on preceding privatized releases within the same clip and are bounded below by history-uniform calibrated minima, and accounts for their joint release via adaptive Rényi Differential Privacy (RDP) composition. The clip-level RDP guarantee is converted to $(\varepsilon,δ)$-differential privacy, while soft quantization and subsequent temporal processing are applied only after privatization as DP post-processing. Experiments on facial expression datasets show reduced identity leakage under evaluated attacks, usable perceptual quality and temporal stability, and high facial expression recognition utility relative to the evaluated privacy-preserving baselines. TADP formally satisfies clip-level DP and empirically provides a favorable privacy–utility trade-off under the evaluated threat models and datasets.
[47] When Depth Lies: Benchmarking Vision-Language Models on Mirror-Induced RGB-D Ambiguity cs.CVPDF
Hao Yin, Tianchen Guo, Heming Du, Yan Ke, Yanbin Liu
TL;DR: 论文关注镜面反射导致RGB-D深度信息失真的问题,构建了以镜面场景为核心的RGB-D数据集,包含525个真实场景的1011张图像,并标注反射面及直接观测/镜像反射物体。基于此提出MIRA-Bench,包含8个子任务、14140个QA对,覆盖反射感知、空间推理和场景级决策三个认知层级,用于评估先进视觉语言模型在镜面诱导RGB-D歧义下的空间推理能力。
Details
Motivation: 现有视觉语言模型常假设深度数据能真实反映物理距离,但镜面会复制物体外观并扭曲深度读数,使反射物体难以与真实物体区分。现有基准很少覆盖此类反射场景,因此需要系统评估VLM在镜面诱导RGB-D歧义下的表现。
Result: 论文在MIRA-Bench上评估先进VLM,发现模型在多个任务上一致失败:难以判断物体是直接观测还是镜面反射,无法将反射实例与真实对应物匹配,并产生错误空间推理。值得注意的是,提供深度数据反而降低推理性能,表明VLM无法识别镜面场景中的错误深度。
Insight: 论文的创新点在于构建镜面反射场景的RGB-D数据集与多层级MIRA-Bench,细粒度标注反射面和反射/真实物体,并系统揭示VLM对镜面诱导深度歧义的感知与推理盲点。客观来看,该工作将“深度并非总是可靠”转化为可评测问题,为多模态空间推理和反射感知研究提供了基准与诊断工具。
Abstract: Vision-language models (VLMs) have made significant progress in spatial reasoning, particularly by integrating RGB images with depth data, which is often assumed to faithfully reflect physical distances. However, mirrors violate this assumption: they distort depth readings while reproducing object appearances, making reflections difficult to distinguish from real objects. As existing benchmarks rarely consider such reflective scenarios, VLM performance in mirror scenes remains largely unexplored. In this work, we introduce a mirror-centric RGB-D dataset from reflective scenes to evaluate the spatial reasoning capabilities of state-of-the-art VLMs in these scenarios. Specifically, we collect 1,011 RGB-D images from 525 real-world scenes, spanning diverse indoor and outdoor environments, six reflective surface types, and both daytime and nighttime conditions. All images are annotated with reflective surface region masks and mask annotations for both directly observed and mirror-reflected objects. Then, we curate a benchmark named MIRA-Bench (Mirror-Induced Reasoning Ambiguity Benchmark), comprising 14,140 QA pairs across eight sub-tasks organized into three cognitive levels: Reflection-aware Perception, Spatial Reasoning, and Scene-level Decision-making. We evaluate state-of-the-art VLMs on MIRA-Bench and find consistent failures across tasks: models misclassify whether objects are directly observed or reflected, fail to match reflected instances to their real-world counterparts, and produce erroneous spatial inferences. Remarkably, providing depth data actually degrades reasoning performance, implying that VLMs cannot identify incorrect depth in mirror scenes. The code and data are available at https://github.com/EvaYin-cloud/MIRA-Bench
[48] A Self-Supervised Pose Estimation Approach for Construction Machines cs.CVPDF
Alaa Alshubbak, Daniel Görges
TL;DR: 论文针对工程机械(如挖掘机)的姿态估计,提出一种基于对比学习的自监督三阶段模型:先在大规模无标注ACID数据集上用SimCLR预训练,再通过迁移学习和下游任务两阶段微调,从而在少量仿真数据甚至仅原始数据3.5%的小数据集上学习并迁移到真实图像。
Details
Motivation: 传统工程机械姿态估计多依赖监督学习,需要大量标注数据;论文希望减少对人工标注图像的依赖,利用大规模无标注数据和仿真数据完成骨架/姿态估计。
Result: 在真实图像上的重型工程机械姿态估计中,所提方法相比普通自监督方法,在使用ResNet-50和HRNet-W32两个骨干网络时分别提升11%和13%的准确率。
Insight: 创新点在于将SimCLR对比预训练、迁移学习微调和下游姿态估计任务组织为三阶段自监督框架,并结合大规模无标注ACID数据与仿真器生成的小数据集进行知识迁移;客观上降低了标注成本,并展示了跨域/小样本迁移到真实工程机械图像的潜力。
Abstract: Pose estimation is a computer vision task used to estimate a skeleton of dynamic systems to predict future movements. Most of the research in this direction is based on a supervised learning approach which requires a massive amount of labeled datasets. In this paper, a self-supervised three-stage model based on a contrastive learning approach is introduced for estimating a skeleton of dynamic construction machines; such as excavators without using any labeled images for the first stage. The whole model structure is divided into three stages: the pre-train stage using the SimCLR contrastive approach, and two fine-tuning stages for the transfer learning and the downstream task. The model can leverage the features and learn from a huge unlabeled dataset called ACID to two small datasets generated from NVIDIA Isaac and MATLAB Simscape simulators as well as transfer the knowledge to a smaller dataset with a ratio of 3.5% from the original ACID dataset. The results show that the proposed approach can improve the accuracy of pose estimation for heavy construction machines in real images by 11% and 13% in comparison to the normal self-supervised approach with two backbones ResNet-50 and HRNet-W32, respectively.
[49] Nereus: A Large-Scale Underwater Dataset for Fine-Grained Attribute Understanding and Grounded Counting Perception cs.CVPDF
Jingtao Deng, Zihao Jin, Yuehe Chen, Feixiang He, Shuiwang Li
TL;DR: 论文提出 Nereus,一个面向水下细粒度感知的大规模数据集,同时支持细粒度物体属性理解与基于空间定位的计数感知。数据集包含 88K 图像、2.68M 问答对,覆盖 136K 个海洋生物对象、2690 个物种和 1515 个属,并提供 65K 点标注与 67K 定位计数问答对,用于推动多模态模型在水下生态场景中的细粒度分析与推理。
Details
Motivation: 现有水下数据集多提供粗类别标签、通用描述或总数量监督,难以支持生物个体级属性分析和基于空间证据的生态推理。论文旨在构建能同时支持细粒度属性理解与定位计数感知的水下大规模数据集。
Result: 实验表明,使用 Nereus 可使代表性多模态模型在下游任务适配中优于现有水下领域数据,并为后续细粒度海洋感知研究提供基础;摘要未给出具体 benchmark 分数或明确 SOTA 结论。
Insight: 创新点在于将细粒度生物属性理解与 grounded counting 联合建模,并提供大规模、多层级物种标注、点标注和定位计数问答数据。客观来看,其价值主要在于任务设定与数据资源构建,为水下多模态感知和生态推理提供更细粒度的监督信号。
Abstract: Underwater imagery records rich ecological information and supports applications such as science communication, fishery monitoring, ecological conservation, and oceanographic research. Beyond general scene understanding, these applications require models to recognize fine-grained biological traits and reason over spatially grounded evidence. However, existing underwater datasets often provide coarse category labels, generic captions, or total-count supervision, which limits their use for organism-level analysis and ecological reasoning. We introduce Nereus, a large-scale dataset for Underwater Fine-grained Perception that jointly supports Fine-Grained Object Attribute Understanding and Grounded Counting Perception. Nereus contains 88K images and 2.68M question-answer pairs, including annotations for 136K marine organism objects across 2,690 species and 1,515 genera, as well as 65K point annotations and 67K grounded counting QA pairs. Experiments with representative multimodal models show that Nereus improves downstream adaptation over existing underwater-domain data and provides a foundation for future research on fine-grained marine perception.
[50] SAR-TS-BERT: Physics-Guided Self-Supervised Pre-Training With Polarimetric Features for SAR Time-Series Crop Classification cs.CVPDF
Jieqiu Wang, Yuhan Zhang, Bo Zheng, Shaoming Zhang
TL;DR: 论文提出SAR-TS-BERT,一种物理引导的BERT式自监督预训练框架,用于SAR时间序列作物分类。该框架将后向散射强度与极化分解特征结合为六波段输入,并通过物理引导的跨度掩码策略和双分支跨模态融合编码器进行预训练。在多个数据集上取得高精度,并显著提升少样本分类性能。
Details
Motivation: 现有多时相SAR作物分类依赖大量标注样本,而现有自监督预训练方法主要针对光学图像或通用SAR场景,未能充分利用SAR的时间后向散射动态和极化散射机制,导致标签依赖问题未有效缓解。
Result: 在爱沙尼亚(EuroCropsML)和美国(California CDL)测试集上总体精度分别达到96.78%和87.29%,上海崇明岛平均月精度达99.37%。在爱沙尼亚基准上,全监督下预训练比从头训练提升OA和Macro F1 1.17和1.59个百分点,仅用1%标注数据时提升2.58和5.19个百分点。
Insight: 创新点包括:1)物理引导的跨度掩码策略(SpanPhysics),基于时间后向散射变化率自适应掩码关键物候过渡;2)双分支跨模态融合编码器,分别建模后向散射和极化分解特征并通过双向交叉注意力融合。客观分析认为,将SAR散射物理(如Cloude-Pottier分解参数)融入自监督预训练,有效增强了模型对时间动态和极化信息的捕捉,提升了分类精度和少样本迁移能力。
Abstract: Multitemporal synthetic aperture radar (SAR) imagery provides continuous, cloud-free observations of crop phenology for agricultural monitoring in cloudy and rainy regions, yet high-accuracy classification typically requires extensive labeled samples. Although self-supervised pre-training mitigates label dependency in remote sensing, existing frameworks primarily target optical imagery or generic spatial SAR scenes, underutilizing temporal backscatter dynamics and polarimetric scattering mechanisms. We propose SAR-TS-BERT, a physics-guided BERT-style self-supervised pre-training framework for SAR time-series crop classification. SAR-TS-BERT integrates backscatter intensities and polarimetric ratio features (VV, VH, and VV/VH) with dual-polarimetric Cloude-Pottier decomposition parameters ($H/α/A$) into a six-band input representation through two core designs: 1) a physics-guided span masking strategy (SpanPhysics) that adaptively selects contiguous masked spans based on temporal backscatter change rates to prioritize key phenological transitions; and 2) a dual-branch cross-modal fusion encoder that models backscatter and polarimetric decomposition features through separate Transformer branches followed by bidirectional cross-attention fusion. On independent test sets in Estonia (EuroCropsML) and the USA (California CDL), SAR-TS-BERT achieves overall accuracies (OAs) of 96.78% and 87.29%, respectively, and attains a mean monthly OA of 99.37% on Chongming Island, Shanghai. On the Estonia benchmark, pre-training improves OA and Macro F1 by 1.17 and 1.59 percentage points over training from scratch under full supervision, and by 2.58 and 5.19 percentage points using only 1% labeled data. Incorporating SAR scattering physics into temporal self-supervised pre-training effectively enhances crop classification accuracy, parcel-level mapping consistency, and few-shot transferability.
[51] WorkIng: A Real-World and Synthetic Highway Work-Zone Benchmark Dataset for Autonomous Navigation cs.CVPDF
Xujun Xie, Dominik Rößle, Adithya Mohan, Daniel Cremers, Torsten Schön
TL;DR: 论文提出 WorkIng,一个面向高速公路作业区的双域基准与训练资源,包含真实世界 360° 多模态数据集 WorkIng-Highway 和法规感知的仿真数据集 WorkIng-CARLA,用于自动驾驶 3D 检测与端到端规划等任务评测。真实部分覆盖 32 km、12,000+ 标注帧和 227,399 个跟踪 3D 边界框;仿真部分覆盖 24 个法规感知场景、6 类典型拓扑和 6 种国际交通标准。
Details
Motivation: 高速公路作业区是自动驾驶关键长尾场景,具有非典型几何结构和地区性法规差异。现有基准难以提供 360° 多模态全场景 3D 标注,且单一真实采集站点难以覆盖多样布局与法规变体。
Result: 对代表性 3D 检测和端到端规划架构的评测表明,真实数据仍是性能提升的主要驱动因素;按场景分层的合成数据增强可在 2D 识别、3D 检测和轨迹规划上改善模型表现。
Insight: 主要创新在于构建真实与仿真互补的双域作业区基准,强调 360° 多模态全场景 3D 标注、法规感知场景生成和场景分层合成增强,为作业区自动驾驶泛化研究提供可复用评测与训练资源。
Abstract: Highway work zones are critical edge cases for autonomous vehicles (AVs), combining atypical geometries and jurisdiction-specific regulations that existing benchmarks fail to capture: real-world datasets lack 360° multi-modal, full-scene 3D work zone annotation, and no single recorded site can span the layouts and regulatory variants needed for generalizable models. We present WorkIng, a dual-domain benchmark and training resource. WorkIng-Highway provides a 360° multi-modal work zone benchmark with synchronized 128-channel LiDAR, six RGB cameras, and RTK-GPS across 32 km, yielding 12,000+ annotated frames and 227,399 tracked 3D bounding boxes. WorkIng-CARLA complements this with procedural simulation spanning 24 regulation-aware scenarios across six canonical topologies and six international traffic standards, populated by 100+ custom 3D assets. Benchmarking representative 3D detection and end-to-end planning architectures shows that real-world data remains the primary performance driver, and that scenario-stratified synthetic augmentation improves architectures across 2D recognition, 3D detection, and trajectory planning. Resources: https://github.com/cvims/WorkIng
[52] Hard Sample-Conditioned Decorrelated Attention for Robust Face Recognition cs.CVPDF
Tahar Chettaoui, Guray Ozgur, Eduarda Caldeira, Naser Damer, Fadi Boutros
TL;DR: 论文提出 ViT-HarDis,一种面向鲁棒人脸识别的硬样本条件化解耦注意力方法。它根据样本与负类中心的余弦相似度衡量难度,对高相似度难样本施加更强的注意力头解耦正则,使不同注意力头在身份歧义最大的区域学习更专精的模式。方法无需修改网络结构或增加推理开销,并在多个基准上验证有效性。
Details
Motivation: ViT 多头注意力缺乏显式正则时容易学习冗余模式;现有相关性惩罚对所有样本均匀正则,忽视人脸识别中靠近竞争身份的难样本对优化和泛化更关键。论文希望按样本难度条件化地促进注意力头解耦。
Result: 实验在多数基准上优于基线及均匀解耦方法。WebFace4M 训练下,在 IJB-B、IJB-C、TinyFace Rank-1 上达到 ViT 类方法 SOTA;MS1MV3 训练下取得最佳 IJB-B 和 TinyFace 结果,且不增加推理成本。
Insight: 创新点在于用样本难度条件化注意力头解耦:难样本获得更强头去相关约束,以提升身份歧义区域的特化;分析还表明只对最后若干 Transformer 层施加解耦优于正则化整个网络,且无需架构修改或额外推理开销。
Abstract: Multi-head attention in Vision Transformers (ViTs) allows independent attention patterns across heads, but without explicit regularization, heads may learn redundant patterns rather than complementary roles. Correlation-based penalties encourage head disentanglement, yet apply uniform regularization across all training samples. In face recognition (FR), hard samples whose embeddings lie close to competing identities are critical for optimization and generalization. Motivated by this, we propose ViT-HarDis, which conditions attention head disentanglement on sample difficulty: samples with high cosine similarity to negative class centers receive stronger head decorrelation, enforcing specialization where identity ambiguity is highest. Experiments show improvements over both the baseline and uniform disentanglement on most benchmarks. Our analysis shows that ViT-HarDis reduces the similarity of hard samples to competing class centers while decreasing inter-head attention similarity, and that applying disentanglement only to the final transformer layers outperforms regularizing the entire network. Trained on WebFace4M, ViT-HarDis achieves SOTA results among ViT-based methods on IJB-B, IJB-C, and TinyFace Rank-1, while achieving the best IJB-B and TinyFace results when trained on MS1MV3, improving FR without architectural modifications or additional inference cost. Code: https://github.com/TaharChettaoui/Hard-Sample-Conditioned-Attention-for-FR.
[53] A Patch is More Than a Token cs.CVPDF
Zixiong Wang, Haiping Ma
TL;DR: 论文指出Vision Transformer通常用一个规范token表示每个图像patch,但这只是patch在某一观测条件下的单一状态。作者提出PMTT,将patch表示扩展为以规范状态为中心的表示状态族,并构建patch级局部特征空间;通过稠密轨道视图、可见到隐藏预测、内在几何分析和流形组织,在冻结的DINOv3主干上无需神经后训练地提取更丰富的patch信息。
Details
Motivation: 标准ViT的patch token是强大但单一的接口,可能掩盖同一patch在保持身份不变条件下的其他合法状态及隐藏结构。论文旨在挖掘预训练视觉主干中未被单token接口直接暴露的更丰富patch信息。
Result: 在冻结DINOv3 S/B/L/H+上,Manifold在两个4096 anchor队列中解释70.8-79.6%的Hidden变化,高于直接Observed的55.0-72.3%,并在各尺度超过95% anchor上胜出;六项视觉能力、四个尺度下,Manifold在23/24条件下超过canonical Center,在16/24下超过Observed;MegaDepth物理对应任务中各尺度均满足Center < Observed < Manifold。
Insight: 创新点在于将patch表示从单一canonical token重构为规范中心化的表示状态族,并用可见到隐藏预测与解析流形组织来区分“获取更多信息”和“组织信息”。客观看,其价值在于揭示预训练backbone存在单token接口的信息瓶颈,并提供无需神经后训练的结构化patch表示分析框架。
Abstract: Vision Transformers represent each image patch through a canonical token produced under one observation condition. That token is a powerful interface, but it is still one realized state of the patch. We introduce PMTT, which establishes two foundations: a patch representation object can extend from one canonical state to a canonical-centered Representation State Family, and identity-preserving variation opens Patch-Level Local Feature Space. We develop four parts of this representation program. Patch Information Acquisition and Dense Orbital Views expose lawful same-patch states. Visible-to-Hidden prediction tests whether observed states reveal structure beyond themselves. Intrinsic analysis identifies low-dimensional, energy-concentrated, patch-specific, predictive geometry. Representation Organization separates acquiring more information from organizing that information, with an analytic Manifold as the central structured representation. We realize PMTT on frozen DINOv3 S/B/L/H+ backbones without neural post-training of the representation. Across two independent 4,096-anchor cohorts, Manifold explains 70.8-79.6% of Hidden variation, compared with 55.0-72.3% for directly Observed states, and wins on more than 95% of anchors at every scale. Effective rank falls from about 2.5 to 2.0 while dominant energy concentration rises with backbone scale. Across six visual capabilities and four scales, Observed exceeds the canonical Center in 21 of 24 conditions, Manifold exceeds Center in 23, and Manifold exceeds Observed in 16. The strongest current consequence appears in physical correspondence, where MegaDepth satisfies Center < Observed < Manifold at every scale. These results show that pretrained visual backbones contain richer patch information than the standard single-token interface directly exposes. Code and experimental assets are available at https://github.com/zixiongwang-ai/PMTT.
[54] Open-Weight MLLMs Cannot Compare What They Cannot Name cs.CVPDF
Wenxin Su, Anna Kreshuk
TL;DR: 论文研究开放权重多模态大模型能否用于生物图像的视觉比较,以合成蚂蚁图像中身体部位大小比较作为简单基线。实验发现SOTA模型对某些身体部位可以比较,但对另一些部位失败;不同上下文学习策略和相关物体识别分析表明,失败源于模型无法识别结构时的识别驱动接地失败。作者进一步分析视觉编码器,认为细粒度视觉结构在缺乏语义支持时可能保存较差,揭示语义识别会门控本不需要语义理解的简单视觉比较。
Details
Motivation: 随着显微图像数据规模远超人工分析能力,论文希望探索MLLM自动化生物图像分析,尤其是复杂表型的视觉比较;为此先检验一个视觉上简单的基线任务,即比较合成蚂蚁身体部位大小。
Result: 在合成蚂蚁身体部位大小比较任务上,SOTA开放权重MLLM对部分身体部位失败、对另一些成功,尽管任务本身视觉上很简单;不同ICL策略和相关物体识别任务分析支持识别驱动的接地失败,但摘要未给出具体定量指标或benchmark数值。
Insight: 论文宣称的创新点是指出语义识别会门控简单视觉比较,并提出识别驱动接地失败的解释;客观看,其价值在于用受控合成任务系统揭示MLLM的细粒度视觉接地与比较缺陷,并将物体识别、ICL策略和视觉编码器分析联系起来,为生物图像自动化分析提供警示。
Abstract: Multimodal Large Language Models (MLLMs) are increasingly capable of understanding images and performing complex visual reasoning, suggesting the possibility of using them to automate biological image analysis. This need is becoming increasingly urgent as advances in technology continue to generate ever larger and more diverse microscopy datasets, already exceeding the scale at which manual visual analysis is feasible. With the final aim to use MLLMs for visual comparison of complex phenotypes, we investigate a very simple baseline: comparison of the size of body parts in synthetic images of ants. We find that state-of-the-art open-weight MLLMs fail for some body parts, but not for others, even though the comparison itself is visually trivial for all of them. We evaluate different in-context learning strategies and analyze performance on the related object recognition task. Our results suggest a recognition-driven grounding failure: structures that the models cannot recognize are instead associated with a recognizable semantic concept, impairing the grounding and comparison of both. Further analysis of the vision encoder is consistent with the hypothesis that fine-grained visual structure may be less well preserved without semantic support. Together, our results reveal a surprising limitation of current MLLMs: semantic recognition can gate simple visual comparisons that do not intrinsically require semantic understanding. Our code and data are available at https://github.com/kreshuklab/Open-Weight-MLLMs-Cannot-Compare-What-They-Cannot-Name
[55] TwinPast: Benchmarking Historical Evidence Use in Streaming Video Understanding with Evidence Contrastive Video Pairs cs.CVPDF
Haotian Yang, Yong Chen, Cheng Peng, Haobo Wang, Zhongle Xie
TL;DR: 论文提出 TwinPast,一个用证据对比视频对检验流式视频模型是否真正使用历史视觉证据的基准。它包含350个人工审核的视频对,覆盖五类历史任务;每对固定近期视频上下文、问题和选项顺序,只改变关键历史证据及正确答案,并用PairAcc要求两种历史下都答对。作者评估34种模型/输入配置,发现现有流式系统在历史证据推理上明显弱于通用模型。
Details
Motivation: 现有流式视频基准的高准确率不能证明模型使用了视觉历史:近期帧足以回答部分历史问题,模型也可能仅凭问题和选项答对。论文希望排除这些捷径,检验模型能否随关键历史证据变化而正确改变答案。
Result: 最强评估流式系统仅达到49.14% PairAcc,而评估的通用模型为66.57%–91.43%,且差距存在于全部五类任务。Qwen3-VL-8B和DeepSeek-Flash在加入历史输入后TwinPast性能提升,并在现有基准上逆转近期帧优势。
Insight: 创新点在于用证据对比视频对控制近期上下文、问题和选项,只改变关键历史证据,从而更严格地隔离“是否使用历史证据”。案例还揭示模型即使保留相关证据,也会选错时间状态或违背事件顺序,说明需将历史记忆/检索与依据时间指代和事件关系的推理连接起来。
Abstract: High accuracy on streaming video benchmarks does not establish whether a model uses visual history. Our review of OVO-Bench and StreamingBench finds that recent frames suffice for 32% of sampled historical questions, while models also answer many questions correctly from the question and options alone. We introduce TwinPast to test whether answers correctly follow changes in history, using 350 human-reviewed video pairs spanning five historical tasks. Each pair holds the recent video context, question, and ordered options fixed while changing critical historical evidence and the correct answer. Each video is evaluated independently, and PairAcc requires correct answers for both histories. We evaluate 34 model and input configurations. For Qwen3-VL-8B and DeepSeek-Flash, historical inputs improve performance on TwinPast, reversing the recent-frame advantage on existing benchmarks. The strongest evaluated streaming system reaches 49.14% PairAcc, compared with 66.57-91.43% for the evaluated general-purpose models. This gap persists across all five tasks. Case studies further show that models select states from the wrong time or contradict recorded event order despite retaining the relevant evidence. These findings highlight the need to connect historical memory and retrieval with reasoning that uses evidence according to the question’s temporal references and event relations.
[56] FoAViS: Analyzing Learned Fixation Strategies in a Foveated Active Vision System cs.CVPDF
Valentin Hassler, Jonathan Henrich, Yohaï-Eliel Berreby, Haoze Zhu, Alexander Ecker
TL;DR: 论文提出FoAViS,一个采用对数极坐标中央凹采样和二维空间记忆的主动视觉系统,注视点由与任务联合训练的二维策略生成。系统无需改变架构即可从同一记忆全局读出做分类,或密集读出做语义分割。作者系统分析学习到的注视策略,发现其选择更具信息量且任务相关的位置,并能改善任务表示、呈现与人类注视行为相似的特征。
Details
Motivation: 动机是构建像人类视觉一样中央凹化且主动的机器系统,在组件和目标已知的受控环境中研究注视行为,进而帮助理解生物视觉。现有方法需要同时支持中央凹采样、从分类到密集预测的多任务,以及与注视策略联合训练。
Result: 摘要未给出具体benchmark和定量数值;定性上,学习到的注视策略比非学习启发式选择更有信息量、更任务相关的位置。分类和分割中,与策略联合训练的模型优于使用随机位置的模型,即使后者获得学习到的扫描路径;学习扫描路径还表现出与人类注视行为相似的经验扫描路径特征。
Insight: 创新点在于将对数极坐标中央凹采样、二维空间记忆和任务联合训练的注视策略统一起来,使同一记忆可支持分类与语义分割而无需架构改动。客观看,其价值还包括用可训练策略塑造任务表示,并以生物启发方式替代被动单遍架构,为研究目标导向注视和信息表示提供平台。
Abstract: Human vision is foveated and active: resolution is highest at the fovea and falls off towards the periphery, and fixations are actively directed to locations of interest. Building a machine system that processes information in the same way lets us study gaze behavior in a controlled setting where the system’s components and objectives are known. The emergence of gaze characteristics can therefore be examined systematically, which may in turn inform our understanding of biological vision. This requires a model that can sample like the fovea, handle various tasks from classification to dense prediction, and be trained concurrently with a fixation policy. In our study, we present such a system and study its capabilities and the fixation policies that emerge from it. Our system, FoAViS, samples visual input by adopting the established log-polar foveation and accumulates information in a 2D spatial memory. Due to its 2D structure, the same memory can be read out globally for classification or densely for semantic segmentation, without any architectural change. Fixations are drawn from a 2D map by a policy trained concurrently with the task. We find that learned fixation policies select more informative, task-specific locations than non-learned heuristics. These policies also shape the task representations in a beneficial way: for both classification and segmentation, models trained concurrently with a policy outperform models trained with random locations, even when the latter are provided with learned scanpaths. Moreover, these scanpaths exhibit similarities to human gaze behavior by capturing aspects of empirical scanpath characteristics. Our work marks an important step away from standard passive, single-pass architectures, towards biologically grounded models of vision. It opens a path for studying goal-directed gaze behavior and information representation in the visual system.
[57] MoETok: Sparse Expert Routing for 1D Visual Tokenizers cs.CVPDF
Jiahao Guo, Bao Tang, Haoxiang Cao, Jingfeng Yao, Bo Li
TL;DR: 论文提出MoETok,一种面向一维视觉分词器的稀疏专家路由方法。作者发现查询式1D分词器中的可学习查询在语义监督下会分化为少量内容查询和大量细节查询,分别负责高层语义与细粒度重建;MoETok通过稀疏激活专家为不同角色token分配专门处理路径,并联合重建与语义目标训练。实验表明其在重建、语义表示和自回归生成上优于稠密分词器,并在ImageNet及统一理解生成模型中取得1D分词器/统一离散模型中的SOTA生成表现。
Details
Motivation: 现有查询式1D视觉分词器虽灵活,但不同可学习查询如何编码图像信息尚不清楚,且所有token采用相同稠密处理难以适配语义与细节的异质角色。论文旨在显式建模查询的内容/细节分工,用稀疏专家路由提升分词器的表示与重建效率。
Result: 在多种模型规模和token预算下,MoETok在匹配监督与激活计算量时持续优于稠密分词器;ImageNet上在1D分词器家族中实现强重建、语义表示与SOTA类条件生成。扩展到统一自回归理解生成模型后,在DPG-Bench 87.61、GenEval2 40.04 GM、WISE 0.49等基准上达到统一离散模型SOTA生成,并可匹配或超过BAGEL、Qwen-Image等扩散模型。
Insight: 创新点在于揭示1D分词器中查询自发形成内容/细节互补角色,并用稀疏专家路由按token角色进行专业化处理。客观看,将MoE稀疏激活引入视觉分词器,在不显著增加激活计算下适配异质token,并通过统一自回归框架验证理解与生成协同,具有较好可扩展性和借鉴价值。
Abstract: Modern visual tokenizers aim to preserve fine-grained visual details while learning high-level semantic representations for downstream generation and understanding. Query-based 1D visual tokenizers encode images into flexible latent sequences, yet how individual learnable queries gather and encode image information remains poorly understood. We reveal that these queries spontaneously develop complementary roles under semantic supervision: a small set of Content queries captures high-level semantic information, while a larger set of Detail queries preserves fine-grained visual details for reconstruction. To adapt token processing to these heterogeneous roles, we introduce MoETok, a 1D visual tokenizer that routes tokens through sparsely activated experts. Jointly optimized with reconstruction and semantic objectives, the router learns role-specialized processing paths adapted to each token’s representation. Across model scales and token budgets, MoETok consistently improves reconstruction, semantic representation, and autoregressive generation over dense tokenizers under matched supervision and active computation. On ImageNet, it combines strong reconstruction and semantic representation with state-of-the-art class-conditional generation within the 1D tokenizer family. We further validate MoETok at scale in a unified autoregressive model jointly trained for understanding and generation with a next-token cross-entropy objective. The resulting model combines competitive visual understanding with state-of-the-art generation among unified discrete models, matching or surpassing diffusion-based models such as BAGEL and Qwen-Image on benchmarks including DPG-Bench (87.61), GenEval2 (40.04 GM), and WISE (0.49).
[58] GeoFlip: Benchmarking Geometry-Sensitive Physics in Video Generation cs.CVPDF
Yuqi Li, Junming Liu, Bowen Hu, Ruize Niu, Huiran Duan
TL;DR: 论文提出 GeoFlip,一个用于评测视频生成模型是否理解初始几何变化所引发物理结果翻转的基准。它通过控制支撑范围、物体位置和机制配置,构建成对场景与参考视频,覆盖六类物理现象,并评测 16 个图像到视频生成模型在事件正确性、结构完整性和参考相似性上的表现。
Details
Motivation: 现有视频生成模型能生成看似真实的运动,但未必能根据首帧几何差异正确预测物理演化;论文旨在检验并推动模型从视觉真实走向几何条件下的物理一致性。
Result: 基准包含 482 个场景对和 964 个参考视频;在 442 个所有模型共享条件下,最高平均事件正确率仅 27.98%,显示物理生成能力明显不足;不同物理机制上模型强弱不一,且视觉语言裁判的事件正确性和结构完整性排名相关性分别达 0.906 和 0.929。
Insight: 创新点在于用受控几何干预构造成对反事实场景,结合程序化场景构建、物理仿真与数值验证提供可验证参考结果,并从事件正确性、结构完整性和参考相似性多维度评测几何条件下的物理一致性。
Abstract: Video generation models synthesize realistic motion, but can they respond correctly to changes in initial geometry? We introduce GeoFlip, a video generation benchmark centered on physical outcome flips induced by geometry. With shared physical settings and textual prompts held fixed, GeoFlip applies controlled interventions to support extent, object placement, and mechanism configuration, constructing paired scenes with contrasting reference events, such as remaining stable versus falling or tipping. These pairs test whether models translate geometric differences in the first frame into correct physical evolution. The benchmark comprises 482 scene pairs and 964 reference videos across six physical categories, with reference outcomes established through procedural scene construction, physics simulation, and numerical validation and verification of physical events. We evaluate 16 models for image to video generation along three dimensions: event correctness, structural integrity, and similarity to reference videos. We examine evaluation consistency with independent vision language judges. On the 442 conditions shared by all models, the highest event correctness averaged across tasks is 27.98%, revealing substantial limitations in physical generation conditioned on geometry. Model strengths vary across physical mechanisms, and the three evaluation dimensions capture distinct capabilities. Typical failures include ignoring geometric changes, generating incorrect object motion, bending rigid objects, and introducing additional components. Rank correlations between judges reach 0.906 for event correctness and 0.929 for structural integrity. By combining controlled geometric interventions, verifiable physical references, and multidimensional evaluation, GeoFlip assesses progress from visual realism toward physical consistency conditioned on geometry.
[59] Any Forcing: Any-Step Generality and One-Step Stability for Autoregressive Video Generation cs.CVPDF
Jiahua Xu, Yijie Fang, Nannan Wang, Xunhao Lin, Li Cao
TL;DR: 论文提出 Any Forcing,一个面向自回归视频生成的统一框架,旨在同时实现任意采样步数的通用性与一步生成的稳定性。方法通过统一对抗判别器联合进行真实分布对齐、步间一致性对齐和首帧质量增强,并结合免训练的自适应长期记忆选择机制,缓解一步生成质量下降和长序列误差累积问题。实验表明该方法可实现高质量、稳定的一步自回归视频生成,并在单张 H100 上达到较高推理帧率。
Details
Motivation: 现有自回归视频生成方法通常受限于固定采样步数,难以灵活平衡生成质量与效率;同时一步生成存在视觉质量下降和长序列误差累积,容易导致内容漂移甚至生成崩溃。
Result: 实验表明 Any Forcing 能实现高质量且稳定的一步自回归视频生成,在单张 H100 GPU 上短视频和长视频生成分别达到 20.9 FPS 和 24.7 FPS;摘要未提及具体 benchmark 分数或明确 SOTA 对比。
Insight: 创新点在于提出统一对抗判别器,将真实分布对齐、步间一致性对齐和首帧质量增强联合优化,以兼顾灵活采样与一步生成质量;同时提出免训练的自适应长期记忆选择机制,基于全局语义相关性和局部价值贡献选择历史状态并重映射时间位置,从而稳定长视频一步生成。
Abstract: Autoregressive video generation enables efficient inference through streaming prediction. However, existing methods are typically restricted to fixed sampling steps, limiting the flexibility to balance generation quality and efficiency, while one-step generation suffers from degraded visual quality and error accumulation over long sequences, leading to content drift and generation collapse. To address these issues, we propose Any Forcing, a unified framework for achieving Any-Step Generality and One-Step Stability. Specifically, we introduce a unified adversarial discriminator that jointly performs real-distribution alignment, inter-step consistency alignment, and first-frame quality enhancement, enabling flexible sampling while preserving one-step generation quality. Furthermore, we propose a training-free adaptive long-term memory selection mechanism that selects historical states based on global semantic relevance and local value contribution, while remapping their temporal positions to preserve temporal relationships within the memory cache, thereby enabling stable one-step long-video generation. Experiments demonstrate that Any Forcing achieves high-quality and stable one-step autoregressive video generation, achieving 20.9 FPS and 24.7 FPS for short and long video generation on a single H100 GPU, respectively.
[60] Think in 360°: Benchmarking and Bootstrapping Panoramic Video Reasoning cs.CVPDF
Lidong Lu, Lidong Zou, Xinyi Mao, Jiaxin Lv, Zhenquan Li
TL;DR: 论文提出首个手工标注的全景视频理解基准 PanoVideoBench,包含 1000 个覆盖五类、15 种题型的问题及诊断性标注,用于评估模型跨视角与跨时间整合证据的能力。作者进一步构建基于真实全景视频的推理数据集 PanoVideoCoT(13,465 个 QA-CoT 对),并训练 PanoVideoReasoner,通过全景 CoT 监督微调和证据特权在线自蒸馏提升全景推理能力。
Details
Motivation: 全景视频中的关键证据可能出现在不同方向和不同时间,模型需要跨视角、跨时间进行证据关联与整合;现有模型在跨视角跟踪和球面-时间证据整合方面能力较弱,且缺乏专门基准与推理数据。
Result: PanoVideoReasoner 在 PanoVideoBench 上达到 55.7%,相比骨干模型提升 12.5 个点,并超过此前 SOTA 2.6 个点;性能提升主要集中在基准所揭示的空间、跨视角和时间理解能力上。
Insight: 创新点在于构建带诊断标注的全景视频理解基准,并利用真实全景视频中的已验证观测和成功智能体探索合成球面-时间 grounded 的 CoT 数据;训练上采用证据特权在线自蒸馏,用验证过的透视视角作为教师信号指导学生轨迹,从而将全景推理能力注入通用视频 MLLM。
Abstract: A panoramic video captures all directions around the camera, but relevant evidence can appear in different directions and at different times. Understanding it requires models to connect observations across the viewing sphere and time. We introduce PanoVideoBench, the first manually annotated benchmark for comprehensive panoramic video understanding. Its 1,000 questions span five categories and 15 question types, with diagnostic annotations for target search and panoramic capabilities. Evaluation of 11 proprietary, open-source, and panoramic-specialized models identifies cross-view tracking and sphere-time evidence integration as their weakest capabilities. To strengthen these capabilities, we construct PanoVideoCoT, the first reasoning dataset grounded in real panoramic video, through sphere-time-grounded synthesis that converts verified panoramic observations and successful agentic explorations into 13,465 QA-CoT pairs. Starting from a general video MLLM, PanoVideoReasoner learns from this corpus through Panoramic CoT SFT and Evidence-Privileged On-Policy Self-Distillation, a panoramic instantiation of Vision-OPD’s OPSD in which the teacher uses verified perspective views to guide student-generated trajectories. It reaches 55.7% on PanoVideoBench, improving the backbone by 12.5 points and the previous SOTA by 2.6. The gains concentrate on the same spatial, cross-view, and temporal capabilities exposed by the benchmark, showing that grounded sphere-time supervision can equip a general video MLLM with panoramic reasoning.
[61] Rethinking Asymmetry in Self-Supervised 3D Geometry Learning cs.CVPDF
Gyeongjin Kang, Freeman Cheng, Botao Ye, Weijie Lyu, Ming-Hsuan Yang
TL;DR: 论文重新审视自监督3D几何学习中的不对称性设计,提出不再在输入空间限制视图交互,而是在3D表示空间施加不对称:共享骨干使用所有视图建立对应,仅将留出视图排除在重建场景之外。论文还利用稀疏与稠密上下文特征对齐,以及冻结视觉基础模型特征对齐,作为无需几何标注的额外训练信号。
Details
Motivation: 现有方法为避免渲染留出视图时的平凡解,通常在输入空间通过独立分支或掩码注意力限制视图交互,且优化目标侧重渲染质量。论文希望以几何精度为主要目标,重新设计不对称性以更好学习深度和相机位姿。
Result: 摘要未给出具体数值或benchmark,但声称该方法在深度和位姿估计上优于输入空间不对称设计及先前自监督方法;各组件通过控制实验验证,且自监督预训练还能提升后续监督训练。
Insight: 创新点在于将不对称性从输入空间迁移到3D表示空间,并利用视图密度差异构造稀疏-稠密特征对齐信号,同时引入冻结视觉基础模型特征对齐;这些信号均无需几何标注,为自监督3D几何学习提供了新思路。
Abstract: Self-supervised learning relies on asymmetry between what a model observes and what it must explain. Recent methods learn 3D geometry from unposed images by predicting cameras and scene structure, then rendering held-out views. To avoid trivial solutions, they impose this asymmetry in the input space, restricting how views interact through separate branches or masked attention. We revisit this design with geometric accuracy, rather than rendering quality, as the main objective. When a held-out view lies between context views, the context already constrains the geometry it observes. We therefore impose the asymmetry in the 3D representation space, excluding held-out views only from the reconstructed scene while a shared backbone uses all views for correspondence. This improves depth and pose estimation over input-space designs. We further exploit view density: sparse and dense observations depict the same scene but provide different amounts of correspondence. Aligning sparse-context features with dense-context features supplies a geometric training signal alongside rendering, without a pretrained geometric teacher. We additionally align features with a frozen vision foundation model, which brings further gains. None of these signals requires geometric annotation, and the resulting model improves geometry estimation over prior self-supervised methods. Controlled experiments isolate the contribution of each component, and self-supervised pretraining also improves subsequent supervised training.
[62] A Better Critic Matters for Distribution Matching Distillation cs.CV | cs.LGPDF
Zongze Wu, Xianbang Wang, Zhengyang Geng, Mingdeng Cao, Zhoutong Zhang
TL;DR: 论文研究DMD少步蒸馏中critic需要多大容量和何种信息才能提供可靠训练信号。作者发现小critic配大teacher会导致学生质量下降甚至训练崩溃,提出critic与teacher联合扩展(joint scaling),并提出按学生生成步条件化的step-aware critic(STAC)。二者结合在ImageNet类条件生成、SDXL/FLUX.2文生图和Wan2.1文生视频上一致改进少步蒸馏。
Details
Motivation: DMD利用teacher score与online critic估计的学生分布score之差训练少步student,因此critic的可靠性至关重要;现有critic可能容量不足,或被迫建模与步无关的学生输出混合分布,限制大teacher收益和蒸馏质量。
Result: 在七个SiT模型规模的ImageNet实验上,固定小critic而增大teacher会降低学生质量并导致训练崩溃;联合扩展可避免崩溃并让学生受益于更大teacher。联合扩展+STAC在多个生成任务上一致提升少步蒸馏,但摘要未给出具体数值或明确SOTA对比。
Insight: 论文宣称的创新是joint scaling与STAC;客观看,其创新在于将DMD训练信号瓶颈拆解为critic容量与信息条件两个维度,并用步条件化score估计替代步无关混合建模,属于有针对性的诊断式改进。
Abstract: Distribution Matching Distillation (DMD) trains a few-step student using the difference between a teacher’s score and an online critic’s estimate of the student distribution’s score. We investigate what the critic needs, in capacity and in information, to provide a reliable training signal. On the capacity side, controlled experiments across seven SiT model sizes on ImageNet show that increasing teacher size while keeping a small critic fixed can degrade student quality and cause training collapse. Scaling the critic alongside the teacher, a strategy we call joint scaling, prevents collapse in our experiments and allows the student to benefit from larger teachers. On the information side, rather than forcing the critic to model a step-agnostic mixture of student outputs, we introduce a step-aware critic (STAC) that conditions on the student’s generation step to estimate step-specific scores. Together, joint scaling and STAC consistently improve few-step distillation across class-conditional ImageNet generation, text-to-image generation with SDXL and FLUX.2, and text-to-video generation with Wan2.1.
[63] ReLift: Dense 3D Tracking as Lifted Trajectory Repair cs.CVPDF
Assaf Singer, Maroun Khriesh, Noam Rotstein, Yoni Kasten, Or Litany
TL;DR: 论文提出ReLift,一个高效前馈框架,将单目视频的密集3D跟踪重新表述为几何轨迹修复与补全。它先利用预训练2D跟踪器和3D几何模型生成提升轨迹,再通过轨迹校正器(沿轨迹和跨轨迹交替注意力)修复对应和深度误差、补全遮挡运动;仅用模拟提升误差的3D轨迹训练,可即插即用地改进不同跟踪器和几何模型。在WorldTrack上达到SOTA。
Details
Motivation: 现有密集3D跟踪方法联合学习点对应、深度和遮挡轨迹,但4D监督稀缺导致学习困难;虽然预训练2D跟踪器和3D几何模型生成的提升轨迹已具竞争力,但其继承对应和深度误差,且遮挡下深度来自遮挡物而非被跟踪点,因此需要专门修复。
Result: 在WorldTrack上,ReLift取得state-of-the-art平均准确率,所有点APD比最强基线高2.1,动态点高3.0;更快配置仍与领先3D跟踪器竞争,且运行速度快于4RC。
Insight: 创新点在于将密集3D跟踪分解为“提升+轨迹修复”,利用预训练2D/3D模型作为基础,仅用几何轨迹训练轨迹校正器,解耦图像和特定模型,实现可重用且能受益于未来组件进步的修复模块;客观看,其沿/跨轨迹交替注意力设计有效集中4D监督于轨迹细化,避免了联合学习的难题。
Abstract: Dense 3D tracking from monocular video requires establishing point correspondences over time, estimating their depth, and inferring their trajectories during occlusion. Existing methods typically learn these tasks jointly, a demanding objective given the scarcity of 4D supervision. We show that strong pretrained 2D trackers and 3D geometry models provide an effective foundation: lifted trajectories, with occluded segments simply interpolated, already rival dedicated 3D trackers using the same geometry. However, lifted trajectories inherit correspondence and depth errors, and under occlusion, their depth comes from the occluder rather than the tracked point. We introduce ReLift, an efficient feed-forward framework that recasts dense 3D tracking as geometric trajectory repair and completion. At its core, a trajectory corrector alternates attention along and across trajectories to preserve reliable observations, repair corrupted positions, and complete missing motion. Using only geometric inputs allows us to train directly on 3D trajectories with simulated lifting errors, decoupling training from images and from any specific tracker or geometry model. Concentrating 4D supervision on trajectory refinement yields a reusable model that consistently improves lifted trajectories across trackers and geometry providers without retraining, so it benefits directly from future advances in either component. On WorldTrack, ReLift achieves state-of-the-art mean accuracy, improving on the strongest baselines by 2.1 APD on all points and 3.0 on dynamic points, with faster configurations remaining competitive with leading 3D trackers while running faster than 4RC. Visit our supplementry page for more examples and code.
[64] Latent Reasoning in Continuous Space for Unified Multimodal Models cs.CV | cs.CL | cs.LGPDF
Byungwoo Jeon, Yoonwoo Jeong, Hyunseok Lee, Minsu Cho, Jinwoo Shin
TL;DR: 论文提出面向统一多模态模型的连续空间潜在推理框架LARC,将图像生成中的推理从离散token空间迁移到连续隐状态空间,以缓解token级瓶颈。方法包含课程式监督微调和信息增益强化学习两个阶段,先逐步把token级推理转化为潜在推理,再在无真值监督下自演化推理能力。
Details
Motivation: 现有统一多模态模型难以将语言链式思维推理有效用于图像生成,导致推理增强模型无法继承大语言模型中的测试时扩展和细粒度可控性。作者将其归因于模态不匹配:推理发生在离散token空间,而图像生成发生在连续空间。
Result: 在多个文本到图像生成基准上,LARC持续提升生成质量、提示词遵循和组合保真度;在GenEval上达到SOTA,相较BAGEL提升7个百分点,并展现出明显的测试时扩展行为,在并行和串行测试时扩展策略下均优于基于token的生成方法。
Insight: 创新点在于用连续隐状态进行潜在推理,以桥接离散推理与连续图像生成之间的模态鸿沟;训练上结合课程式SFT实现token推理到潜在推理的渐进转换,并用信息增益RL在无真值监督下自演化潜在推理。客观看,其可借鉴之处是将推理空间与生成空间对齐,并通过两阶段训练兼顾迁移与自提升。
Abstract: Unified Multimodal Models (UMMs) achieve remarkable success on diverse multimodal understanding and generation tasks, yet they still struggle to effectively leverage language-based chain-of-thought reasoning for image generation. As a result, existing reasoning-augmented UMMs often fail to inherit the test-time scaling and fine-grained controllability observed in large language models. We attribute this failure to a fundamental modality mismatch: reasoning is performed in a discrete token space, whereas image generation operates in a continuous space. To bridge this gap, we introduce a Latent Reasoning framework in Continuous space (LARC), which alleviates the token-level bottleneck by reasoning over continuous hidden states. Specifically, the training framework consists of two stages: (i) curriculum supervised fine-tuning (SFT), and (ii) information-gain reinforcement learning (RL). First, the curriculum SFT stage gradually converts token-level reasoning into latent reasoning. Subsequently, the information-gain RL stage self-evolves latent reasoning without ground truth supervision. Experiments across diverse text-to-image generation benchmarks show that LARC consistently improves generation quality, prompt following, and compositional fidelity. Notably, LARC achieves state-of-the-art performance on GenEval with a 7%p gain over BAGEL and demonstrates clear test-time scaling behavior, outperforming token-based generation under both parallel and sequential test-time scaling strategies.
cs.AI [Back]
[65] Incongruence-Aware Evaluation of Multimodal Speech Emotion Recognition in Audio Large Language Models cs.AI | cs.CL | cs.MM | cs.SD | eess.ASPDF
Fengji Jing, Guy Laban
TL;DR: 论文研究音频大语言模型在语音情感识别中,音频内容与文本内容不一致是否会提高错误率。作者将文本和音频映射到共享的效价-唤醒-支配空间,用双组分高斯混合把两者距离转换为连续、无需阈值的不一致分数,并在eNTERFACE’05、MELD和CMU-MOSEI上评估三个具备音频能力的模型。结果表明,在控制真实情感和特征空间距离后,只要存在音频,不一致分数在各语料上均提高错误几率;而监督分类器在同特征上表现相反,说明问题主要在于模型如何融合原始证据。
Details
Motivation: 语音交互大模型同时接收说话内容和表达方式,但音频与文本两个通道并不总是一致。论文关注这种跨模态不一致是否会削弱语音情感识别,并进一步分析音频大模型融合多模态证据时的脆弱性。
Result: 在eNTERFACE’05、MELD和CMU-MOSEI三个数据集上评估三个音频能力模型;不一致分数在音频存在时显著提高错误几率,且在控制真实情感和特征空间距离后仍成立。监督分类器在同特征上呈相反趋势;无情感词典重建的分数复现核心结果,并在六种更简单替代中成为唯一在三个语料上都能预测错误的无标签分数。
Insight: 创新点在于提出基于共享VAD空间和双组分高斯混合的连续、无阈值不一致分数,并用人工判断验证。客观来看,论文的价值还在于揭示音频大模型的错误更多来自原始多模态证据的组合方式而非信号本身,并指出纯文本效应不稳定,强调需将音频-文本一致性纳入多模态情感识别评测。
Abstract: Voice-interactive large language models process both what a speaker says and how they say it, yet the two channels do not always agree. We ask whether that disagreement degrades speech emotion recognition. Mapping text and audio into a shared valence-arousal-dominance space, we convert the distance between them into a continuous, threshold-free incongruence score with a two-component Gaussian mixture, and validate the score against human judgement. We evaluate three audiocapable models on acted, conversational and spontaneous speech (eNTERFACE’05, MELD and CMU-MOSEI). With the model held constant across modalities, incongruence raises the odds of error whenever audio is present, in every corpus, after controlling for the ground-truth emotion and for distance in the feature space. Supervised classifiers trained on the same features behave in the opposite way, which suggests that the difficulty lies in how these models combine raw evidence rather than in the signal itself. Text-only effects are unreliable: swapping only the model that reads the transcript reverses their sign on one corpus. Rebuilding the score without the emotion lexicon reproduces the central result, and among six simpler alternatives it is the only label-free score that predicts error in all three corpora. Code, data tables and prompts are released.
[66] SciGraphSelect: Auditable Scientific Data Selection through Knowledge–Reasoning Graphs cs.AI | cs.CLPDF
MiHyeon Kim, Ji-Eun Han, Eunkyeong Lee, Jeehyun Lim, Hwijung Ryu
TL;DR: 论文提出 SciGraphSelect,一个基于参考知识图谱的可审计科学数据选择框架。它将领域知识、可复用推理操作和参考文献支撑关系组织为类型化知识-推理图,并利用概念覆盖、关系与路径共支持以及方向对齐对候选文档打分。该方法强调暴露每次选择背后的参考证据,以提升科学训练语料构建的可追溯性,并与下游科学验证形成互补。
Details
Motivation: 现有科学数据选择通常依赖相关性或效用分数,难以解释文档为何被保留。论文关注上游科学训练语料构建中缺乏可审计性和可追溯性的问题。
Result: 摘要未提及定量实验结果或 benchmark 对比,仅提出框架与评分机制,并宣称可提升科学训练语料构建的可追溯性、补充下游科学验证。
Insight: 创新点在于用类型化知识-推理图组织领域知识、可复用推理操作和参考支撑关系,并通过概念覆盖、关系与路径共支持、方向对齐进行可审计选择。客观看,将参考侧丰富关系结构与候选侧保守证据处理相结合,可降低对文档嵌入的过度解释,具有借鉴意义。
Abstract: Scientific data selection is typically driven by relevance or utility scores, providing limited insight into why a document is retained. We introduce SciGraphSelect, a reference-grounded framework for auditable scientific data selection. SciGraphSelect organizes domain-specific knowledge, reusable reasoning operations, and reference-attested relations into a typed knowledge–reasoning graph, then scores candidate documents using concept coverage, relation and path co-support, and directional alignment. To avoid over-interpreting document embeddings, rich relational structure is preserved on the reference side while candidate-side evidence is treated conservatively without assuming relation entailment. By exposing the reference-grounded evidence behind each selection, SciGraphSelect brings traceability to the upstream construction of scientific training corpora and complements downstream scientific verification.
[67] ReuScope: Auditing Data Reuse and Evaluation Sensitivity in Capsule Vision 2024 cs.AI | cs.CVPDF
Eunseob Choi, Kyeonghun Kim, Hyuk-Jae Lee, Nam-Joon Kim
TL;DR: 论文提出 ReuScope 审计协议,以 Capsule Vision 2024 为案例,联合分析数据复用与评估敏感性。研究发现训练集与验证集存在字节相同文件及共享来源录音,移除 Kvasir-Capsule 相关训练样本会使公共验证集平衡准确率从 82.5% 降至 61.2%,但在其他来源验证集上同一操作反而提升 1.38 个百分点,说明该下降反映的是对来源移除的敏感性而非泄漏导致的性能膨胀。
Details
Motivation: 针对医学影像/胶囊内镜数据集中可能存在的训练-验证数据复用和评估偏差,论文希望建立可审计流程,区分“观察到的数据复用”与“条件性能敏感性”,避免将跨划分重叠简单解释为泄漏导致的性能虚高。
Result: 在 Capsule Vision 2024 上发现字节相同文件跨越公共训练-验证边界,且所有 41 个贡献帧的 Kvasir-Capsule 录音同时出现在两个划分;十组配对种子下,移除 Kvasir-Capsule 训练行及 5 条近匹配行使公共验证集平衡准确率由 82.5% 降至 61.2%,但在其他来源验证行上同一移除操作使平衡准确率提升 1.38 个百分点。匹配对照表明样本量和类别计数不能解释该差异;探索性双重差分估计为真类概率 -0.355 个百分点,准确率的配对 t 区间跨零。
Insight: 创新点在于提出一套审计协议,将数据关系测量与两种受控比较结合:固定预测改变评估总体,以及固定评估改变训练数据;并强调每项结论需明确总体、类别支持和不确定性单位,从而区分数据复用与条件性能敏感性。
Abstract: We develop ReuScope, a protocol for auditing data reuse and evaluation sensitivity, using Capsule Vision 2024 as a case study. It combines measurements of data relationships with two comparisons: changing the evaluation population while holding predictions fixed, and changing training data while holding evaluation fixed. Byte-identical files cross the public training-validation boundary, and all 41 Kvasir-Capsule recordings that contribute frames occur in both partitions. Shared source recordings establish cross-split overlap; missing cross-recording patient identifiers prevent counting unique shared patients. Across ten paired seeds, removing all training rows attributed to Kvasir-Capsule, plus five rows from other sources flagged as near-matches to it, lowers public-validation balanced accuracy from 82.5% to 61.2%. Evaluating the same fitted models on the validation rows of the other sources reverses the comparison: removal improves balanced accuracy by 1.38 percentage points. The public-validation drop measures sensitivity to source removal, not an amount of leakage-induced inflation. Matched controls show that sample size and class counts alone do not explain the observed score differences. An exploratory experiment removes the training neighbors of validation frames that have close training matches, or equally many distant training rows, and yields a difference-in-differences estimate, relative to matched unflagged frames, of -0.355 percentage points in true-class probability; the paired-t interval for accuracy spans zero. The resulting procedure distinguishes observed reuse from conditional performance sensitivity and specifies the population, class support and uncertainty unit for each conclusion.
[68] DAMI: Dual-Axis Memory Initialization for Enhanced Long-Context Retrieval in Gated Linear Attention cs.AI | cs.CLPDF
Zhangchi Feng, Harry Jiang, Yangyifei Luo, Ray Zhang, Richong Zhang
TL;DR: 论文提出DAMI双轴记忆初始化方法,用于增强门控线性注意力中的长上下文检索能力。DAMI在训练前联合配置门控参数A_log和dt_bias:跨注意力头分配几何时间尺度以覆盖不同检索距离,跨层设计低-高-低输入敏感度剖面,并用少量激活样本校准GDN族实现,且不增加可训练参数。在GDN、KDA、GDN2、Mamba-2和RWKV-7上,DAMI提升了RULER式检索准确率,同时语言建模、常识推理和真实检索性能与基线大致相当。
Details
Motivation: 线性注意力随序列长度线性扩展,但其有限状态记忆限制长上下文检索。门控线性递归中记忆保留依赖状态衰减,而状态衰减由门参数A_log、dt_bias和输入激活共同决定;受控GDN实验发现训练后这些参数仍与初值高度相关,说明初始化十分关键。
Result: 在GDN、KDA、GDN2、Mamba-2和RWKV-7上,相比对应随机初始化基线,DAMI将平均RULER式检索准确率提升5.29至16.71个百分点;语言建模、常识推理和2K真实世界检索性能与这些基线总体相当。
Insight: 摘要宣称的创新点是双轴记忆初始化:跨头设置几何时间尺度以匹配不同检索距离,跨层设置低-高-低输入敏感度剖面,并通过小激活样本校准GDN族实现且不增加可训练参数。客观来看,该工作把门控线性注意力的记忆衰减初始化提升为长上下文检索的关键设计维度,具有无需改变架构或参数量的可迁移性。
Abstract: Linear attention scales linearly with sequence length, but its finite-state memory can limit long-context retrieval. In gated linear recurrences, memory retention depends on state decay, which is shaped jointly by gate parameters A_log and dt_bias and by input activations. In controlled Gated DeltaNet (GDN) experiments, both parameters remain highly correlated with their initial values after training, highlighting the importance of initialization. Therefore, we introduce Dual-Axis Memory Initialization (DAMI), which jointly configures these parameters before training. DAMI assigns geometric timescales across heads for varied retrieval distances and a low-high-low input-sensitivity profile across layers. A small activation sample calibrates the GDN-family implementation to match both behavioral targets without adding trainable parameters. Across GDN, KDA, GDN2, Mamba-2, and RWKV-7, DAMI improves average RULER-style retrieval accuracy by 5.29 to 16.71 percentage points over the corresponding random-initialization baselines. Performance on language modeling, commonsense reasoning, and 2K real-world retrieval remains broadly comparable to those baselines. Our code is available at https://speedai.com/research/dami.
cs.CR [Back]
[69] Safin-1.5: Safety-Native Architecture and Verifiable Reasoning for Recursive Safety Improvement cs.CR | cs.AI | cs.CLPDF
Jingnan Zheng, Cheng Jin, Ming Zhang, Huan Sun, Zhongtian Ma
TL;DR: 论文提出 Safin-1.5,一个以安全为核心架构、并通过递归安全改进训练的大语言模型,同时配备推理时形式化推理框架。方法包括持久安全状态与危险知识解耦、发现-评估-修复的安全递归提升循环,以及结合符号推理与验证的形式化推理组件。
Details
Motivation: 递归自我改进可提升模型能力,但也可能引入新风险并削弱已有安全防护。论文希望在保持通用能力和限制过度拒答的同时,让安全作为模型的内在能力随能力共同发展。
Result: 摘要称实验表明 Safin-1.5 在安全性和推理性能上均取得表现,并优于前沿模型;但未给出具体 benchmark 名称或定量指标。
Insight: 创新点在于将持久安全状态与可移除的危险知识专家解耦,以较小通用能力代价移除危险知识;用发现-评估-修复闭环递归提升安全并约束过度拒答;再通过证明义务引导的语义桥接,使符号求解器能构造可验证证明。
Abstract: Recursive self-improvement (RSI) enables large language models to advance their capabilities through successive updates, but these updates can also introduce new risks and weaken existing safeguards. In line with the AI-45° Law, we pursue Safety RSI, in which safety develops as an intrinsic capability alongside general model capabilities through continual risk discovery and repair, while preserving utility and limiting over-refusal. We introduce Safin-1.5, a model built on a safety architecture and trained through recursive safety improvement, with an inference-time harness that extends its capabilities to formal reasoning. Our approach comprises three components: i) The safety architecture combines persistent safety states with hazardous-knowledge disentanglement. The former retains and selectively invokes safety knowledge, while the latter relocates targeted hazardous knowledge into removable experts, enabling its removal with minimal impact on general capabilities. ii) Recursive safety improvement establishes a discovery-evaluation-repair loop. The loop identifies new vulnerabilities and uses experimental feedback to develop repair strategies. It accepts updates only when they improve safety while preserving general capabilities and limiting over-refusal. iii) The formal reasoning harness combines the model’s language understanding with symbolic inference and verification. Proof-obligation-guided semantic bridging recovers missing relations in formal representations, enabling symbolic solvers to construct checkable proofs under formalized premises and accepted semantic bridges. Experimental results demonstrate that Safin-1.5 achieves safety and reasoning performance, outperforming frontier models.
cs.LG [Back]
[70] FARO: Function-Space Adaptive Return-to-Risk Optimization via Cross-Block Equal-Risk-Contribution Budget Allocation cs.LG | cs.CLPDF
Xiaojun Wu, Cehao Yang, Xuhui Jiang, Chengjin Xu, Jia Li
TL;DR: 论文提出 FARO,一种面向大语言模型训练的函数空间自适应优化器。它不再手工规定注意力/MLP矩阵与其余参数分别使用哪种优化几何,而是把参数分块作为决策单位:按每个块实际反事实回报选择更新几何,并在冻结路由后依据投资组合理论中的等风险贡献(ERC)原则跨块重新分配更新预算。在 FineWeb 上训练的 GPT 风格模型上,FARO 能匹配最佳手工设计优化器,并通过 ERC 获得稳定的小幅损失改进。
Details
Motivation: 现有 LLM 训练配方依赖人工先验为不同层类型指定不同优化器几何,并且所有参数块共享一个固定全局更新预算,缺乏自适应性和风险分配视角。论文希望自动决定参数块如何更新以及预算如何分配,从而减少手工调参并提升训练稳健性。
Result: 在 FineWeb 上训练的 GPT 风格模型上,FARO-Adaptive 达到 3.2780 nats,匹配最佳手工设计配方 NorMuonAndAdam 的 3.2794 nats;FARO-ERC 在 6 个随机种子下均降低干净数据损失,平均降低 0.0136 nats,且显著性检验和置信区间支持该增益,未增加时间或内存开销。在损坏数据流下仍能降低平均损失并在多数种子上胜出。
Insight: 创新点在于把参数分块作为优化决策单元,用函数空间中的实际反事实回报自动为块分配更新几何,再以 ERC 风险平价原则重新分配冻结的块级预算;其可借鉴之处是用组合风险思想做优化器预算分配,并通过削减高风险块而非扩大安全块来获得稳健增益。
Abstract: Training a large language model is compute-bound, and current recipes divide the work between optimizers by hand: orthogonalized momentum for attention and MLP matrices, Adam-style updates for the rest. This division rests on human priors about which layer type deserves which geometry, and it fixes a single global update budget that every parameter block shares. We introduce FARO, a function-space adaptive optimizer that makes the partition of parameters into blocks the unit of decision. FARO gives each block a geometry by its realized counterfactual return, then freezes the routes and reallocates the frozen per-block budget by cross-block equal risk contribution (ERC), the risk-parity principle from portfolio theory. On a GPT-style model trained on FineWeb, the framework gives two results. First, its FARO-Adaptive arm reaches 3.2780 nats at the full schedule, matching NorMuonAndAdam (3.2794), the best hand-designed recipe: it recovers the human prior without hand-setting per-layer geometry. Second, FARO-ERC improves on that arm: it lowers the clean-data loss at every one of 6 seeds, by 0.0136 nats on average, a modest but robust gain. Its sign test, paired test, and confidence interval all agree, and it exceeds the measured run-to-run noise at no added time or memory. It works by trimming the riskiest blocks rather than expanding the safest, and it holds up under a corrupted stream, still lowering the mean loss and winning the seed majority. The source code is available at https://github.com/wxj630/FARO.
[71] No Channels Left Behind: Understanding Diffusion Acceleration through Residual Gates cs.LG | cs.CVPDF
Junha Park, Jaemyung Yu, Jaehui Hwang, Jong-Seok Lee, Dongyoon Han
TL;DR: 论文重新审视扩散Transformer训练加速方法,发现REPA和分散正则等方法会显著重组AdaLN-Zero残差门控α,并提出加速部分源于残差门控通路重塑而非仅辅助监督或特征几何。作者引入归一化参与率nPR刻画门控的通道级组织,发现更广的门控参与与更好生成质量相关,并提出一种无需教师的径向目标,通过特征范数正则促进门控参与。在ImageNet 256×256的SiT-B/2上,该方法相比分散损失改善FID并提升门参与度。
Details
Motivation: 现有扩散Transformer加速方法多被解释为额外监督、空间先验或特征几何改进,论文关注并验证这些方法对AdaLN-Zero残差门控通路的内部重组,试图从门控组织角度解释训练加速。
Result: 论文提出无需教师的径向目标,在ImageNet 256×256的SiT-B/2上相比dispersive loss改善FID,同时提升跨时间步和类别的平均门参与度;摘要未给出具体FID数值,但结果支持门控组织与加速相关。
Insight: 创新点在于识别AdaLN-Zero残差门控α的重组是扩散训练加速的可测量轴,提出nPR指标刻画门控通道组织,并设计简单无教师特征范数正则来促进更广门控参与;客观上为理解加速方法提供了门控通路视角和可操作正则方向。
Abstract: Training acceleration methods for diffusion transformers, such as REPA and dispersive regularization, are commonly explained by additional supervision, spatial-structure priors, or improved feature geometry. We revisit this view and uncover a striking internal change: the AdaLN-Zero residual gate $α$, which directly controls how conditioned updates enter the residual stream, is substantially reorganized by these methods. This observation motivates our hypothesis that acceleration arises, at least in part, from reshaping the residual gating pathway rather than solely from the feature structures explicitly imposed by auxiliary objectives. Supporting this view, accelerated models are markedly more sensitive to perturbations of $α$, and their gate geometry reflects semantic relationships among classes. We further introduce the normalized participation ratio (nPR) to characterize the channel-wise organization of the gates and find that, across REPA teachers, projector architectures, and radial-objective targets, broader gate participation accompanies better generation quality. Guided by this insight, we propose a simple teacher-free radial objective that promotes broader gate participation through feature-norm regularization, without pairwise feature comparisons. On ImageNet $256\times256$ with SiT-B/2, our method improves FID over dispersive loss while increasing mean gate participation over timesteps and classes, further supporting the link between gate organization and training acceleration. Our results identify residual-gate reorganization as a measurable axis of diffusion training acceleration and suggest a simple direction for further acceleration.
[72] Do Language Models Know What Compelling Scientific Evidence Looks Like? cs.LG | cs.AI | cs.CLPDF
Conor Igoe, Gabriel Grand, Zane Koch, Albert Bou, Eli Moss
TL;DR: 论文提出 EDB-Bio,一个包含 9.6 万个 episode 的科学推理基准,覆盖药物扰动转录组学、CRISPR 筛选和人类 GWAS 三个生物领域;每个 episode 给出一系列已发表实验结果,要求模型预测目标假设为真的概率。作者发现现有开放权重和前沿模型普遍过度不自信,Brier 分数仅与朴素基础率预测器相当,而按严格适当评分规则进行后训练能显著提升校准与区分度,并在下游序贯实验选择中取得更高效率。
Details
Motivation: 论文关注语言模型在驱动自主发现系统时,是否具备像专家科学家一样权衡多个来源证据的能力;现有模型面对统计可靠性和认知相关性不同的实证结果时,证据判断与不确定性校准可能不足。
Result: 在 EDB-Bio 上,现成开放权重和前沿模型表现出系统性过度不自信,Brier 分数与简单基础率预测器相当;经严格适当评分规则后训练后,校准和区分度显著改善。下游序贯实验选择任务中,Ibis0 模型比最佳前沿模型效率高 2.5 倍。
Insight: 论文的创新点在于构建基于真实生物医学数据的科学证据推理基准,将校准训练和不确定性分析扩展到现实科学预测场景,并用 Bayes factor 量化证据权重以分析模型信念更新、与专家相关性判断的一致性及偏差;其基准设计、严格适当评分后训练和下游实验选择评估具有借鉴价值。
Abstract: Expert scientists excel at weighing empirical findings from multiple sources that vary in both statistical reliability and epistemic relevance. To what extent are language models, which are increasingly driving autonomous discovery systems, capable of such evidence deliberation? We introduce EDB-Bio, a scientific reasoning benchmark curated from real-world data in three biological domains: drug-perturbation transcriptomics, CRISPR screens, and human genome-wide association studies (GWAS). Each of EDB-Bio’s 96k episodes presents a sequence of experimental results from published studies and asks the model to predict the probability that a target hypothesis is true. Out-of-the-box, open-weight and frontier models alike demonstrate systematic underconfidence and attain Brier scores comparable to a trivial base-rate forecaster. Post-training against strictly proper scoring rules significantly improves both calibration and discrimination. This calibration pays off downstream: in a sequential experiment selection task, our Ibis0 model is 2.5 times more efficient than the best frontier model. Throughout, quantifying evidence weights as Bayes factors reveals how models update their beliefs over time, whether they agree with expert biologists on relevance, and where they exhibit inconsistencies and biases. Our work extends calibration training and uncertainty analysis to a realistic scientific forecasting setting with applications to human cancer biology. We release all code, data, and trained models.
[73] Does Reasoning Make Language Models Agree? cs.LG | cs.CL | cs.CYPDF
Mila Gorecki, Moritz Hardt
TL;DR: 论文研究推理是否让语言模型更一致。作者区分‘推理作为干预’与‘推理作为观察行为’,在九个模型、两个开发者和三个分类任务上比较模型间一致性。结果发现,受控提高推理投入会增加同一批样本上的模型一致性,但观察到推理轨迹更长的样本反而跨模型预测更不一致,且在控制准确率差异后仍成立。
Details
Motivation: 语言模型既可能因同质化而犯相关错误,也可能因多重性而在同等准确时做出冲突决策;推理模型的兴起使推理究竟促进还是削弱模型一致性成为待解张力。
Result: 在九个模型、两个开发者和三个分类任务上,启用推理提高了同一样本上的模型间一致性;但推理轨迹更长的样本跨模型预测一致性更低。两类模式在控制准确率差异后仍存在,摘要未给出具体数值或SOTA比较。
Insight: 主要创新是区分推理作为干预与作为观察行为,并将模型一致性与准确率预期对照;客观看,该设计揭示‘要求更多推理’与‘观察到更长推理’对一致性具有相反信号,提醒评估推理模型时需区分因果干预和样本选择效应。
Abstract: Language models face two opposing concerns: monoculture, in which models err in correlated ways, and multiplicity, in which equally accurate models make conflicting decisions. The rise of reasoning models has complicated this debate: On the one hand, reasoning may increase model agreement if models anchor on the same salient features or follow similar reasoning paths. On the other hand, stochastic reasoning traces may introduce additional branch points leading to more diverse predictions. We resolve this tension by distinguishing reasoning as an intervention from reasoning as an observed behavior. We compare agreement under controlled changes in reasoning effort with agreement across observed reasoning lengths, measuring both against the agreement expected from model accuracy. Across nine models from two developers and three classification tasks, enabling reasoning increases agreement between models evaluated on the same individuals. Yet the individuals who elicit longer reasoning traces receive less consistent predictions across models. Both patterns persist after accounting for differences in accuracy. Asking models to reason more thus promotes agreement, while observing them reason longer signals disagreement.
[74] UpLoop: Training Looped Transformers via Upcycling at No Extra Pretraining Compute cs.LG | cs.CLPDF
Cheng Liang, Tengfei Zhang, Ya Zhang, Yanfeng Wang, Zhouhan Lin
TL;DR: 论文提出 UpLoop,一种在预训练阶段通过 upcycling 训练循环 Transformer 的调度方案:先训练浅层 Transformer,再复制已学层逐步增加深度,并把早期节省的计算用于后期重复执行共享中间块。该方法在最终参数量、总训练 token 数和训练 FLOPs 上与标准稠密基线完全对齐的情况下提升语言建模和下游任务表现。
Details
Motivation: 循环语言模型可通过共享权重重复计算增强推理能力,但额外循环会增加训练 FLOPs,导致性能收益成本较高。论文旨在不增加额外预训练计算的前提下训练循环 Transformer。
Result: 在多个模型规模上,UpLoop 提升了语言建模和下游性能;在 1.3B 参数下,22 个下游任务平均分从 22.99 提升到 27.52,且在监督微调后数学推理收益也可延续。
Insight: 创新点在于用 upcycling 和计算重分配实现“无额外预训练计算”的循环 Transformer 训练:通过早期浅层训练节省计算,后期复制层增长深度并重复执行共享中间块,同时保持与稠密基线相同的参数量、token 数和 FLOPs。
Abstract: Looped language models can strengthen reasoning through repeated computation with shared weights, increasing execution depth without adding independent parameters. However, each additional pass increases training floating-point operations (FLOPs), making these gains more expensive to obtain. To address this cost, we propose UpLoop, a pretraining schedule that uses upcycling to train looped Transformers while exactly matching a standard dense baseline in final parameter count, total training tokens, and training FLOPs. Specifically, UpLoop begins with a shallow Transformer, progressively grows its depth by copying learned layers, and uses the computation saved early to support repeated execution of a shared middle block later in training. Experiments across model scales show improvements in language modeling and downstream performance, with the average score across 22 downstream tasks increasing from 22.99 to 27.52 at 1.3B parameters. These gains also extend to mathematical reasoning after supervised fine-tuning.
cs.RO [Back]
[75] Mechanistic Interpretability for End-to-End Self-Driving cs.RO | cs.CV | cs.LGPDF
Manav Gagvani, Benjamin Namikas, Sivamurugan Velmurugan, Nysa Kumar, Shrey Sharma
TL;DR: 论文将机制可解释性方法引入端到端自动驾驶,在单摄像头端到端路径规划模型的内部激活上训练稀疏自编码器(SAE),发现SAE潜在变量可解释地对应路线意图、车速和刹车行为。作者进一步通过反事实图像编辑验证特征的视觉 grounding,并展示操控单个SAE潜在变量可单调改变规划输出,最后用基于SAE潜在变量的轻量适配器降低轨迹选择误差。
Details
Motivation: 端到端自动驾驶模型规划性能强但缺乏可观测性和可信度,论文希望打开其黑箱、理解内部表示,并验证可解释特征是否能实际提升下游规划。
Result: 无监督聚类可恢复路线意图且准确率达98.3%;基于SAE潜在变量的轻量适配器将轨迹选择误差降低3.44%,优于不使用SAE潜在变量的同等适配器。
Insight: 创新点在于将SAE机制可解释性系统用于端到端驾驶规划模型,并证明特定稀疏特征与场景因素和规划输出存在因果式对应关系;客观看,其价值在于把可解释性从可视化分析推进到可控干预和下游性能验证。
Abstract: End-to-end (E2E) autonomous driving models achieve strong planning performance but remain largely opaque, limiting observability and trust. We apply mechanistic interpretability to E2E driving by training Sparse Autoencoders (SAEs) on the internal activations of a simple, single-camera end-to-end path planning model. Unsupervised clustering of SAE latents reveals interpretable structure aligned with route intent, vehicle speed, and braking behavior, recovering route intent with 98.3% accuracy without supervision. Through paired counterfactual image edits, we further show that specific SAE features are visually grounded and systematically respond to scene changes such as traffic light state and stop sign presence. Steering individual SAE latents produces predictable, monotonic changes in planner outputs including lateral displacement, curvature, and braking. Finally, we demonstrate a practical application of these representations: we build a lightweight adapter trained on SAE latents that reduces trajectory selection error by 3.44%, outperforming an equivalent adapter that omits SAE latents, confirming that the sparse representation captures useful information for interpretability. Code and visualizations are available at https://mgagvani.github.io/mechanistic-interpretability-for-e2e-driving/
[76] TinyAstral: Transferring Self-Evolving Harnesses to Tiny Generalist Manipulation Agents cs.RO | cs.CVPDF
Kehong Gong, Xiaoxu Zheng, Guanli Hou, Seunghun Lee, Mengqi Guo
TL;DR: 论文提出TinyAstral,一种两阶段方法:先冻结开放权重教师模型,通过递归自我改进(RSI)让前沿LLM智能体根据教师轨迹提出harness编辑,并用配对统计检验筛选有效编辑;再冻结最终harness,采集教师带推理与失败恢复的轨迹,蒸馏出4B小模型并在同一harness中替代教师。该方法旨在把大教师模型的harness改进迁移到低推理成本的紧凑机器人操作智能体。
Details
Motivation: 基于视觉语言模型的机器人智能体不仅依赖规划器,还依赖其harness(工具、反馈、记忆和恢复逻辑)。论文关注能否将大教师模型的harness改进迁移到紧凑智能体,以降低推理计算成本并保持操作能力。
Result: 在RoboTwin上,harness进化将教师成功率从33.4%提升到63.6%,最终4B学生达到62.8%;四个harness版本上蒸馏学生均接近对应教师,且后续版本学生更强。最终4B学生在相同episode上优于开放4B智能体、开放27B的SenseNova-RoboRSI以及教师模型配三个竞争者harness的表现;在三个训练benchmark上接近教师,在三个未用于RSI或训练的benchmark上接近运行同一harness的552B规划器;部署加速器时间约为教师的五分之一。
Insight: 主要创新在于将harness自我进化与模型蒸馏解耦:用LLM智能体递归编辑harness并以配对统计检验保证改进可靠,再把进化后的harness和教师轨迹蒸馏给紧凑学生,证明harness收益可迁移到小模型且可跨其他冻结规划器迁移而无需重训练。
Abstract: Robot agents built on vision-language models depend on both the planner and its harness: the tools, feedback, memory, and recovery logic around it. Can improvements to a large teacher’s harness be carried into a compact agent with lower inference compute cost? We present TinyAstral, a two-stage approach. We first freeze an open-weight teacher and improve its harness through recursive self-improvement (RSI): frontier LLM agents read the teacher’s trajectories and propose harness edits, and each edit is accepted only if it passes a paired statistical test. We then freeze the final harness, collect teacher trajectories with reasoning and failure recovery, and distill a small student that replaces the teacher in the same harness. On RoboTwin, harness evolution raises the teacher’s success from 33.4% to 63.6%; the final 4B student reaches 62.8%. Across four harness versions, distilled 4B students score close to their corresponding teachers, and later versions yield stronger students, showing that harness evolution gains carry into compact models. The harness gains also transfer to other frozen planners without retraining. On identical episodes, the final 4B student outperforms open 4B agents, the open 27B harness-RSI model SenseNova-RoboRSI, and our own teacher model placed in three competitors’ harnesses. It stays close to its teacher on all three training benchmarks, and on three benchmarks never used for RSI or training it scores close to a 552B planner running the same harness. On the measured deployment episodes, it uses about one fifth of its teacher’s accelerator time.
[77] Joint-Health-Aware Reinforcement Learning for Robust Robot Control Under Actuator Degradation cs.RO | cs.CVPDF
Haifa Zhang, Yijing Wang, Zhiqiang Zuo
TL;DR: 论文提出一种关节健康感知的强化学习框架,用于执行器权限退化下的鲁棒机器人控制。方法用连续关节健康向量表示各关节执行器状态,将退化建模为执行动作的乘性衰减,并通过 HealthNet 从近期观测和动作估计逐关节健康度,使策略能在健康与退化关节间重新分配控制努力。
Details
Motivation: 现有鲁棒方法多依赖离散故障标签、广泛域随机化或全局潜变量,未显式建模每个关节连续残余执行能力;执行器权限退化会导致按名义动力学训练的强化学习策略失效。
Result: 在 Walker、Go2、Cassie 和 Panda/ManiSkill 操控任务的多退化掩码与严重程度下,方法优于名义训练和无健康感知的退化训练基线。相对名义基线,任务级运动回报最高提升 250.1%,操控成功率最高提升 15.2 个百分点;高退化 Cassie 组回报从 3.94 增至 40.50(+928.0%),消融支持逐关节健康估计、动作退化建模和健康条件控制的作用。
Insight: 创新点在于用连续逐关节健康向量显式表征执行器残余能力,结合动作乘性衰减建模与 HealthNet 在线估计,并让策略按健康状态重分配控制。客观看,将细粒度关节健康感知引入鲁棒强化学习,比离散故障标签或全局潜变量更具表达力和可解释性。
Abstract: Robots deployed in unstructured environments are susceptible to actuator-authority degradation, which reduces the effective control authority of individual actuators and can cause reinforcement learning policies trained under nominal dynamics to fail. Existing robustness approaches rely on discrete fault labels, broad domain randomization, or global latent variables, often without explicitly modeling continuous per-joint residual actuation capability. We propose a joint-health-aware reinforcement learning framework for robust robot control under simulated actuator degradation. Our framework encodes actuator condition in a continuous joint-health vector and models degradation as multiplicative attenuation of executed actions. A dedicated HealthNet estimator infers per-joint health from recent observations and actions, allowing the policy to redistribute control effort across healthy and degraded joints. We evaluate the method on Walker, Go2, Cassie, and Panda-based ManiSkill manipulation tasks under diverse degradation masks and severity levels. The method outperforms nominal and degradation-trained no-health baselines on the main evaluated metrics. Relative to the nominal baseline, task-level locomotion returns improve by up to 250.1% and manipulation success rates by up to 15.2 percentage points; in the high-degradation Cassie group, return increases from 3.94 to 40.50 (+928.0%). Ablations support the contribution of per-joint health estimation, action-degradation modeling, and health-conditioned control.
[78] CalibSwitch: Observation-Driven Camera-LiDAR Extrinsic Recovery and Selective Maintenance for Multimodal Perception cs.RO | cs.CVPDF
Haifa Zhang, Yijing Wang, Zhiqiang Zuo
TL;DR: 论文提出CalibSwitch框架,将相机-LiDAR绝对外参恢复与运行时选择性维护分离。CAEG无需数值初始变换即可从图像-点云观测恢复完整SE(3)候选,SRCP结合观测质量与时间历史决定何时替换冻结多模态检测器所用的工作几何。该方法在nuScenes和实车平台上验证了对外参漂移与观测波动的鲁棒性。
Details
Motivation: 移动机器人和智能车辆部署中,安装差异导致初始外参不确定,观测质量变化又使逐帧外参估计波动,影响多模态感知可靠性。论文旨在解决绝对外参恢复与运行中外参状态稳定维护的问题。
Result: 在nuScenes上,CAEG平均平移误差0.248 m、旋转误差5.884°;持续几何变化下SRCP平均复合误差1.46,候选提交率9.70%,优于相近提交率下的质量门控和时间一致性门控。相比无条件逐帧更新,冻结检测器mAP和NDS分别提升1.43和0.92点,并展示13.5 FPS的Jetson单相机运行能力。
Insight: 创新点在于将外参校准拆分为绝对候选恢复和选择性状态维护,避免逐帧更新带来的波动与开销。CAEG采用无投影双向跨模态特征交互与几何正则学习,无需数值初始变换;SRCP将观测质量与时间历史结合做候选提交决策,更符合长期部署中的几何状态维护需求。
Abstract: Camera–LiDAR perception on mobile robots and intelligent vehicles depends on reliable sensor geometry throughout deployment. Installation variations create uncertainty in the initial extrinsics, while changing observation quality makes frame-wise estimates fluctuate during operation. This paper presents CalibSwitch, an observation-driven framework that separates absolute extrinsic recovery from selective state maintenance. A cross-modal absolute extrinsic generator (CAEG) combines projection-free bidirectional feature interaction with geometry-regularized learning to recover complete SE(3) candidates from image–point observations and coarse view context, without a supplied numerical initial transform. A selective recurrent calibration policy (SRCP) combines observation quality and temporal history to determine when a candidate should replace the working geometry consumed by a frozen multimodal detector. On nuScenes, CAEG achieves mean translation and rotation errors of 0.248 m and 5.884$^\circ$. Under sustained geometric transitions, SRCP reaches a mean composite error of 1.46 with 9.70% candidate commitments, outperforming quality-based and temporal-consistency gating at comparable commitment rates. Relative to unconditional per-frame updating, it improves frozen-detector mAP and NDS by 1.43 and 0.92 points. Paired geometry–observation uncertainty tests, two real-vehicle platforms, and 13.5-FPS single-camera Jetson execution demonstrate the system value of selective geometric-state maintenance for continuous perception.
[79] Movens-Zero: Scaling Humanoid Motion Foundation Models with Large-Scale Human Videos cs.RO | cs.CVPDF
Wendong Bu, Tinghong Ye, Qizhou Wang, Jiacheng Li, Zhiqi Ge
TL;DR: 论文面向人形机器人运动基础模型的数据稀缺与多模态实时闭环控制难题,构建了从互联网视频中提取并对齐的Movens-5M数据集,包含1500万人体运动序列和500万可物理执行的人形运动序列,总时长超过1万小时。基于该数据提出Movens-Zero统一理解-生成-执行架构,通过任务混合流匹配训练单一动作专家支持文本、语音、音乐、视频到运动的生成,并用多模态实时分块实现连续闭环执行。论文还提出Movens-Intent四轨基准,验证了数据规模、动作专家容量和跨模态监督带来的收益。
Details
Motivation: 现有方法受限于多样且机器人可执行的数据不足,以及难以从异构多模态输入中联合学习并支持实时闭环控制,因此论文希望利用大规模人类视频扩展人形运动基础模型。
Result: 论文构建了超过1万小时、含1500万人体运动序列和500万可执行人形运动序列的数据集,并发布含8192个专家校验条件-运动对的Movens-Intent基准;实验显示将执行验证数据从1%扩展到100%、动作专家从64M扩展到911M参数时,文本、语音、音乐、视频四种模态上均获得一致提升,并表明数据质量与跨模态监督可改善生成和执行。
Insight: 创新点在于大规模互联网视频到可执行人形运动的数据构建与物理验证流程、统一理解-生成-执行且支持多模态联合学习的架构、跨七类任务的任务混合流匹配训练,以及面向持续指令和机器人状态更新的多模态实时分块闭环执行机制;客观看,其核心贡献是数据管线、统一动作专家和实时闭环分块策略的系统性结合。
Abstract: Scaling humanoid motion foundation models is constrained by the scarcity of diverse, robot-executable data and by the difficulty of jointly learning from heterogeneous multimodal inputs while supporting real-time closed-loop control. We present Movens-5M, constructed from raw Internet videos through clip generation, quality filtering, semantic annotation, motion recovery, and physical refinement with humanoid retargeting and execution verification. It contains 15 million human-motion sequences and 5 million physically executable humanoid-motion sequences spanning over 10,000 hours, aligned with text, RGB video, speech, and music. Building on this data, we introduce Movens-Zero, a scalable understanding-generation-execution architecture that supports joint learning across modalities and real-time closed-loop execution. Task-mixed flow-matching training jointly optimizes a single action expert across seven generation tasks, enabling direct motion generation from text, speech, music, and video. Multimodal Real-Time Chunking connects overlapping chunks under continuously updated instructions and robot states for continuous closed-loop execution. We further introduce Movens-Intent, a four-track benchmark with 8,192 expert-verified condition-motion pairs across text, speech, music, and video. Experiments show consistent gains across all four modalities when scaling execution-verified data from 1% to 100% and action-expert capacity from 64M to 911M parameters, while demonstrating that data quality and cross-modal supervision improve generation and execution.
[80] REA: A Routed Embodied Agent Runtime for Asynchronous Brain-Cerebellum Control cs.RO | cs.CVPDF
Pengfei Zhou, Yixing Ma, Zhengfeiyang Zhang, Haoyuan Chen, Zhenglin Wan
TL;DR: 论文提出 REA(Routed Embodied Agent),一个统一的具身智能体运行时,将负责高层推理的 System 2、负责路由的 System 1 和快速反应策略 System 0 协同起来,实现异步的“脑-小脑”控制。REA 通过路由记忆、边界触发判断门和共享预算升级阶梯,避免快速策略在运行时等待大模型,从而兼顾推理能力与实时控制。
Details
Motivation: 基础模型高层推理可靠但太慢、成本高,难以作为细粒度控制器;VLA 和全身控制器等快速策略反应高效但泛化不足。将审慎“大脑”与反应“小脑”结合虽自然,但现有方法栈不兼容,且异步协调困难,容易出现智能体循环阻塞快速执行、VLA 反复触发回退或漏检静默失败等问题。
Result: 在 60 个 LIBERO-Pro 任务的 3000 个测试回合上,REA 达到 95.9% 成功率,超过其策略池中最佳单智能体基线 94.7%,并减少 25% 控制步数和 17% 控制时间;在 RoboTwin 2.0 的 100 回合研究中,VLA 策略达 84%、VLA+代码达 86%,优于最强单 VLA 基线 MEDex 的 80%;还迁移到 BEHAVIOR-1K 和真实 Unitree G1 人形机器人,在对比方法中取得最高观测成功率。
Insight: 创新点在于统一运行时下让 System 2、System 1 和 System 0 策略协作,并遵循“运行中的策略不等待大脑”的原则;通过预先存储策略成功率的 route memory、在执行边界触发大脑判断的 judge gate,以及有界恢复的共享预算升级阶梯,实现异步协调和故障恢复。客观看,其价值在于把学习策略与程序化策略纳入同一运行时,并系统解决高层推理与快速反应策略之间的调度与回退问题。
Abstract: Foundation models are becoming remarkably reliable at high-level reasoning, yet they remain too slow and costly to serve as fine-grained controllers. Conversely, fast policies like VLAs and whole-body controllers offer more efficient reflexes, but struggle to generalize. While pairing a deliberative brain (foundation models) with a reactive cerebellum (fast policies) seems natural for embodied systems, two obstacles remain: methods are built and scored on incompatible stacks, and their asynchronous coordination is difficult. For example, an agent that calls the VLA in the agentic loop can stall the fast execution, while a VLA that calls the agent on errors tends to re-trigger fallbacks repeatedly or completely miss silent failures. To address this, we propose REA (Routed Embodied Agent), a unified runtime where a deliberative System 2, an embodied router System 1 and reactive System 0 policies cooperate under a single golden rule - no running policy waits on the brain. Instead of querying the brain at each failure, REA employs a route memory that stores each policy’s measured success ahead of time, a boundary-triggered judge gate that consults the brain on policy execution, and a shared-budget escalation ladder for bounded recovery. Evaluated on 3,000 test episodes across 60 LIBERO-Pro tasks, REA reaches 95.9% success and beats the best single-agent baseline in its own policy pool (94.7%) while reducing control steps by 25% and control time by 17%. In a separate 100-episode RoboTwin 2.0 study, REA achieves 84% success with VLA policies and 86% with VLA+code, compared with 80% for the strongest single VLA baseline MEDex. These results demonstrate learned and programmatic policies operating within the same runtime. REA also transfers to BEHAVIOR-1K and a real Unitree G1 humanoid, achieving the highest observed success among the compared methods.
[81] RoboArchitect: Automating the Design of Robotic Agents with Persistent Memory cs.RO | cs.AI | cs.CVPDF
Shengguang Wu, Suning Huang, Cherie Ho, Ria Doshi, Hao Zhu
TL;DR: 论文提出 RoboArchitect,用代码模型观察冻结机器人策略的 rollout、诊断失败并迭代修改智能体外壳(harness),从而自动化机器人智能体设计。该方法无需更新任何模型权重,通过进化外壳及其持久记忆提升在未见场景和任务上的成功率。
Details
Motivation: 现有机器人智能体外壳通常依赖人工设计,并需针对每个新机器人或 benchmark 重新设计;论文希望自动化这一过程,作为新设定下微调策略的替代方案。
Result: 在多个 benchmark 和策略上,RoboArchitect 将未见场景平均成功率从 9.7% 提升到 43.3%(4.5 倍),未见任务从 12.7% 提升到 40.2%(3.2 倍);在远离策略训练数据的设定下收益最大,进化外壳甚至常优于微调版本。
Insight: 创新点在于将 harness 设计视为可自动优化对象:代码模型基于 rollout 诊断失败并迭代改写控制代码,同时进化持久记忆以积累抓取深度等经验。客观看,其价值在于不更新权重即可释放冻结策略潜力,并自动发现抬升卡住机械臂、遇上方障碍改水平接近等控制策略。
Abstract: Recent work shows that a frozen robot policy performs far better inside an agent harness, where control code decides when to query the policy, consult vision tools, or command the arm directly. Yet today these harnesses are built by hand and redesigned for each new robot or benchmark. We show that harness design can be automated, offering a strong alternative to finetuning on a novel setup. In RoboArchitect, a coding model observes a robot’s rollouts, diagnoses failures, and iteratively revises the harness around the frozen policy. The harness evolves on a set of episodes for its optimization and is evaluated on held-out scenes and tasks. Without updating any model weights, RoboArchitect raises average success from 9.7% to 43.3% on unseen scenes (4.5x) and from 12.7% to 40.2% on unseen tasks (3.2x), across multiple benchmarks and policies. The gains are largest where the benchmark is far from the policy’s training data: wrapping a general-purpose policy, the evolved harness yields a specialist that often outperforms even the finetuned version. Intuitive control strategies are automatically designed, such as lifting a jammed arm clear or reaching in horizontally when an obstacle blocks the way from above. The harness also evolves its own persistent memory, carrying learned experience, such as effective grasp depths, into later decisions. These results suggest that substantial headroom in robotic manipulation lies in the agent harness, and its design can be effectively automated and optimized without weight updates.
cs.SD [Back]
[82] Apriel Voice 27B Preview: An Open Model for Speech, Reasoning, and Tool Use cs.SD | cs.CL | cs.LG | eess.ASPDF
Aman Tiwari, Akshay Kalkunte, Akintunde Oladipo, Jash Shah, Tara Bogavelli
TL;DR: 论文提出 Apriel Voice 27B Preview,一个开源的语音语言模型,在共享推理骨干中统一语音理解、推理、对话和工具调用能力,同时保留文本与图像输入。模型基于后训练的 Gemma 4 26B-A4B checkpoint,可直接处理语音并生成回复或工具调用,无需独立对话模型。作者采用两阶段训练适配语音,并结合自动构造的语音对话数据与高效在线蒸馏完成训练。
Details
Motivation: 现有语音助手不能只做语音识别,还需要理解请求、基于上下文推理、维持多轮对话并通过工具执行动作。论文希望用一个统一的语音-语言模型替代分离的语音识别、对话和工具调用模块,降低系统复杂性并提升综合交互能力。
Result: 在 thinking 模式下,Apriel Voice 在 Big Bench Audio 上得分 98.90,在 Speech-BFCL 上得分 81.13,在 τ-Voice 上任务成功率为 65.51%,平均延迟 1.91 秒;同时取得 51.27% 的 English EVA 准确率和 74.13% 的 experience pass@1。摘要未给出与其他模型的直接对比结论。
Insight: 主要创新在于将语音接口接入已后训练的多模态推理骨干,使语音、文本、图像、推理和工具调用共享同一模型,并避免单独对话模型。训练上提出两阶段适配、自动构造语音条件对话以及结合本地教师服务、师生计算重叠和融合全词表损失的高效在线蒸馏,具有较好的工程与训练效率借鉴价值。
Abstract: A capable voice assistant must do more than recognize words; it must understand requests, reason over context, sustain conversations, and act through tools. We present Apriel Voice 27B Preview, an open speech-language model that unifies these capabilities within a shared reasoning backbone while retaining text and image input modalities. Built on the post-trained Gemma 4 26B-A4B checkpoint, Apriel Voice directly processes speech and generates both responses and tool calls without requiring a separate dialogue model. We introduce a two-stage training procedure to adapt the model to speech. The first stage establishes the speech interface, while the second jointly adapts the projector and backbone through supervised fine-tuning, text replay, and source-dependent cross-entropy and forward Kullback–Leibler divergence supervision. Automatically constructed speech-conditioned dialogues provide conversational training data, while local teacher serving, overlapping teacher and student computation, and fused full-vocabulary losses enable efficient online distillation. In thinking mode, Apriel Voice achieves a score of 98.90 on Big Bench Audio, 81.13 on Speech-BFCL, and a task success rate of 65.51% on $τ$-Voice, with a mean latency of 1.91 seconds. It also achieves 51.27% English EVA accuracy and 74.13% experience pass@1. Finally, we deploy Apriel Voice 27B Preview on a custom serving platform to support interactive voice interactions.
cs.SE [Back]
[83] One Verdict Is Not Enough: Regression Testing Learned World Models and Their Test Harnesses cs.SE | cs.AI | cs.CVPDF
Junyu Huo, Zihao Wan, Ziqi Mao, Gouri Ginde
TL;DR: 论文提出 TetraState,一种面向学习型世界模型的回归测试协议,用成对的可能/不可能视频测试,并在四种固定模型原生配置下运行每个测试,根据正确判定数划分四类结果;同时单独报告不确定测试和执行失败,并按逐测试结果转移比较版本,且将测试框架也纳入被测系统进行版本管理。
Details
Motivation: 学习型世界模型在版本、预处理和测试框架变化时,仅靠准确率检查会掩盖逐样本回归;且测试可能在不同配置下通过/失败不一致,测试框架故障也可能在不崩溃时产生错误分数和错误判定。
Result: 在 IntPhys 2 Main 的全部 506 个测试及 Cosmos 3 Nano、V-JEPA 2 两个异构系统上,两个 V-JEPA 2 测试框架修订仅使众数准确率变化 0.2 和 2.6 个百分点,却改变了 11 和 214 个测试标签;其中第二次准确率上升但正确判定测试数减少,且 43 个回归仍保持众数判定,说明仅检查众数判定翻转会漏检。
Insight: 创新点在于将回归测试从单一判定扩展到多配置结果类别与逐测试结果转移,并把测试框架版本化作为被测系统的一部分;客观看,其价值在于暴露配置级和测试框架级回归,推动学习型组件配置级回归判据的研究议程。
Abstract: Learned world models ship as software components whose model versions, preprocessing, and test harnesses change between releases. Work on model updates already counts per-sample regressions, or negative flips, that an accuracy check hides. Two problems remain: a test can pass under one configuration and fail under another, and a harness fault can produce erroneous scores and wrong verdicts without a crash. We present TetraState, a regression-testing protocol for paired possible/impossible video tests. Each test runs under four fixed model-native configurations; its number of correct verdicts assigns one of four outcome classes. Inconclusive tests and failed executions are reported separately alongside the verdict rate. Revisions are compared by per-test outcome transitions, and the harness is versioned as part of the system under test. On all 506 IntPhys 2 Main tests and two heterogeneous systems, Cosmos 3 Nano and V-JEPA 2, two V-JEPA 2 harness revisions move modal accuracy by only 0.2 and 2.6 points yet change 11 and 214 test labels; in the second, accuracy rises although fewer tests are judged correctly, and 43 regressions keep the modal verdict, so a modal-verdict flip check misses them. These results outline a research agenda on configuration-level regression oracles for learned components.
[84] Lineage Is Memory: Parent Inheritance and Feedback Disclosure in Short-Budget LLM-Guided Optimization cs.SE | cs.AI | cs.CLPDF
Shanglin Yang, Zhan Shi
TL;DR: 论文在九个提示与程序合成基准上进行了1750次预注册实验,分离LLM引导优化中的“父代继承”与“每轮反馈披露”两个因素。研究发现,贪心父代继承是优化进展的主要来源;种子重置重采样即使每轮获得全失败聚类摘要或使用更大提议模型,仍不如父代继承。错误聚类等方法只提升单步重写和搜索初期速度,随着父代继承保留已解决错误模式并过滤后续失败列表,差距会迅速缩小。成本分解还显示,GEPA中提议模型反思只占小部分开销,验证与小批量门控几乎消耗全部预算。
Details
Motivation: 现有迭代语言模型优化器多将性能限制归因于单轮反思范围窄,并引入全失败聚类、定向小批量或更大提议模型来增加每轮可见失败模式;但这些评估都在保留谱系的搜索中进行,混淆了父代继承与反馈广度的贡献。论文旨在匹配评估预算下厘清二者作用,寻找短预算下更高效的优化机制。
Result: 在1750次预注册运行、九个基准上,贪心父代继承驱动绝大多数优化进展,并优于种子重置重采样,即使后者每轮接收所有失败的聚类摘要或使用更大提议模型。全失败聚类和多样失败选择能改善从种子的单步重写并加速搜索初期,但父代继承会快速追平;GEPA的token开销分解显示反思只占小部分,验证与小批量门控消耗近全部预算。
Insight: 论文的创新点是将“谱系记忆”与“反馈披露”解耦,在严格匹配预算下证明父代继承本身是主要收益来源,而非更宽的单轮反馈。可借鉴之处在于:持续继承候选可保留已解决错误模式并在后续失败列表中过滤,简单贪心继承原始失败反馈即可用更低计算获得复杂反思循环的收益,并需重视验证与小批量门控的成本。
Abstract: Iterative language-model optimizers refine prompts and programs by evaluating a candidate, showing a proposer model a sample of failing cases, and keeping revisions that improve performance. Recent work attributes the limits of this loop to narrow single-round reflection, introducing full-set error clustering, targeted minibatches, or larger proposer models so the optimizer sees more failure modes per step. Yet every existing evaluation tests these mechanisms exclusively inside lineage-preserving search, where each revision edits a surviving parent, leaving the contribution of parent inheritance confounded with the breadth of per-round feedback. Across 1,750 pre-registered runs spanning nine prompt and program synthesis benchmarks, we separate parent inheritance from feedback disclosure under matched evaluation budgets. We find that greedy parent inheritance drives the vast majority of optimization progress, outperforming seed-reset resampling even when seed-reset proposals receive a clustered summary of all failures in every round or use a larger proposer model. While full-set error clustering and diverse failure selection improve single-step rewrites from the seed and accelerate the opening rounds of search, sequential parent inheritance rapidly closes this gap as solved error modes are retained in the candidate and filtered from subsequent failure lists. Moreover, decomposing token expenditure in GEPA reveals that proposer reflection accounts for only a small fraction of total cost whereas validation and minibatch gating consume nearly the entire budget, showing that greedy parent inheritance over raw failures captures the gains of complex reflective loops at a fraction of their compute.
[85] SCRBench: Beyond Equivalence Checking for Semantic Change Reasoning in Code cs.SE | cs.CLPDF
Omid Saghatchian, Ali Mahzoun, Khashayar Etemadi, Saeed Reza Kheradpisheh
TL;DR: 论文提出了SCRBench基准,包含67,658个Python程序对,用于评估语言模型对代码语义变化的理解。除了传统的语义等价检测,还引入了诊断语义变化和重建代码更改序列两个新任务。实验表明,高等价准确率会掩盖模型在理解语义变化原因和序列上的弱点。
Details
Motivation: 现有评估通常只通过判断两个程序是否语义等价来评估模型对代码更改影响的理解,但二分类答案无法揭示模型是否知道具体哪个更改导致了语义差异,或更改序列如何产生修改后的程序。论文旨在更直接地诊断模型对代码语义变化的推理能力。
Result: 在4,500对分层子集上评估五个语言模型。最强模型在语义等价检测上接近上限,但一个等价准确率94.71%的模型在诊断语义变化上仅54.44%。重建代码更改序列更难:最佳模型仅39.27%的配对能精确重建,当应用五个更改时没有模型超过1.11%。
Insight: 论文的创新点在于超越了传统的等价检查,设计了诊断语义变化和重建更改序列两个新任务,并构建了相应的基准SCRBench。客观分析,其创新在于更细粒度地评估模型对代码更改语义影响的理解,揭示了等价准确率作为评估指标的局限性,为代码推理评估提供了新视角。
Abstract: Understanding how code changes affect program semantics is a core aspect of code reasoning in language models. It is commonly evaluated by asking whether two programs are semantically equivalent, but a yes/no answer does not reveal whether a model knows which code change caused a semantic difference, or which sequence of code changes produced the modified program. We study two tasks that ask these questions directly, diagnosing semantic change and reconstructing sequences of code changes, alongside detecting semantic equivalence. We build SCRBench, 67,658 Python program pairs generated by applying sequences of code changes that preserve or alter program semantics, and evaluate five language models on a stratified subset of 4,500 pairs. Detecting semantic equivalence exposes weaker models, whose accuracy declines as changes accumulate, but the strongest models remain near the ceiling. Diagnosing semantic change reveals weaknesses that equivalence accuracy hides: a model with 94.71% equivalence accuracy diagnoses only 54.44% of semantic changes. Reconstructing sequences of code changes is much harder: the best model reconstructs the exact sequence for only 39.27% of pairs, and no model exceeds 1.11% when five changes are applied. High equivalence accuracy therefore overestimates how well models reason about the semantic effects of code changes.
eess.IV [Back]
[86] Inter-Frame Predictive Contour Coding for Semantic Map Compression eess.IV | cs.CVPDF
Runyu Yang, Hyomin Choi, Fabien Racapé, Ivan V. Bajić
TL;DR: 论文提出一种面向动态语义图无损压缩的帧间预测轮廓编码框架:先利用参考帧预测当前帧的斑块位置和标签索引,再基于参考轮廓中的对应链码单元进行轮廓预测,并通过模式选择在帧内与帧间编码间切换。该方法在VSPW动态语义分割数据集上取得优于现有方法的压缩性能,较近期上下文自适应扩展链码框架平均码率降低3.0%,且平均编码时间无显著增加。
Details
Motivation: 动态语义图的无损压缩对降低存储和传输成本很重要,但现有基于轮廓的方法主要利用单帧内空间冗余,忽略了相邻语义图之间强时间相关性,压缩效率仍有提升空间。
Result: 在VSPW动态语义分割数据集上,所提方法一致优于当前最先进的无损语义图压缩方法;与近期上下文自适应扩展链码框架相比,平均码率降低3.0%,平均编码时间没有显著增加。
Insight: 创新点在于将帧间预测思想引入基于轮廓的语义图无损压缩,联合预测斑块位置、标签索引和轮廓链码,并用编码成本估计驱动帧内/帧间模式选择;客观看,其价值在于有效利用时间轮廓冗余,同时保留轮廓表示优势并平衡压缩性能与计算复杂度。
Abstract: Dynamic semantic maps have become an important intermediate representation for many computer vision applications. Efficient lossless compression of such maps is essential for reducing storage and transmission costs. Existing contour-based semantic map compression methods mainly exploit spatial redundancy within individual frames while largely ignoring the strong temporal correlation between adjacent semantic maps. In this paper, we propose an inter-frame predictive contour coding framework for lossless semantic map compression. The proposed framework first predicts blob positions and label indexes from the reference frame and then performs contour prediction using corresponding chain-code units in the reference contours. A mode selection mechanism is further employed to choose between intra and inter coding according to the estimated coding cost. By exploiting temporal contour redundancy while preserving the advantages of contour-based representation, the proposed framework significantly improves coding efficiency. Experimental results on the VSPW dynamic semantic segmentation dataset demonstrate that the proposed method consistently outperforms state-of-the-art lossless semantic map compression methods. Compared with the recent context-adaptive extended chain coding framework, the proposed inter-frame coding achieves an average bitrate reduction of 3.0% without significant increase in the average encoding time, demonstrating an effective trade-off between compression performance and computational complexity. The source code is made publicly available at https://github.com/InterDigitalInc/LosslessSegmentationMapCompression.
[87] TopoHyperClinicalGraph: A Geometry and Topology-Aware Multimodal Framework for Cancer Survival Prediction eess.IV | cs.CVPDF
Noaman Khan, Zubair Shah
TL;DR: 论文提出TopoHyperClinicalGraph,一个融合双曲组织几何、持久拓扑和临床图学习的多模态癌症生存预测框架。方法先用PCA降维UNI2-h病理特征,构建结合双曲邻域与Lorentz质心注意力池化的病理骨干,再用切片级持久描述符校正生存logits,并通过五节点临床图融合年龄、性别、分期、T和N进行进一步校正。作者在8个TCGA/CPTAC队列、2364名患者上做五折患者级评估,并分别分析几何、拓扑和临床信息的贡献。
Details
Motivation: 动机是提升癌症生存预测中对组织病理空间几何结构、拓扑特征和临床变量信息的综合利用,并支持对几何、拓扑与临床贡献进行分离评估。
Result: 在多个TCGA队列中,TopoHyperClinicalGraph的C-index高于病理-only的PatchGCN,如COADREAD 0.7563 vs 0.7386、HNSC 0.6220 vs 0.5998、BRCA 0.6755 vs 0.6535、STAD 0.6409 vs 0.6176;CPTAC上HNSC/PDA/CCRCC/LUAD分别为0.5984/0.6060/0.8055/0.7192。几何带来大部分病理-only增益,持久拓扑增益较小;临床图融合并非在所有队列都优于对照,且所有结果均为验证集选择,需独立测试验证泛化。
Insight: 创新点在于将双曲组织几何、持久拓扑描述符和临床图学习整合到生存预测框架中,并采用Lorentz质心注意力池化与模块化设计分别评估几何、拓扑和临床贡献。客观看,其价值主要在于多模态分解评估和几何拓扑建模思路,但临床融合收益不稳定,仍需独立外部验证。
Abstract: We developed TopoHyperClinicalGraph, a multimodal framework combining hyperbolic tissue geometry, persistent topology and clinical graph learning for cancer survival prediction. Pre-extracted UNI2-h features are reduced from 1,536 to 1,024 dimensions using training-fold PCA. A pathology backbone combines hyperbolic neighborhoods with Lorentz-centroid attention pooling. Once frozen, its survival logits are adjusted using persistence descriptors from up to 64 landmarks per slide and sampling controls. A five-node clinical graph then learns a further correction from age, sex, stage, T and N. Five-fold patient-level evaluation covered eight TCGA and CPTAC cohorts with 2,364 patients. Mean validation-selected C-indices for TopoHyperClinicalGraph versus pathology-only spatial PatchGCN were 0.7563 versus 0.7386 in COADREAD, 0.6220 versus 0.5998 in HNSC, 0.6755 versus 0.6535 in BRCA and 0.6409 versus 0.6176 in STAD. CPTAC scores were 0.5984 in HNSC, 0.6060 in PDA, 0.8055 in CCRCC and 0.7192 in LUAD. CPTAC-HNSC used TCGA-initialized adaptation; other CPTAC backbones were trained within their cohorts. Geometry accounted for most pathology-only gains, with smaller persistence effects. Clinical graph fusion exceeded its matched node-wise control in five cohorts, tied in one and scored lower in two; clinical-only prediction exceeded fusion in three CPTAC cohorts. The framework supports separate evaluation of geometric, topological and clinical contributions. All scores were validation-selected; independent testing is needed to establish generalizability.
[88] IGDA-ICMH: an Importance Guided Feature and Network Dual Adaptation Framework towards Image Compression for Machine and Human Vision eess.IV | cs.CVPDF
Yanbo Gao, Xinyu Zhao, Shuai Li, Xun Cai, Mao Ye
TL;DR: 论文提出IGDA-ICMH框架,用于面向机器和人类视觉的图像压缩(ICMH)。该框架通过定量分析不同特征和网络层对机器视觉任务的重要性,设计重要性引导的特征和网络双重适应,基于隐式神经表示,包含IGNA模块(低秩适应调整关键网络层)和IGFA模块(低秩Kronecker积变换减少参数),在图像分类和目标检测任务上显著优于现有INR-based方法。
Details
Motivation: 现有ICMH研究主要关注任务特定适应或多任务联合优化,但忽略不同特征和网络层对下游机器视觉任务的不平等贡献,导致适应效率不高。论文旨在解决如何根据重要性进行高效适应的问题。
Result: 在图像分类和目标检测等代表性机器视觉任务上,实验表明IGDA-ICMH框架显著优于现有的基于隐式神经表示的ICMH方法,但摘要未给出具体定量数值。
Insight: 创新点包括:定量研究特征和网络层对机器视觉性能的影响;提出重要性引导的双重适应框架,结合网络权重适应和特征适应;IGNA模块使用低秩适应策略选择性调整关键网络层;IGFA模块使用低秩Kronecker积变换探索特征非局部性以减少参数。客观分析:将重要性引导与低秩适应结合,针对ICMH任务有效提升压缩后机器视觉性能,同时控制适应参数。
Abstract: With the surge of machine vision applications, image compression for both machine and human vision (ICMH) has become an important research area. Existing studies mainly focus on task-specific adaptation or multi-task joint optimization, yet they often overlook the unequal contributions of different features and network layers to downstream machine vision tasks. To address this issue, we first quantitatively investigate the impact of different feature and network layers on machine vision performance, providing a principled understanding of their importance. Based on this analysis, an Importance Guided feature and network Dual Adaptation (IGDA) framework is developed for ICMH. It adopts implicit neural representation as the basic architecture and designs an efficient adaptation structure with both feature and network weight adaptation. An Importance-Guided Network Adaptation (IGNA) module is developed with a low-rank adaptation strategy to selectively adjust key network layers according to their importance in different tasks. An Importance-Guided Feature Adaptation (IGFA) module is developed with a low-rank and Kronecker product-based transform to explore the non-local property of the features in order to reduce the number of adaptation parameters. Extensive experiments on representative machine vision tasks, including image classification and object detection, demonstrate that our IGDA-ICMH framework achieves significantly better performance than the existing implicit neural representation-based methods for ICMH.
physics.flu-dyn [Back]
[89] ART of PIV: Agentic Real-Time Particle Image Velocimetry for Active Flow Control physics.flu-dyn | cs.AI | cs.CVPDF
Antonio Terpin, Francesco Banelli, Alan Bonomi, Raffaello D’Andrea
TL;DR: 论文提出 ART of PIV,一种面向实时主动流动控制的智能体式粒子图像测速框架。它让学习到的智能体针对每一对图像,在给定速度预算和测量目标下,从候选配置库中自动选择 PIV 预处理、流场估计和后处理配置,从而适应变化工况并兼顾精度与速度。
Details
Motivation: 传统 PIV 流程中的预处理、流估计和后处理选择依赖专家调参,固定配置在工况变化时精度下降;而实时控制反馈要求测量既准确又快速,因此需要按图像对自适应选择 PIV 流程配置。
Result: 在互相关、经典光流和学习型估计器上,ART of PIV 在匹配速度下将平均端点误差 AEPE 相对最佳单一估计器降低 12.2%–22.4%,增益延伸至 1000–4000 图像对/秒;在仿真局部速度调节任务中,将平均归一化闭环成本相对调优固定估计器降低 13.24–13.59 个百分点,并在桌面水槽的流体弹球实验中展示了物理世界闭环流场跟踪。
Insight: 创新点在于将 PIV 流程配置选择建模为按图像对进行的智能体决策,在速度预算和测量目标约束下从配置库中自动组合跨类别估计器与调参策略;客观看,其可借鉴之处是将传统测量流程的专家调参转化为面向下游控制性能的在线自适应选择,并验证了从仿真到物理闭环控制的可行性。
Abstract: Particle image velocimetry (PIV) measures fluid motion from images of tracer particles, but its accuracy depends on choices throughout the estimation pipeline, including image preprocessing, flow estimation, and postprocessing. These choices are typically left to expert tuning, and a fixed configuration can become less accurate as conditions change. This matters especially when PIV supplies feedback for real-time control, where measurements must be both accurate and fast, and flow conditions vary. We introduce ART of PIV – short for Agentic Real-Time and also a nod to the art of tuning a PIV pipeline – which automates pipeline choice for each image pair. A learned agent selects a configuration according to a chosen measurement objective from a library assembled for a specified speed budget. Across cross-correlation, classical optical-flow, and learning-based estimators, ART of PIV reduces average end-point error (AEPE) by 12.2-22.4 % relative to the best single estimator at matched speed, with gains extending to 1000-4000 image pair/s. In a simulated local velocity regulation task, it reduces mean normalized closed-loop cost by 13.24-13.59 percentage points relative to a tuned fixed estimator, with the same classical controller in both cases. In a tabletop water channel, we illustrate two ways to tune ART of PIV for a fluidic pinball of three rotating cylinders: selecting between complementary optical-flow tunings for accuracy, and selecting regions of interest with coherent rotation or shear using a common flow estimator. For the first variant, we train a DreamerV3 reinforcement learning controller directly in the physical world and demonstrate tracking of a prescribed flow field in the wake of a fluidic pinball.
q-bio.QM [Back]
[90] A small HER2 reader for routine H&E slides: real-world deployment q-bio.QM | cs.CV | cs.LG | cs.SEPDF
Thang Tran, Lan Dang
TL;DR: 论文构建了一个面向常规H&E切片的轻量级HER2状态预读系统,作为病理医生的辅助基线。方法采用22M参数的ViT-S/16图块编码器,在TCGA公开切片上通过自监督、从H-optimus-0蒸馏及两轮JEPA精炼训练,并用五个交叉验证的注意力切片分类器集成输出概率和判定。系统在多家外部医院数据上验证,可在无GPU环境下5–6秒/切片运行,并部署于病理平台优先提示IHC 2+等不确定病例是否需ISH。
Details
Motivation: HER2状态决定患者能否接受HER2靶向治疗;临床依赖IHC评分,而IHC 2+等不确定病例需等待ISH确认。论文希望利用常规H&E切片提供一个端到端、轻量且可部署的HER2预读工具,帮助病理医生分诊和优先处理不确定病例。
Result: 在配对十种子协议下,基于该编码器的分类器在TCGA-BRCA到Yale-HER2外部验证中7个模型中排名第一,AUC 0.812,与参数量大4倍和50倍的编码器相当;在重建的PathBench基准上TCGA-BRCA AUC 0.689排名第一,Yale-HER2 AUC 0.897与H-optimus-0、Phikon相当。部署后在Yale-HER2、HEROHE和BCNB活检上AUC分别为0.82、0.75和0.63;HEROHE的85例IHC 2+中区分ISH扩增的AUC为0.76。
Insight: 创新点在于用小型ViT编码器结合自监督、大模型蒸馏和JEPA迭代精炼,在H&E切片上构建HER2预读器;通过多折注意力分类器集成和训练集固定操作点提升稳定性,并以公开权重、逐切片预测和可复现实验推动临床前部署研究。
Abstract: HER2 status decides who receives HER2-targeted therapy. Pathologists score HER2 protein by immunohistochemistry (IHC); equivocal IHC 2+ tumours, their hardest calls, wait for in situ hybridisation (ISH). We built a HER2 reader for routine H&E slides, owned end to end, as a baseline for pathologists. A 22M-parameter ViT-S/16 tile encoder was trained on public TCGA slides by self-supervision, distillation from H-optimus-0 and two rounds of JEPA refinement; five cross-validated attention-based slide classifiers average their probabilities; and an operating point is fixed on training data. Compared under one paired ten-seed protocol, readers on our encoder rank first of 7 on external validation from TCGA-BRCA to Yale-HER2 (AUC 0.812), level with readers on encoders 4 and 50 times larger. We also rebuilt PathBench’s encoder benchmark on public cohorts; our replica reproduces its published order of reference encoders, and under it our encoder ranks first of 7 on TCGA-BRCA (0.689) and sits level with H-optimus-0 and Phikon on Yale-HER2 (0.897). On three external hospitals our deployed reader reaches AUC 0.82 on Yale-HER2 (about 0.81 with training slides balanced for tissue size), 0.75 on HEROHE and 0.63 on BCNB biopsies, and among HEROHE’s 85 IHC 2+ cases it ranks ISH-amplified above non-amplified tumours at 0.76, best among our TCGA-trained readers. Per-laboratory score normalisation and a laboratory’s own slides improve it further on a new site. Without a GPU our reader needs 5-6 s per slide. It serves as a research-use pre-read in a pathology platform, where it proposes a probability and a call beside IHC and helps prioritise ISH for equivocal cases. Weights, per-slide predictions and the data behind every figure are public at https://huggingface.co/cloudkites/her2-reader.