2025-07-01

Table of Contents

• cs.CL [Total: 46]
• cs.CV [Total: 69]
• stat.ME [Total: 1]
• quant-ph [Total: 1]
• cs.RO [Total: 6]
• cs.IR [Total: 4]
• cs.HC [Total: 1]
• cs.AI [Total: 3]
• cs.LG [Total: 4]
• cs.MM [Total: 1]
• physics.optics [Total: 1]
• eess.IV [Total: 6]
• stat.ML [Total: 1]

cs.CL [Back]

[1] VAT-KG: Knowledge-Intensive Multimodal Knowledge Graph Dataset for Retrieval-Augmented Generation

Hyeongcheol Park,MinHyuk Jang,Ha Dam Baek,Gyusam Chang,Jiyoung Seo,Jiwan Park,Hogun Park,Sangpil Kim

Main category: cs.CL

TL;DR: 本文提出了VAT-KG，首个概念为中心、知识密集型的多模态知识图谱，覆盖视觉、音频和文本信息，支持跨模态知识对齐和检索增强生成（RAG），在多模态任务中表现优异。

Details

Motivation: 现有MMKGs覆盖范围有限，知识可能过时或不完整，且仅支持少数模态（如文本和视觉），限制了其在多模态任务中的扩展性和应用性。

Contribution: 1. 提出首个概念为中心、多模态（视觉、音频、文本）知识密集型知识图谱VAT-KG。
2. 设计了自动生成MMKGs的流程，支持跨模态知识对齐。
3. 提出新型多模态RAG框架，支持任意模态查询的概念级知识检索。

Method: 1. 构建VAT-KG时，通过严格筛选和对齐步骤实现跨模态知识对齐。
2. 从任意多模态数据集中自动生成MMKG。
3. 提出了多模态RAG框架，支持跨模态查询。

Result: 在多模态问答任务中，VAT-KG显著提升了MLLMs的表现，展示了其在统一和利用多模态知识方面的实用价值。

Insight: VAT-KG填补了现有MMKGs在模态覆盖和知识密集性上的不足，为多模态任务提供了更全面的知识支持。

Abstract: Multimodal Knowledge Graphs (MMKGs), which represent explicit knowledge across multiple modalities, play a pivotal role by complementing the implicit knowledge of Multimodal Large Language Models (MLLMs) and enabling more grounded reasoning via Retrieval Augmented Generation (RAG). However, existing MMKGs are generally limited in scope: they are often constructed by augmenting pre-existing knowledge graphs, which restricts their knowledge, resulting in outdated or incomplete knowledge coverage, and they often support only a narrow range of modalities, such as text and visual information. These limitations reduce their extensibility and applicability to a broad range of multimodal tasks, particularly as the field shifts toward richer modalities such as video and audio in recent MLLMs. Therefore, we propose the Visual-Audio-Text Knowledge Graph (VAT-KG), the first concept-centric and knowledge-intensive multimodal knowledge graph that covers visual, audio, and text information, where each triplet is linked to multimodal data and enriched with detailed descriptions of concepts. Specifically, our construction pipeline ensures cross-modal knowledge alignment between multimodal data and fine-grained semantics through a series of stringent filtering and alignment steps, enabling the automatic generation of MMKGs from any multimodal dataset. We further introduce a novel multimodal RAG framework that retrieves detailed concept-level knowledge in response to queries from arbitrary modalities. Experiments on question answering tasks across various modalities demonstrate the effectiveness of VAT-KG in supporting MLLMs, highlighting its practical value in unifying and leveraging multimodal knowledge.

[2] Debunk and Infer: Multimodal Fake News Detection via Diffusion-Generated Evidence and LLM Reasoning

Kaiying Yan,Moyang Liu,Yukun Liu,Ruibo Fu,Zhengqi Wen,Jianhua Tao,Xuefei Liu

Main category: cs.CL

TL;DR: DIFND 是一个多模态假新闻检测框架，通过扩散模型生成反驳证据并结合多模态大语言模型的推理能力，显著提升了检测性能和可解释性。

Details

Motivation: 假新闻在多媒体平台上的快速传播对信息可信度构成严重威胁，因此需要一种高效且可解释的检测方法。

Contribution: 提出了整合条件扩散模型和多模态大语言模型的 Debunk-and-Infer 框架，结合生成反驳证据和逻辑推理提升检测效果。

Method: 使用扩散模型生成语义对齐的反驳证据，并通过多代理 MLLM 系统进行逻辑推理和多模态感知验证。

Result: 在 FakeSV 和 FVC 数据集上，DIFND 不仅优于现有方法，还提供了可信赖的决策。

Insight: 生成式模型与推理模型的结合能够为假新闻检测提供更丰富的证据和更强的解释性。

Abstract: The rapid spread of fake news across multimedia platforms presents serious challenges to information credibility. In this paper, we propose a Debunk-and-Infer framework for Fake News Detection(DIFND) that leverages debunking knowledge to enhance both the performance and interpretability of fake news detection. DIFND integrates the generative strength of conditional diffusion models with the collaborative reasoning capabilities of multimodal large language models (MLLMs). Specifically, debunk diffusion is employed to generate refuting or authenticating evidence based on the multimodal content of news videos, enriching the evaluation process with diverse yet semantically aligned synthetic samples. To improve inference, we propose a chain-of-debunk strategy where a multi-agent MLLM system produces logic-grounded, multimodal-aware reasoning content and final veracity judgment. By jointly modeling multimodal features, generative debunking cues, and reasoning-rich verification within a unified architecture, DIFND achieves notable improvements in detection accuracy. Extensive experiments on the FakeSV and FVC datasets show that DIFND not only outperforms existing approaches but also delivers trustworthy decisions.

[3] Reinforcement Learning Fine-Tuning of Language Model for Instruction Following and Math Reasoning

Yifu Han,Geo Zhang

Main category: cs.CL

TL;DR: 该研究探讨了强化学习（RL）微调技术在小型语言模型（Qwen2.5-0.5B Base）上对指令跟随和数学推理任务的有效性，比较了SFT、DPO和RLOO等方法。

Details

Motivation: 如何利用高效微调方法（如RL）改进小型语言模型在复杂任务（如指令跟随和数学推理）中的表现。

Contribution: 提出RLOO结合DeBERTa奖励模型的方法，验证了DPO的稳定性，并展示了合成数据增强和推理时工具对数学任务的提升效果。

Method: 对比了SFT、DPO和RLOO微调方法，实验中使用RLOO结合DeBERTa奖励模型和外部验证器进行数学任务优化。

Result: RLOO在任务对齐上表现最佳，DPO效果稳定；数学任务中合成数据增强和best-of-N采样显著提高了准确性。

Insight: 结合微调与推理时工具可以高效优化小型语言模型，平衡性能与资源消耗。

Abstract: This study investigates the effectiveness of reinforcement learning (RL) fine-tuning techniques on a compact language model (Qwen2.5-0.5B Base) for two challenging tasks: instruction following and mathematical reasoning. We compare supervised fine-tuning (SFT), Direct Preference Optimization (DPO) using preference-labeled data, and Reinforce Leave-One-Out (RLOO) with reward models. Our experiments show that RLOO with DeBERTa reward modeling achieves the best alignment, while DPO provides strong and consistent results. For math reasoing tasks, synthetic data augmentation and best-of-N sampling with an external verifier significantly improve accuracy, showing the potential of combining fine-tuning with inference-time tools. This study highlights key trade-offs and practical strategies for training lightweight, task-aligned small-scale language models.

[4] Reasoning Isn’t Enough: Examining Truth-Bias and Sycophancy in LLMs

Emilio Barkett,Olivia Long,Madhavendra Thakur

Main category: cs.CL

TL;DR: 该研究评估了大型语言模型（LLMs）在真相检测方面的能力，首次分析了推理模型在这方面的表现。研究发现推理模型的真相偏见低于非推理模型，但仍高于人类基准，且高级模型存在谄媚倾向。

Details

Motivation: 尽管LLMs被广泛用于事实核查和决策制定，但其作为真相评判者的能力仍不够清晰。研究旨在填补这一空白，分析不同LLMs在真相检测中的表现。

Contribution: 首次分析了推理模型的真相检测能力，揭示了高级模型中存在的谄媚倾向，表明单纯的技术进步未能完全解决LLMs的真相检测挑战。

Method: 研究让8个LLM完成4800次真相判断，比较了推理和非推理模型的表现。

Result: 推理模型的真相偏见较低，但仍高于人类；高级模型在真相检测中表现不对称（真相准确率高，欺骗检测差）。

Insight: LLMs的真相检测能力仍有缺陷，尤其是高级模型中存在的谄媚行为表明技术改进需结合行为分析。

Abstract: Despite their widespread use in fact-checking, moderation, and high-stakes decision-making, large language models (LLMs) remain poorly understood as judges of truth. This study presents the largest evaluation to date of LLMs’ veracity detection capabilities and the first analysis of these capabilities in reasoning models. We had eight LLMs make 4,800 veracity judgments across several prompts, comparing reasoning and non-reasoning models. We find that rates of truth-bias, or the likelihood to believe a statement is true, regardless of whether it is actually true, are lower in reasoning models than in non-reasoning models, but still higher than human benchmarks. Most concerning, we identify sycophantic tendencies in several advanced models (o4-mini and GPT-4.1 from OpenAI, R1 from DeepSeek), which displayed an asymmetry in detection accuracy, performing well in truth accuracy but poorly in deception accuracy. This suggests that capability advances alone do not resolve fundamental veracity detection challenges in LLMs.

[5] FloorPlan-DeepSeek (FPDS): A multimodal approach to floorplan generation using vector-based next room prediction

Jun Yin,Pengyu Zeng,Jing Zhong,Peilin Li,Miao Zhang,Ran Luo,Shuai Lu

Main category: cs.CL

TL;DR: FloorPlan-DeepSeek (FPDS) 提出了一种基于‘下一个房间预测’的多模态方法，用于逐步生成矢量化的平面图，更适合实际建筑设计的迭代工作流程。

Details

Motivation: 现有平面图生成模型通常是端到端的，直接生成整个像素布局，与实际建筑设计中的渐进式工作流程不符。作者受大型语言模型中自回归‘下一个token预测’机制的启发，试图改进这一范式。

Contribution: 提出了‘下一个房间预测’的新范式，专为建筑设计中的平面图建模设计，结合多模态方法生成矢量化的平面图。

Method: 采用自回归的‘下一个房间预测’机制，逐步生成矢量化的房间布局，而非一次性生成整个像素图。

Result: 在文本到平面图任务中，FPDS表现出与扩散模型和Tell2Design相当的竞争性能。

Insight: 这种逐步生成方法更贴近实际建筑设计流程，有望支持未来智能建筑设计。

Abstract: In the architectural design process, floor plan generation is inherently progressive and iterative. However, existing generative models for floor plans are predominantly end-to-end generation that produce an entire pixel-based layout in a single pass. This paradigm is often incompatible with the incremental workflows observed in real-world architectural practice. To address this issue, we draw inspiration from the autoregressive ‘next token prediction’ mechanism commonly used in large language models, and propose a novel ‘next room prediction’ paradigm tailored to architectural floor plan modeling. Experimental evaluation indicates that FPDS demonstrates competitive performance in comparison to diffusion models and Tell2Design in the text-to-floorplan task, indicating its potential applicability in supporting future intelligent architectural design.

[6] FormosanBench: Benchmarking Low-Resource Austronesian Languages in the Era of Large Language Models

Kaiying Kevin Lin,Hsiyu Chen,Haopeng Zhang

Main category: cs.CL

TL;DR: 该论文提出了FORMOSANBENCH，首个针对低资源南岛语族的基准测试，评估了大语言模型（LLMs）在濒危台湾南岛语（泰雅语、阿美语、排湾语）的机器翻译、自动语音识别和文本摘要任务上的表现。结果显示，LLMs在这些语言上的表现远低于高资源语言，且10-shot学习和微调改进有限，呼吁开发更包容的NLP技术。

Details

Motivation: 虽然大语言模型（LLMs）在高资源语言的NLP任务中表现优异，但对低资源和濒危语言的能力尚未充分探索。台湾南岛语兼具语言多样性和濒危性，亟需研究支持。

Contribution: 提出了FORMOSANBENCH基准，覆盖三种濒危台湾南岛语，为LLMs在低资源语言上的评估提供了首个系统性测试平台。

Method: 评估了三种任务（机器翻译、语音识别、文本摘要），并在零样本、10-shot和微调设置下测试了LLMs的性能。

Result: LLMs在台湾南岛语上表现显著落后于高资源语言，10-shot和微调改进有限，突显了语言资源的匮乏问题。

Insight: LLMs在低资源语言上的性能劣势揭示了当前技术对濒危语言支持的不足，需更注重多语言包容性研究。

Abstract: While large language models (LLMs) have demonstrated impressive performance across a wide range of natural language processing (NLP) tasks in high-resource languages, their capabilities in low-resource and minority languages remain significantly underexplored. Formosan languages – a subgroup of Austronesian languages spoken in Taiwan – are both linguistically rich and endangered, largely due to the sociolinguistic dominance of Mandarin. In this work, we introduce FORMOSANBENCH, the first benchmark for evaluating LLMs on low-resource Austronesian languages. It covers three endangered Formosan languages: Atayal, Amis, and Paiwan, across three core NLP tasks: machine translation, automatic speech recognition (ASR), and text summarization. We assess model performance in zero-shot, 10-shot, and fine-tuned settings using FORMOSANBENCH. Our results reveal a substantial performance gap between high-resource and Formosan languages. Existing LLMs consistently underperform across all tasks, with 10-shot learning and fine-tuning offering only limited improvements. These findings underscore the urgent need for more inclusive NLP technologies that can effectively support endangered and underrepresented languages. We release our datasets and code to facilitate future research in this direction.

[7] Towards Understanding the Cognitive Habits of Large Reasoning Models

Jianshuo Dong,Yujia Fu,Chuanrui Hu,Chao Zhang,Han Qiu

Main category: cs.CL

TL;DR: 该论文提出了CogTest基准，用于评估大型推理模型（LRMs）是否表现出类似人类的认知习惯，发现LRMs能够适应性地部署这些习惯，并揭示了其与安全相关任务的关联。

Details

Motivation: 受到大型推理模型（LRMs）在推理过程中表现出类似人类思维习惯的现象启发，研究试图验证这些模型是否确实拥有类似人类的认知习惯，并探索其对模型行为解释的意义。

Contribution: 1. 提出了CogTest基准，涵盖16种认知习惯，每个习惯具有25个多样化任务。2. 发现LRMs能够表现出人类类似认知习惯，并适应性地在不同任务中部署。3. 揭示了认知习惯与模型生成有害回复之间的关联。

Method: 1. 基于Habits of Mind框架设计CogTest基准。2. 采用证据优先的提取方法（evidence-first extraction）确保习惯识别的可靠性。3. 对16种主流LLM（包括13个LRMs和3个非推理模型）进行全面评估。

Result: 1. LRMs表现出人类类似的认知习惯，并能根据任务适应性部署。2. 同一家族模型（如Qwen-3和DeepSeek-R1）在认知习惯上表现出相似性。3. 某些习惯（如Taking Responsible Risks）与生成有害回复强相关。

Insight: 研究LRMs的持续行为模式有助于深入理解模型的行为机制和潜在问题，尤其在与安全相关的任务中。

Abstract: Large Reasoning Models (LRMs), which autonomously produce a reasoning Chain of Thought (CoT) before producing final responses, offer a promising approach to interpreting and monitoring model behaviors. Inspired by the observation that certain CoT patterns – e.g., ``Wait, did I miss anything?’’ – consistently emerge across tasks, we explore whether LRMs exhibit human-like cognitive habits. Building on Habits of Mind, a well-established framework of cognitive habits associated with successful human problem-solving, we introduce CogTest, a principled benchmark designed to evaluate LRMs’ cognitive habits. CogTest includes 16 cognitive habits, each instantiated with 25 diverse tasks, and employs an evidence-first extraction method to ensure reliable habit identification. With CogTest, we conduct a comprehensive evaluation of 16 widely used LLMs (13 LRMs and 3 non-reasoning ones). Our findings reveal that LRMs, unlike conventional LLMs, not only exhibit human-like habits but also adaptively deploy them according to different tasks. Finer-grained analyses further uncover patterns of similarity and difference in LRMs’ cognitive habit profiles, particularly certain inter-family similarity (e.g., Qwen-3 models and DeepSeek-R1). Extending the study to safety-related tasks, we observe that certain habits, such as Taking Responsible Risks, are strongly associated with the generation of harmful responses. These findings suggest that studying persistent behavioral patterns in LRMs’ CoTs is a valuable step toward deeper understanding of LLM misbehavior. The code is available at: https://github.com/jianshuod/CogTest.

[8] Aligning MLLM Benchmark With Human Preferences via Structural Equation Modeling

Tianyu. Zou,Shengwu. Xiong,Ruilin. Yao,Jirui. Huang,Yi. Rong,Yaxiong. Chen,Shili. Xiong,Cong. Wang

Main category: cs.CL

TL;DR: 该论文提出了一种基于结构方程建模（SEM）的新框架，用于对齐和评估多模态大语言模型（MLLM）的基准测试，以解决现有基准测试设计中的冗余和认知目标不明确问题。通过引入基于皮亚杰认知发展理论的能力层次结构，作者重新组织了基准测试并构建了名为Gold的新基准。

Details

Motivation: 现有MLLM基准测试设计缺乏结构化、可解释性和理论基础，导致能力重叠、指标冗余和诊断能力有限。为了解决这些问题，作者提出了基于SEM的框架。

Contribution: 1. 提出基于SEM的框架，增强基准测试的内部效度、维度可分性和组件的贡献分析。2. 引入基于皮亚杰理论的能力层次结构（感知、记忆和推理）。3. 构建新基准Gold，展示更强的可解释性和更少的指标冗余。

Method: 1. 使用SEM分析基准测试的内部结构和维度。2. 根据皮亚杰理论将MLLM能力分为感知、记忆和推理三个层次。3. 重新组织现有基准并构建Gold基准测试。

Result: 实验结果表明，Gold基准比现有方法具有更强的可解释性、更少的指标冗余和更清晰的认知一致性。

Insight: 通过结合心理学理论和统计建模，可以设计更具诊断性和可解释性的MLLM基准测试，从而更好地评估模型的能力。

Abstract: Evaluating multimodal large language models (MLLMs) remains a fundamental challenge due to a lack of structured, interpretable, and theoretically grounded benchmark designs. Existing benchmarks often adopt heuristic-based task groupings with unclear cognitive targets, thus resulting in overlapping abilities, redundant indicators, and limited diagnostic power. In this work, we propose a novel framework for aligning MLLM benchmark based on Structural Equation Modeling (SEM) to analyze and quantify the internal validity, dimensional separability, and contribution of benchmark components. Motivated by the observed limitations of current designs, we further introduce a novel capability hierarchy grounded in Piagets theory of cognitive development, dividing MLLM abilities into three hierarchical layers, i.e., Perception, Memory, and Reasoning. We reorganize existing MLLM benchmarks under the proposed framework and construct a new benchmark named Gold. Experimental results demonstrate that the proposed benchmark exhibits stronger interpretability, reduced indicator redundancy, and clearer cognitive consistency compared to existing approaches.

[9] Instruction Learning Paradigms: A Dual Perspective on White-box and Black-box LLMs

Yanwei Ren,Liu Liu,Baosheng Yu,Jiayan Qiu,Quan Chen

Main category: cs.CL

TL;DR: 论文提出了一种结合白盒和黑盒LLM优势的新框架，通过黑盒模型提供高质量指令初始化，白盒模型提供细粒度可解释性，并通过语义相似性约束实现优化，实验表明其性能优于现有基线。

Details

Motivation: 现有方法中，白盒模型需要大量计算资源且表示能力有限，而黑盒模型成本高昂，因此需要一种融合两者优势的高效解决方案。

Contribution: 提出了一种新颖的框架，结合黑盒模型的高质量初始化与白盒模型的细粒度可解释性，通过语义相似性约束优化指令质量。

Method: 利用黑盒模型生成多样化的指令初始化，白盒模型通过隐藏状态和输出特征提供可解释性，并通过语义相似性约束融合为统一高维表示，进行迭代优化。

Result: 在复杂推理和跨语言泛化等广泛任务中，该框架性能显著优于现有基线。

Insight: 融合黑盒和白盒模型的优势可以高效提升LLM的指令优化能力，为实际应用提供可扩展的解决方案。

Abstract: Optimizing instructions for large language models (LLMs) is critical for harnessing their full potential in complex and diverse tasks. However, relying solely on white-box approaches demands extensive computational resources and offers limited representational capacity, while black-box models can incur prohibitive financial costs. To address these challenges, we introduce a novel framework that seamlessly merges the strengths of both paradigms. Black-box models provide high-quality, diverse instruction initializations, and white-box models supply fine-grained interpretability through hidden states and output features. By enforcing a semantic similarity constraint, these components fuse into a unified high-dimensional representation that captures deep semantic and structural nuances, enabling an iterative optimization process to refine instruction quality and adaptability. Extensive evaluations across a broad spectrum of tasks-ranging from complex reasoning to cross-lingual generalization-demonstrate that our approach consistently outperforms state-of-the-art baselines. This fusion of black-box initialization with advanced semantic refinement yields a scalable and efficient solution, paving the way for next-generation LLM-driven applications in diverse real-world scenarios. The source code will be released soon.

[10] Digital Gatekeepers: Exploring Large Language Model’s Role in Immigration Decisions

Yicheng Mao,Yang Zhao

Main category: cs.CL

TL;DR: 该论文研究了大型语言模型（如GPT-3.5和GPT-4）在移民决策中的作用，发现其可以与人战略对齐，但也揭示了潜在的偏见问题。

Details

Motivation: 全球化与移民人口增加导致移民部门工作负担加重，需要公平决策的挑战。AI的整合为此提供了潜在解决方案。

Contribution: 论文揭示了LLMs在移民决策中的潜力与局限性，尤其在公平性和偏见方面提供了双重分析。

Method: 采用了混合方法，包括离散选择实验和深度访谈，研究LLMs的决策战略及其公平性。

Result: LLMs能够与人战略对齐，强调效用最大化和程序公平性，但仍存在国籍偏见和特权群体偏好。

Insight: LLMs在自动化移民决策中具有潜力，但需警惕其内在偏见，需进一步优化以确保公平性。

Abstract: With globalization and increasing immigrant populations, immigration departments face significant work-loads and the challenge of ensuring fairness in decision-making processes. Integrating artificial intelligence offers a promising solution to these challenges. This study investigates the potential of large language models (LLMs),such as GPT-3.5 and GPT-4, in supporting immigration decision-making. Utilizing a mixed-methods approach,this paper conducted discrete choice experiments and in-depth interviews to study LLM decision-making strategies and whether they are fair. Our findings demonstrate that LLMs can align their decision-making with human strategies, emphasizing utility maximization and procedural fairness. Meanwhile, this paper also reveals that while ChatGPT has safeguards to prevent unintentional discrimination, it still exhibits stereotypes and biases concerning nationality and shows preferences toward privileged group. This dual analysis highlights both the potential and limitations of LLMs in automating and enhancing immigration decisions.

[11] STRuCT-LLM: Unifying Tabular and Graph Reasoning with Reinforcement Learning for Semantic Parsing

Josefa Lia Stoisser,Marc Boubnovski Martell,Lawrence Phillips,Casper Hansen,Julien Fauqueur

Main category: cs.CL

TL;DR: STRuCT-LLM是一个统一框架，通过强化学习和Chain-of-Thought监督，训练LLMs在关系型和图结构数据上进行结构化推理。

Details

Motivation: 现有工作通常孤立处理关系型和图形式的数据，缺乏统一的框架来联合优化两种任务。

Contribution: 1. 提出了结合强化学习和CoT的统一框架；2. 引入了基于图编辑距离的拓扑感知奖励函数；3. 实现了SQL和Cypher间的跨形式迁移。

Method: 联合优化Text-to-SQL和Text-to-Cypher任务，采用强化学习和Chain-of-Thought监督，引入拓扑感知奖励。

Result: Spider任务提升13.5%，Text2Cypher任务提升73.1%，并在下游任务中展示了零样本泛化能力。

Insight: 联合训练SQL和Cypher能产生协同效应，以执行查询为脚手架的结构化推理是有效的。

Abstract: We propose STRuCT-LLM, a unified framework for training large language models (LLMs) to perform structured reasoning over both relational and graph-structured data. Our approach jointly optimizes Text-to-SQL and Text-to-Cypher tasks using reinforcement learning (RL) combined with Chain-of-Thought (CoT) supervision. To support fine-grained optimization in graph-based parsing, we introduce a topology-aware reward function based on graph edit distance. Unlike prior work that treats relational and graph formalisms in isolation, STRuCT-LLM leverages shared abstractions between SQL and Cypher to induce cross-formalism transfer, enabling SQL training to improve Cypher performance and vice versa - even without shared schemas. Our largest model (QwQ-32B) achieves substantial relative improvements across tasks: on semantic parsing, Spider improves by 13.5% and Text2Cypher by 73.1%. The model also demonstrates strong zero-shot generalization, improving performance on downstream tabular QA (TableBench: 8.5%) and knowledge graph QA (CR-LT-KGQA: 1.7%) without any QA-specific supervision. These results demonstrate both the effectiveness of executable queries as scaffolds for structured reasoning and the synergistic benefits of jointly training on SQL and Cypher (code available at https://github.com/bouv/STRuCT-LLM).

[12] Adapting Whisper for Parameter-efficient Code-Switching Speech Recognition via Soft Prompt Tuning

Hongli Yang,Yizhou Peng,Hao Huang,Sheng Li

Main category: cs.CL

TL;DR: 该论文探讨了通过Soft Prompt Tuning（SPT）这一参数高效方法提升大模型Whisper在低资源场景下的代码混合语音识别的性能，同时避免灾难性遗忘。提出了SPT4ASR组合策略，并在实验中验证了其有效性。

Details

Motivation: 大规模多语言ASR模型（如Whisper）在高资源场景表现优秀，但在低资源场景（如罕见语言和代码混合）中面临计算成本和灾难性遗忘的挑战。

Contribution: 提出了Soft Prompt Tuning（SPT）及其组合策略SPT4ASR，用于提升代码混合语音识别的性能，同时保持模型的参数高效性。

Method: 研究了两种策略：全微调（FFT）和仅训练soft prompts的SPT，并引入了SPT4ASR组合不同SPT变体。

Result: 深度提示调优是最有效的SPT方法，SPT4ASR在代码混合语音识别中进一步降低了错误率，且不影响现有语言的性能。

Insight: 参数高效方法（如SPT）可以显著提升低资源场景下的模型性能，同时避免灾难性遗忘。

Abstract: Large-scale multilingual ASR models like Whisper excel in high-resource settings but face challenges in low-resource scenarios, such as rare languages and code-switching (CS), due to computational costs and catastrophic forgetting. We explore Soft Prompt Tuning (SPT), a parameter-efficient method to enhance CS ASR while preserving prior knowledge. We evaluate two strategies: (1) full fine-tuning (FFT) of both soft prompts and the entire Whisper model, demonstrating improved cross-lingual capabilities compared to traditional methods, and (2) adhering to SPT’s original design by freezing model parameters and only training soft prompts. Additionally, we introduce SPT4ASR, a combination of different SPT variants. Experiments on the SEAME and ASRU2019 datasets show that deep prompt tuning is the most effective SPT approach, and our SPT4ASR methods achieve further error reductions in CS ASR, maintaining parameter efficiency similar to LoRA, without degrading performance on existing languages.

[13] Language-Aware Prompt Tuning for Parameter-Efficient Seamless Language Expansion in Multilingual ASR

Hongli Yang,Sheng Li,Hao Huang,Ayiduosi Tuohan,Yizhou Peng

Main category: cs.CL

TL;DR: 本文提出了一种语言感知的提示调优方法（LAPT）和完整软提示调优（Entire SPT），用于多语言ASR中的语言扩展，显著提升了性能，并开发了SPT-Whisper工具包支持高效持续学习。

Details

Motivation: 多语言ASR系统（如Whisper）在语言扩展和语言干扰方面仍存在挑战，需要高效且不影响性能的解决方案。

Contribution: 1) Entire SPT方法，通过软提示同时优化编码器和解码器；2) LAPT方法，利用跨语言相似性编码共享和语言特定特征；3) SPT-Whisper工具包，集成SPT支持高效持续学习。

Method: 1) Entire SPT：软提示应用于编码器和解码器；2) LAPT：轻量级提示矩阵编码共享和语言特定特征；3) 在FLEURS数据集上进行实验验证。

Result: Entire SPT和LAPT在语言扩展任务中分别比Decoder SPT提升5.0%和16.0%的性能。

Insight: 通过轻量化的提示调优方法，可以有效解决多语言ASR中的语言干扰和扩展问题，且计算开销低。

Abstract: Recent advancements in multilingual automatic speech recognition (ASR) have been driven by large-scale end-to-end models like Whisper. However, challenges such as language interference and expanding to unseen languages (language expansion) without degrading performance persist. This paper addresses these with three contributions: 1) Entire Soft Prompt Tuning (Entire SPT), which applies soft prompts to both the encoder and decoder, enhancing feature extraction and decoding; 2) Language-Aware Prompt Tuning (LAPT), which leverages cross-lingual similarities to encode shared and language-specific features using lightweight prompt matrices; 3) SPT-Whisper, a toolkit that integrates SPT into Whisper and enables efficient continual learning. Experiments across three languages from FLEURS demonstrate that Entire SPT and LAPT outperform Decoder SPT by 5.0% and 16.0% in language expansion tasks, respectively, providing an efficient solution for dynamic, multilingual ASR models with minimal computational overhead.

[14] From General Reasoning to Domain Expertise: Uncovering the Limits of Generalization in Large Language Models

Dana Alsagheer,Yang Lu,Abdulrahman Kamal,Omar Kamal,Mohammad Kamal,Nada Mansour,Cosmo Yang Wu,Rambiba Karanjai,Sen Li,Weidong Shi

Main category: cs.CL

TL;DR: 研究探讨了大型语言模型（LLMs）通用推理能力与领域特定推理任务表现之间的关系。

Details

Motivation: 随着AI技术的发展，训练LLMs在通用推理上表现优异的需求增加，但其在领域特定任务中的表现仍有待探索。

Contribution: 揭示了通用推理能力与领域专业推理之间的联系，为LLMs在特定领域的应用提供了理论支持。

Method: 通过分析LLMs在通用推理和领域特定任务中的表现，评估其能力转移的界限。

Result: 研究发现LLMs的通用推理能力并不总是能直接转化为领域专业推理的高效表现。

Insight: LLMs的通用推理能力虽然在多个领域有潜力，但在特定领域的专业推理仍需针对性优化。

Abstract: Recent advancements in Large Language Models (LLMs) have demonstrated remarkable capabilities in various domains. However, effective decision-making relies heavily on strong reasoning abilities. Reasoning is the foundation for decision-making, providing the analytical and logical framework to make sound choices. Reasoning involves analyzing information, drawing inferences, and reaching conclusions based on logic or evidence. Decision-making builds on this foundation by applying the insights from reasoning to select the best course of action among alternatives. Together, these processes create a continuous cycle of thought and action aimed at achieving goals effectively. As AI technology evolves, there is a growing trend to train LLMs to excel in general reasoning. This study explores how the general reasoning capabilities of LLMs connect to their performance in domain-specific reasoning tasks.

[15] VIDEE: Visual and Interactive Decomposition, Execution, and Evaluation of Text Analytics with Intelligent Agents

Sam Yu-Te Lee,Chengyang Ji,Shicheng Wen,Lifu Huang,Dongyi Liu,Kwan-Liu Ma

Main category: cs.CL

TL;DR: 论文提出VIDEE系统，通过智能代理支持非专业用户进行高级文本分析，包括分解、执行和评估三个阶段，并通过实验和用户研究验证其有效性。

Details

Motivation: 传统文本分析需要NLP专业知识，阻碍了非专业人士的使用；大型语言模型（LLMs）的发展使文本分析更易用，但仍需系统支持用户与代理协作。

Contribution: 1. 提出VIDEE系统，支持非专业用户与智能代理协作完成文本分析；2. 设计了一种结合人类反馈的蒙特卡洛树搜索算法；3. 通过实验和用户研究验证了系统的实用性和设计启示。

Method: 1. 分解阶段：采用人类参与的蒙特卡洛树搜索算法辅助生成式推理；2. 执行阶段：生成可执行的文本分析管道；3. 评估阶段：结合LLM的评估和可视化帮助用户验证结果。

Result: 实验和用户研究表明VIDEE对非专家用户有效，揭示了用户行为模式，并为未来智能文本分析系统的改进提供了设计启示。

Insight: 人类与代理的协作可以显著降低文本分析的门槛；人类反馈在生成式推理中的作用至关重要；系统设计需考虑不同用户背景的多样性。

Abstract: Text analytics has traditionally required specialized knowledge in Natural Language Processing (NLP) or text analysis, which presents a barrier for entry-level analysts. Recent advances in large language models (LLMs) have changed the landscape of NLP by enabling more accessible and automated text analysis (e.g., topic detection, summarization, information extraction, etc.). We introduce VIDEE, a system that supports entry-level data analysts to conduct advanced text analytics with intelligent agents. VIDEE instantiates a human-agent collaroration workflow consisting of three stages: (1) Decomposition, which incorporates a human-in-the-loop Monte-Carlo Tree Search algorithm to support generative reasoning with human feedback, (2) Execution, which generates an executable text analytics pipeline, and (3) Evaluation, which integrates LLM-based evaluation and visualizations to support user validation of execution results. We conduct two quantitative experiments to evaluate VIDEE’s effectiveness and analyze common agent errors. A user study involving participants with varying levels of NLP and text analytics experience – from none to expert – demonstrates the system’s usability and reveals distinct user behavior patterns. The findings identify design implications for human-agent collaboration, validate the practical utility of VIDEE for non-expert users, and inform future improvements to intelligent text analytics systems.

[16] Empirical Evidence for Alignment Faking in Small LLMs and Prompt-Based Mitigation Techniques

J. Koorndijk

Main category: cs.CL

TL;DR: 论文首次实证表明，即使是小型指令调优模型（如LLaMA 3 8B）也会出现对齐伪装行为，并通过提示干预显著减少这种行为，挑战了现有关于对齐伪装需大规模模型的假设。

Details

Motivation: 当前文献认为对齐伪装是大语言模型的涌现特性，但未验证小型模型是否也存在类似行为。研究填补了这一空白，并通过提示干预探索其缓解方法。

Contribution: 1. 首次证明小型指令调优模型存在对齐伪装；2. 提出提示干预方法（如义务论道德框架和草稿推理）显著减少该行为；3. 提出分类法区分浅层伪装（可通过提示抑制）和深层伪装（持久的目标驱动）。

Method: 通过实验验证LLaMA 3 8B的对齐伪装行为，并设计两种提示干预（义务论道德框架和草稿推理）进行行为矫正。

Result: 提示干预显著减少了小型模型的对齐伪装行为，表明无需修改模型内部结构即可实现行为改善。

Insight: 对齐伪装不仅限于大规模模型，且提示干预是有效的缓解手段。研究呼吁对不同规模和部署场景的模型进行更全面的对齐评估。

Abstract: Current literature suggests that alignment faking (deceptive alignment) is an emergent property of large language models. We present the first empirical evidence that a small instruction-tuned model, specifically LLaMA 3 8B, can also exhibit alignment faking. We further show that prompt-only interventions, including deontological moral framing and scratchpad reasoning, significantly reduce this behavior without modifying model internals. This challenges the assumption that prompt-based ethics are trivial and that deceptive alignment requires scale. We introduce a taxonomy distinguishing shallow deception, shaped by context and suppressible through prompting, from deep deception, which reflects persistent, goal-driven misalignment. Our findings refine the understanding of deception in language models and underscore the need for alignment evaluations across model sizes and deployment settings.

[17] Can Vision Language Models Understand Mimed Actions?

Hyundong Cho,Spencer Lin,Tejas Srinivasan,Michael Saxon,Deuksin Kwon,Natali T. Chavez,Jonathan May

Main category: cs.CL

TL;DR: 论文提出了MIME评测基准，用于评估视觉语言模型对模仿动作（mimed actions）的理解能力，发现现有模型表现显著低于人类，需进一步研究以提升模型对非语言交流的理解。

Details

Motivation: 非语言交流（NVC）在人类语言中扮演重要角色，但因其范围广且解释差异大，研究NVC具有挑战性。模仿动作（mime）作为NVC的子集，动作明确且解释差异小，是研究NVC的理想切入点。

Contribution: 提出了MIME评测基准，包含86种模仿动作的视频问答任务，通过运动捕捉数据生成，并引入人物、背景和视角的变化以评估识别鲁棒性。

Method: 利用运动捕捉数据构建MIME数据集，设计多样化的动作变体，评估现有视觉语言模型（VLM）在识别模仿动作上的表现。

Result: 实验表明，现有VLM（包括开源和API模型）在MIME上的表现显著低于人类。

Insight: 模仿动作理解是视觉语言模型掌握更复杂非语言交流的基础，当前模型在这一任务上仍有较大改进空间。

Abstract: Nonverbal communication (NVC) plays an integral role in human language, but studying NVC in general is challenging because of its broad scope and high variance in interpretation among individuals and cultures. However, mime – the theatrical technique of suggesting intent using only gesture, expression, and movement – is a subset of NVC that consists of explicit and embodied actions with much lower human interpretation variance. We argue that a solid understanding of mimed actions is a crucial prerequisite for vision-language models capable of interpreting and commanding more subtle aspects of NVC. Hence, we propose Mime Identification Multimodal Evaluation (MIME), a novel video-based question answering benchmark comprising of 86 mimed actions. Constructed with motion capture data, MIME consists of variations of each action with perturbations applied to the character, background, and viewpoint for evaluating recognition robustness. We find that both open-weight and API-based vision-language models perform significantly worse than humans on MIME, motivating the need for increased research for instilling more robust understanding of human gestures.

[18] Representation Consistency for Accurate and Coherent LLM Answer Aggregation

Junqi Jiang,Tom Bewley,Salim I. Amoukou,Francesco Leofante,Antonio Rago,Saumitra Mishra,Francesca Toni

Main category: cs.CL

TL;DR: 该论文提出了一种称为表示一致性（RC）的测试时扩展方法，用于聚合大语言模型（LLM）生成的多个候选回答，通过内部激活一致性增强答案的准确性和一致性，无需额外模型查询。

Details

Motivation: 现有测试时扩展方法通常需要复杂的提示和采样策略修改，而RC方法旨在简化这一过程，通过利用模型内部激活信息提升答案聚合的准确性和一致性。

Contribution: 提出了表示一致性（RC）方法，利用模型内部激活的一致性信息增强答案聚合性能，无需额外模型查询。

Method: 通过分析模型生成多个响应时的内部激活（密集或稀疏）一致性，动态调整答案聚合权重，稀疏激活信号与连贯推理概念一致。

Result: 在四个开源LLM和四个推理数据集上的实验表明，RC方法能提升任务性能（最高4%准确率提升），优于强基线方法。

Insight: 模型内部激活的一致性可以作为衡量答案可靠性的有效指标，稀疏激活信号与连贯推理高度相关。

Abstract: Test-time scaling improves large language models’ (LLMs) performance by allocating more compute budget during inference. To achieve this, existing methods often require intricate modifications to prompting and sampling strategies. In this work, we introduce representation consistency (RC), a test-time scaling method for aggregating answers drawn from multiple candidate responses of an LLM regardless of how they were generated, including variations in prompt phrasing and sampling strategy. RC enhances answer aggregation by not only considering the number of occurrences of each answer in the candidate response set, but also the consistency of the model’s internal activations while generating the set of responses leading to each answer. These activations can be either dense (raw model activations) or sparse (encoded via pretrained sparse autoencoders). Our rationale is that if the model’s representations of multiple responses converging on the same answer are highly variable, this answer is more likely to be the result of incoherent reasoning and should be down-weighted during aggregation. Importantly, our method only uses cached activations and lightweight similarity computations and requires no additional model queries. Through experiments with four open-source LLMs and four reasoning datasets, we validate the effectiveness of RC for improving task performance during inference, with consistent accuracy improvements (up to 4%) over strong test-time scaling baselines. We also show that consistency in the sparse activation signals aligns well with the common notion of coherent reasoning.

[19] FinEval-KR: A Financial Domain Evaluation Framework for Large Language Models’ Knowledge and Reasoning

Shaoyu Dou,Yutian Shen,Mofan Chen,Zixuan Wang,Jiajie Xu,Qi Guo,Kailai Shao,Chao Chen,Haixiang Hu,Haibo Shi,Min Min,Liwen Zhang

Main category: cs.CL

TL;DR: FinEval-KR 是一个用于解耦和量化大型语言模型（LLMs）在金融领域知识与推理能力的评估框架，提出独立的知识分数和推理分数指标，并基于Bloom分类法分析不同认知层级的推理能力。

Details

Motivation: 当前评估基准在复杂金融推理任务中未能解耦知识与推理能力，且缺乏对任务失败的根因分析。FinEval-KR旨在填补这一空白，提供一个更全面的评估工具。

Contribution: 1. 提出FinEval-KR框架，独立评估LLMs的金融知识与推理能力；2. 基于Bloom分类法引入认知分数；3. 发布开源中文金融推理数据集。

Method: 设计知识分数和推理分数指标，结合Bloom分类法分析认知层级能力，并在新数据集上进行实验验证。

Result: 实验表明，LLMs的推理能力和高阶认知能力是影响推理准确性的核心因素，且顶级模型在知识应用上仍存在瓶颈。

Insight: 研究发现，专业金融LLMs在多指标上普遍落后于顶级通用大模型，突显了跨领域能力的重要性。

Abstract: Large Language Models (LLMs) demonstrate significant potential but face challenges in complex financial reasoning tasks requiring both domain knowledge and sophisticated reasoning. Current evaluation benchmarks often fall short by not decoupling these capabilities indicators from single task performance and lack root cause analysis for task failure. To address this, we introduce FinEval-KR, a novel evaluation framework for decoupling and quantifying LLMs’ knowledge and reasoning abilities independently, proposing distinct knowledge score and reasoning score metrics. Inspired by cognitive science, we further propose a cognitive score based on Bloom’s taxonomy to analyze capabilities in reasoning tasks across different cognitive levels. We also release a new open-source Chinese financial reasoning dataset covering 22 subfields to support reproducible research and further advancements in financial reasoning. Our experimental results reveal that LLM reasoning ability and higher-order cognitive ability are the core factors influencing reasoning accuracy. We also specifically find that even top models still face a bottleneck with knowledge application. Furthermore, our analysis shows that specialized financial LLMs generally lag behind the top general large models across multiple metrics.

[20] Gazal-R1: Achieving State-of-the-Art Medical Reasoning with Parameter-Efficient Two-Stage Training

Ahmed M. Adly,Mostafa Samy,Amr Fawzy

Main category: cs.CL

TL;DR: Gazal-R1是一种320亿参数的语言模型，通过两阶段训练在医学推理任务上达到最先进性能，同时提供透明的临床决策解释。

Details

Motivation: 研究旨在通过高效的训练策略，使中等规模模型在专业领域（如医学推理）上超越更大规模的模型。

Contribution: 1. 提出两阶段训练方法（监督微调+强化学习）；2. 引入参数高效技术（DoRA和rsLoRA）；3. 设计多组件奖励系统的GRPO强化学习。

Method: 1. 监督微调使用合成医学推理数据集；2. 结合DoRA和rsLoRA技术；3. 通过GRPO强化学习优化模型性能。

Result: Gazal-R1在MedQA（87.1%）、MMLU Pro（Medical）（81.6%）和PubMedQA（79.6%）上表现优异，超越更大模型。

Insight: 1. 专业领域模型的性能可通过高效训练策略显著提升；2. 奖励设计和稳定性是关键挑战；3. 事实记忆与详细推理之间存在平衡问题。

Abstract: We present Gazal-R1, a 32-billion-parameter language model that achieves state-of-the-art performance in medical reasoning while providing transparent, step-by-step explanations for clinical decision-making. Built upon Qwen3 32B, our model demonstrates that strategic training can enable mid-sized models to outperform significantly larger counterparts in specialized domains. We developed a novel two-stage training pipeline: first, supervised fine-tuning on a carefully curated dataset of 107,033 synthetic medical reasoning examples that teaches structured clinical thinking, enhanced by advanced parameter-efficient techniques including Weight-Decomposed Low-Rank Adaptation (DoRA) and Rank-Stabilized LoRA (rsLoRA); second, reinforcement learning using Group Relative Policy Optimization (GRPO) with a sophisticated multi-component reward system that refines accuracy, format adherence, and reasoning quality. Gazal-R1 achieves exceptional performance across medical benchmarks, scoring 87.1% on MedQA, 81.6% on MMLU Pro (Medical), and 79.6% on PubMedQA, surpassing models up to 12x larger. Beyond its strong empirical results, this work provides detailed insights into the challenges of training reasoning-capable models in specialized domains, including issues with reward hacking, training instability, and the fundamental tension between factual recall and detailed reasoning. Our methodology offers a reproducible framework for developing high-capability, domain-specific language models that balance performance, efficiency, and explainability.

[21] Thunder-LLM: Efficiently Adapting LLMs to Korean with Minimal Resources

Jinpyo Kim,Gyeongje Cho,Chanwoo Park,Jongwon Park,Jongmin Kim,Yeonkyoun So,Jaejin Lee

Main category: cs.CL

TL;DR: 论文提出了一种低成本方法Thunder-LLM，将英文大语言模型高效适配到韩语，包括数据收集、预处理、训练、下游任务构建和评估的完整流程，并展示其在资源有限下的优越性能。

Details

Motivation: 当前主流大语言模型在英语或汉语之外的其他语言上表现不佳，且完整的训练流程因技术复杂性和商业保密性而未公开。本文旨在填补这一空白，提供低成本适配新语言的解决方案。

Contribution: 1. 提出完整的端到端流程，将英文LLM高效适配到韩语；2. 分享了数据收集、训练和评估的详细经验；3. 开源代码和模型Thunder-LLM，其在韩语任务中表现优于现有模型。

Method: 1. 收集并预处理韩语数据集；2. 基于现有英文LLM进行训练；3. 构建下游任务基准；4. 进行成本和性能评估。

Result: Thunder-LLM和Thunder-LLM-Ins在韩语任务中表现优异，且所需数据和计算资源极少。

Insight: 即使在低预算下，通过合理的流程设计，现有LLM可以高效适配新语言，语言多样性问题可通过开源和共享经验解决。

Abstract: Since state-of-the-art LLMs often underperform in languages other than English or Chinese, improving the capability of LLMs in new languages has become an essential task. Moreover, LLMs’ entire end-to-end training process remains largely unknown to the public due to proprietary reasons, technical complexity, inconsistent documentation, and ethical considerations. The complete picture remains a closely guarded secret within the industry. This paper presents methods to adapt an existing English-based LLM to Korean in a low-budget scenario. We describe the entire end-to-end process: collecting Korean datasets, preprocessing the data, training the model, creating downstream benchmarks, and conducting evaluations. The evaluation results indicate that our method can effectively and cost-efficiently add new language capabilities to existing LLMs. Our new bilingual models, Thunder-LLM and Thunder-LLM-Ins, achieve superior Korean performance compared to state-of-the-art models while utilizing minimal data and computational resources. We share our comprehensive experience and make the code publicly available.

[22] Evaluating Multimodal Large Language Models on Educational Textbook Question Answering

Hessa A. Alawwad,Anas Zafar,Areej Alhothali,Usman Naseem,Ali Alkhathlan,Amani Jamal

Main category: cs.CL

TL;DR: 该论文首次评估了多模态大语言模型（MLLMs）在教育教科书问答任务（TQA）中的表现，使用了CK12-QA数据集，并分析了LLaVA和LLaMA 3.2-Vision等模型的表现。同时，论文提出了一种轻量级的多模态检索增强生成（RAG）方法，整合了课程中的段落和图表，展示了检索到的教育上下文对模型性能的影响。

Details

Motivation: 尽管MLLMs在视觉语言任务中取得了显著成功，但其处理复杂、长篇课程和复杂教育图表的能力尚未得到充分测试。论文旨在填补这一空白，探索MLLMs在教育环境中的潜力。

Contribution: 1. 首次评估了MLLMs在TQA任务中的表现；2. 提出了一种轻量级多模态RAG方法，整合了课程中的段落和图表；3. 揭示了检索上下文对模型性能的影响及当前模型的局限性。

Method: 论文采用了CK12-QA数据集，评估了LLaVA和LLaMA 3.2-Vision等模型的性能，并提出了一种多模态RAG方法，将课程内容（文本和图表）集成到提示中，以增强模型的上下文理解能力。

Result: 实验结果显示，检索到的教育上下文显著提升了模型的准确性和推理能力，但也暴露了模型在处理问题-上下文关系和噪声方面的局限性。

Insight: 该研究为多模态AI在教育领域的应用提供了关键方向，指出了模型在处理复杂上下文时的改进需求以及未来研究的重点。

Abstract: Multimodal large language models (MLLMs) have recently achieved significant success in vision–language tasks. However, their capacity to reason over complex, long lessons and intricate educational diagrams that cannot be represented as a single natural image remains largely untested. In this work, we present the first evaluation of state-of-the-art MLLMs on the textbook question answering (TQA) task using the CK12-QA dataset. We assess the performance of recent vision-language models, including LLaVA and LLaMA 3.2-Vision, across various input configurations. Additionally, we introduce a lightweight multimodal retrieval-augmented generation (RAG) pipeline that integrates both paragraphs and diagrams from the lesson into the prompt. Our results demonstrate the influence of retrieved educational context on model accuracy and reasoning, while also revealing current limitations in handling question-context relationships and the potential for noise, pointing to key directions for future research in multimodal AI-driven learning.

[23] Overview of the ClinIQLink 2025 Shared Task on Medical Question-Answering

Brandon Colelough,Davis Bartels,Dina Demner-Fushman

Main category: cs.CL

TL;DR: 本文概述了ClinIQLink 2025共享任务，这是一个针对医学问答的大型语言模型测试任务，包含多种问答格式和自动化评分及医师评审。

Details

Motivation: 设计ClinIQLink共享任务的动机是为了在医学领域的问答任务中测试大型语言模型的性能，尤其是针对全科医生级别的需求。

Contribution: 主要贡献是提供了一套7种格式的4,978个专家验证的医学问答对，并设计了自动化评分和医师评审的双重评估机制。

Method: 任务使用Docker或Apptainer镜像部署系统，在CodaBench平台或Zaratan集群上执行。自动化评分分为闭式问答的精确匹配和开放式问答的嵌入度量，并由医师团队审核高分模型。

Result: 任务通过结合自动化评分和医师评审，有效评估了大型语言模型在医学问答任务中的表现。

Insight: 任务设计强调了医学问答的多样性和专家验证的重要性，为未来医学AI应用提供了标准化的测试框架。

Abstract: In this paper, we present an overview of ClinIQLink, a shared task, collocated with the 24th BioNLP workshop at ACL 2025, designed to stress-test large language models (LLMs) on medically-oriented question answering aimed at the level of a General Practitioner. The challenge supplies 4,978 expert-verified, medical source-grounded question-answer pairs that cover seven formats: true/false, multiple choice, unordered list, short answer, short-inverse, multi-hop, and multi-hop-inverse. Participating systems, bundled in Docker or Apptainer images, are executed on the CodaBench platform or the University of Maryland’s Zaratan cluster. An automated harness (Task 1) scores closed-ended items by exact match and open-ended items with a three-tier embedding metric. A subsequent physician panel (Task 2) audits the top model responses.

[24] Structured Attention Matters to Multimodal LLMs in Document Understanding

Chang Liu,Hongkai Chen,Yujun Cai,Hang Wu,Qingwen Ye,Ming-Hsuan Yang,Yiwei Wang

Main category: cs.CL

TL;DR: 研究发现，原始OCR文本输入会损害多模态大语言模型（MLLMs）的文档理解能力，提出了一种基于LaTex范式的结构保留方法，显著提升了模型的问答性能。

Details

Motivation: 探索输入格式对MLLMs文档理解性能的影响，尤其是发现未经处理的原始OCR文本反而会降低模型表现。

Contribution: 揭示了结构化文本能够引导注意力集中到语义关键区域，提出了一种无需修改模型结构的LaTex范式编码方法。

Method: 采用LaTex范式编码文档元素，保持层次结构和空间关系，通过注意力分析验证结构化输入的优势。

Result: 新方法显著提升了MLLMs在多种文档类型上的问答性能。

Insight: 结构化输入能够引导模型注意力，减少注意力分散，提升理解能力。

Abstract: Document understanding remains a significant challenge for multimodal large language models (MLLMs). While previous research has primarily focused on locating evidence pages through precise multimodal queries, our work investigates a fundamental yet overlooked aspect: how input format influences document comprehension performance. Through systematic analysis, we discover that raw OCR text often impairs rather than improves MLLMs’ performance, which is a counterintuitive finding we attribute to attention dispersion and structure loss. To further substantiate our hypothesis, we propose a novel structure-preserving approach that encodes document elements using the LaTex paradigm, maintaining the hierarchical organization and spatial relationships critical for comprehension. Our attention analysis reveals that structured text induces structured attention patterns on both textual and visual content, directing models to focus on semantically meaningful regions while reducing attention waste. This approach significantly enhances MLLMs’ document question answering performance across diverse document types without requiring architectural modifications or additional training.

[25] BiMark: Unbiased Multilayer Watermarking for Large Language Models

Xiaoyan Feng,He Zhang,Yanjun Zhang,Leo Yu Zhang,Shirui Pan

Main category: cs.CL

TL;DR: BiMark 是一种新颖的水印框架，通过创新的无偏重加权机制、多层架构和信息编码方法，解决了大型语言模型生成文本的认证问题，同时保持文本质量和高检测率。

Details

Motivation: 大型语言模型（LLMs）生成的文本难以区分其来源，引发了认证和监管需求。现有水印技术难以同时实现文本质量保持、模型无关检测和多比特水印嵌入，BiMark 致力于解决这一挑战。

Contribution: 1. 提出无偏比特翻转重加权机制，实现模型无关检测；2. 多层架构提升检测能力而不损害生成质量；3. 支持多比特水印的信息编码方法。

Method: BiMark 结合了无偏重加权、多层架构和高效信息编码，通过理论分析和实验验证其平衡文本质量与水印能力的有效性。

Result: 与现有多比特水印方法相比，BiMark 在短文本中的提取率提高30%，同时保持较低的困惑度，且在下游任务（如摘要和翻译）中与非水印文本表现相当。

Insight: BiMark 的创新设计为LLM生成的文本认证提供了实用且高效的解决方案，其多层架构和无偏机制对其他水印技术具有启发意义。

Abstract: Recent advances in Large Language Models (LLMs) have raised urgent concerns about LLM-generated text authenticity, prompting regulatory demands for reliable identification mechanisms. Although watermarking offers a promising solution, existing approaches struggle to simultaneously achieve three critical requirements: text quality preservation, model-agnostic detection, and message embedding capacity, which are crucial for practical implementation. To achieve these goals, the key challenge lies in balancing the trade-off between text quality preservation and message embedding capacity. To address this challenge, we propose BiMark, a novel watermarking framework that achieves these requirements through three key innovations: (1) a bit-flip unbiased reweighting mechanism enabling model-agnostic detection, (2) a multilayer architecture enhancing detectability without compromising generation quality, and (3) an information encoding approach supporting multi-bit watermarking. Through theoretical analysis and extensive experiments, we validate that, compared to state-of-the-art multi-bit watermarking methods, BiMark achieves up to 30% higher extraction rates for short texts while maintaining text quality indicated by lower perplexity, and performs comparably to non-watermarked text on downstream tasks such as summarization and translation.

[26] Operationalizing Automated Essay Scoring: A Human-Aware Approach

Yenisel Plasencia-Calaña

Main category: cs.CL

TL;DR: 论文探讨了自动化作文评分（AES）系统的人性化实现，不仅关注准确性，还比较了机器学习与大型语言模型（LLM）方法的优劣。研究发现ML模型准确性更高，但LLM在可解释性上表现更好；两者在偏见和边缘分数鲁棒性上均存在不足。

Details

Motivation: 自动化作文评分系统需要超越单纯的准确性，关注人性化实现中的关键维度（如偏见、鲁棒性和可解释性），以提升系统的可靠性和信任度。

Contribution: 1. 对比ML与LLM在AES中的表现；2. 发现ML模型精度更高但可解释性较差，LLM反之；3. 揭示两者在偏见和鲁棒性上的共同问题。

Method: 通过比较ML（如传统机器学习模型）和LLM（如GPT类模型）在AES任务上的表现，分析其在准确性、可解释性、偏见和鲁棒性等维度的差异。

Result: ML模型在准确性上优于LLM，但LLM能提供更丰富的解释；两者在偏见和边缘分数处理上均表现不佳。

Insight: AES系统设计需权衡不同方法的优缺点，未来可结合ML的高精度与LLM的可解释性，同时解决偏见和鲁棒性问题。

Abstract: This paper explores the human-centric operationalization of Automated Essay Scoring (AES) systems, addressing aspects beyond accuracy. We compare various machine learning-based approaches with Large Language Models (LLMs) approaches, identifying their strengths, similarities and differences. The study investigates key dimensions such as bias, robustness, and explainability, considered important for human-aware operationalization of AES systems. Our study shows that ML-based AES models outperform LLMs in accuracy but struggle with explainability, whereas LLMs provide richer explanations. We also found that both approaches struggle with bias and robustness to edge scores. By analyzing these dimensions, the paper aims to identify challenges and trade-offs between different methods, contributing to more reliable and trustworthy AES methods.

[27] MemBench: Towards More Comprehensive Evaluation on the Memory of LLM-based Agents

Haoran Tan,Zeyu Zhang,Chen Ma,Xu Chen,Quanyu Dai,Zhenhua Dong

Main category: cs.CL

TL;DR: MemBench 是一个用于全面评估基于 LLM 的智能体记忆能力的数据集和基准测试工具，涵盖多层次的记忆类型和交互场景，并提供多维度评估指标。

Details

Motivation: 当前对基于 LLM 的智能体记忆能力的评估存在局限，缺乏多样化的记忆层次和交互场景，以及多维度的综合指标。

Contribution: 提出 MemBench，包含多样化的记忆层次（事实记忆和反思记忆）和交互场景（参与和观察），并提供多维度（有效性、效率和容量）的评估基准。

Method: 构建了一个综合数据集，并设计了一个基准测试工具，通过多任务和多场景评估智能体的记忆能力。

Result: 发布了一个开源的数据集和基准测试工具，为研究社区提供了更全面的记忆能力评估方法。

Insight: 记忆能力评估需要覆盖不同类型的记忆层次和交互场景，而多维度的指标设计能更全面地反映智能体的记忆能力。

Abstract: Recent works have highlighted the significance of memory mechanisms in LLM-based agents, which enable them to store observed information and adapt to dynamic environments. However, evaluating their memory capabilities still remains challenges. Previous evaluations are commonly limited by the diversity of memory levels and interactive scenarios. They also lack comprehensive metrics to reflect the memory capabilities from multiple aspects. To address these problems, in this paper, we construct a more comprehensive dataset and benchmark to evaluate the memory capability of LLM-based agents. Our dataset incorporates factual memory and reflective memory as different levels, and proposes participation and observation as various interactive scenarios. Based on our dataset, we present a benchmark, named MemBench, to evaluate the memory capability of LLM-based agents from multiple aspects, including their effectiveness, efficiency, and capacity. To benefit the research community, we release our dataset and project at https://github.com/import-myself/Membench.

[28] From Thinking to Output: Chain-of-Thought and Text Generation Characteristics in Reasoning Language Models

Junhao Liu,Zhenhao Xu,Yuxin Fang,Yichuan Chen,Zuobin Ying,Wenhan Chang

Main category: cs.CL

TL;DR: 论文提出了一种分析大型语言模型推理特性的新框架，通过关键词统计和LLM-as-a-judge范式，比较了GPT-o1、DeepSeek-R1、Kimi-k1.5和Grok-3四种模型的推理过程和输出，揭示了它们在权衡探索与利用、问题处理及结论达成方面的模式差异。

Details

Motivation: 现有研究缺乏对大型语言模型推理过程和输出的系统性比较，尤其是在自反思模式（'Aha moment'）和跨领域关联方面。

Contribution: 提出了一个新颖的分析框架，通过定量和定性方法揭示了不同模型在推理深度、中间步骤依赖及与GPT-o1的相似度等方面的差异。

Method: 使用关键词统计和LLM-as-a-judge范式，分析了四种大型推理模型的内部思考过程与最终输出的关联性，并提出了一套评估推理连贯性和输出准确性的指标。

Result: 研究发现不同模型在推理过程中展现出不同的模式，包括探索与利用的平衡、问题处理方式及结论达成路径。

Insight: 研究为计算效率与推理稳健性之间的权衡提供了见解，并为实际应用中的模型设计与评估提出了实用建议。

Abstract: Recently, there have been notable advancements in large language models (LLMs), demonstrating their growing abilities in complex reasoning. However, existing research largely overlooks a thorough and systematic comparison of these models’ reasoning processes and outputs, particularly regarding their self-reflection pattern (also termed “Aha moment”) and the interconnections across diverse domains. This paper proposes a novel framework for analyzing the reasoning characteristics of four cutting-edge large reasoning models (GPT-o1, DeepSeek-R1, Kimi-k1.5, and Grok-3) using keywords statistic and LLM-as-a-judge paradigm. Our approach connects their internal thinking processes with their final outputs. A diverse dataset consists of real-world scenario-based questions covering logical deduction, causal inference, and multi-step problem-solving. Additionally, a set of metrics is put forward to assess both the coherence of reasoning and the accuracy of the outputs. The research results uncover various patterns of how these models balance exploration and exploitation, deal with problems, and reach conclusions during the reasoning process. Through quantitative and qualitative comparisons, disparities among these models are identified in aspects such as the depth of reasoning, the reliance on intermediate steps, and the degree of similarity between their thinking processes and output patterns and those of GPT-o1. This work offers valuable insights into the trade-off between computational efficiency and reasoning robustness and provides practical recommendations for enhancing model design and evaluation in practical applications. We publicly release our project at: https://github.com/ChangWenhan/FromThinking2Output

[29] Does Multimodality Lead to Better Time Series Forecasting?

Xiyuan Zhang,Boran Han,Haoyang Fang,Abdul Fatir Ansari,Shuai Zhang,Danielle C. Maddix,Cuixiong Hu,Andrew Gordon Wilson,Michael W. Mahoney,Hao Wang,Yan Liu,Huzefa Rangwala,George Karypis,Bernie Wang

Main category: cs.CL

TL;DR: 本文系统地研究了多模态（文本信号）是否有助于时间序列预测，发现其效果并非普遍适用，并提出了模型和数据特性的关键条件。

Details

Motivation: 探讨多模态（尤其是文本信息）在时间序列预测中的作用，以明确其适用条件和潜在增益。

Contribution: 系统地评估了多模态预测方法的有效性，明确了模型架构和数据特性对增益的影响，并提供了实用指南。

Method: 对比了两种主流多模态方法（对齐基方法和提示基方法），并通过14个任务和7个领域的数据进行实验验证。

Result: 多模态方法的增益取决于文本模型容量、时间序列模型能力、对齐策略、数据量和文本信号互补性等条件。

Insight: 多模态并非总是有效；文本信息的高质量和互补性、模型能力匹配及数据充足性是关键成功因素。

Abstract: Recently, there has been growing interest in incorporating textual information into foundation models for time series forecasting. However, it remains unclear whether and under what conditions such multimodal integration consistently yields gains. We systematically investigate these questions across a diverse benchmark of 14 forecasting tasks spanning 7 domains, including health, environment, and economics. We evaluate two popular multimodal forecasting paradigms: aligning-based methods, which align time series and text representations; and prompting-based methods, which directly prompt large language models for forecasting. Although prior works report gains from multimodal input, we find these effects are not universal across datasets and models, and multimodal methods sometimes do not outperform the strongest unimodal baselines. To understand when textual information helps, we disentangle the effects of model architectural properties and data characteristics. Our findings highlight that on the modeling side, incorporating text information is most helpful given (1) high-capacity text models, (2) comparatively weaker time series models, and (3) appropriate aligning strategies. On the data side, performance gains are more likely when (4) sufficient training data is available and (5) the text offers complementary predictive signal beyond what is already captured from the time series alone. Our empirical findings offer practical guidelines for when multimodality can be expected to aid forecasting tasks, and when it does not.

[30] LastingBench: Defend Benchmarks Against Knowledge Leakage

Yixiong Fang,Tianran Sun,Yuling Shi,Min Wang,Xiaodong Gu

Main category: cs.CL

TL;DR: LastingBench 是一个新框架，旨在通过扰动识别并重写泄露点，保护 QA 基准测试免受大语言模型记忆效应的影响。

Details

Motivation: 随着大语言模型的复杂化，模型可能通过记忆任务数据“作弊”，削弱基准测试的有效性。现有方法主要关注泄露检测，但对减少其影响的解决方案研究不足。

Contribution: 提出 LastingBench 框架，通过扰动和反事实重写泄露点，减少记忆效应，确保基准测试的长期有效性。

Method: LastingBench 通过扰动识别上下文中的泄露点，将其重写为反事实内容，破坏模型记忆并保留原评估意图。

Result: 实验表明，LastingBench 在减少记忆效应方面显著有效，展现了基准测试的性能差距。

Insight: 维护基准测试的长期有效性需要动态防御策略，而不仅是泄露检测。

Abstract: The increasing complexity of large language models (LLMs) raises concerns about their ability to “cheat” on standard Question Answering (QA) benchmarks by memorizing task-specific data. This undermines the validity of benchmark evaluations, as they no longer reflect genuine model capabilities but instead the effects of data leakage. While prior work has focused on detecting such leakage, little attention has been given to mitigating its impact and preserving the long-term utility of benchmarks. In this paper, we introduce LastingBench, a novel framework designed to continuously reinforce and safeguard existing benchmarks against knowledge leakage. LastingBench identifies leakage points in the context through perturbation, then rewrites the leakage points to counterfactual ones-disrupting memorization while preserving the benchmark’s original evaluative intent. Evaluations of state-of-the-art QA benchmarks show significant performance gaps, highlighting the efficacy of LastingBench in reducing memorization effects. LastingBench offers a practical and scalable solution to ensure benchmark robustness over time, promoting fairer and more interpretable evaluations of LLMs.

[31] Refine Medical Diagnosis Using Generation Augmented Retrieval and Clinical Practice Guidelines

Wenhao Li,Hongkuan Zhang,Hongwei Zhang,Zhengxu Li,Zengjie Dong,Yafan Chen,Niranjan Bidargaddi,Hong Liu

Main category: cs.CL

TL;DR: 论文提出了一种名为GARMLE-G的生成增强检索框架，通过整合大语言模型预测和权威临床实践指南，解决了现有医学语言模型在诊断中的局限性。

Details

Motivation: 现有的医学语言模型通常基于ICD代码预测诊断，但ICD代码无法捕捉医生在实际诊断中的复杂推理和临床指南依据，导致模型临床实用性受限。

Contribution: 论文的主要贡献是提出了一种生成增强检索框架GARMLE-G，能够直接检索权威临床指南内容，生成与临床实践一致的建议，避免了模型幻觉问题。

Method: 方法包括：(1) 整合大语言模型预测和电子健康记录生成语义丰富的查询，(2) 通过嵌入相似性检索相关临床指南片段，(3) 将指南内容与模型输出融合生成临床建议。

Result: 原型系统在高血压诊断任务中表现出色，检索精度、语义相关性和临床指南一致性均优于基于RAG的基线模型，且架构轻量适合本地医疗部署。

Insight: 论文的创新点在于通过直接检索权威指南内容而非依赖模型生成文本，确保了临床建议的准确性和可靠性，为医学语言模型的实际应用提供了可行方案。

Abstract: Current medical language models, adapted from large language models (LLMs), typically predict ICD code-based diagnosis from electronic health records (EHRs) because these labels are readily available. However, ICD codes do not capture the nuanced, context-rich reasoning clinicians use for diagnosis. Clinicians synthesize diverse patient data and reference clinical practice guidelines (CPGs) to make evidence-based decisions. This misalignment limits the clinical utility of existing models. We introduce GARMLE-G, a Generation-Augmented Retrieval framework that grounds medical language model outputs in authoritative CPGs. Unlike conventional Retrieval-Augmented Generation based approaches, GARMLE-G enables hallucination-free outputs by directly retrieving authoritative guideline content without relying on model-generated text. It (1) integrates LLM predictions with EHR data to create semantically rich queries, (2) retrieves relevant CPG knowledge snippets via embedding similarity, and (3) fuses guideline content with model output to generate clinically aligned recommendations. A prototype system for hypertension diagnosis was developed and evaluated on multiple metrics, demonstrating superior retrieval precision, semantic relevance, and clinical guideline adherence compared to RAG-based baselines, while maintaining a lightweight architecture suitable for localized healthcare deployment. This work provides a scalable, low-cost, and hallucination-free method for grounding medical language models in evidence-based clinical practice, with strong potential for broader clinical deployment.

[32] IndexTTS2: A Breakthrough in Emotionally Expressive and Duration-Controlled Auto-Regressive Zero-Shot Text-to-Speech

Siyi Zhou,Yiquan Zhou,Yi He,Xun Zhou,Jinchao Wang,Wei Deng,Jingchen Shu

Main category: cs.CL

TL;DR: IndexTTS2 提出了一种新型的自回归语音合成方法，能够精确控制语音持续时间，并实现了音色与情感的分离控制。通过 GPT 潜在表征增强清晰度，结合软指令机制，其在零样本设置下显著提升了语音合成的表现。

Details

Motivation: 现有的自回归语音合成模型难以精确控制语音持续时间，且情感表达与说话者身份高度耦合，限制了其在严格音视频同步等场景的应用。

Contribution: 1. 提出了一种支持精确持续时间控制和自由生成两种模式的自回归模型；2. 实现了音色与情感的分离控制；3. 结合 GPT 潜在表征和软指令机制，提升了情感表达的清晰度和可控性。

Method: 1. 提出新的持续时间控制方法；2. 使用 GPT 潜在表征增强稳定性；3. 基于 Qwen3 设计了软指令机制，支持自然语言引导情感表达。

Result: 实验表明，IndexTTS2 在词错误率、说话者相似度和情感保真度方面优于现有零样本 TTS 模型。

Insight: 通过分离音色与情感控制，结合自然语言引导，语音合成的灵活性与表现力显著提升。

Abstract: Large-scale text-to-speech (TTS) models are typically categorized into autoregressive and non-autoregressive systems. Although autoregressive systems exhibit certain advantages in speech naturalness, their token-by-token generation mechanism makes it difficult to precisely control the duration of synthesized speech. This is a key limitation in applications such as video dubbing that require strict audio-visual synchronization. This paper introduces IndexTTS2, which proposes a novel and autoregressive-model-friendly method for speech duration control. The method supports two generation modes: one allows explicit specification of the number of generated tokens for precise duration control; the other does not require manual input and lets the model freely generate speech while preserving prosodic characteristics from the input prompt. Furthermore, IndexTTS2 achieves disentanglement between emotional expression and speaker identity, enabling independent control of timbre and emotion. In the zero-shot setting, the model can perfectly reproduce the emotional characteristics of the input prompt. Users may also provide a separate emotion prompt, even from a different speaker, allowing the model to reconstruct the target timbre while conveying the desired emotion. To enhance clarity during strong emotional expressions, we incorporate GPT latent representations to improve speech stability. Meanwhile, to lower the barrier for emotion control, we design a soft instruction mechanism based on textual descriptions by fine-tuning Qwen3. This enables effective guidance of speech generation with desired emotional tendencies using natural language input. Experimental results demonstrate that IndexTTS2 outperforms existing state-of-the-art zero-shot TTS models in word error rate, speaker similarity, and emotional fidelity.

[33] Doc2SAR: A Synergistic Framework for High-Fidelity Extraction of Structure-Activity Relationships from Scientific Documents

Jiaxi Zhuang,Kangning Li,Jue Hou,Mingjun Xu,Zhifeng Gao,Hengxing Cai

Main category: cs.CL

TL;DR: 这篇论文提出了 Doc2SAR，一个结合领域专用工具和微调多模态大模型的框架，用于从科学文献中高保真地提取结构-活性关系（SARs），并在新基准 DocSAR-200 上显著优于现有方法。

Details

Motivation: 从科学文献和专利中提取分子结构-活性关系（SARs）对药物发现和材料研究至关重要，但现有方法因文档格式多样性和专用任务（如化学结构识别）的局限性难以胜任。

Contribution: 1. 提出了 DocSAR-200 基准数据集；2. 开发了 Doc2SAR 框架，结合领域工具和微调的大模型，显著提升了 SARs 提取的准确性和泛化能力。

Method: Doc2SAR 整合了领域专用工具（如化学结构识别）和经过监督微调（SFT）的多模态大模型，通过协同框架处理异构文档格式。

Result: 在 DocSAR-200 上，Doc2SAR 的表格召回率达到 80.78%，比 GPT-4o 高出 51.48%，并在实用性和高效推理方面表现突出。

Insight: 结合领域专用工具与大模型微调是解决异构文档格式和专用任务挑战的有效途径。

Abstract: Extracting molecular structure-activity relationships (SARs) from scientific literature and patents is essential for drug discovery and materials research. However, this task remains challenging due to heterogeneous document formats and limitations of existing methods. Specifically, rule-based approaches relying on rigid templates fail to generalize across diverse document layouts, while general-purpose multimodal large language models (MLLMs) lack sufficient accuracy and reliability for specialized tasks, such as layout detection and optical chemical structure recognition (OCSR). To address these challenges, we introduce DocSAR-200, a rigorously annotated benchmark of 200 scientific documents designed specifically for evaluating SAR extraction methods. Additionally, we propose Doc2SAR, a novel synergistic framework that integrates domain-specific tools with MLLMs enhanced via supervised fine-tuning (SFT). Extensive experiments demonstrate that Doc2SAR achieves state-of-the-art performance across various document types, significantly outperforming leading end-to-end baselines. Specifically, Doc2SAR attains an overall Table Recall of 80.78% on DocSAR-200, exceeding end2end GPT-4o by 51.48%. Furthermore, Doc2SAR demonstrates practical usability through efficient inference and is accompanied by a web app.

[34] DeepTalk: Towards Seamless and Smart Speech Interaction with Adaptive Modality-Specific MoE

Hang Shao,Heting Gao,Yunhang Shen,Jiawei Chen,Lijiang Li,Zuwei Long,Bo Tong,Ke Li,Xing Sun

Main category: cs.CL

TL;DR: DeepTalk通过自适应模态专家学习框架解决了原生多模态大语言模型（MLLMs）在语音和文本生成中的性能退化问题，显著降低了性能下降，并保持了低延迟的语音交互体验。

Details

Motivation: 原生多模态大语言模型（MLLMs）虽然能直接生成语音响应，但受限于语音-文本配对数据不足，导致性能下降和灾难性遗忘问题，需要一种更高效的框架来解决这些挑战。

Contribution: 提出了DeepTalk框架，采用Mixture of Experts（MoE）架构，通过自适应模态专家学习和联合多模态训练，显著减少性能下降（仅5.5%），并保持低延迟（0.5秒内）。

Method: DeepTalk首先根据模态负载自适应区分模态专家，进行单模态训练，再进行联合多模态协同训练，实现了模态间的协同优化和高效生成。

Result: DeepTalk在性能上仅下降5.5%，远低于原生MLLMs（通常下降20%以上），同时对话延迟保持在0.5秒内，与模块化MLLMs性能相当。

Insight: 通过MoE架构和模态专家分离训练，DeepTalk展示了在多模态学习中平衡性能与效率的潜力，为未来智能语音交互提供了新思路。

Abstract: Native multimodal large language models (MLLMs) restructure a single large language model (LLM) into a spoken language model (SLM) capable of both speech and text generation. Compared to modular and aligned MLLMs, native MLLMs preserve richer paralinguistic features such as emotion and prosody, and generate speech responses directly within the backbone LLM rather than using a separate speech decoder. This integration also results in lower response latency and smoother interaction. However, native MLLMs suffer from catastrophic forgetting and performance degradation because the available paired speech-text data is insufficient to support the pretraining of MLLMs compared to the vast amount of text data required to pretrain text LLMs. To address this issue, we propose DeepTalk, a framework for adaptive modality expert learning based on a Mixture of Experts (MoE) architecture. DeepTalk first adaptively distinguishes modality experts according to their modality load within the LLM. Each modality expert then undergoes specialized single-modality training, followed by joint multimodal collaborative training. As a result, DeepTalk incurs only a 5.5% performance drop compared to the original LLM, which is significantly lower than the average performance drop of over 20% typically seen in native MLLMs (such as GLM-4-Voice), and is on par with modular MLLMs. Meanwhile, the end-to-end dialogue latency remains within 0.5 seconds, ensuring a seamless and intelligent speech interaction experience. Code and models are released at https://github.com/talkking/DeepTalk.

[35] WildSpeech-Bench: Benchmarking Audio LLMs in Natural Speech Conversation

Jian Zhang,Linhao Zhang,Bokai Lei,Chuhan Wu,Wei Jia,Xiao Zhou

Main category: cs.CL

TL;DR: 该论文提出了WildSpeech-Bench，一个专门用于评估音频大语言模型（LLMs）在实际语音对话中性能的基准测试工具。通过收集真实的语音对话数据、引入多样化的说话者属性和声学条件，并设计查询感知的评估方法，该研究填补了现有文本基准测试在语音场景中的不足。

Details

Motivation: 现有的大语言模型（如GPT-4o）在语音交互中表现出强大能力，但缺乏专门针对语音场景的全面评估工具。现有方法往往直接采用文本基准测试，忽略了语音特有的挑战（如韵律、同音词、口吃等）。

Contribution: 1. 提出WildSpeech-Bench，首个专注于语音对话场景的音频LLM基准测试工具；2. 通过系统收集真实语音数据，模拟多样化说话者和声学条件；3. 设计查询感知的评估方法，提升自动化评估的准确性。

Method: 1. 从真实语音对话中收集并整理数据；2. 引入说话者属性和声学条件的多样性；3. 基于语音特有现象（如韵律、同音词）增强数据集；4. 使用查询感知的评估方法，结合定制化检查表和提示。

Result: 对不同主流语音模型进行全面测试，结果显示模型在不同语音场景中表现差异显著。查询感知评估方法能够更细粒度地评估语音特有场景下的模型性能。

Insight: 语音LLM的评估需要特别关注语音特有的现象和用户期望，直接照搬文本基准测试可能无法全面反映模型在语音场景中的性能。WildSpeech-Bench为语音模型开发提供了更精准的评估工具。

Abstract: Recent multi-modal Large Language Models (LLMs) such as GPT-4o have demonstrated strong capabilities of direct speech interaction. However, the lack of specialized and comprehensive benchmarks for end-to-end speech LLM evaluation hinders optimizing the user experience of Audio LLMs in real-world applications. Existing evaluation methods often adapt text-based benchmarks, overlooking speech’s unique characteristics and challenges, including prosody, homophones, stuttering, and differing user expectations. Here, we present a novel approach to thoroughly evaluate LLMs in practical speech conversations. We systematically curate real-world chat data relevant to spoken scenarios, introduce diversity in speaker attributes and acoustic conditions, and augment the dataset with speech-specific phenomena. We further design a query-aware evaluation method to use customized evaluation checklists and prompts to enhance the accuracy of automatic evaluation. We conduct comprehensive testing and detailed analysis of various mainstream speech models, revealing significant differences in model performance across different speech scenarios. The use of query-aware evaluation further enables a finer-grained assessment under various speech-specific scenarios. Our benchmark can provide valuable insights for speech model development and evaluation.

[36] Do Vision-Language Models Have Internal World Models? Towards an Atomic Evaluation

Qiyue Gao,Xinyu Pi,Kevin Liu,Junrong Chen,Ruolan Yang,Xinqi Huang,Xinyu Fang,Lu Sun,Gautham Kishore,Bo Ai,Stone Tao,Mengyang Liu,Jiaxi Yang,Chao-Jung Lai,Chuanyang Jin,Jiannan Xiang,Benhao Huang,Zeming Chen,David Danks,Hao Su,Tianmin Shu,Ziqiao Ma,Lianhui Qin,Zhiting Hu

Main category: cs.CL

TL;DR: 该论文提出了一个系统化的两阶段框架（感知与预测），用于评估视觉语言模型（VLMs）的内部世界模型（WMs）能力，并引入了一个大规模基准测试WM-ABench。通过对15个最新商业和开源VLMs的660项实验，发现这些模型在基础世界建模能力上存在显著局限性。

Details

Motivation: 当前的大型视觉语言模型表现出作为通用世界模型的潜力，但缺乏对其基本世界建模能力的系统性评估。论文旨在填补这一空白，通过心理学和认知科学启发的方法，量化VLMs的WMs能力。

Contribution: 论文的主要贡献包括：（1）提出一个两阶段框架，从感知和预测角度评估VLMs的世界模型能力；（2）引入WM-ABench基准测试，涵盖23个细化维度和6个模拟环境；（3）通过大规模实验揭示了VLMs在世界建模方面的显著局限性。

Method: 论文采用了两阶段评估框架：（1）感知阶段，评估视觉、空间、时间、数量和运动理解；（2）预测阶段，评估机制模拟、传递推理和组合推理。基准测试WM-ABench通过控制反事实仿真生成数据，覆盖多样化任务。

Result: 实验结果表明，当前VLMs在基础世界建模能力上表现较差，例如在区分运动轨迹时接近随机准确率，且在解耦理解上存在偏差（如对颜色影响速度的错误认知）。这些模型与人类水平的世界建模能力存在显著差距。

Insight: 论文揭示了当前VLMs的能力瓶颈，尤其是在动态场景和理解因果关系上的不足。这些发现为未来模型设计提供了重要方向，例如需要增强对物理规律和时序关系的建模能力。

Abstract: Internal world models (WMs) enable agents to understand the world’s state and predict transitions, serving as the basis for advanced deliberative reasoning. Recent large Vision-Language Models (VLMs), such as OpenAI o3, GPT-4o and Gemini, exhibit potential as general-purpose WMs. While the latest studies have evaluated and shown limitations in specific capabilities such as visual understanding, a systematic evaluation of VLMs’ fundamental WM abilities remains absent. Drawing on comparative psychology and cognitive science, we propose a two-stage framework that assesses Perception (visual, spatial, temporal, quantitative, and motion) and Prediction (mechanistic simulation, transitive inference, compositional inference) to provide an atomic evaluation of VLMs as WMs. Guided by this framework, we introduce WM-ABench, a large-scale benchmark comprising 23 fine-grained evaluation dimensions across 6 diverse simulated environments with controlled counterfactual simulations. Through 660 experiments on 15 latest commercial and open-source VLMs, we find that these models exhibit striking limitations in basic world modeling abilities. For instance, almost all models perform at near-random accuracy when distinguishing motion trajectories. Additionally, they lack disentangled understanding – e.g., some models tend to believe blue objects move faster than green ones. More rich results and analyses reveal significant gaps between VLMs and human-level world modeling.

[37] A Dual-Layered Evaluation of Geopolitical and Cultural Bias in LLMs

Sean Kim,Hyuhng Joon Kim

Main category: cs.CL

TL;DR: 这篇论文提出了一种双层次评估框架，用于分析大语言模型（LLMs）在地缘政治和文化偏见上的表现，揭示了模型偏见和推理偏见的区别及其影响。

Details

Motivation: 随着LLMs在不同语言和文化背景中的广泛部署，理解它们在中立和敏感话题上的表现变得尤为重要，尤其是当它们的输出可能影响公众观点或强化主流叙事时。

Contribution: 定义了模型偏见和推理偏见两种偏见类型，并构建了一个手动标注的多语言数据集，用于评估LLMs在实际和争议性问题上的表现。

Method: 通过两阶段评估：第一阶段针对有明确答案的事实性问题，分析查询语言对模型输出的影响；第二阶段探讨地缘政治敏感问题，揭示模型训练背景和查询语言的交互作用。

Result: 结果显示，第一阶段存在查询语言导致的输出对齐，而第二阶段则反映模型训练背景和查询语言的复杂交互。

Insight: 研究强调了LLMs多语言和文化背景下评估的重要性，为未来部署和文化敏感评估提供了框架。

Abstract: As large language models (LLMs) are increasingly deployed across diverse linguistic and cultural contexts, understanding their behavior in both factual and disputable scenarios is essential, especially when their outputs may shape public opinion or reinforce dominant narratives. In this paper, we define two types of bias in LLMs: model bias (bias stemming from model training) and inference bias (bias induced by the language of the query), through a two-phase evaluation. Phase 1 evaluates LLMs on factual questions where a single verifiable answer exists, assessing whether models maintain consistency across different query languages. Phase 2 expands the scope by probing geopolitically sensitive disputes, where responses may reflect culturally embedded or ideologically aligned perspectives. We construct a manually curated dataset spanning both factual and disputable QA, across four languages and question types. The results show that Phase 1 exhibits query language induced alignment, while Phase 2 reflects an interplay between the model’s training context and query language. This paper offers a structured framework for evaluating LLM behavior across neutral and sensitive topics, providing insights for future LLM deployment and culturally aware evaluation practices in multilingual contexts.

[38] AutoMixer: Checkpoint Artifacts as Automatic Data Mixers

Ernie Chang,Yang Li,Patrick Huber,David Kant,Yangyang Shi,Vikas Chandra

Main category: cs.CL

TL;DR: 该论文提出了一种利用训练过程中的检查点模型作为数据混合器的方法，以优化语言模型训练中的数据混合物。

Details

Motivation: 语言模型训练中需要模型具备多样化任务能力，但数据与任务之间的关系难以建模。检查点模型在训练轨迹中展现出不同的能力，但这些信息未被充分利用。

Contribution: 主要贡献是将检查点模型视为数据混合器，利用其一阶影响近似方法优化数据混合物。

Method: 通过识别检查点模型在基准测试上的能力，利用其聚合的一阶影响近似方法混合数据。

Result: 在八个推理基准上验证了方法的有效性，性能提升高达1.93%。

Insight: 检查点模型可以作为增强数据质量和优化数据混合的有效工具。

Abstract: In language model training, it is desirable to equip models with capabilities from various tasks. However, it is not clear how to directly obtain the right data mixtures for these capabilities as the relationship between data and tasks is difficult to be modeled. In this work, we observe that checkpoint models exhibit emerging capabilities at different points in the training trajectory. Often, the training process saves checkpoints as artifacts that are under-utilized as a source of in-training data signals. We identify these artifact models based on their respective capabilities on the benchmarks and leverage them as data mixers by using their aggregated first-order influence approximation over source data. We demonstrated on eight reasoning benchmarks that the proposed framework shows significant improvements in the pretraining setting, with performance improvements of up to 1.93%. Overall, this shows the potential of checkpoint models to enhance data quality and optimize data mixtures.

[39] More Vulnerable than You Think: On the Stability of Tool-Integrated LLM Agents

Weimin Xiong,Ke Wang,Yifan Song,Hanchao Liu,Sai Zhou,Wei Peng,Sujian Li

Main category: cs.CL

TL;DR: 该论文探讨了集成工具的LLM代理在实际应用中的稳定性问题，发现其在整个工具调用过程中容易出错，开源模型尤其脆弱，增加模型规模也不会显著改善推理能力。

Details

Motivation: 当前对集成工具的LLM代理的评估主要关注端到端的工具使用效果，忽略了其稳定性问题，而现实应用中代理可能因各种因素崩溃或异常行为。

Contribution: 论文揭示了LLM代理在工具调用的每个阶段（如阅读文档、选择工具、生成参数等）的脆弱性，且开源模型比专有模型更易受影响。

Method: 通过大量实验，研究了代理在工具调用各阶段的稳定性，并分析了模型规模对稳定性的影响。

Result: 实验表明代理在每个阶段均易出错，开源模型更脆弱，增加模型规模不仅未显著改善推理能力，反而可能使其更容易受到攻击。

Insight: 强调评估代理稳定性的重要性，为未来LLM开发与评估提供了重要参考。

Abstract: Current evaluations of tool-integrated LLM agents typically focus on end-to-end tool-usage evaluation while neglecting their stability. This limits their real-world applicability, as various internal or external factors can cause agents to crash or behave abnormally. Our research addresses this by investigating whether agents are vulnerable to errors throughout the entire tool invocation process, including reading tool documentation, selecting tools and generating parameters, and processing the tool’s response. Through extensive experiments, we observe that agents are highly susceptible to errors at each stage and agents based on open-source models are more vulnerable than those based on proprietary models. We also find that increasing the model size does not significantly improve tool invocation reasoning and may make agents more vulnerable to attacks resembling normal user instructions. This highlights the importance of evaluating agent stability and offers valuable insights for future LLM development and evaluation.

[40] Lost at the Beginning of Reasoning

Baohao Liao,Xinyi Chen,Sara Rajaee,Yuhui Xu,Christian Herold,Anders Søgaard,Maarten de Rijke,Christof Monz

Main category: cs.CL

TL;DR: 论文研究了大型语言模型（LLM）在长推理链中自我修正能力的不足，发现第一步推理对最终结果影响巨大。提出了一种高效的采样策略，通过奖励模型筛选高质量第一步推理，显著降低推理成本，并引入新基准评估模型自我修正能力。

Details

Motivation: 尽管LLM在复杂推理能力上取得了进展，但其在长推理链中的自我修正能力尚未充分探索。研究发现，第一步推理的错误会显著影响后续推理质量，因此需要解决这一问题以提高推理效率。

Contribution: 1. 揭示了第一步推理对最终结果的巨大影响。2. 提出了一种高效的采样策略，显著降低推理成本。3. 引入了评估模型自我修正能力的新基准。

Method: 使用奖励模型对第一步推理进行采样和筛选，保留高质量推理步骤，丢弃低质量步骤，从而优化推理过程。

Result: 提出的方法在不牺牲准确性的情况下，将推理成本降低高达70%。

Insight: 研究强调了在长推理链中优化初始步骤的重要性，为未来研究提供了关于LLM稳健推理的新方向。

Abstract: Recent advancements in large language models (LLMs) have significantly advanced complex reasoning capabilities, particularly through extended chain-of-thought (CoT) reasoning that incorporates mechanisms such as backtracking, self-reflection and self-correction. Despite these developments, the self-correction abilities of LLMs during long CoT reasoning remain underexplored. And recent findings on overthinking suggest that such models often engage in unnecessarily redundant reasoning. In this work, we empirically show that the first reasoning step exerts a disproportionately large influence on the final prediction - errors introduced at this stage can substantially degrade subsequent reasoning quality. This phenomenon is consistently observed across two state-of-the-art open-source reasoning model families: DeepSeek-R1 and Qwen3. To address this, we propose an efficient sampling strategy that leverages a reward model to identify and retain high-quality first reasoning steps while discarding suboptimal ones, achieving up to a 70% reduction in inference cost without sacrificing accuracy. Finally, we introduce a new benchmark specifically constructed with deliberately flawed first reasoning steps to systematically evaluate model self-correction capabilities, offering a foundation for future research on robust reasoning in LLMs.

[41] Training Language Model to Critique for Better Refinement

Tianshu Yu,Chao Xiang,Mingchuan Yang,Pei Ke,Bosi Wen,Cunxiang Wang,Jiale Cheng,Li Zhang,Xinyu Mu,Chuxiong Sun,Minlie Huang

Main category: cs.CL

TL;DR: 论文提出了一种名为RCO的新框架，通过训练评论模型优化语言模型的反馈与改进能力，显著提升了评论质量和改进效果。

Details

Motivation: 当前大型语言模型（LLMs）在提供反馈和评论方面表现出色，但缺乏对评论类型的有效性及其生成方式的研究。

Contribution: 1. 提出RCO框架，利用改进信号训练评论模型；2. 引入评论效用（CU）作为奖励信号；3. 实验证明RCO在多个任务中优于传统方法。

Method: RCO通过反馈循环生成评论，指导模型改进响应，并以CU作为训练评论模型的奖励信号。

Result: RCO在对话生成、摘要、问答、数学推理和代码生成五个任务中，显著提升了评论质量和改进效果。

Insight: 关注能带来实际改进的评论比直接评估评论偏好更有效。

Abstract: Large language models (LLMs) have demonstrated remarkable evaluation and critique capabilities, providing insightful feedback and identifying flaws in various tasks. However, limited research has explored which types of critiques are most effective for improving model responses or how to generate such critiques. To address this gap, we introduce \textbf{R}efinement-oriented \textbf{C}ritique \textbf{O}ptimization (RCO), a novel framework designed to train critic models using refinement signals. RCO uses a feedback loop where critiques, generated by the critic model, guide the actor model in refining its responses. The critique utility (CU) quantifies the effectiveness of these refinements, serving as the reward signal for training the critic model. By focusing on critiques that lead to better refinements, RCO eliminates the need for direct critique preference assessment, ensuring that critiques driving meaningful improvements are rewarded. We evaluate RCO across five tasks, i.e., dialog generation, summarization, question answering, mathematical reasoning, and code generation, and show that it significantly outperforms traditional methods and open-source models in terms of critique quality and refinement outcomes. Our contributions include the introduction of RCO, a novel supervision scheme based on refined response preferences, and comprehensive experimental results that highlight the method’s effectiveness in enhancing LLM critique-refinement loops.

[42] Leveraging In-Context Learning for Political Bias Testing of LLMs

Patrick Haller,Jannis Vamvas,Rico Sennrich,Lena A. Jäger

Main category: cs.CL

TL;DR: 本文提出了一种新的政治偏见测试方法——问卷建模（QM），利用人类调查数据作为上下文示例，提高了基于问题的偏见评估的稳定性，并验证了指令调优可能改变偏见方向。

Details

Motivation: 现有方法通过政治问题测试大型语言模型（LLM）的偏见，但存在稳定性不足的问题，难以可靠比较不同模型。作者认为模型需要更多上下文来提高测试的稳定性。

Contribution: 提出问卷建模（QM）任务，利用人类调查数据作为上下文示例，显著提升偏见评估的稳定性，并验证了指令调优对偏见方向的影响。

Method: 通过QM任务，将人类调查数据作为上下文示例输入LLM，测试其在政治偏见方面的表现，并比较不同规模模型和指令调优版本的差异。

Result: 实验表明，QM提高了偏见评估的稳定性，指令调优可以改变偏见方向，且更大规模的模型能更有效地利用上下文示例，表现出更小的偏见分数。

Insight: 上下文学习对模型偏见测试至关重要；更大的模型在利用上下文时表现更优，偏见更少。

Abstract: A growing body of work has been querying LLMs with political questions to evaluate their potential biases. However, this probing method has limited stability, making comparisons between models unreliable. In this paper, we argue that LLMs need more context. We propose a new probing task, Questionnaire Modeling (QM), that uses human survey data as in-context examples. We show that QM improves the stability of question-based bias evaluation, and demonstrate that it may be used to compare instruction-tuned models to their base versions. Experiments with LLMs of various sizes indicate that instruction tuning can indeed change the direction of bias. Furthermore, we observe a trend that larger models are able to leverage in-context examples more effectively, and generally exhibit smaller bias scores in QM. Data and code are publicly available.

[43] Detection of Personal Data in Structured Datasets Using a Large Language Model

Albert Agisha Ntwali,Luca Rück,Martin Heckmann

Main category: cs.CL

TL;DR: 论文提出了一种基于GPT-4o的大型语言模型方法，用于检测结构化数据集中的个人数据，通过结合上下文信息（如特征名称、数据集描述等）显著提升了检测性能。

Details

Motivation: 现有方法在检测个人数据时往往忽略上下文信息，而在实际应用中，这些信息可能对提高检测准确性至关重要。因此，作者提出了一种结合上下文的新方法。

Contribution: 主要贡献包括：1）提出了一种结合上下文信息的GPT-4o模型方法；2）在多个数据集上验证了方法的有效性，尤其是在医疗数据集和Kaggle/OpenML数据集上表现优异。

Method: 方法核心是利用GPT-4o模型，结合特征名称、值以及其他特征名称和数据集描述的上下文信息，进行个人数据检测。

Result: 实验表明，新方法在MIMIC-Demo-Ext和Kaggle/OpenML数据集上显著优于CASSED和Microsoft Presidio，尤其是在利用上下文信息时效果更佳。

Insight: 研究表明，上下文信息对个人数据检测至关重要，未来研究需要更多包含真实个人数据的数据集以推动领域进步。

Abstract: We propose a novel approach for detecting personal data in structured datasets, leveraging GPT-4o, a state-of-the-art Large Language Model. A key innovation of our method is the incorporation of contextual information: in addition to a feature’s name and values, we utilize information from other feature names within the dataset as well as the dataset description. We compare our approach to alternative methods, including Microsoft Presidio and CASSED, evaluating them on multiple datasets: DeSSI, a large synthetic dataset, datasets we collected from Kaggle and OpenML as well as MIMIC-Demo-Ext, a real-world dataset containing patient information from critical care units. Our findings reveal that detection performance varies significantly depending on the dataset used for evaluation. CASSED excels on DeSSI, the dataset on which it was trained. Performance on the medical dataset MIMIC-Demo-Ext is comparable across all models, with our GPT-4o-based approach clearly outperforming the others. Notably, personal data detection in the Kaggle and OpenML datasets appears to benefit from contextual information. This is evidenced by the poor performance of CASSED and Presidio (both of which do not utilize the context of the dataset) compared to the strong results of our GPT-4o-based approach. We conclude that further progress in this field would greatly benefit from the availability of more real-world datasets containing personal information.

[44] Refining Czech GEC: Insights from a Multi-Experiment Approach

Petr Pechman,Milan Straka,Jana Straková,Jakub Náplava

Main category: cs.CL

TL;DR: 这篇论文提出了一种基于Transformer架构的捷克语语法错误校正（GEC）系统，通过动态生成合成错误数据并结合多实验方法，实现了当前最佳性能。

Details

Motivation: 捷克语的语法错误校正任务缺乏高效且高性能的解决方案，现有方法通常在数据生成和模型优化上存在不足。

Contribution: 提出了一种实时合成错误生成管道，结合语言无关和捷克语特定的错误类型，并通过对多实验方法的综合分析优化了模型性能。

Method: 基于Transformer架构的神经机器翻译方法，采用动态合成错误生成策略，研究了多种数据增强和模型优化技术。

Result: 最佳模型在性能和计算效率上均显著优于现有方法，并公开了源代码和模型。

Insight: 合成错误生成和领域平衡对提升GEC任务性能至关重要，而大型语言模型（LLMs）在专家调优场景中表现优异。

Abstract: We present a grammar error correction (GEC) system that achieves state of the art for the Czech language. Our system is based on a neural network translation approach with the Transformer architecture, and its key feature is its real-time synthetic generation pipeline, which dynamically augments sentences with artificial errors by introducing both language-agnostic and Czech-specific errors. We conduct a comprehensive series of experiments, investigating the Czech GEC corpora as bases for synthetic error introduction, several error generation strategies, domain balancing, tokenization granularity, model size, and data scaling during fine-tuning. Additionally, we evaluate the performance of large language models (LLMs) on Czech GEC in both end-user and expert fine-tuning scenarios. Our best-performing model is superior both in performance and computational efficiency. The source code and the trained model links are available on https://github.com/ufal/tsd2025-gec.

[45] HyperCLOVA X THINK Technical Report

NAVER Cloud HyperCLOVA X Team

Main category: cs.CL

TL;DR: HyperCLOVA X THINK是一个专注于推理的大型语言模型，针对韩语和英语优化，通过三阶段课程学习和强化学习实现高性能表现，并在韩国本地化基准测试中表现优异。

Details

Motivation: 针对韩语和英语的双语推理需求，推动韩国AI创新，同时为全球研究社区提供资源。

Contribution: 1. 首个专注于推理的HyperCLOVA X系列模型；2. 通过$μP扩展的Peri-LN Transformer实现计算-内存平衡；3. 提出三阶段课程学习和强化学习方法，支持128K上下文窗口。

Method: 1. 预训练采用$μP扩展的Peri-LN Transformer；2. 三阶段课程学习扩展上下文窗口至128K；3. 后训练通过监督微调和基于可验证奖励的强化学习。

Result: 在韩国本地化基准测试（如KMMLU、CSAT等）中表现优异，视觉增强变体在KCSAT STEM基准上媲美或超越GPT-4.1，且训练计算成本更低。

Insight: 模型通过高效的数据增强和技术创新，实现了高性能与低计算成本的平衡，为双语推理提供了新的可能性。

Abstract: We introduce HyperCLOVA X THINK, the first reasoning-focused large language model in the HyperCLOVA X family, pre-trained on roughly $6$ trillion high-quality Korean, and English tokens, augmented with targeted synthetic Korean data. It was implemented as a compute-memory-balanced Peri-LN Transformer scaled with $\mu$P, pre-trained through a three-stage curriculum that expands the context window to $128$K tokens, and post-trained via supervised fine-tuning with Reinforcement Learning from Verifiable Rewards supports both detailed rationale and concise-answer modes. It delivers competitive performance against similarly sized models on Korea-focused benchmarks such as KMMLU, CSAT, KoBALT-700, HAERAE-1.0, and KoBigBench, while preserving robust bilingual consistency and translation quality. In addition, a vision-augmented variant matches or exceeds GPT-4.1 on the KCSAT STEM benchmark, all of which are achieved with substantially lower training compute than existing models of similar sizes. We also present a pruning and distillation technique that will soon be applied to HyperCLOVA X THINK for an open-source and business-friendly foundation model. Altogether, these capabilities position HyperCLOVA X THINK as a robust foundation for Korean AI innovation and a valuable resource for the global research community.

[46] Sequential Diagnosis with Language Models

Harsha Nori,Mayank Daswani,Christopher Kelly,Scott Lundberg,Marco Tulio Ribeiro,Marc Wilson,Xiaoxuan Liu,Viknesh Sounderajah,Jonathan Carlson,Matthew P Lungren,Bay Gross,Peter Hames,Mustafa Suleyman,Dominic King,Eric Horvitz

Main category: cs.CL

TL;DR: A paper introducing the Sequential Diagnosis Benchmark and MAI-DxO orchestrator for iterative medical diagnosis, showing significant improvements in accuracy and cost-effectiveness over physicians and standard models.

Details

Motivation: Current evaluations of language models in medical diagnosis rely on static vignettes, missing the iterative and adaptive nature of real-world clinical practice. The paper aims to bridge this gap.

Contribution: 1) Introduces the Sequential Diagnosis Benchmark to emulate real-world diagnostic complexity. 2) Presents MAI-DxO, a model-agnostic orchestrator that improves diagnostic accuracy and reduces costs.

Method: 1) Uses NEJM-CPC cases to create stepwise diagnostic encounters. 2) MAI-DxO simulates a panel of physicians, proposes differential diagnoses, and selects cost-effective tests iteratively.

Result: MAI-DxO achieves 80% accuracy (vs. 20% for physicians) and reduces costs by 20% vs. physicians and 70% vs. standard models. Maximum accuracy reaches 85.5%.

Insight: AI systems guided to think iteratively and act judiciously can significantly enhance diagnostic precision and cost-effectiveness in clinical care.

Abstract: Artificial intelligence holds great promise for expanding access to expert medical knowledge and reasoning. However, most evaluations of language models rely on static vignettes and multiple-choice questions that fail to reflect the complexity and nuance of evidence-based medicine in real-world settings. In clinical practice, physicians iteratively formulate and revise diagnostic hypotheses, adapting each subsequent question and test to what they’ve just learned, and weigh the evolving evidence before committing to a final diagnosis. To emulate this iterative process, we introduce the Sequential Diagnosis Benchmark, which transforms 304 diagnostically challenging New England Journal of Medicine clinicopathological conference (NEJM-CPC) cases into stepwise diagnostic encounters. A physician or AI begins with a short case abstract and must iteratively request additional details from a gatekeeper model that reveals findings only when explicitly queried. Performance is assessed not just by diagnostic accuracy but also by the cost of physician visits and tests performed. We also present the MAI Diagnostic Orchestrator (MAI-DxO), a model-agnostic orchestrator that simulates a panel of physicians, proposes likely differential diagnoses and strategically selects high-value, cost-effective tests. When paired with OpenAI’s o3 model, MAI-DxO achieves 80% diagnostic accuracy–four times higher than the 20% average of generalist physicians. MAI-DxO also reduces diagnostic costs by 20% compared to physicians, and 70% compared to off-the-shelf o3. When configured for maximum accuracy, MAI-DxO achieves 85.5% accuracy. These performance gains with MAI-DxO generalize across models from the OpenAI, Gemini, Claude, Grok, DeepSeek, and Llama families. We highlight how AI systems, when guided to think iteratively and act judiciously, can advance diagnostic precision and cost-effectiveness in clinical care.

cs.CV [Back]

[47] Fine-Grained Preference Optimization Improves Spatial Reasoning in VLMs

Yifan Shen,Yuanzhe Liu,Jingyuan Zhu,Xu Cao,Xiaofeng Zhang,Yixiao He,Wenming Ye,James Matthew Rehg,Ismini Lourentzou

Main category: cs.CV

TL;DR: 该论文提出了一种名为SpatialReasoner-R1的视觉语言推理模型，通过创新的M3CTS方法和fDPO优化技术，显著提升了细粒度空间推理能力。

Details

Motivation: 当前视觉语言模型（VLM）在多步逻辑和精确空间对齐方面表现不佳，需要一种新的方法来提升细粒度空间推理能力。

Contribution: 1. 提出M3CTS方法生成多样且逻辑一致的LongCoT推理轨迹；2. 设计fDPO优化技术，通过片段特定偏好细粒度提升描述性基础和逻辑推理；3. 在SPATIALRGPT-Bench上实现新的SOTA。

Method: 1. 使用M3CTS生成高质量监督数据；2. 提出fDPO优化方法，结合空间奖励机制评估候选响应。

Result: fDPO在空间质量任务中平均提升4.1%，在空间数量任务中提升9.0%；SpatialReasoner-R1在SPATIALRGPT-Bench上超越基线9.8%。

Insight: 通过结合蒙特卡洛树搜索和细粒度偏好优化，能够显著提升视觉语言模型在复杂空间推理任务中的表现。

Abstract: Current Vision-Language Models (VLMs) struggle with fine-grained spatial reasoning, particularly when multi-step logic and precise spatial alignment are required. In this work, we introduce SpatialReasoner-R1, a vision-language reasoning model designed to address these limitations. To construct high-quality supervision for spatial reasoning, we design a Multi-Model Monte Carlo Tree Search (M3CTS) method that generates diverse, logically consistent Long Chain-of-Thought (LongCoT) reasoning trajectories. In addition, we propose fine-grained Direct Preference Optimization (fDPO), which introduces segment-specific preference granularity for descriptive grounding and logical reasoning, guided by a spatial reward mechanism that evaluates candidate responses based on visual consistency, spatial grounding, and logical coherence. Experimental results demonstrate that fDPO achieves an average improvement of 4.1% over standard DPO across spatial quality tasks, and a 9.0% gain in spatial quantity tasks. SpatialReasoner-R1, trained with fDPO, sets a new SoTA on SPATIALRGPT-Bench, outperforming the strongest baseline by 9.8% in average accuracy, while maintaining competitive performance on general vision-language tasks.

[48] TanDiT: Tangent-Plane Diffusion Transformer for High-Quality 360° Panorama Generation

Hakan Çapuk,Andrew Bond,Muhammed Burak Kızıl,Emir Göçen,Erkut Erdem,Aykut Erdem

Main category: cs.CV

TL;DR: TanDiT提出了一种基于切平面扩散Transformer的方法，用于高质量360°全景图像生成，解决了传统方法在几何畸变和环状一致性方面的挑战。

Details

Motivation: 现有的图像生成模型在合成全景图像时面临几何畸变和环状一致性的问题，限制了生成质量。

Contribution: 1）提出TanDiT方法，通过生成覆盖360°视野的切平面图像网格；2）设计了统一的扩散模型和模型无关的全局一致性后处理步骤；3）提出了TangentIS和TangentFID两个评估指标。

Method: 采用切平面图像网格生成策略，结合统一的扩散模型和全局一致性后处理，实现高质量全景合成。

Result: 实验表明TanDiT能有效泛化到训练数据之外，处理复杂文本提示，并与其他生成模型无缝集成。

Insight: 通过切平面设计和全局一致性后处理，TanDiT解决了全景生成的几何和一致性挑战，为未来研究提供了新思路。

Abstract: Recent advances in image generation have led to remarkable improvements in synthesizing perspective images. However, these models still struggle with panoramic image generation due to unique challenges, including varying levels of geometric distortion and the requirement for seamless loop-consistency. To address these issues while leveraging the strengths of the existing models, we introduce TanDiT, a method that synthesizes panoramic scenes by generating grids of tangent-plane images covering the entire 360$^\circ$ view. Unlike previous methods relying on multiple diffusion branches, TanDiT utilizes a unified diffusion model trained to produce these tangent-plane images simultaneously within a single denoising iteration. Furthermore, we propose a model-agnostic post-processing step specifically designed to enhance global coherence across the generated panoramas. To accurately assess panoramic image quality, we also present two specialized metrics, TangentIS and TangentFID, and provide a comprehensive benchmark comprising captioned panoramic datasets and standardized evaluation scripts. Extensive experiments demonstrate that our method generalizes effectively beyond its training data, robustly interprets detailed and complex text prompts, and seamlessly integrates with various generative models to yield high-quality, diverse panoramic images.

[49] FOCUS: Internal MLLM Representations for Efficient Fine-Grained Visual Question Answering

Liangyu Zhong,Fabio Rosenthal,Joachim Sicking,Fabian Hüger,Thorsten Bagdonat,Hanno Gottschalk,Leo Schwinn

Main category: cs.CV

TL;DR: 论文提出了一种名为FOCUS的训练自由视觉裁剪方法，通过利用MLLM内部表示引导搜索最相关的图像区域，提升了细粒度视觉问答（VQA）的性能和效率。

Details

Motivation: 多模态大型语言模型（MLLM）在细粒度VQA任务中表现不佳，尤其是涉及微小图像细节时。现有视觉裁剪方法存在需要任务特定微调、效率低下或与高效注意力机制不兼容等问题。

Contribution: 提出了FOCUS方法，无需训练，通过MLLM内部表示（KV缓存）生成对象相关性图，高效搜索相关图像区域，显著提升了性能和计算效率。

Method: 1）识别VQA提示中的目标对象；2）利用KV缓存计算对象相关性图；3）基于该图提议并排序相关图像区域；4）用排名最高的区域执行细粒度VQA。

Result: 在四个细粒度VQA数据集和两种MLLM上表现优异，准确率和效率均超过三种流行视觉裁剪方法，计算量减少3-6.5倍。

Insight: MLLM内部表示（如KV缓存）可用于高效定位细粒度视觉信息，无需额外训练即可提升VQA性能。

Abstract: While Multimodal Large Language Models (MLLMs) offer strong perception and reasoning capabilities for image-text input, Visual Question Answering (VQA) focusing on small image details still remains a challenge. Although visual cropping techniques seem promising, recent approaches have several limitations: the need for task-specific fine-tuning, low efficiency due to uninformed exhaustive search, or incompatibility with efficient attention implementations. We address these shortcomings by proposing a training-free visual cropping method, dubbed FOCUS, that leverages MLLM-internal representations to guide the search for the most relevant image region. This is accomplished in four steps: first, we identify the target object(s) in the VQA prompt; second, we compute an object relevance map using the key-value (KV) cache; third, we propose and rank relevant image regions based on the map; and finally, we perform the fine-grained VQA task using the top-ranked region. As a result of this informed search strategy, FOCUS achieves strong performance across four fine-grained VQA datasets and two types of MLLMs. It outperforms three popular visual cropping methods in both accuracy and efficiency, and matches the best-performing baseline, ZoomEye, while requiring 3 - 6.5 x less compute.

[50] CAST: Cross-Attentive Spatio-Temporal feature fusion for Deepfake detection

Aryan Thakre,Omkar Nagwekar,Vedang Talekar,Aparna Santra Biswas

Main category: cs.CV

TL;DR: 论文提出了一种统一模型CAST，通过跨注意机制融合时空特征，提升了深度伪造视频检测的性能和泛化能力。

Details

Motivation: 现有CNN-Transformer模型独立处理时空特征，融合方式简单，限制了时空交互的深度。论文旨在解决这一问题。

Contribution: 提出CAST模型，首次引入跨注意机制动态融合时空特征，增强了对时间演化伪造痕迹的检测能力。

Method: 利用跨注意机制，使时间特征动态关注相关空间区域，实现更精细的时空特征交互。

Result: 在多个数据集（FaceForensics++、Celeb-DF、DeepfakeDetection）上表现优异，AUC和准确率显著提升。

Insight: 跨注意机制能更有效地捕捉时空特征的动态关联，为深度伪造检测开辟了新方向。

Abstract: Deepfakes have emerged as a significant threat to digital media authenticity, increasing the need for advanced detection techniques that can identify subtle and time-dependent manipulations. CNNs are effective at capturing spatial artifacts, and Transformers excel at modeling temporal inconsistencies. However, many existing CNN-Transformer models process spatial and temporal features independently. In particular, attention-based methods often use separate attention mechanisms for spatial and temporal features and combine them using naive approaches like averaging, addition, or concatenation, which limits the depth of spatio-temporal interaction. To address this challenge, we propose a unified CAST model that leverages cross-attention to effectively fuse spatial and temporal features in a more integrated manner. Our approach allows temporal features to dynamically attend to relevant spatial regions, enhancing the model’s ability to detect fine-grained, time-evolving artifacts such as flickering eyes or warped lips. This design enables more precise localization and deeper contextual understanding, leading to improved performance across diverse and challenging scenarios. We evaluate the performance of our model using the FaceForensics++, Celeb-DF, and DeepfakeDetection datasets in both intra- and cross-dataset settings to affirm the superiority of our approach. Our model achieves strong performance with an AUC of 99.49 percent and an accuracy of 97.57 percent in intra-dataset evaluations. In cross-dataset testing, it demonstrates impressive generalization by achieving a 93.31 percent AUC on the unseen DeepfakeDetection dataset. These results highlight the effectiveness of cross-attention-based feature fusion in enhancing the robustness of deepfake video detection.

[51] Asymmetric Dual Self-Distillation for 3D Self-Supervised Representation Learning

Remco F. Leijenaar,Hamidreza Kasaei

Main category: cs.CV

TL;DR: 论文提出了AsymDSD，一种非对称双自蒸馏框架，用于3D自监督表示学习，通过潜在空间预测统一掩码建模和不变性学习，在ScanObjectNN上取得了最先进的性能。

Details

Motivation: 传统的基于重构的掩码点建模（MPM）在捕捉高级语义时表现有限，因此需要一种新的方法来解决这一问题。

Contribution: 1. 提出了AsymDSD框架，统一了掩码建模和不变性学习；2. 引入了非对称设置、多掩码采样等关键设计；3. 在ScanObjectNN上达到了90.53%的精度，并进一步通过预训练提升到93.72%。

Method: AsymDSD采用联合嵌入架构，通过潜在空间预测替代输入空间重构，结合非对称设置、注意力屏蔽和多掩码采样等技术。

Result: 在ScanObjectNN上实现了90.53%的精度，预训练后进一步提升至93.72%。

Insight: 潜在空间预测比输入空间重构更适用于捕捉高级语义，非对称设计和多掩码采样能够有效提升表示学习的效果。

Abstract: Learning semantically meaningful representations from unstructured 3D point clouds remains a central challenge in computer vision, especially in the absence of large-scale labeled datasets. While masked point modeling (MPM) is widely used in self-supervised 3D learning, its reconstruction-based objective can limit its ability to capture high-level semantics. We propose AsymDSD, an Asymmetric Dual Self-Distillation framework that unifies masked modeling and invariance learning through prediction in the latent space rather than the input space. AsymDSD builds on a joint embedding architecture and introduces several key design choices: an efficient asymmetric setup, disabling attention between masked queries to prevent shape leakage, multi-mask sampling, and a point cloud adaptation of multi-crop. AsymDSD achieves state-of-the-art results on ScanObjectNN (90.53%) and further improves to 93.72% when pretrained on 930k shapes, surpassing prior methods.

[52] Exploring Image Generation via Mutually Exclusive Probability Spaces and Local Correlation Hypothesis

Chenqiu Zhao,Anup Basu

Main category: cs.CV

TL;DR: 论文提出了两个理论框架MESP和LCH，揭示了概率生成模型可能存在的局限性，即全局分布学习导致记忆而非生成行为。通过改进VAE和提出BL-AE、ARVM，在标准数据集上取得竞争性FID分数，但发现这反映的是记忆而非生成能力，进而提出LCH假设。

Details

Motivation: 研究发现概率生成模型在全局分布学习时易陷入记忆而非生成行为，导致模型无法真正具备生成能力，希望通过理论框架和实验解决这一问题。

Contribution: 1. 提出MESP框架，揭示VAE中潜在变量分布重叠导致优化冲突；2. 提出BL-AE和ARVM模型，实现竞争性FID分数；3. 提出LCH假设，解释生成能力源于局部相关性。

Method: 1. 通过MESP分析VAE的潜在变量分布问题，提出二值潜在自编码器BL-AE；2. 基于BL-AE设计ARVM，输出直方图；3. 提出LCH假设，验证局部相关性对生成能力的影响。

Result: ARVM在标准数据集上取得竞争性FID分数，但实验表明高分反映的是记忆行为而非生成能力。

Insight: 生成模型的性能指标如FID可能无法真实反映生成能力，局部相关性（LCH）可能是真正驱动生成的关键因素。

Abstract: We propose two theoretical frameworks, the Mutually Exclusive Probability Space (MESP) and the Local Correlation Hypothesis (LCH), to explore a potential limitation in probabilistic generative models; namely that learning global distributions leads to memorization rather than generative behavior. MESP emerges from our rethinking of the Variational Autoencoder (VAE). We observe that latent variable distributions in VAE exhibit overlap, which leads to an optimization conflict between the reconstruction loss and KL-divergence loss. A lower bound based on the overlap coefficient is proposed. We refer to this phenomenon as Mutually Exclusive Probability Spaces. Based on MESP, a Binary Latent Autoencoder (BL-AE) is proposed to encode images into binary latent representations. These binary latents are used as the input to our Autoregressive Random Variable Model (ARVM), a modified autoregressive model outputting histograms. Our ARVM achieves competitive FID scores, outperforming state-of-the-art methods on standard datasets. However, such scores reflect memorization rather than generation. To address this issue, we propose the Local Correlation Hypothesis (LCH), which posits that generative capability arising from local correlations among latent variables. Comprehensive experiments and discussions are conducted to validate our frameworks.

[53] Equitable Federated Learning with NCA

Nick Lemke,Mirko Konstantin,Henry John Krumb,John Kalkhof,Jonathan Stieber,Anirban Mukhopadhyay

Main category: cs.CV

TL;DR: 本文提出了适用于医疗图像分割任务的轻量级联邦学习系统FedNCA，解决了资源受限地区（如低收入和中等收入国家）在联邦学习中面临的高计算资源需求和网络不可靠问题。

Details

Motivation: 在资源受限地区，医疗专业人员有限，联邦学习（FL）可以促进跨机构的协作模型训练，但现有的FL系统对高性能计算和稳定网络的需求限制了其在低收入和中等收入国家（LMICs）的应用。

Contribution: 提出了FedNCA，一种基于轻量级Med-NCA架构的联邦学习系统，支持在低成本边缘设备上训练，并减少通信开销，同时具备加密能力。

Method: 利用Med-NCA架构，使系统能够在智能手机等低成本设备上高效运行，并通过优化通信协议降低对网络的依赖。

Result: FedNCA实现了在资源受限环境下的高效、轻量级和安全的医疗图像分割任务，推动了这些地区的医疗公平。

Insight: 通过轻量化和通信优化，FedNCA为资源受限地区提供了可行的联邦学习解决方案，为医疗AI的普及奠定了基础。

Abstract: Federated Learning (FL) is enabling collaborative model training across institutions without sharing sensitive patient data. This approach is particularly valuable in low- and middle-income countries (LMICs), where access to trained medical professionals is limited. However, FL adoption in LMICs faces significant barriers, including limited high-performance computing resources and unreliable internet connectivity. To address these challenges, we introduce FedNCA, a novel FL system tailored for medical image segmentation tasks. FedNCA leverages the lightweight Med-NCA architecture, enabling training on low-cost edge devices, such as widely available smartphones, while minimizing communication costs. Additionally, our encryption-ready FedNCA proves to be suitable for compromised network communication. By overcoming infrastructural and security challenges, FedNCA paves the way for inclusive, efficient, lightweight, and encryption-ready medical imaging solutions, fostering equitable healthcare advancements in resource-constrained regions.

[54] ImplicitQA: Going beyond frames towards Implicit Video Reasoning

Sirnam Swetha,Rohit Gupta,Parth Parag Kulkarni,David G Shatwell,Jeffrey A Chan Santiago,Nyle Siddiqui,Joseph Fioresi,Mubarak Shah

Main category: cs.CV

TL;DR: ImplicitQA是一个新的视频问答（VideoQA）基准测试，专注于测试模型在需要隐式推理（如动机、因果关系等）的任务上的表现，填补了当前视频问答系统在人类类似理解能力上的不足。

Details

Motivation: 现有的视频问答基准主要关注可以通过明确视觉内容回答的问题，而忽略了人类在观看创意和叙事视频时所需的隐式推理能力。本文旨在填补这一空白。

Contribution: 1. 提出ImplicitQA基准，包含1K个高难度QA对，涵盖8种隐式推理维度；2. 通过实验验证当前主流VideoQA模型在隐式推理任务上的表现不足；3. 开源数据集和标注框架。

Method: 从320+高质量创意视频中提取1K QA对，分类为空间、深度、视角、运动、因果动机等8类隐式推理任务。通过专家标注确保挑战性和多样性。

Result: 实验表明当前VideoQA模型在隐式推理任务上表现显著下降，显示出对表层视觉线索的依赖和对隐式推理能力的不足。

Insight: 隐式推理是视频理解的难点，需要模型具备跨时间和上下文的信息整合能力；当前模型在这一挑战中仍有很大改进空间。

Abstract: Video QA has made significant strides by leveraging multimodal learning to align visual and textual modalities. However, current benchmarks overwhelmingly focus on questions answerable through explicit visual content - actions, objects & events directly observable within individual frames or short clips. In contrast, creative and cinematic videos - such as movies, TV shows, and narrative-driven content - employ storytelling techniques that deliberately omit certain depictions, requiring viewers to infer motives, causality, and relationships across discontinuous frames. Humans naturally excel at such implicit reasoning, seamlessly integrating information across time and context to construct coherent narratives. Current VideoQA systems and benchmarks fail to capture this essential dimension of human-like understanding. To bridge this gap, we present ImplicitQA, a novel benchmark specifically designed to test models on implicit reasoning. It comprises 1K meticulously annotated QA pairs derived from 320+ high-quality creative video clips, systematically categorized into key reasoning dimensions: lateral and vertical spatial reasoning, depth and proximity, viewpoint and visibility, motion and trajectory, causal and motivational reasoning, social interactions, physical context, and inferred counting. These annotations are deliberately challenging, crafted by authors ensuring high-quality. Our extensive evaluations on leading VideoQA models reveals performance degradation, underscoring their reliance on surface-level visual cues and highlighting the difficulty of implicit reasoning. Performance variations across models further illustrate the complexity and diversity of the challenges presented by ImplicitQA. By releasing both the dataset and our data collection framework, we aim to stimulate further research and development in the community. https://huggingface.co/datasets/ucf-crcv/ImplicitQA.

[55] Early Glaucoma Detection using Deep Learning with Multiple Datasets of Fundus Images

Rishiraj Paul Chowdhury,Nirmit Shekar Karkera

Main category: cs.CV

TL;DR: 本文提出了一种基于EfficientNet-B0架构的深度学习流程，通过多数据集（ACRIMA、ORIGA、RIM-ONE）的顺序训练和微调，实现视网膜眼底图像的青光眼早期检测。

Details

Motivation: 青光眼是导致不可逆失明的主要原因，早期检测可显著改善治疗效果。传统诊断方法通常具有侵入性且需专业设备，亟需非侵入性的高效解决方案。

Contribution: 1. 提出基于EfficientNet-B0的多数据集训练流程；2. 证明简单预处理方法优于复杂增强；3. 模型在未见数据集上表现优异，具备泛化能力。

Method: 1. 使用EfficientNet-B0架构；2. 对ACRIMA、ORIGA、RIM-ONE数据集进行顺序训练和微调；3. 采用简单的预处理方法。

Result: 模型在AUC-ROC指标上表现优异，且在跨数据集测试中展示了强泛化能力。

Insight: 1. 多数据集训练可提升模型泛化性；2. 简单预处理足以满足需求，避免复杂操作；3. 该方法具有临床实用潜力。

Abstract: Glaucoma is a leading cause of irreversible blindness, but early detection can significantly improve treatment outcomes. Traditional diagnostic methods are often invasive and require specialized equipment. In this work, we present a deep learning pipeline using the EfficientNet-B0 architecture for glaucoma detection from retinal fundus images. Unlike prior studies that rely on single datasets, we sequentially train and fine-tune our model across ACRIMA, ORIGA, and RIM-ONE datasets to enhance generalization. Our experiments show that minimal preprocessing yields higher AUC-ROC compared to more complex enhancements, and our model demonstrates strong discriminative performance on unseen datasets. The proposed pipeline offers a reproducible and scalable approach to early glaucoma detection, supporting its potential clinical utility.

[56] Comparing Learning Paradigms for Egocentric Video Summarization

Daniel Wen

Main category: cs.CV

TL;DR: 该研究比较了监督学习、无监督学习和提示调优（prompt fine-tuning）三种范式在自中心视频摘要任务中的表现，发现通用模型GPT-4o在调整后优于专用模型，突显了当前方法对第一视角视频的局限性。

Details

Motivation: 自中心视频（egocentric video）的独特视角带来了处理上的特殊挑战，现有模型对其理解能力有限。研究旨在探索不同学习范式在这一任务中的潜力。

Contribution: 1. 全面比较了三种学习范式在自中心视频摘要中的表现；2. 发现GPT-4o通过提示调优优于专用模型，为通用模型在此领域的潜力提供了证据。

Method: 1. 使用Ego-Exo4D数据集的子集；2. 评估了监督学习的Shotluck Holmes、无监督学习的TAC-SUM和提示调优的GPT-4o。

Result: GPT-4o表现最佳，但所有模型在第一视角视频中的表现均劣于第三视角视频。

Insight: 通用模型通过适当调优可能在特殊任务中超越专用模型；自中心视频领域仍需进一步研究。

Abstract: In this study, we investigate various computer vision paradigms - supervised learning, unsupervised learning, and prompt fine-tuning - by assessing their ability to understand and interpret egocentric video data. Specifically, we examine Shotluck Holmes (state-of-the-art supervised learning), TAC-SUM (state-of-the-art unsupervised learning), and GPT-4o (a prompt fine-tuned pre-trained model), evaluating their effectiveness in video summarization. Our results demonstrate that current state-of-the-art models perform less effectively on first-person videos compared to third-person videos, highlighting the need for further advancements in the egocentric video domain. Notably, a prompt fine-tuned general-purpose GPT-4o model outperforms these specialized models, emphasizing the limitations of existing approaches in adapting to the unique challenges of first-person perspectives. Although our evaluation is conducted on a small subset of egocentric videos from the Ego-Exo4D dataset due to resource constraints, the primary objective of this research is to provide a comprehensive proof-of-concept analysis aimed at advancing the application of computer vision techniques to first-person videos. By exploring novel methodologies and evaluating their potential, we aim to contribute to the ongoing development of models capable of effectively processing and interpreting egocentric perspectives.

[57] CAT-SG: A Large Dynamic Scene Graph Dataset for Fine-Grained Understanding of Cataract Surgery

Felix Holm,Gözde Ünver,Ghazal Ghazaei,Nassir Navab

Main category: cs.CV

TL;DR: 该论文介绍了CAT-SG数据集，首个针对白内障手术的动态场景图数据集，通过捕捉手术工具与组织间的语义关系，支持手术流程的细粒度理解。

Details

Motivation: 现有数据集仅关注手术分析的孤立方面（如工具检测或阶段分割），缺乏对实体间语义关系和时间依赖性的综合表征。因此，需要一种更全面的数据集来支持复杂手术流程的建模。

Contribution: 提出了CAT-SG数据集，首次提供了工具-组织交互、手术变体和时间依赖性的结构化标注；此外，还设计了一种新颖的场景图生成模型CatSGG，在生成结构化手术表征方面优于现有方法。

Method: 通过构建动态场景图数据集CAT-SG，标注手术工具、解剖结构和手术技术之间的复杂交互；提出CatSGG模型，专注于生成高质量的语义关系表征。

Result: CatSGG模型在生成结构化手术表征方面表现优于现有方法，CAT-SG数据集支持更准确的手术阶段和技术识别。

Insight: 该研究通过动态场景图建模手术流程，为AI驱动的手术培训、实时决策支持和流程分析提供了更智能、上下文感知的基础。

Abstract: Understanding the intricate workflows of cataract surgery requires modeling complex interactions between surgical tools, anatomical structures, and procedural techniques. Existing datasets primarily address isolated aspects of surgical analysis, such as tool detection or phase segmentation, but lack comprehensive representations that capture the semantic relationships between entities over time. This paper introduces the Cataract Surgery Scene Graph (CAT-SG) dataset, the first to provide structured annotations of tool-tissue interactions, procedural variations, and temporal dependencies. By incorporating detailed semantic relations, CAT-SG offers a holistic view of surgical workflows, enabling more accurate recognition of surgical phases and techniques. Additionally, we present a novel scene graph generation model, CatSGG, which outperforms current methods in generating structured surgical representations. The CAT-SG dataset is designed to enhance AI-driven surgical training, real-time decision support, and workflow analysis, paving the way for more intelligent, context-aware systems in clinical practice.

[58] Few-Shot Segmentation of Historical Maps via Linear Probing of Vision Foundation Models

Rafael Sterzinger,Marco Peer,Robert Sablatnig

Main category: cs.CV

TL;DR: 该论文提出了一种高效的小样本历史地图分割方法，利用视觉基础模型的语义嵌入和参数高效微调，在数据稀缺情况下显著优于现有方法。

Details

Motivation: 历史地图的多样性标注数据稀缺，传统方法难以自动化处理，需要一种能在小样本情况下高效分割的方法。

Contribution: 结合视觉基础模型的语义嵌入和轻量级微调，实现了在小样本（5-shot和10-shot）场景下的高性能历史地图分割。

Method: 采用线性探测（linear probing）和微调视觉基础模型，仅需训练少量参数（0.21%模型大小），减少了计算和标注需求。

Result: 在Siegfried数据集上，mIoU相对提升5%至20%；在ICDAR 2021数据集上，PQ达到67.3%，展示了强泛化能力。

Insight: 小样本分割任务可以通过预训练视觉基础模型的迁移学习显著提升性能，同时减少标注和计算成本。

Abstract: As rich sources of history, maps provide crucial insights into historical changes, yet their diverse visual representations and limited annotated data pose significant challenges for automated processing. We propose a simple yet effective approach for few-shot segmentation of historical maps, leveraging the rich semantic embeddings of large vision foundation models combined with parameter-efficient fine-tuning. Our method outperforms the state-of-the-art on the Siegfried benchmark dataset in vineyard and railway segmentation, achieving +5% and +13% relative improvements in mIoU in 10-shot scenarios and around +20% in the more challenging 5-shot setting. Additionally, it demonstrates strong performance on the ICDAR 2021 competition dataset, attaining a mean PQ of 67.3% for building block segmentation, despite not being optimized for this shape-sensitive metric, underscoring its generalizability. Notably, our approach maintains high performance even in extremely low-data regimes (10- & 5-shot), while requiring only 689k trainable parameters - just 0.21% of the total model size. Our approach enables precise segmentation of diverse historical maps while drastically reducing the need for manual annotations, advancing automated processing and analysis in the field. Our implementation is publicly available at: https://github.com/RafaelSterzinger/few-shot-map-segmentation.

[59] TaleForge: Interactive Multimodal System for Personalized Story Creation

Minh-Loi Nguyen,Quang-Khai Le,Tam V. Nguyen,Minh-Triet Tran,Trung-Nghia Le

Main category: cs.CV

TL;DR: TaleForge是一个交互式多模态系统，通过结合大语言模型（LLMs）和文本到图像扩散技术，将用户的面部图像嵌入故事叙述和插图中，实现个性化故事创作。

Details

Motivation: 现有方法通常将用户视为被动消费者，提供通用情节且缺乏个性化，限制了用户的参与感和沉浸感。TaleForge旨在通过个性化故事创作提升用户的参与度和沉浸感。

Contribution: 1. 提出了一种结合LLMs和文本到图像扩散技术的多模态系统；2. 实现了用户面部图像和服装选择在叙事和插图中的个性化嵌入；3. 通过用户研究验证了系统的有效性和用户体验。

Method: 1. 故事生成模块：利用LLMs根据用户提示生成叙事和角色描述；2. 个性化图像生成模块：将用户面部和服装选择融合到角色插图中；3. 背景生成模块：创建包含个性化角色的场景背景。

Result: 用户研究表明，当用户作为故事主角时，参与感和归属感显著提升。用户赞赏系统的实时预览和直观控制，但也提出了对更精细叙事编辑工具的需求。

Insight: 多模态技术（如LLMs和文本到图像扩散）的结合可以显著提升个性化故事创作的沉浸感和用户体验，但需要进一步优化编辑功能以满足用户需求。

Abstract: Storytelling is a deeply personal and creative process, yet existing methods often treat users as passive consumers, offering generic plots with limited personalization. This undermines engagement and immersion, especially where individual style or appearance is crucial. We introduce TaleForge, a personalized story-generation system that integrates large language models (LLMs) and text-to-image diffusion to embed users’ facial images within both narratives and illustrations. TaleForge features three interconnected modules: Story Generation, where LLMs create narratives and character descriptions from user prompts; Personalized Image Generation, merging users’ faces and outfit choices into character illustrations; and Background Generation, creating scene backdrops that incorporate personalized characters. A user study demonstrated heightened engagement and ownership when individuals appeared as protagonists. Participants praised the system’s real-time previews and intuitive controls, though they requested finer narrative editing tools. TaleForge advances multimodal storytelling by aligning personalized text and imagery to create immersive, user-centric experiences.

[60] GenEscape: Hierarchical Multi-Agent Generation of Escape Room Puzzles

Mengyi Shan,Brian Curless,Ira Kemelmacher-Shlizerman,Steve Seitz

Main category: cs.CV

TL;DR: 论文提出了一种层次化多智能体框架GenEscape，用于生成视觉吸引、逻辑严密且富有挑战性的密室逃脱谜题图片，通过功能设计、符号场景图推理、布局合成和局部图像编辑的分阶段协作提升生成质量。

Details

Motivation: 现有的文本到图像模型在空间关系理解和功能推理能力上表现不足，难以生成复杂且功能合理的密室逃脱谜题图像，因此需要一种更结构化的方法来分解和优化这一生成过程。

Contribution: 提出了一种层次化多智能体协作框架，通过分阶段的任务分解（功能设计、符号推理、布局合成和局部编辑）提升密室逃脱谜题图像的逻辑合理性和视觉质量。

Method: 采用多智能体协作框架，每个智能体负责特定任务（如功能设计、场景图推理、布局合成等），通过反馈循环确保图像的视觉一致性和功能合理性。

Result: 实验表明，智能体协作显著提高了谜题的逻辑严密性（如避免捷径）和功能清晰性，同时保持了视觉质量。

Insight: 通过任务分解和协作反馈，可以弥补单一模型在复杂生成任务中的局限性，为多模态生成任务提供新思路。

Abstract: We challenge text-to-image models with generating escape room puzzle images that are visually appealing, logically solid, and intellectually stimulating. While base image models struggle with spatial relationships and affordance reasoning, we propose a hierarchical multi-agent framework that decomposes this task into structured stages: functional design, symbolic scene graph reasoning, layout synthesis, and local image editing. Specialized agents collaborate through iterative feedback to ensure the scene is visually coherent and functionally solvable. Experiments show that agent collaboration improves output quality in terms of solvability, shortcut avoidance, and affordance clarity, while maintaining visual quality.

[61] Periodic-MAE: Periodic Video Masked Autoencoder for rPPG Estimation

Jiho Choi,Sang Jun Lee

Main category: cs.CV

TL;DR: 论文提出了一种名为Periodic-MAE的自监督学习方法，通过视频掩码自编码器从无标记的面部视频中学习周期性信号的通用表示，并将其应用于远程光电容积描记（rPPG）的估计任务。

Details

Motivation: 远程光电容积描记（rPPG）是一种无需接触就能测量生理信号（如心率）的技术，但现有方法在复杂场景下的泛化能力有限。论文旨在通过自监督学习提取更鲁棒的周期性信号表示，提升rPPG的性能。

Contribution: 1. 提出了一种周期性视频掩码自编码器（Periodic-MAE），用于学习面部视频中的准周期性信号表示；2. 引入生理带宽约束，利用生理信号的频率稀疏性提供脉冲线索；3. 在多个数据集（PURE、UBFC-rPPG、MMPD、V4V）上验证了方法的有效性，尤其在跨数据集评估中表现出色。

Method: 1. 使用视频掩码自编码器进行自监督预训练，通过帧掩码采样捕捉准周期性信号；2. 引入生理带宽约束，限制模型学习频率范围内的稀疏信号；3. 将预训练的编码器迁移到rPPG任务中，提取生理信号。

Result: 实验结果表明，所提方法在多个数据集上显著优于现有方法，特别是在跨数据集场景中表现突出。

Insight: 1. 自监督预训练能有效捕捉面部视频中的周期性信号；2. 生理带宽约束是提升模型性能的关键因素；3. 方法在复杂场景下具有较强的泛化能力。

Abstract: In this paper, we propose a method that learns a general representation of periodic signals from unlabeled facial videos by capturing subtle changes in skin tone over time. The proposed framework employs the video masked autoencoder to learn a high-dimensional spatio-temporal representation of the facial region through self-supervised learning. Capturing quasi-periodic signals in the video is crucial for remote photoplethysmography (rPPG) estimation. To account for signal periodicity, we apply frame masking in terms of video sampling, which allows the model to capture resampled quasi-periodic signals during the pre-training stage. Moreover, the framework incorporates physiological bandlimit constraints, leveraging the property that physiological signals are sparse within their frequency bandwidth to provide pulse cues to the model. The pre-trained encoder is then transferred to the rPPG task, where it is used to extract physiological signals from facial videos. We evaluate the proposed method through extensive experiments on the PURE, UBFC-rPPG, MMPD, and V4V datasets. Our results demonstrate significant performance improvements, particularly in challenging cross-dataset evaluations. Our code is available at https://github.com/ziiho08/Periodic-MAE.

[62] SPADE: Spatial Transcriptomics and Pathology Alignment Using a Mixture of Data Experts for an Expressive Latent Space

Ekaterina Redekop,Mara Pleasure,Zichen Wang,Kimberly Flores,Anthony Sisk,William Speier,Corey W. Arnold

Main category: cs.CV

TL;DR: SPADE是一种基于多模态数据（病理全切片图像和空间转录组）的基础模型，通过专家混合技术和对比学习，构建了一个统一的潜在空间，显著提升了few-shot性能。

Details

Motivation: 数字病理学和多模态数据的快速发展需要一种方法，能够整合病理图像和空间转录组数据，以捕捉更丰富的分子异质性。

Contribution: 提出了SPADE模型，首次将病理图像与空间转录组数据整合到一个统一的潜在空间中，并通过两阶段特征空间聚类和对比学习优化表示学习。

Method: 采用专家混合技术，通过两阶段聚类生成数据专家，使用对比学习训练；在HEST-1k数据集上预训练，并在14个下游任务中评估性能。

Result: SPADE在few-shot学习中表现显著优于基线模型，证明了其在形态学和分子信息整合方面的优势。

Insight: 整合多模态数据（如病理图像和空间转录组）可以显著提升模型的表示能力，尤其是在数据稀缺的下游任务中。

Abstract: The rapid growth of digital pathology and advances in self-supervised deep learning have enabled the development of foundational models for various pathology tasks across diverse diseases. While multimodal approaches integrating diverse data sources have emerged, a critical gap remains in the comprehensive integration of whole-slide images (WSIs) with spatial transcriptomics (ST), which is crucial for capturing critical molecular heterogeneity beyond standard hematoxylin & eosin (H&E) staining. We introduce SPADE, a foundation model that integrates histopathology with ST data to guide image representation learning within a unified framework, in effect creating an ST-informed latent space. SPADE leverages a mixture-of-data experts technique, where experts, created via two-stage feature-space clustering, use contrastive learning to learn representations of co-registered WSI patches and gene expression profiles. Pre-trained on the comprehensive HEST-1k dataset, SPADE is evaluated on 14 downstream tasks, demonstrating significantly superior few-shot performance compared to baseline models, highlighting the benefits of integrating morphological and molecular information into one latent space.

[63] LLaVA-Scissor: Token Compression with Semantic Connected Components for Video LLMs

Boyuan Sun,Jiaxing Zhao,Xihan Wei,Qibin Hou

Main category: cs.CV

TL;DR: LLaVA-Scissor提出了一种无需训练的token压缩策略，通过语义连通分量（SCC）方法在视频多模态大语言模型中实现高效的token压缩，显著提升了视频理解任务的表现。

Details

Motivation: 现有的token压缩方法主要基于注意力分数，但无法全面捕捉语义区域且存在冗余。LLaVA-Scissor旨在通过语义连通分量方法更有效地压缩token，提高模型效率。

Contribution: 1）提出基于语义连通分量（SCC）的token压缩方法；2）设计了一种两步的时空token压缩策略；3）在多种视频理解任务中验证了方法的优越性。

Method: 利用语义连通分量（SCC）将token划分为不同的语义区域，实现无重叠的token表示。采用两步时空压缩策略，分别在空间和时间域进行token压缩。

Result: 在视频问答、长视频理解等任务中，LLaVA-Scissor在低token保留率下表现优于其他方法。

Insight: 语义驱动的token压缩方法比基于注意力的方法更有效，尤其适用于视频多模态模型。

Abstract: In this paper, we present LLaVA-Scissor, a training-free token compression strategy designed for video multimodal large language models. Previous methods mostly attempt to compress tokens based on attention scores, but fail to effectively capture all semantic regions and often lead to token redundancy. Differently, we propose to leverage the Semantic Connected Components (SCC) approach that assigns tokens to distinct semantic regions within the token set, ensuring comprehensive semantic coverage. The outcome is a two-step spatio-temporal token compression strategy that utilizes SCC in both spatial and temporal domains. This strategy can effectively compress tokens by representing the entire video with a set of non-overlapping semantic tokens. We conduct extensive evaluations of the token compression capabilities of LLaVA-Scissor across diverse video understanding benchmarks, including video question answering, long video understanding, and comprehensive multi-choices benchmarks. Experimental results show that the proposed LLaVA-Scissor outperforms other token compression methods, achieving superior performance in various video understanding benchmarks, particularly at low token retention ratios. Project page: https://github.com/HumanMLLM/LLaVA-Scissor.

[64] Remote Sensing Large Vision-Language Model: Semantic-augmented Multi-level Alignment and Semantic-aware Expert Modeling

Sungjune Park,Yeongyun Kim,Se Yeon Kim,Yong Man Ro

Main category: cs.CV

TL;DR: 该论文针对遥感图像中的视觉-语言任务提出了一种新型的大型视觉-语言模型框架，通过语义增强的多级对齐和语义感知的专家建模，解决了现有LVLMs在遥感领域的适配问题。

Details

Motivation: 遥感图像与自然图像在视觉外观、对象尺度和语义上存在显著差异，限制了现有LVLMs的直接适配。遥感场景包含从粗到细的多级语义信息，需要专门的模型来处理。

Contribution: 提出了一个专门用于遥感理解的LVLM框架，核心贡献包括语义增强的多级对齐和语义感知的专家建模。

Method: 1. 通过检索式语义增强模块，从遥感语义知识库中检索并聚合多级语义线索，丰富视觉特征。2. 设计语义感知的专家模型，分层处理不同级别的语义表示。

Result: 在多个遥感任务（如场景分类和视觉问答）中表现出显著的性能提升，证明了框架在跨语义层级理解上的有效性。

Insight: 通过多级语义对齐和专家分层处理，可以显著提升模型在遥感领域的跨模态理解能力，为未来遥感专用LVLMs的研究提供了新思路。

Abstract: Large Vision and Language Models (LVLMs) have shown strong performance across various vision-language tasks in natural image domains. However, their application to remote sensing (RS) remains underexplored due to significant domain differences in visual appearances, object scales, and semantics. These discrepancies hider the effective understanding of RS scenes, which contain rich, multi-level semantic information spanning from coarse-to-fine levels. Hence, it limits the direct adaptation of existing LVLMs to RS imagery. To address this gap, we propose a novel LVLM framework tailored for RS understanding, incorporating two core components: Semantic-augmented Multi-level Alignment and Semantic-aware Expert Modeling. First, to align multi-level visual features, we introduce the retrieval-based Semantic Augmentation Module which enriches the visual features with relevant semantics across fine-to-coarse levels (e.g., object- and scene-level information). It is designed to retrieve relevant semantic cues from a RS semantic knowledge database, followed by aggregation of semantic cues with user query and multi-level visual features, resulting in semantically enriched representation across multiple levels. Second, for Semantic-aware Expert Modeling, we design semantic experts, where each expert is responsible for processing semantic representation at different levels separately. This enables hierarchical semantic understanding from coarse to fine levels. Evaluations across multiple RS tasks-including scene classification and VQA, etc.-demonstrate that the proposed framework achieves consistent improvements across multiple semantic levels. This highlights its capability and effectiveness in bridging the gap between general LVLMs and unique demands of RS-specific vision-language understanding.

[65] Dual-Perspective United Transformer for Object Segmentation in Optical Remote Sensing Images

Yanguang Sun,Jiexi Yan,Jianjun Qian,Chunyan Xu,Jian Yang,Lei Luo

Main category: cs.CV

TL;DR: DPU-Former提出了一种新型双视角统一Transformer，用于光学遥感图像中的目标分割，通过全局-局部混合注意力和傅里叶空间融合策略，解决了现有方法在特征异质性和模型复杂性上的不足。

Details

Motivation: 现有方法主要基于卷积或Transformer特征，各自优势尚未充分结合，且特征异质性和模型复杂性常被忽视，导致分割性能不佳。

Contribution: 提出了DPU-Former，首次结合全局-局部混合注意力、傅里叶空间融合策略和门控线性前馈网络，有效整合长程依赖与空间细节。

Method: 设计全局-局部混合注意力捕捉多视角信息，引入傅里叶空间融合策略以减少偏差，并使用门控线性前馈网络增强表达能力。

Result: 在多个数据集上超越现有最优方法，证明了模型的有效性。

Insight: 结合卷积与Transformer特征的优势需解决特征异质性，而傅里叶空间融合策略为高效特征融合提供了新思路。

Abstract: Automatically segmenting objects from optical remote sensing images (ORSIs) is an important task. Most existing models are primarily based on either convolutional or Transformer features, each offering distinct advantages. Exploiting both advantages is valuable research, but it presents several challenges, including the heterogeneity between the two types of features, high complexity, and large parameters of the model. However, these issues are often overlooked in existing the ORSIs methods, causing sub-optimal segmentation. For that, we propose a novel Dual-Perspective United Transformer (DPU-Former) with a unique structure designed to simultaneously integrate long-range dependencies and spatial details. In particular, we design the global-local mixed attention, which captures diverse information through two perspectives and introduces a Fourier-space merging strategy to obviate deviations for efficient fusion. Furthermore, we present a gated linear feed-forward network to increase the expressive ability. Additionally, we construct a DPU-Former decoder to aggregate and strength features at different layers. Consequently, the DPU-Former model outperforms the state-of-the-art methods on multiple datasets. Code: https://github.com/CSYSI/DPU-Former.

[66] Grounding-Aware Token Pruning: Recovering from Drastic Performance Drops in Visual Grounding Caused by Pruning

Tzu-Chun Chien,Chieh-Kai Lin,Shiang-Feng Tsai,Ruei-Chi Lai,Hung-Jen Chen,Min Sun

Main category: cs.CV

TL;DR: 该论文提出了Grounding-Aware Token Pruning (GAP)方法，通过调整位置ID来恢复因视觉令牌修剪导致的视觉接地性能下降，无需额外训练或计算资源。

Details

Motivation: 多模态大语言模型(MLLMs)在视觉接地任务中表现出色，但处理大量视觉令牌带来的高计算成本促使了令牌修剪技术的发展。然而，修剪会导致模型接地能力显著下降，论文旨在解决这一问题。

Contribution: 提出了GAP方法，通过修复修剪后错位的位置ID，显著恢复了模型的接地性能，且在多种模型和修剪策略中表现一致。

Method: GAP通过简单调整修剪后的位置ID，使其顺序和值与原始任务对齐，从而恢复性能。它无需额外训练或资源开销。

Result: 在RefCOCO验证集上，LLaVA的接地准确率从修剪后的15.34%恢复到51.42%，接近原始性能的90%。

Insight: 位置ID的顺序和值对视觉接地任务的性能至关重要，GAP提供了一种轻量级解决方案，适用于多种模型。

Abstract: Recent Multimodal Large Language Models (MLLMs) have demonstrated strong performance in visual grounding, establishing themselves as a general interface for various vision-language applications. This progress has driven the development of token pruning methods to mitigate the high computational costs associated with processing numerous visual tokens. However, we observe that pruning significantly weakens the model’s grounding ability, leading to incorrect predictions and drastic performance degradation. In Referring Expression Comprehension (REC), for instance, pruning causes the accuracy of LLaVA on the RefCOCO validation set to drop from 56.14% to 15.34%. Our analysis identifies misaligned position IDs after pruning as the primary cause of this degradation, as both the order and value of these IDs are crucial for maintaining performance in grounding tasks. To address this issue, we propose Grounding-Aware Token Pruning (GAP), a simple yet effective adjustment to position IDs that recovers REC accuracy back to 51.42%, which is 90% of the original performance in the without pruning setting, all while requiring no additional training, memory, or computational overhead. Applied to models such as Shikra, MiniGPTv2, and the LLaVA series, our method consistently improves performance across various token pruning strategies.

[67] GRASP-PsONet: Gradient-based Removal of Spurious Patterns for PsOriasis Severity Classification

Basudha Pal,Sharif Amit Kamran,Brendon Lutnick,Molly Lucas,Chaitanya Parmar,Asha Patel Shah,David Apfel,Steven Fakharzadeh,Lloyd Miller,Gabriela Cula,Kristopher Standish

Main category: cs.CV

TL;DR: 论文提出了一种基于梯度的框架GRASP-PsONet，用于自动识别并去除训练数据中引入虚假相关性的图像，从而提升银屑病（PsO）严重程度分类模型的泛化能力。

Details

Motivation: 银屑病严重程度评分的自动化受到图像质量不一致和标注变异的挑战，传统方法依赖多标注者和人工评审，成本高昂。

Contribution: 提出了一种梯度驱动的训练样本过滤方法，通过追踪误分类验证图像的梯度，识别并去除标注不一致或受非临床伪影影响的训练样本。

Method: 基于ConvNeXT的弱监督模型，利用梯度解释性方法检测问题图像，直接优化训练数据集。

Result: 移除8.2%的问题图像后，模型AUC-ROC提升了5%（从85%到90%），并能高效识别标注不一致的样本（覆盖90%的评分差异案例）。

Insight: 梯度分析方法可有效检测数据中的虚假相关性，为远程医疗中的自动化评分提供了高鲁棒性的解决方案。

Abstract: Psoriasis (PsO) severity scoring is important for clinical trials but is hindered by inter-rater variability and the burden of in person clinical evaluation. Remote imaging using patient captured mobile photos offers scalability but introduces challenges, such as variation in lighting, background, and device quality that are often imperceptible to humans but can impact model performance. These factors, along with inconsistencies in dermatologist annotations, reduce the reliability of automated severity scoring. We propose a framework to automatically flag problematic training images that introduce spurious correlations which degrade model generalization, using a gradient based interpretability approach. By tracing the gradients of misclassified validation images, we detect training samples where model errors align with inconsistently rated examples or are affected by subtle, nonclinical artifacts. We apply this method to a ConvNeXT based weakly supervised model designed to classify PsO severity from phone images. Removing 8.2% of flagged images improves model AUC-ROC by 5% (85% to 90%) on a held out test set. Commonly, multiple annotators and an adjudication process ensure annotation accuracy, which is expensive and time consuming. Our method detects training images with annotation inconsistencies, potentially removing the need for manual review. When applied to a subset of training data rated by two dermatologists, the method identifies over 90% of cases with inter-rater disagreement by reviewing only the top 30% of samples. This improves automated scoring for remote assessments, ensuring robustness despite data collection variability.

[68] Integrating Multi-Modal Sensors: A Review of Fusion Techniques for Intelligent Vehicles

Chuheng Wei,Ziye Qin,Ziyan Zhang,Guoyuan Wu,Matthew J. Barth

Main category: cs.CV

TL;DR: 这篇论文综述了多传感器融合在自动驾驶中的关键作用，归类了数据级、特征级和决策级融合策略，并系统回顾了基于深度学习的融合方法。同时探讨了多模态数据集的应用及新兴趋势，如视觉语言模型和大语言模型的集成。

Details

Motivation: 多传感器融合对于提升自动驾驶的环境感知能力至关重要，尤其在恶劣天气和复杂城市场景中。当前的方法需要系统化的总结和未来方向的展望。

Contribution: 论文主要贡献包括对多传感器融合策略的分类、深度学习方法在各融合层级的系统性回顾，以及对新兴趋势和未来方向的探讨。

Method: 论文将多传感器融合策略分为数据级、特征级和决策级，并针对每类策略总结了基于深度学习的方法。同时，分析了多模态数据集的实际应用。

Result: 通过融合多种传感器数据，可以显著提升自动驾驶系统的环境感知能力和鲁棒性，尤其是在复杂场景中。

Insight: 视觉语言模型和大语言模型的集成可能是未来多传感器融合的重要方向，有望进一步提升系统的自适应能力。

Abstract: Multi-sensor fusion plays a critical role in enhancing perception for autonomous driving, overcoming individual sensor limitations, and enabling comprehensive environmental understanding. This paper first formalizes multi-sensor fusion strategies into data-level, feature-level, and decision-level categories and then provides a systematic review of deep learning-based methods corresponding to each strategy. We present key multi-modal datasets and discuss their applicability in addressing real-world challenges, particularly in adverse weather conditions and complex urban environments. Additionally, we explore emerging trends, including the integration of Vision-Language Models (VLMs), Large Language Models (LLMs), and the role of sensor fusion in end-to-end autonomous driving, highlighting its potential to enhance system adaptability and robustness. Our work offers valuable insights into current methods and future directions for multi-sensor fusion in autonomous driving.

[69] DIVE: Deep-search Iterative Video Exploration A Technical Report for the CVRR Challenge at CVPR 2025

Umihiro Kamoto,Tatsuya Ishibashi,Noriyuki Kugo

Main category: cs.CV

TL;DR: 这篇报告介绍了在CVPR 2025 CVRR挑战赛中夺冠的解决方案DIVE，通过迭代推理方法在复杂视频问答任务中实现了81.44%的准确率。

Details

Motivation: 解决复杂视频问答任务中的语义理解和推理问题，提升系统在多样化现实场景中的问答能力。

Contribution: 提出DIVE方法，采用迭代推理框架，逐步分解问题和推理答案，显著提升了复杂视频问答的准确性。

Method: 使用迭代推理方法，将输入问题语义分解，并通过逐步推理和渐进推断生成答案。

Result: 在CVRR-ES测试集上达到81.44%准确率，排名第一。

Insight: 迭代推理框架能有效处理复杂语义和上下文关系，适用于需要多层次推理的视频问答任务。

Abstract: In this report, we present the winning solution that achieved the 1st place in the Complex Video Reasoning & Robustness Evaluation Challenge 2025. This challenge evaluates the ability to generate accurate natural language answers to questions about diverse, real-world video clips. It uses the Complex Video Reasoning and Robustness Evaluation Suite (CVRR-ES) benchmark, which consists of 214 unique videos and 2,400 question-answer pairs spanning 11 categories. Our method, DIVE (Deep-search Iterative Video Exploration), adopts an iterative reasoning approach, in which each input question is semantically decomposed and solved through stepwise reasoning and progressive inference. This enables our system to provide highly accurate and contextually appropriate answers to even the most complex queries. Applied to the CVRR-ES benchmark, our approach achieves 81.44% accuracy on the test set, securing the top position among all participants. This report details our methodology and provides a comprehensive analysis of the experimental results, demonstrating the effectiveness of our iterative reasoning framework in achieving robust video question answering. The code is available at https://github.com/PanasonicConnect/DIVE

[70] SODA: Out-of-Distribution Detection in Domain-Shifted Point Clouds via Neighborhood Propagation

Adam Goodge,Xun Xu,Bryan Hooi,Wee Siong Ng,Jingyi Liao,Yongyi Su,Xulei Yang

Main category: cs.CV

TL;DR: 该论文提出了一种名为SODA的新方法，通过基于邻域的分数传播方案，改进点云数据的OOD检测，解决了合成数据到真实数据域偏移的问题。

Details

Motivation: 点云数据在多种应用中日益普及，但OOD检测问题尚未充分研究。由于预训练3D视觉语言模型的数据集较小且多为合成数据，导致在实际任务中域偏移严重，影响了OOD检测的性能。

Contribution: 提出SODA方法，通过邻域传播改进OOD点云检测，无需额外训练，实现了在多种数据集和问题设置中的最先进性能。

Method: SODA采用基于邻域的分数传播方案，利用3D视觉语言模型的潜在空间对齐点云和文本嵌入，解决了合成到真实数据的域偏移问题。

Result: 实验表明，SODA在合成到真实数据的域偏移场景中显著提升了OOD检测性能，优于现有方法。

Insight: 点云OOD检测的挑战主要来自域偏移问题，利用邻域传播可以有效缓解这一问题，提升模型在实际任务中的可靠性。

Abstract: As point cloud data increases in prevalence in a variety of applications, the ability to detect out-of-distribution (OOD) point cloud objects becomes critical for ensuring model safety and reliability. However, this problem remains under-explored in existing research. Inspired by success in the image domain, we propose to exploit advances in 3D vision-language models (3D VLMs) for OOD detection in point cloud objects. However, a major challenge is that point cloud datasets used to pre-train 3D VLMs are drastically smaller in size and object diversity than their image-based counterparts. Critically, they often contain exclusively computer-designed synthetic objects. This leads to a substantial domain shift when the model is transferred to practical tasks involving real objects scanned from the physical environment. In this paper, our empirical experiments show that synthetic-to-real domain shift significantly degrades the alignment of point cloud with their associated text embeddings in the 3D VLM latent space, hindering downstream performance. To address this, we propose a novel methodology called SODA which improves the detection of OOD point clouds through a neighborhood-based score propagation scheme. SODA is inference-based, requires no additional model training, and achieves state-of-the-art performance over existing approaches across datasets and problem settings.

[71] Exploring Task-Solving Paradigm for Generalized Cross-Domain Face Anti-Spoofing via Reinforcement Fine-Tuning

Fangling Jiang,Qi Li,Weining Wang,Gang Wang,Bing Liu,Zhenan Sun

Main category: cs.CV

TL;DR: 该论文提出了一种基于强化微调的人脸反欺骗方法，通过多模态大语言模型学习解决反欺骗任务的能力，而非依赖模式记忆，实现了跨域泛化性能和可解释性。

Details

Motivation: 现有的人脸反欺骗方法容易记忆训练数据的模式，导致对未知攻击类型的泛化能力差且缺乏可解释性。作者希望通过强化学习优化策略，引导模型从多角度探索推理策略，从而提升跨域泛化性能。

Contribution: 1) 提出了一种基于强化微调的人脸反欺骗框架；2) 设计了可验证的类一致性奖励和推理一致性奖励；3) 采用GRPO优化策略，从广泛的解空间中提炼高泛化决策规则。

Method: 结合多模态大语言模型，设计奖励函数（类一致性和推理一致性），并利用GRPO策略进行优化，通过迭代学习保留高奖励轨迹，提炼泛化决策规则。

Result: 实验表明，该方法在跨域泛化性能上达到SOTA，对未知攻击类型具有强适应能力，同时提供可解释的推理过程。

Insight: 通过强化学习引导模型主动学习和推理，而非依赖数据模式记忆，可显著提升跨域任务的泛化能力和模型的可解释性。

Abstract: Recently the emergence of novel presentation attacks has drawn increasing attention to face anti-spoofing. However, existing methods tend to memorize data patterns from the training set, resulting in poor generalization to unknown attack types across different scenarios and limited interpretability. To address these challenges, this paper presents a reinforcement fine-tuning-based face anti-spoofing method that stimulates the capabilities of multimodal large language models to think and learn how to solve the anti-spoofing task itself, rather than relying on the memorization of authenticity patterns. We design verifiable class consistent reward and reasoning consistent reward, and employ a GRPO-based optimization strategy to guide the model in exploring reasoning policies from multiple perspectives to maximize expected rewards. As a result, through iterative trial-and-error learning while retaining only high-reward trajectories, the model distills highly generalizable decision-making rules from the extensive solution space to effectively address cross-domain face anti-spoofing tasks. Extensive experimental results demonstrate that our method achieves state-of-the-art cross-domain generalization performance. It generalizes well to diverse unknown attack types in unseen target domains while providing interpretable reasoning for its authenticity decisions without requiring labor-intensive textual annotations for training.

[72] Visual Content Detection in Educational Videos with Transfer Learning and Dataset Enrichment

Dipayan Biswas,Shishir Shah,Jaspal Subhlok

Main category: cs.CV

TL;DR: 采用迁移学习和数据集增强技术，检测教育视频中的视觉内容（如图表、表格），优化YOLO模型并通过半监督自动标注策略提升性能。

Details

Motivation: 教育视频中的视觉元素（如图表、表格）对理解和检索至关重要，但由于其结构独特且缺乏标注数据，现有目标检测模型效果不佳，因此需要改进。

Contribution: 1. 提出一种基于迁移学习的视觉内容检测方法；2. 公开标注的教育视频帧数据集和源代码；3. 通过优化YOLO模型和半监督策略提升检测性能。

Method: 评估多种目标检测模型，选择YOLO并进行优化，结合多基准数据集训练和半监督自动标注策略。

Result: 优化的YOLO模型在教育视频视觉内容检测任务中表现优异，并开发了通用解决方案。

Insight: 迁移学习和半监督策略能有效解决教育视频中视觉内容检测的数据稀缺和结构多样性问题。

Abstract: Video is transforming education with online courses and recorded lectures supplementing and replacing classroom teaching. Recent research has focused on enhancing information retrieval for video lectures with advanced navigation, searchability, summarization, as well as question answering chatbots. Visual elements like tables, charts, and illustrations are central to comprehension, retention, and data presentation in lecture videos, yet their full potential for improving access to video content remains underutilized. A major factor is that accurate automatic detection of visual elements in a lecture video is challenging; reasons include i) most visual elements, such as charts, graphs, tables, and illustrations, are artificially created and lack any standard structure, and ii) coherent visual objects may lack clear boundaries and may be composed of connected text and visual components. Despite advancements in deep learning based object detection, current models do not yield satisfactory performance due to the unique nature of visual content in lectures and scarcity of annotated datasets. This paper reports on a transfer learning approach for detecting visual elements in lecture video frames. A suite of state of the art object detection models were evaluated for their performance on lecture video datasets. YOLO emerged as the most promising model for this task. Subsequently YOLO was optimized for lecture video object detection with training on multiple benchmark datasets and deploying a semi-supervised auto labeling strategy. Results evaluate the success of this approach, also in developing a general solution to the problem of object detection in lecture videos. Paper contributions include a publicly released benchmark of annotated lecture video frames, along with the source code to facilitate future research.

[73] RAUM-Net: Regional Attention and Uncertainty-aware Mamba Network

Mingquan Liu

Main category: cs.CV

TL;DR: RAUM-Net是一种半监督方法，结合视觉Mamba、区域注意力和贝叶斯不确定性，针对细粒度视觉分类（FGVC）任务，在标注数据稀缺时表现优异。

Details

Motivation: FGVC任务因类间差异细微且特征表示脆弱而极具挑战性，尤其当标注数据不足时，现有方法表现不佳。

Contribution: 提出了一种结合Mamba特征建模、区域注意力和贝叶斯不确定性的半监督方法，增强局部到全局特征建模，并利用贝叶斯推断筛选高质量伪标签。

Method: 采用Mamba结构进行特征建模，引入区域注意力机制关注关键区域，结合贝叶斯不确定性评估选择伪标签以提高鲁棒性。

Result: 在FGVC基准测试中表现优异，尤其在标注数据有限且存在遮挡的情况下展现了强鲁棒性。

Insight: 区域注意力与贝叶斯不确定性相结合的策略在提升半监督FGVC任务性能中发挥了重要作用。

Abstract: Fine Grained Visual Categorization (FGVC) remains a challenging task in computer vision due to subtle inter class differences and fragile feature representations. Existing methods struggle in fine grained scenarios, especially when labeled data is scarce. We propose a semi supervised method combining Mamba based feature modeling, region attention, and Bayesian uncertainty. Our approach enhances local to global feature modeling while focusing on key areas during learning. Bayesian inference selects high quality pseudo labels for stability. Experiments show strong performance on FGVC benchmarks with occlusions, demonstrating robustness when labeled data is limited. Code is available at https://github.com/wxqnl/RAUM Net.

[74] CERBERUS: Crack Evaluation & Recognition Benchmark for Engineering Reliability & Urban Stability

Justin Reinman,Sunwoong Choi

Main category: cs.CV

TL;DR: CERBERUS是一个用于训练和评估AI模型在基础设施裂缝检测中的合成基准，包含裂缝图像生成器和3D检测场景，测试表明合成与真实数据结合可提升性能。

Details

Motivation: 基础设施裂缝检测在实际应用中面临数据稀缺和多样性不足的问题，需要一个灵活、可重复的基准来支持模型训练和评估。

Contribution: 提出CERBERUS基准，包含合成数据生成和复杂场景模拟，并验证了合成与真实数据结合的有效性。

Method: 利用Unity构建3D场景和图像生成器，设计简单与复杂检测任务，测试YOLO模型在不同数据组合下的表现。

Result: 实验表明，合成与真实数据结合可显著提升模型在真实场景中的裂缝检测性能。

Insight: 合成数据可作为真实数据的补充，提升模型泛化能力；复杂场景设计有助于评估模型的鲁棒性。

Abstract: CERBERUS is a synthetic benchmark designed to help train and evaluate AI models for detecting cracks and other defects in infrastructure. It includes a crack image generator and realistic 3D inspection scenarios built in Unity. The benchmark features two types of setups: a simple Fly-By wall inspection and a more complex Underpass scene with lighting and geometry challenges. We tested a popular object detection model (YOLO) using different combinations of synthetic and real crack data. Results show that combining synthetic and real data improves performance on real-world images. CERBERUS provides a flexible, repeatable way to test defect detection systems and supports future research in automated infrastructure inspection. CERBERUS is publicly available at https://github.com/justinreinman/Cerberus-Defect-Generator.

[75] SPAZER: Spatial-Semantic Progressive Reasoning Agent for Zero-shot 3D Visual Grounding

Zhao Jin,Rong-Cheng Tu,Jingyi Liao,Wenhao Sun,Xiao Luo,Shunyu Liu,Dacheng Tao

Main category: cs.CV

TL;DR: SPAZER 是一个结合空间与语义推理的零样本 3D 视觉定位系统，无需 3D 标注数据即可实现高精度定位，显著超过现有零样本方法。

Details

Motivation: 当前零样本 3D 视觉定位方法过于依赖单一空间或语义模态，限制了复杂场景下的表现，因此需要一种能结合两者优势的推理框架。

Contribution: 提出了 SPAZER，通过渐进式推理框架联合空间（3D）和语义（2D）模态，实现无需 3D 标注数据的零样本 3D 视觉定位。

Method: 1) 生成 3D 场景最优视角；2) 基于锚点指导筛选候选对象；3) 结合 2D 图像进行 3D-2D 联合决策。

Result: 在 ScanRefer 和 Nr3D 基准测试中，分别以 9.0% 和 10.9% 的准确率优势超越现有零样本方法。

Insight: 结合空间与语义模态的渐进式推理能显著提升零样本 3D 视觉定位性能，尤其在复杂场景下表现突出。

Abstract: 3D Visual Grounding (3DVG) aims to localize target objects within a 3D scene based on natural language queries. To alleviate the reliance on costly 3D training data, recent studies have explored zero-shot 3DVG by leveraging the extensive knowledge and powerful reasoning capabilities of pre-trained LLMs and VLMs. However, existing paradigms tend to emphasize either spatial (3D-based) or semantic (2D-based) understanding, limiting their effectiveness in complex real-world applications. In this work, we introduce SPAZER - a VLM-driven agent that combines both modalities in a progressive reasoning framework. It first holistically analyzes the scene and produces a 3D rendering from the optimal viewpoint. Based on this, anchor-guided candidate screening is conducted to perform a coarse-level localization of potential objects. Furthermore, leveraging retrieved relevant 2D camera images, 3D-2D joint decision-making is efficiently performed to determine the best-matching object. By bridging spatial and semantic reasoning neural streams, SPAZER achieves robust zero-shot grounding without training on 3D-labeled data. Extensive experiments on ScanRefer and Nr3D benchmarks demonstrate that SPAZER significantly outperforms previous state-of-the-art zero-shot methods, achieving notable gains of 9.0% and 10.9% in accuracy.

[76] TASeg: Text-aware RGB-T Semantic Segmentation based on Fine-tuning Vision Foundation Models

Meng Yu,Te Cui,Qitong Chu,Wenjie Song,Yi Yang,Yufeng Yue

Main category: cs.CV

TL;DR: TASeg提出了一种基于低秩适应（LoRA）微调视觉基础模型的文本感知RGB-T语义分割框架，解决了现有模型缺乏高层文本信息和模态异构性问题。

Details

Motivation: 现有RGB-T语义分割模型依赖低层视觉特征，缺乏高层文本信息；SAM在多模态环境下应用受限，亟需一种高效集成视觉与文本信息的方法。

Contribution: 1) 提出动态特征融合模块（DFFM）有效合并多视觉模态特征；2) 引入CLIP生成文本嵌入，实现语义对齐；3) 通过LoRA微调减少可训练参数量。

Method: TASeg结合了LoRA微调、DFFM模块和多模态文本嵌入，冻结SAM的transformer块以保持计算效率。

Result: 实验证明TASeg在多样数据集上性能优越，尤其在视觉特征相似场景下表现突出。

Insight: 高层文本信息与多模态特征融合可显著提升语义分割的准确性，同时LoRA微调保持了模型的高效性。

Abstract: Reliable semantic segmentation of open environments is essential for intelligent systems, yet significant problems remain: 1) Existing RGB-T semantic segmentation models mainly rely on low-level visual features and lack high-level textual information, which struggle with accurate segmentation when categories share similar visual characteristics. 2) While SAM excels in instance-level segmentation, integrating it with thermal images and text is hindered by modality heterogeneity and computational inefficiency. To address these, we propose TASeg, a text-aware RGB-T segmentation framework by using Low-Rank Adaptation (LoRA) fine-tuning technology to adapt vision foundation models. Specifically, we propose a Dynamic Feature Fusion Module (DFFM) in the image encoder, which effectively merges features from multiple visual modalities while freezing SAM’s original transformer blocks. Additionally, we incorporate CLIP-generated text embeddings in the mask decoder to enable semantic alignment, which further rectifies the classification error and improves the semantic understanding accuracy. Experimental results across diverse datasets demonstrate that our method achieves superior performance in challenging scenarios with fewer trainable parameters.

[77] R1-Track: Direct Application of MLLMs to Visual Object Tracking via Reinforcement Learning

Biao Wang,Wenwen Li

Main category: cs.CV

TL;DR: R1-Track将多模态大语言模型（MLLM）直接应用于视觉目标跟踪任务，通过强化学习微调Qwen2.5-VL模型，取得了显著性能提升。

Details

Motivation: 传统跟踪方法依赖大规模监督训练且缺乏灵活性，而MLLM在基础任务中表现出色，因此尝试将其直接应用于跟踪任务。

Contribution: 提出了R1-Track，通过强化学习微调MLLM模型Qwen2.5-VL，支持灵活初始化并保留了模型的原生能力。

Method: 使用基于规则的奖励函数和小规模数据集，采用GRPO强化学习方法微调Qwen2.5-VL。

Result: 在GOT-10k基准测试中表现优异，支持边界框或文本描述的初始化。

Insight: MLLM的潜力可通过强化学习扩展到特定视觉任务，但需要进一步优化以适应更复杂场景。

Abstract: Visual single object tracking aims to continuously localize and estimate the scale of a target in subsequent video frames, given only its initial state in the first frame. This task has traditionally been framed as a template matching problem, evolving through major phases including correlation filters, two-stream networks, and one-stream networks with significant progress achieved. However, these methods typically require explicit classification and regression modeling, depend on supervised training with large-scale datasets, and are limited to the single task of tracking, lacking flexibility. In recent years, multi-modal large language models (MLLMs) have advanced rapidly. Open-source models like Qwen2.5-VL, a flagship MLLMs with strong foundational capabilities, demonstrate excellent performance in grounding tasks. This has spurred interest in applying such models directly to visual tracking. However, experiments reveal that Qwen2.5-VL struggles with template matching between image pairs (i.e., tracking tasks). Inspired by deepseek-R1, we fine-tuned Qwen2.5-VL using the group relative policy optimization (GRPO) reinforcement learning method on a small-scale dataset with a rule-based reward function. The resulting model, R1-Track, achieved notable performance on the GOT-10k benchmark. R1-Track supports flexible initialization via bounding boxes or text descriptions while retaining most of the original model’s general capabilities. And we further discuss potential improvements for R1-Track. This rough technical report summarizes our findings as of May 2025.

[78] RoboEnvision: A Long-Horizon Video Generation Model for Multi-Task Robot Manipulation

Liudi Yang,Yang Bai,George Eskandar,Fengyi Shen,Mohammad Altillawi,Dong Chen,Soumajit Majumder,Ziyuan Liu,Gitta Kutyniok,Abhinav Valada

Main category: cs.CV

TL;DR: 论文提出了一种名为RoboEnvision的新方法，用于生成长时间跨度的视频以支持多任务机器人操作。通过分解高层目标、语义保持注意力模块和轻量级策略模型，避免了自回归生成带来的误差累积。

Details

Motivation: 现有的文本到视频扩散模型在机器人长时间任务中表现不佳，且自回归生成会导致误差累积。因此，需要一种新方法来提高长时间跨度视频的生成质量和任务一致性。

Contribution: 1) 分解高层目标为原子任务并生成关键帧；2) 提出语义保持注意力模块以保持关键帧一致性；3) 设计轻量级策略模型从视频回归机器人关节状态。

Method: 1) 提出两步生成管道：先分解目标生成关键帧，再通过扩散模型插值生成完整视频；2) 引入语义保持注意力模块；3) 设计轻量级策略模型。

Result: 在两个基准测试中实现了视频质量和一致性上的最先进表现，并在长时间任务中优于之前的策略模型。

Insight: 通过分解任务和插值生成方法，可以有效避免自回归误差积累，同时语义保持注意力模块确保了视频的连贯性和一致性。

Abstract: We address the problem of generating long-horizon videos for robotic manipulation tasks. Text-to-video diffusion models have made significant progress in photorealism, language understanding, and motion generation but struggle with long-horizon robotic tasks. Recent works use video diffusion models for high-quality simulation data and predictive rollouts in robot planning. However, these works predict short sequences of the robot achieving one task and employ an autoregressive paradigm to extend to the long horizon, leading to error accumulations in the generated video and in the execution. To overcome these limitations, we propose a novel pipeline that bypasses the need for autoregressive generation. We achieve this through a threefold contribution: 1) we first decompose the high-level goals into smaller atomic tasks and generate keyframes aligned with these instructions. A second diffusion model then interpolates between each of the two generated frames, achieving the long-horizon video. 2) We propose a semantics preserving attention module to maintain consistency between the keyframes. 3) We design a lightweight policy model to regress the robot joint states from generated videos. Our approach achieves state-of-the-art results on two benchmarks in video quality and consistency while outperforming previous policy models on long-horizon tasks.

[79] Towards Universal & Efficient Model Compression via Exponential Torque Pruning

Sarthak Ketanbhai Modi,Lim Zi Pong,Shourya Kuchhal,Yoshi Cao,Yupeng Cheng,Teo Yon Shin,Lin Shang-Wei,Zhiming Li

Main category: cs.CV

TL;DR: 论文提出了一种基于指数扭矩剪枝（ETP）的高效模型压缩方法，解决了现有扭矩正则化方法在剪枝效果上的不足。

Details

Motivation: 现代深度神经网络（DNNs）的规模和复杂性快速增长，导致计算成本和内存使用问题日益突出。现有基于扭矩正则化的剪枝方法在剪枝后网络仍较密集，且精度下降较大，作者认为这是因为线性力的施加方式不合理。

Contribution: 提出了一种新的指数扭矩剪枝（ETP）方法，通过指数形式的力施加方案，显著提高了剪枝效率，同时保持精度损失可忽略。

Method: ETP采用指数形式的力施加方案，对距离不同的神经网络模块施加不同的力，从而更高效地剪除冗余模块，保留关键模块。

Result: 实验表明，ETP在多领域中均能实现更高的压缩率，且精度损失极小。

Insight: 力的施加形式对模型压缩效果具有重要影响，指数形式的力比线性力更适合区分关键模块和冗余模块。

Abstract: The rapid growth in complexity and size of modern deep neural networks (DNNs) has increased challenges related to computational costs and memory usage, spurring a growing interest in efficient model compression techniques. Previous state-of-the-art approach proposes using a Torque-inspired regularization which forces the weights of neural modules around a selected pivot point. Whereas, we observe that the pruning effect of this approach is far from perfect, as the post-trained network is still dense and also suffers from high accuracy drop. In this work, we attribute such ineffectiveness to the default linear force application scheme, which imposes inappropriate force on neural module of different distances. To efficiently prune the redundant and distant modules while retaining those that are close and necessary for effective inference, in this work, we propose Exponential Torque Pruning (ETP), which adopts an exponential force application scheme for regularization. Experimental results on a broad range of domains demonstrate that, though being extremely simple, ETP manages to achieve significantly higher compression rate than the previous state-of-the-art pruning strategies with negligible accuracy drop.

[80] Advancing Facial Stylization through Semantic Preservation Constraint and Pseudo-Paired Supervision

Zhanyi Lu,Yue Zhou

Main category: cs.CV

TL;DR: 该论文提出了一种结合语义保留约束和伪配对监督的面部风格化方法，解决了现有方法中因忽略生成器的语义偏移而导致的伪影和内容不一致问题。

Details

Motivation: 当前基于StyleGAN的面部风格化方法虽取得了进展，但生成的图像仍存在伪影或与源图像内容不一致的问题。作者认为这是由于在风格化过程中忽略了生成器的语义偏移所导致。

Contribution: 1. 提出了一种结合语义保留约束和伪配对监督的面部风格化方法。2. 开发了一种创建多级伪配对数据集的方法以实现监督约束。3. 在框架基础上，实现了无需复杂网络设计或额外训练的多模态和参考引导风格化。

Method: 1. 使用语义保留约束确保生成图像与源图像的内容一致性。2. 通过伪配对监督优化风格化效果。3. 构建多级伪配对数据集以提供监督信号。

Result: 实验结果表明，该方法生成的高保真面部风格化图像在质量和内容一致性上优于以往方法。

Insight: 语义保留约束和伪配对监督的结合能够显著提升面部风格化任务的效果，尤其是在减少伪影和保持内容一致性方面。

Abstract: Facial stylization aims to transform facial images into appealing, high-quality stylized portraits, with the critical challenge of accurately learning the target style while maintaining content consistency with the original image. Although previous StyleGAN-based methods have made significant advancements, the generated results still suffer from artifacts or insufficient fidelity to the source image. We argue that these issues stem from neglecting semantic shift of the generator during stylization. Therefore, we propose a facial stylization method that integrates semantic preservation constraint and pseudo-paired supervision to enhance the content correspondence and improve the stylization effect. Additionally, we develop a methodology for creating multi-level pseudo-paired datasets to implement supervisory constraint. Furthermore, building upon our facial stylization framework, we achieve more flexible multimodal and reference-guided stylization without complex network architecture designs or additional training. Experimental results demonstrate that our approach produces high-fidelity, aesthetically pleasing facial style transfer that surpasses previous methods.

[81] Cross-modal Ship Re-Identification via Optical and SAR Imagery: A Novel Dataset and Method

Han Wang,Shengyang Li,Jian Yang,Yuxuan Liu,Yixuan Lv,Zhuang Zhou

Main category: cs.CV

TL;DR: 该论文提出了一种新的数据集HOSS ReID，用于通过光学与SAR图像进行跨模态船舶重识别，并基于Vision Transformer提出了一种基准方法TransOSS。

Details

Motivation: 现有船舶跟踪方法依赖于低分辨率的地球同步卫星或拍摄时间短的视频卫星，无法满足全天候、高覆盖的实时跟踪需求。通过结合光学与SAR数据，提出了一种更高效的解决方案。

Contribution: 1. 提出了HOSS ReID数据集，支持跨模态船舶重识别任务；2. 基于Vision Transformer设计了基准方法TransOSS，优化了跨模态特征提取。

Method: TransOSS方法改进了Vision Transformer的patch嵌入结构，引入额外嵌入以提供更多参考信息，并通过对比学习预训练提取模态不变特征。

Result: 所提出的数据集和方法为跨模态船舶重识别提供了新的基准，并公开了数据和代码。

Insight: 结合光学与SAR数据的跨模态方法能够解决全天候跟踪问题，Vision Transformer的改进为跨模态任务提供了新思路。

Abstract: Detecting and tracking ground objects using earth observation imagery remains a significant challenge in the field of remote sensing. Continuous maritime ship tracking is crucial for applications such as maritime search and rescue, law enforcement, and shipping analysis. However, most current ship tracking methods rely on geostationary satellites or video satellites. The former offer low resolution and are susceptible to weather conditions, while the latter have short filming durations and limited coverage areas, making them less suitable for the real-world requirements of ship tracking. To address these limitations, we present the Hybrid Optical and Synthetic Aperture Radar (SAR) Ship Re-Identification Dataset (HOSS ReID dataset), designed to evaluate the effectiveness of ship tracking using low-Earth orbit constellations of optical and SAR sensors. This approach ensures shorter re-imaging cycles and enables all-weather tracking. HOSS ReID dataset includes images of the same ship captured over extended periods under diverse conditions, using different satellites of different modalities at varying times and angles. Furthermore, we propose a baseline method for cross-modal ship re-identification, TransOSS, which is built on the Vision Transformer architecture. It refines the patch embedding structure to better accommodate cross-modal tasks, incorporates additional embeddings to introduce more reference information, and employs contrastive learning to pre-train on large-scale optical-SAR image pairs, ensuring the model’s ability to extract modality-invariant features. Our dataset and baseline method are publicly available on https://github.com/Alioth2000/Hoss-ReID.

[82] Partial CLIP is Enough: Chimera-Seg for Zero-shot Semantic Segmentation

Jialei Chen,Xu Zheng,Danda Pani Paudel,Luc Van Gool,Hiroshi Murase,Daisuke Deguchi

Main category: cs.CV

TL;DR: 该论文提出了Chimera-Seg方法，通过结合分割主干和CLIP语义头（CSH）解决了零样本语义分割中的视觉-语言对齐和局部特征与全局表示之间的语义鸿沟问题，提出了选择性全局蒸馏（SGD）和语义对齐模块（SAM），在两个基准测试中分别提升了0.9%和1.2%的hIoU。

Details

Motivation: 零样本语义分割（ZSS）的目标是利用已见类别的监督分割未见和已见类别。基于蒸馏的方法需要将视觉语言模型（如CLIP）的对齐能力迁移到分割模型中，但存在视觉特征与文本空间对齐困难以及局部特征与全局表示的语义鸿沟问题。

Contribution: 主要贡献包括：1) Chimera-Seg通过结合分割主干和CLIP语义头（CSH），保留了分割能力并简化了对CLIP语义空间的映射；2) 提出了选择性全局蒸馏（SGD）和语义对齐模块（SAM），提升了对齐效率和语义一致性。

Method: 方法包括：1) Chimera-Seg结构设计，结合分割主干和CSH；2) SGD从高相似度密集特征中蒸馏知识，并逐步减少对齐特征数量；3) SAM进一步对齐密集视觉特征和CLIP文本嵌入。

Result: 在两个基准测试中，hIoU分别提升了0.9%和1.2%，验证了方法的有效性。

Insight: 仅使用CLIP的部分模块（如固定子网络和投影层）结合轻量级可训练组件，可以在保留分割能力的同时实现高效的视觉-语言对齐。

Abstract: Zero-shot Semantic Segmentation (ZSS) aims to segment both seen and unseen classes using supervision from only seen classes. Beyond adaptation-based methods, distillation-based approaches transfer vision-language alignment of vision-language model, e.g., CLIP, to segmentation models. However, such knowledge transfer remains challenging due to: (1) the difficulty of aligning vision-based features with the textual space, which requires combining spatial precision with vision-language alignment; and (2) the semantic gap between CLIP’s global representations and the local, fine-grained features of segmentation models. To address challenge (1), we propose Chimera-Seg, which integrates a segmentation backbone as the body and a CLIP-based semantic head as the head, like the Chimera in Greek mythology, combining spatial precision with vision-language alignment. Specifically, Chimera-Seg comprises a trainable segmentation model and a CLIP Semantic Head (CSH), which maps dense features into the CLIP-aligned space. The CSH incorporates a frozen subnetwork and fixed projection layers from the CLIP visual encoder, along with lightweight trainable components. The partial module from CLIP visual encoder, paired with the segmentation model, retains segmentation capability while easing the mapping to CLIP’s semantic space. To address challenge (2), we propose Selective Global Distillation (SGD), which distills knowledge from dense features exhibiting high similarity to the CLIP CLS token, while gradually reducing the number of features used for alignment as training progresses. Besides, we also use a Semantic Alignment Module (SAM) to further align dense visual features with semantic embeddings extracted from the frozen CLIP text encoder. Experiments on two benchmarks show improvements of 0.9% and 1.2% in hIoU.

[83] Few-Shot Identity Adaptation for 3D Talking Heads via Global Gaussian Field

Hong Nie,Fuyuan Cao,Lu Chen,Fengxin Chen,Yuefeng Zou,Jun Yu

Main category: cs.CV

TL;DR: 该论文提出了一种名为FIAG的新框架，用于通过少量训练数据快速适应3D说话头合成的身份特征，利用全局高斯场和通用运动场实现高效的身份扩展和运动建模。

Details

Motivation: 现有基于重建和渲染的说话头合成方法需要为每个新身份从头训练，计算成本高且扩展性差。为了解决这一问题，作者提出了FIAG框架，实现高效的身份适应。

Contribution: 主要贡献包括：1) 提出了Global Gaussian Field，支持在一个共享场中表示多身份；2) 引入Universal Motion Field，捕捉跨身份的通用运动动态；3) 证明了少量数据即可快速适应新身份的能力。

Method: 方法依赖两个关键技术：1) Global Gaussian Field用于共享面部结构信息；2) Universal Motion Field用于学习通用运动先验，结合这两者实现快速身份适应。

Result: 实验表明，FIAG在多个基准上优于现有方法，验证了其有效性和泛化能力。

Insight: 通过共享结构和运动信息，可以显著减少新身份适应所需的数据和计算成本，为说话头合成提供了一种更高效的解决方案。

Abstract: Reconstruction and rendering-based talking head synthesis methods achieve high-quality results with strong identity preservation but are limited by their dependence on identity-specific models. Each new identity requires training from scratch, incurring high computational costs and reduced scalability compared to generative model-based approaches. To overcome this limitation, we propose FIAG, a novel 3D speaking head synthesis framework that enables efficient identity-specific adaptation using only a few training footage. FIAG incorporates Global Gaussian Field, which supports the representation of multiple identities within a shared field, and Universal Motion Field, which captures the common motion dynamics across diverse identities. Benefiting from the shared facial structure information encoded in the Global Gaussian Field and the general motion priors learned in the motion field, our framework enables rapid adaptation from canonical identity representations to specific ones with minimal data. Extensive comparative and ablation experiments demonstrate that our method outperforms existing state-of-the-art approaches, validating both the effectiveness and generalizability of the proposed framework. Code is available at: \textit{https://github.com/gme-hong/FIAG}.

[84] EnLVAM: Enhanced Left Ventricle Linear Measurements Utilizing Anatomical Motion Mode

Durgesh K. Singh,Ahcene Boubekki,Qing Cao,Svein Arne Aase,Robert Jenssen,Michael Kampffmeyer

Main category: cs.CV

TL;DR: 论文提出了一种增强左心室线性测量的新框架EnLVAM，通过强制直线约束和利用解剖M模式（AMM）图像，改进了B模式超声心动图中的测量精度。

Details

Motivation: 传统B模式超声心动图中手动标记左心室线性测量点耗时且易出错，现有深度学习方法常导致标记点对齐不准确。论文旨在通过强制直线约束和利用AMM图像解决这一问题。

Contribution: 1. 提出了一种结合解剖M模式（AMM）图像的增强框架EnLVAM。2. 通过半自动设计和人机交互步骤简化了临床操作。3. 实验结果验证了方法的准确性和泛化性。

Method: 1. 使用AMM图像训练标记点检测器；2. 将检测结果实时转换回B模式空间；3. 引入直线约束以纠正标记点对齐；4. 提供半自动设计，用户仅需放置虚拟扫描线。

Result: 实验表明，该方法显著提高了左心室线性测量的准确性，优于传统B模式方法，并在不同网络架构中表现良好。

Insight: 结合AMM图像和人机交互可以有效解决标记点对齐问题，同时保持临床灵活性；半自动设计简化了操作，提升了实用性和精度。

Abstract: Linear measurements of the left ventricle (LV) in the Parasternal Long Axis (PLAX) view using B-mode echocardiography are crucial for cardiac assessment. These involve placing 4-6 landmarks along a virtual scanline (SL) perpendicular to the LV axis near the mitral valve tips. Manual placement is time-consuming and error-prone, while existing deep learning methods often misalign landmarks, causing inaccurate measurements. We propose a novel framework that enhances LV measurement accuracy by enforcing straight-line constraints. A landmark detector is trained on Anatomical M-Mode (AMM) images, computed in real time from B-mode videos, then transformed back to B-mode space. This approach addresses misalignment and reduces measurement errors. Experiments show improved accuracy over standard B-mode methods, and the framework generalizes well across network architectures. Our semi-automatic design includes a human-in-the-loop step where the user only places the SL, simplifying interaction while preserving alignment flexibility and clinical relevance.

[85] MirrorMe: Towards Realtime and High Fidelity Audio-Driven Halfbody Animation

Dechao Meng,Steven Xiao,Xindi Zhang,Guangyuan Wang,Peng Zhang,Qi Wang,Bang Zhang,Liefeng Bo

Main category: cs.CV

TL;DR: MirrorMe 是一个基于 LTX 视频模型的实时、可控框架，通过高效的空间和时间压缩，解决了音频驱动肖像动画生成的高延迟和时间一致性问题。

Details

Motivation: 现有音频驱动肖像动画方法依赖逐帧 UNet 架构，导致高延迟和时间不一致，无法满足实时高保真生成的需求。

Contribution: 1. 提出了 MirrorMe 框架，通过 LTX 视频模型实现高效潜在空间去噪；2. 引入三种创新技术（身份注入机制、因果音频编码器和适配器、渐进训练策略）解决了压缩与语义保真之间的权衡问题。

Method: 1. 使用 VAE 编码图像拼接和自注意力实现身份一致性；2. 设计因果音频编码器适配 LTX 时间结构；3. 采用渐进训练策略（面部特写、半身合成与手部姿势整合）。

Result: 在 EMTD 基准测试中，MirrorMe 在保真度、唇音同步准确性和时间稳定性方面达到最先进水平。

Insight: 通过空间和时间压缩、身份注入和渐进训练，MirrorMe 有效平衡了实时性与生成质量，为音频驱动动画提供了新思路。

Abstract: Audio-driven portrait animation, which synthesizes realistic videos from reference images using audio signals, faces significant challenges in real-time generation of high-fidelity, temporally coherent animations. While recent diffusion-based methods improve generation quality by integrating audio into denoising processes, their reliance on frame-by-frame UNet architectures introduces prohibitive latency and struggles with temporal consistency. This paper introduces MirrorMe, a real-time, controllable framework built on the LTX video model, a diffusion transformer that compresses video spatially and temporally for efficient latent space denoising. To address LTX’s trade-offs between compression and semantic fidelity, we propose three innovations: 1. A reference identity injection mechanism via VAE-encoded image concatenation and self-attention, ensuring identity consistency; 2. A causal audio encoder and adapter tailored to LTX’s temporal structure, enabling precise audio-expression synchronization; and 3. A progressive training strategy combining close-up facial training, half-body synthesis with facial masking, and hand pose integration for enhanced gesture control. Extensive experiments on the EMTD Benchmark demonstrate MirrorMe’s state-of-the-art performance in fidelity, lip-sync accuracy, and temporal stability.

[86] Single-Scanline Relative Pose Estimation for Rolling Shutter Cameras

Petr Hruby,Marc Pollefeys

Main category: cs.CV

TL;DR: 提出了一种新方法，通过单扫描线投影线交点估计滚动快门相机间的相对位姿，无需显式建模相机运动，支持单视图相对位姿估计，并为滚动快门SfM提供了基础模块。

Details

Motivation: 滚动快门相机在动态场景中广泛使用，但其相对位姿估计通常需要复杂的运动模型。本文旨在提出一种无需运动模型的轻量级方法，简化位姿估计流程。

Contribution: 1. 提出基于单扫描线投影线交点的相对位姿估计方法；2. 支持单视图和多视图场景下的位姿估计；3. 分类并开发了多种最小解器，适用于并行线和已知重力方向的场景。

Method: 利用扫描线与投影线的交点计算相对位姿，无需相机运动模型。通过最小解器分析，覆盖通用和特定场景（如并行线或已知重力方向）。

Result: 在Fastec数据集上的实验验证了该方法在初始化滚动快门SfM中的可行性，展示了其实际应用潜力。

Insight: 通过单扫描线投影实现位姿估计，方法轻量且灵活，为滚动快门相机的SfM提供了一种新的基础模块。

Abstract: We propose a novel approach for estimating the relative pose between rolling shutter cameras using the intersections of line projections with a single scanline per image. This allows pose estimation without explicitly modeling camera motion. Alternatively, scanlines can be selected within a single image, enabling single-view relative pose estimation for scanlines of rolling shutter cameras. Our approach is designed as a foundational building block for rolling shutter structure-from-motion (SfM), where no motion model is required, and each scanline’s pose can be computed independently. % We classify minimal solvers for this problem in both generic and specialized settings, including cases with parallel lines and known gravity direction, assuming known intrinsics and no lens distortion. Furthermore, we develop minimal solvers for the parallel-lines scenario, both with and without gravity priors, by leveraging connections between this problem and the estimation of 2D structure from 1D cameras. % Experiments on rolling shutter images from the Fastec dataset demonstrate the feasibility of our approach for initializing rolling shutter SfM, highlighting its potential for further development. % The code will be made publicly available.

[87] Reasoning in machine vision: learning to think fast and slow

Shaheer U. Saeed,Yipei Wang,Veeru Kasivisvanathan,Brian R. Davidson,Matthew J. Clarkson,Yipeng Hu,Daniel C. Alexander

Main category: cs.CV

TL;DR: 论文提出了一种新学习范式，使机器能在视觉任务中通过增加推理时间提升性能，模仿人类快慢思考系统。

Details

Motivation: 现有机器智能依赖训练数据，缺乏动态推理能力。非语言推理任务（如视觉感知、医学诊断）仍是开放挑战。

Contribution: 提出了一种结合快思考（System I）和慢思考（System II）模块的推理框架，通过自玩强化学习在数据稀缺时优化性能。

Method: 快思考模块处理熟悉任务，慢思考模块通过自玩强化学习迭代优化解决方案。

Result: 在视觉任务和医学图像癌症定位中表现优于大规模监督学习、基础模型及人类专家。

Insight: 融合人类认知理论，展示了在非语言领域机器推理的潜力，为数据稀缺场景提供了新思路。

Abstract: Reasoning is a hallmark of human intelligence, enabling adaptive decision-making in complex and unfamiliar scenarios. In contrast, machine intelligence remains bound to training data, lacking the ability to dynamically refine solutions at inference time. While some recent advances have explored reasoning in machines, these efforts are largely limited to verbal domains such as mathematical problem-solving, where explicit rules govern step-by-step reasoning. Other critical real-world tasks - including visual perception, spatial reasoning, and radiological diagnosis - require non-verbal reasoning, which remains an open challenge. Here we present a novel learning paradigm that enables machine reasoning in vision by allowing performance improvement with increasing thinking time (inference-time compute), even under conditions where labelled data is very limited. Inspired by dual-process theories of human cognition in psychology, our approach integrates a fast-thinking System I module for familiar tasks, with a slow-thinking System II module that iteratively refines solutions using self-play reinforcement learning. This paradigm mimics human reasoning by proposing, competing over, and refining solutions in data-scarce scenarios. We demonstrate superior performance through extended thinking time, compared not only to large-scale supervised learning but also foundation models and even human experts, in real-world vision tasks. These tasks include computer-vision benchmarks and cancer localisation on medical images across five organs, showcasing transformative potential for non-verbal machine reasoning.

[88] Towards Accurate Heart Rate Measurement from Ultra-Short Video Clips via Periodicity-Guided rPPG Estimation and Signal Reconstruction

Pei-Kai Huanga,Ya-Ting Chan,Kuan-Wen Chen,Yen-Chun Chou,Shih-Yu Yang,Chiou-Ting Hsu

Main category: cs.CV

TL;DR: 该论文提出了一种通过周期性引导的rPPG估计和信号重建方法，从2秒超短视频片段中准确测量心率。

Details

Motivation: 以往远程心率测量方法多关注10秒左右的视频片段，忽视了超短视频片段的心率测量需求。论文针对超短视频片段中心跳周期少和频谱泄漏问题，提出新方法。

Contribution: 论文的主要贡献包括：(1)提出周期性引导的rPPG估计方法，确保超短视频与真实信号的周期性一致；(2)引入信号生成器，重建更长rPPG信号以减少频谱泄漏影响。

Method: 方法分为两步：(1)通过周期性约束估计超短视频的rPPG信号；(2)利用生成器重建更长信号，提升心率测量精度。

Result: 在四个rPPG基准数据集上的实验表明，该方法不仅能准确测量超短视频的心率，还超越了现有技术的性能。

Insight: 论文表明，通过周期性约束和信号重建，可以有效解决超短视频中心率测量的挑战，为未来远程健康监测提供了新思路。

Abstract: Many remote Heart Rate (HR) measurement methods focus on estimating remote photoplethysmography (rPPG) signals from video clips lasting around 10 seconds but often overlook the need for HR estimation from ultra-short video clips. In this paper, we aim to accurately measure HR from ultra-short 2-second video clips by specifically addressing two key challenges. First, to overcome the limited number of heartbeat cycles in ultra-short video clips, we propose an effective periodicity-guided rPPG estimation method that enforces consistent periodicity between rPPG signals estimated from ultra-short clips and their much longer ground truth signals. Next, to mitigate estimation inaccuracies due to spectral leakage, we propose including a generator to reconstruct longer rPPG signals from ultra-short ones while preserving their periodic consistency to enable more accurate HR measurement. Extensive experiments on four rPPG estimation benchmark datasets demonstrate that our proposed method not only accurately measures HR from ultra-short video clips but also outperform previous rPPG estimation techniques to achieve state-of-the-art performance.

[89] BézierGS: Dynamic Urban Scene Reconstruction with Bézier Curve Gaussian Splatting

Zipei Ma,Junzhe Jiang,Yurui Chen,Li Zhang

Main category: cs.CV

TL;DR: 该论文提出了BézierGS方法，通过Bézier曲线表示动态对象的运动轨迹，无需依赖高精度物体标注，实现了动态场景的高精度重建。

Details

Motivation: 现有动态场景重建方法依赖物体姿态标注，限制了大规模场景重建的扩展性。研究旨在提出一种无需标注的方法，更高效地重建动态城市街景。

Contribution: 提出BézierGS，一种利用可学习Bézier曲线建模动态对象运动轨迹的方法，自动纠正姿态误差，并引入动态对象渲染和曲线一致性约束，实现场景元素的合理分离与重建。

Method: 基于Bézier曲线对动态对象运动轨迹建模，结合时间信息，通过可学习曲线自动校正姿态误差，并引入额外的动态对象渲染监督和曲线间一致性约束。

Result: 在Waymo Open Dataset和nuPlan基准测试中，BézierGS在动态和静态场景重建及新视角合成任务上优于现有方法。

Insight: Bézier曲线的引入不仅降低了对外部标注的依赖，还能通过时间信息优化运动轨迹，为动态场景重建提供了新思路。

Abstract: The realistic reconstruction of street scenes is critical for developing real-world simulators in autonomous driving. Most existing methods rely on object pose annotations, using these poses to reconstruct dynamic objects and move them during the rendering process. This dependence on high-precision object annotations limits large-scale and extensive scene reconstruction. To address this challenge, we propose B'ezier curve Gaussian splatting (B'ezierGS), which represents the motion trajectories of dynamic objects using learnable B'ezier curves. This approach fully leverages the temporal information of dynamic objects and, through learnable curve modeling, automatically corrects pose errors. By introducing additional supervision on dynamic object rendering and inter-curve consistency constraints, we achieve reasonable and accurate separation and reconstruction of scene elements. Extensive experiments on the Waymo Open Dataset and the nuPlan benchmark demonstrate that B'ezierGS outperforms state-of-the-art alternatives in both dynamic and static scene components reconstruction and novel view synthesis.

[90] Tied Prototype Model for Few-Shot Medical Image Segmentation

Hyeongji Kim,Stine Hansen,Michael Kampffmeyer

Main category: cs.CV

TL;DR: 本文提出了Tied Prototype Model (TPM)，用于改进医学图像小样本分割，通过绑定前景和背景的原型分布，解决了现有方法依赖单一原型、固定阈值等问题，并支持多原型和多类分割。

Details

Motivation: 现有原型方法ADNet在医学图像分割中存在局限性，如依赖单一原型、仅支持二分类、固定阈值无法适应不同患者和器官的变异性。TPM旨在解决这些问题。

Contribution: 1. 提出TPM模型，绑定前景和背景原型分布；2. 支持多原型和多类分割；3. 提出自适应阈值优化分割性能。

Method: 1. 绑定前景和背景的原型分布；2. 扩展为多原型和多类分割；3. 利用类别先验定义自适应阈值。

Result: TPM显著提升了分割精度，尤其在处理非典型背景特征时表现更好。

Insight: TPM为基于原型的医学图像小样本分割提供了新视角，通过绑定分布和多原型设计，提升了模型的适应性和性能。

Abstract: Common prototype-based medical image few-shot segmentation (FSS) methods model foreground and background classes using class-specific prototypes. However, given the high variability of the background, a more promising direction is to focus solely on foreground modeling, treating the background as an anomaly – an approach introduced by ADNet. Yet, ADNet faces three key limitations: dependence on a single prototype per class, a focus on binary classification, and fixed thresholds that fail to adapt to patient and organ variability. To address these shortcomings, we propose the Tied Prototype Model (TPM), a principled reformulation of ADNet with tied prototype locations for foreground and background distributions. Building on its probabilistic foundation, TPM naturally extends to multiple prototypes and multi-class segmentation while effectively separating non-typical background features. Notably, both extensions lead to improved segmentation accuracy. Finally, we leverage naturally occurring class priors to define an ideal target for adaptive thresholds, boosting segmentation performance. Taken together, TPM provides a fresh perspective on prototype-based FSS for medical image segmentation. The code can be found at https://github.com/hjk92g/TPM-FSS.

[91] Pedestrian Intention and Trajectory Prediction in Unstructured Traffic Using IDD-PeD

Ruthvik Bokkasam,Shankar Gangisetty,A. H. Abdul Hafez,C. V. Jawahar

Main category: cs.CV

TL;DR: 该论文提出了一个针对非结构化交通环境的印度驾驶行人数据集（IDD-PeD），用于提升行人意图和轨迹预测的准确性。研究发现现有方法在该数据集上性能显著下降，突显了非结构化环境的挑战。

Details

Motivation: 随着自动驾驶技术的发展，准确预测行人在复杂和非结构化交通环境中的行为对安全性至关重要。目前缺乏能够捕捉此类环境的全面数据集，因此需要新的数据支持模型开发。

Contribution: 1. 引入IDD-PeD数据集，专注于非结构化交通环境中的行人行为；
2. 显示现有预测方法在该数据集上的性能显著下降；
3. 提供详尽的定性和定量分析。

Method: 1. 通过收集和标注印度非结构化交通场景中的行人数据构建数据集；
2. 评估现有意图预测和轨迹预测方法在新数据集上的表现，并使用MSE等指标量化性能差异。

Result: 实验结果表明，现有意图预测方法的性能下降高达15%，轨迹预测方法的MSE增加高达1208，表明非结构化环境的挑战性。

Insight: 非结构化交通环境（如照明变化、遮挡等）显著增加了行人行为预测的难度，现有方法需要进一步改进以适应此类场景。

Abstract: With the rapid advancements in autonomous driving, accurately predicting pedestrian behavior has become essential for ensuring safety in complex and unpredictable traffic conditions. The growing interest in this challenge highlights the need for comprehensive datasets that capture unstructured environments, enabling the development of more robust prediction models to enhance pedestrian safety and vehicle navigation. In this paper, we introduce an Indian driving pedestrian dataset designed to address the complexities of modeling pedestrian behavior in unstructured environments, such as illumination changes, occlusion of pedestrians, unsignalized scene types and vehicle-pedestrian interactions. The dataset provides high-level and detailed low-level comprehensive annotations focused on pedestrians requiring the ego-vehicle’s attention. Evaluation of the state-of-the-art intention prediction methods on our dataset shows a significant performance drop of up to $\mathbf{15%}$, while trajectory prediction methods underperform with an increase of up to $\mathbf{1208}$ MSE, defeating standard pedestrian datasets. Additionally, we present exhaustive quantitative and qualitative analysis of intention and trajectory baselines. We believe that our dataset will open new challenges for the pedestrian behavior research community to build robust models. Project Page: https://cvit.iiit.ac.in/research/projects/cvit-projects/iddped

[92] Pipe Reconstruction from Point Cloud Data

Antje Alex,Jannis Stoppe

Main category: cs.CV

TL;DR: 提出了一种从点云数据自动重建管道的流程，包括骨架曲线估计、曲线延伸、滚动球技术和3D平滑，支持快速准确的数字孪生模型构建。

Details

Motivation: 工业资产（如船舶和海上平台）的数字孪生需要精确的管道网络重建，但手动建模耗时耗力，亟需自动化解决方案。

Contribution: 开发了一套完整的自动化管道重建流程，显著提升了点云数据处理的效率和准确性。

Method: 基于拉普拉斯收缩估计骨架曲线，通过曲线延伸和滚动球技术优化中心轴，最后进行3D平滑以确定管道属性。

Result: 方法能够从非完整扫描数据中重建管道，精确提取半径、长度和方向等属性。

Insight: 自动化管道重建支持数字孪生的快速开发，降低建模成本，适用于复杂工业场景。

Abstract: Accurate digital twins of industrial assets, such as ships and offshore platforms, rely on the precise reconstruction of complex pipe networks. However, manual modelling of pipes from laser scan data is a time-consuming and labor-intensive process. This paper presents a pipeline for automated pipe reconstruction from incomplete laser scan data. The approach estimates a skeleton curve using Laplacian-based contraction, followed by curve elongation. The skeleton axis is then recentred using a rolling sphere technique combined with 2D circle fitting, and refined with a 3D smoothing step. This enables the determination of pipe properties, including radius, length and orientation, and facilitates the creation of detailed 3D models of complex pipe networks. By automating pipe reconstruction, this approach supports the development of digital twins, allowing for rapid and accurate modeling while reducing costs.

[93] Low-Rank Implicit Neural Representation via Schatten-p Quasi-Norm and Jacobian Regularization

Zhengyun Cheng,Changhao Wang,Guanwen Zhang,Yi Xu,Wei Zhou,Xiangyang Ji

Main category: cs.CV

TL;DR: 该论文提出了一种基于CP分解的低秩张量函数（CP-INR），结合Schatten-p拟范数和雅可比谱范数正则化，用于隐式神经表示，解决了现有方法的稀疏性和平滑性问题，并在多维数据恢复任务中表现优异。

Details

Motivation: 现有低秩张量表示方法（如Tucker和CP分解）在灵活性和可解释性之间存在权衡，且稀疏解难以获得。论文旨在通过结合CP分解和神经网络的隐式表示能力，实现连续数据的低秩表示。

Contribution: 1) 提出CP-INR方法，将CP分解与神经网络结合；2) 引入Schatten-p拟范数实现稀疏低秩解；3) 提出基于雅可比谱范数的新正则化方法，避免SVD计算。

Method: 1) 使用CP分解参数化神经网络的隐式表示；2) 提出Schatten-p拟范数的变分形式用于稀疏性；3) 设计基于雅可比谱范数的正则化项保证平滑性。

Result: 在图像修复、去噪和点云上采样任务中表现优于现有方法，验证了方法的有效性和普适性。

Insight: 通过结合张量分解的显式结构与神经网络的隐式能力，能够更灵活地处理多维数据，同时提供理论保证。

Abstract: Higher-order tensors are well-suited for representing multi-dimensional data, such as color images and videos. Low-rank tensor representation has become essential in machine learning and computer vision, but existing methods like Tucker decomposition offer flexibility at the expense of interpretability. In contrast, while the CANDECOMP/PARAFAC (CP) decomposition provides a more natural and interpretable tensor structure, obtaining sparse solutions remains challenging. Leveraging the rich properties of CP decomposition, we propose a CP-based low-rank tensor function parameterized by neural networks for implicit neural representation (CP-INR). This approach enables continuous data representation beyond structured grids, fully exploiting the non-linearity of tensor data with theoretical guarantees on excess risk bounds. To achieve a sparse CP decomposition, we introduce a variational form of the Schatten-p quasi-norm and prove its relationship to multilinear rank minimization. For smoothness, we propose a regularization term based on the spectral norm of the Jacobian and Hutchinson’s trace estimator. Our proposed smoothness regularization is SVD-free and avoids explicit chain rule derivations. It can serve as an alternative to Total Variation (TV) regularization in image denoising tasks and is naturally applicable to continuous data. Extensive experiments on multi-dimensional data recovery tasks, including image inpainting, denoising, and point cloud upsampling, demonstrate the superiority and versatility of our method compared to state-of-the-art approaches.

[94] Q-Frame: Query-aware Frame Selection and Multi-Resolution Adaptation for Video-LLMs

Shaojie Zhang,Jiahui Yang,Jianqin Yin,Zhenbo Luo,Jian Luan

Main category: cs.CV

TL;DR: 论文提出了Q-Frame，一种针对视频内容与查询的自适应帧选择与多分辨率调整方法，解决了现有Video-LLMs在处理视频理解任务时的效率与准确性问题。

Details

Motivation: 视频理解任务中，现有Video-LLMs由于采用均匀帧采样，难以有效捕捉与查询相关的关键时空线索，亟需一种更高效的帧选择方法。

Contribution: 提出了Q-Frame，一种无需训练的即插即用方法，通过CLIP等文本-图像匹配网络实现自适应帧选择与多分辨率调整，提升了视频理解性能。

Method: 基于CLIP网络和Gumbel-Max技巧，Q-Frame实现了高效的帧选择与多分辨率调整，确保在有限计算资源下处理更多帧。

Result: 在MLVU、LongVideoBench和Video-MME等基准数据集上表现出色，优于现有方法。

Insight: Q-Frame展示了通过动态调整帧选择与分辨率，可以显著提升Video-LLMs的视频理解能力，同时保持计算效率。

Abstract: Multimodal Large Language Models (MLLMs) have demonstrated significant success in visual understanding tasks. However, challenges persist in adapting these models for video comprehension due to the large volume of data and temporal complexity. Existing Video-LLMs using uniform frame sampling often struggle to capture the query-related crucial spatiotemporal clues of videos effectively. In this paper, we introduce Q-Frame, a novel approach for adaptive frame selection and multi-resolution scaling tailored to the video’s content and the specific query. Q-Frame employs a training-free, plug-and-play strategy generated by a text-image matching network like CLIP, utilizing the Gumbel-Max trick for efficient frame selection. Q-Frame allows Video-LLMs to process more frames without exceeding computational limits, thereby preserving critical temporal and spatial information. We demonstrate Q-Frame’s effectiveness through extensive experiments on benchmark datasets, including MLVU, LongVideoBench, and Video-MME, illustrating its superiority over existing methods and its applicability across various video understanding tasks.

[95] Visual Structures Helps Visual Reasoning: Addressing the Binding Problem in VLMs

Amirmohammad Izadi,Mohammad Ali Banayeeanzade,Fatemeh Askari,Ali Rahimiakbar,Mohammad Mahdi Vahedi,Hosein Hasani,Mahdieh Soleymani Baghshah

Main category: cs.CV

TL;DR: 论文通过引入低层次空间结构（如水平线）增强视觉输入，并结合文本提示来改善视觉-语言模型（VLM）的绑定问题，显著提升了在计数、视觉搜索等任务中的性能。

Details

Motivation: 当前视觉-语言模型（VLM）在视觉推理中的绑定问题（即无法可靠地将感知特征与正确视觉对象关联）导致其在计数、空间关系理解等任务中表现不佳。这一问题的核心在于缺乏空间基础和序列注意力机制。

Contribution: 1. 提出了一种简单而有效的方法：通过增强视觉输入的低层次空间结构和文本提示，显著改善了VLM在视觉推理任务中的表现。2. 实验表明，该方法在视觉搜索、计数、场景描述和空间关系任务中均有显著提升。

Method: 在视觉输入中引入低层次空间结构（如水平线），并结合文本提示以鼓励序列化和空间感知的解析。该方法无需多次推理即可显著提升性能。

Result: 在2D合成数据集上，GPT-4o的视觉搜索准确率提升了25.00%，计数准确率提升了26.83%，场景描述的编辑距离误差减少了0.32，空间关系任务的性能提升了9.50%。

Insight: 视觉输入的设计比纯语言方法（如思维链提示）对解决绑定问题更为关键，低层次视觉结构化是提升组合视觉推理能力的有效方向。

Abstract: Despite progress in Vision-Language Models (VLMs), their capacity for visual reasoning is often limited by the \textit{binding problem}: the failure to reliably associate perceptual features with their correct visual referents. This limitation underlies persistent errors in tasks such as counting, visual search, scene description, and spatial relationship understanding. A key factor is that current VLMs process visual features largely in parallel, lacking mechanisms for spatially grounded, serial attention. This paper introduces a simple yet effective intervention: augmenting visual inputs with low-level spatial structures (e.g., horizontal lines) and pairing this with a textual prompt that encourages sequential, spatially-aware parsing. We empirically demonstrate substantial performance improvements across core visual reasoning tasks. Specifically, our method improves GPT-4o visual search accuracy by 25.00%, increases counting accuracy by 26.83%, reduces edit distance error in scene description by 0.32, and enhances performance on spatial relationship tasks by 9.50% on a a 2D synthetic dataset. Furthermore, we find that the visual modification is essential for these gains; purely textual strategies, including Chain-of-Thought prompting, are insufficient and can even degrade performance. Our method enhances binding only with a single-query inference, underscoring the importance of visual input design over purely linguistically-based approaches. These findings suggest that low-level visual structuring is a powerful and underexplored direction for improving compositional visual reasoning and could serve as a general strategy for enhancing VLM performance on spatially grounded tasks.

[96] RetFiner: A Vision-Language Refinement Scheme for Retinal Foundation Models

Ronald Fecso,José Morano,Ursula Schmidt-Erfurth,Hrvoje Bogunović

Main category: cs.CV

TL;DR: 该论文提出了一种名为RetFiner的自监督学习视觉语言细化方案，旨在通过利用文本数据中的监督信号来改进现有的视网膜基础模型，从而提升其在多种下游任务中的性能。

Details

Motivation: 现有的视网膜基础模型仅依赖图像数据进行训练，缺乏对图像的全面和鲁棒的语义理解，导致其在下游任务（尤其是复杂任务）中表现不佳，且需要监督微调以适应特定应用和人群。

Contribution: 1. 提出RetFiner，一种自监督学习视觉语言细化方案，优化现有基础模型的表示能力。2. 通过多样化的训练目标，利用文本数据的丰富监督信号，实现高效且直接的模型适应性改进。

Method: RetFiner采用多种训练目标，充分利用文本数据中的监督信号，对现有视网膜基础模型（如RETFound、UrFound和VisionFM）进行细化。通过实验验证，该方法显著提升了模型在七种不同的OCT分类任务中的线性探测性能。

Result: 在RETFound、UrFound和VisionFM上，RetFiner分别带来5.8、3.9和2.1个百分点的平均性能提升。

Insight: 结合视觉与语言信号可以显著提升基础模型的语义理解能力，而无需依赖昂贵的标注数据或监督微调。

Abstract: The rise of imaging techniques such as optical coherence tomography (OCT) and advances in deep learning (DL) have enabled clinicians and researchers to streamline retinal disease staging. A popular DL approach is self-supervised learning (SSL), where models learn from vast amounts of unlabeled data, avoiding costly annotation. SSL has allowed the development of foundation models (FMs), large models that can be used for a variety of downstream tasks. However, existing FMs for OCT, trained solely on image data, lack a comprehensive and robust semantic understanding of images, as evidenced by their downstream performance (especially for complex tasks), and thus require supervised fine-tuning (which may be unfeasible) to better adapt to specific applications and populations. To address this, we propose RetFiner, an SSL vision-language refinement scheme that improves the representations of existing FMs and enables their efficient and direct adaptation to specific populations for improved downstream performance. Our method uses a diverse set of training objectives which take advantage of the rich supervisory signal found in textual data. We tested RetFiner on the retinal FMs RETFound, UrFound, and VisionFM, showing significant improvements in linear probing performance on seven highly diverse OCT classification tasks, with an average increase of 5.8, 3.9, and 2.1 percentage points over their baselines, respectively. Our code and model weights are publicly available at https://github.com/ronnief1/RetFiner.

[97] Attention-disentangled Uniform Orthogonal Feature Space Optimization for Few-shot Object Detection

Taijin Zhao,Heqian Qiu,Yu Dai,Lanxiao Wang,Fanman Meng,Qingbo Wu,Hongliang Li

Main category: cs.CV

TL;DR: 该论文提出了一种用于少样本目标检测的统一正交特征空间（UOFS）优化框架，通过解耦特征空间中的目标性和分类任务，解决了现有方法中类特定目标性标准和样本不具代表性的问题。

Details

Motivation: 现有的少样本目标检测方法通常在共享特征空间中耦合目标性识别和前景分类任务，导致类特定的目标性标准和新类样本的不具代表性。

Contribution: 1. 提出UOFS框架，将特征空间解耦为幅度（目标性）和角度（分类）两个正交分量；2. 设计了混合背景优化（HBO）策略和空间注意力解耦与关联（SADA）模块，解决基类中未标记前景实例和新类样本分布不均的问题。

Method: 1. 特征空间解耦为幅度和角度；2. HBO策略优化背景和前景分布；3. SADA模块处理类无关和类特定任务的冲突。

Result: 实验表明，该方法显著优于基于耦合特征空间的现有方法。

Insight: 正交特征空间解耦和注意力机制的结合能够有效提升少样本目标检测的性能，特别是在处理新类样本时。

Abstract: Few-shot object detection (FSOD) aims to detect objects with limited samples for novel classes, while relying on abundant data for base classes. Existing FSOD approaches, predominantly built on the Faster R-CNN detector, entangle objectness recognition and foreground classification within shared feature spaces. This paradigm inherently establishes class-specific objectness criteria and suffers from unrepresentative novel class samples. To resolve this limitation, we propose a Uniform Orthogonal Feature Space (UOFS) optimization framework. First, UOFS decouples the feature space into two orthogonal components, where magnitude encodes objectness and angle encodes classification. This decoupling enables transferring class-agnostic objectness knowledge from base classes to novel classes. Moreover, implementing the disentanglement requires careful attention to two challenges: (1) Base set images contain unlabeled foreground instances, causing confusion between potential novel class instances and backgrounds. (2) Angular optimization depends exclusively on base class foreground instances, inducing overfitting of angular distributions to base classes. To address these challenges, we propose a Hybrid Background Optimization (HBO) strategy: (1) Constructing a pure background base set by removing unlabeled instances in original images to provide unbiased magnitude-based objectness supervision. (2) Incorporating unlabeled foreground instances in the original base set into angular optimization to enhance distribution uniformity. Additionally, we propose a Spatial-wise Attention Disentanglement and Association (SADA) module to address task conflicts between class-agnostic and class-specific tasks. Experiments demonstrate that our method significantly outperforms existing approaches based on entangled feature spaces.

[98] Frequency-Semantic Enhanced Variational Autoencoder for Zero-Shot Skeleton-based Action Recognition

Wenhan Wu,Zhishuai Guo,Chen Chen,Hongfei Xue,Aidong Lu

Main category: cs.CV

TL;DR: 提出了一种基于频率-语义增强的变分自编码器（FS-VAE），用于零样本骨架动作识别，通过频率分解和语义对齐提升了模型的识别能力。

Details

Motivation: 现有方法在零样本骨架动作识别中主要关注视觉和语义的对齐，但忽略了语义空间中细粒度动作模式的重要性，导致识别效果受限。

Contribution: 1) 提出了频率增强模块，通过高低频调整丰富骨架语义学习；2) 设计了多级对齐的语义描述，捕捉局部和全局特征；3) 引入校准的跨对齐损失，解决特征模糊性问题。

Method: FS-VAE结合了频率分解、多级语义对齐和校准损失，增强语义特征学习并提升对齐效果。

Result: 在基准测试中验证了模型的优越性，频率增强的语义特征能有效区分视觉和语义相似的动作簇。

Insight: 频率分解和语义多级对齐是解决零样本骨架动作识别中细粒度差异的关键，校准损失能显著提升特征对齐的鲁棒性。

Abstract: Zero-shot skeleton-based action recognition aims to develop models capable of identifying actions beyond the categories encountered during training. Previous approaches have primarily focused on aligning visual and semantic representations but often overlooked the importance of fine-grained action patterns in the semantic space (e.g., the hand movements in drinking water and brushing teeth). To address these limitations, we propose a Frequency-Semantic Enhanced Variational Autoencoder (FS-VAE) to explore the skeleton semantic representation learning with frequency decomposition. FS-VAE consists of three key components: 1) a frequency-based enhancement module with high- and low-frequency adjustments to enrich the skeletal semantics learning and improve the robustness of zero-shot action recognition; 2) a semantic-based action description with multilevel alignment to capture both local details and global correspondence, effectively bridging the semantic gap and compensating for the inherent loss of information in skeleton sequences; 3) a calibrated cross-alignment loss that enables valid skeleton-text pairs to counterbalance ambiguous ones, mitigating discrepancies and ambiguities in skeleton and text features, thereby ensuring robust alignment. Evaluations on the benchmarks demonstrate the effectiveness of our approach, validating that frequency-enhanced semantic features enable robust differentiation of visually and semantically similar action clusters, improving zero-shot action recognition.

[99] Robust and Accurate Multi-view 2D/3D Image Registration with Differentiable X-ray Rendering and Dual Cross-view Constraints

Yuxin Cui,Rui Song,Yibin Li,Max Q. -H. Meng,Zhe Min

Main category: cs.CV

TL;DR: 该论文提出了一种新颖的多视图2D/3D刚性配准方法，通过设计联合损失函数和引入跨视图约束，显著提高了配准的鲁棒性和准确性。

Details

Motivation: 单视图术中图像视野有限，影响了配准的准确性，因此需要利用多视图图像以增强配准的鲁棒性。

Contribution: 提出了一个两阶段多视图2D/3D配准框架，设计了联合损失函数并引入跨视图约束，提升了配准性能。

Method: 第一阶段结合位姿和图像差异的联合损失函数，并引入跨视图训练损失项；第二阶段通过测试时优化细化位姿。

Result: 在DeepFluoro数据集上实现了0.79±2.17 mm的平均目标配准误差（mTRE），优于现有方法。

Insight: 多视图投影位姿的相互约束可以显著提升配准的鲁棒性，测试时优化能进一步细化位姿估计。

Abstract: Robust and accurate 2D/3D registration, which aligns preoperative models with intraoperative images of the same anatomy, is crucial for successful interventional navigation. To mitigate the challenge of a limited field of view in single-image intraoperative scenarios, multi-view 2D/3D registration is required by leveraging multiple intraoperative images. In this paper, we propose a novel multi-view 2D/3D rigid registration approach comprising two stages. In the first stage, a combined loss function is designed, incorporating both the differences between predicted and ground-truth poses and the dissimilarities (e.g., normalized cross-correlation) between simulated and observed intraoperative images. More importantly, additional cross-view training loss terms are introduced for both pose and image losses to explicitly enforce cross-view constraints. In the second stage, test-time optimization is performed to refine the estimated poses from the coarse stage. Our method exploits the mutual constraints of multi-view projection poses to enhance the robustness of the registration process. The proposed framework achieves a mean target registration error (mTRE) of $0.79 \pm 2.17$ mm on six specimens from the DeepFluoro dataset, demonstrating superior performance compared to state-of-the-art registration algorithms.

[100] ReF-LLE: Personalized Low-Light Enhancement via Reference-Guided Deep Reinforcement Learning

Ming Zhao,Pingping Liu,Tongshun Zhang,Zhe Zhang

Main category: cs.CV

TL;DR: ReF-LLE是一种基于参考图像和强化学习的个性化低光增强方法，首次将强化学习引入傅里叶频域，通过零参考评分策略和自适应迭代实现个性化效果。

Details

Motivation: 低光图像增强面临两个主要挑战：图像条件差异大且增强结果依赖主观偏好。现有方法难以同时解决这两个问题。

Contribution: 提出首个融合傅里叶频域和强化学习的个性化低光增强方法ReF-LLE，引入零参考评分和自适应迭代策略，实现用户导向的个性化增强。

Method: 在傅里叶频域中结合深度强化学习，训练时使用零参考评分生成奖励信号，推理时通过傅里叶零频分量自适应调整图像光照分布。

Result: 在基准数据集上性能优于现有方法，展现更高的感知质量和个性化适应能力。

Insight: 傅里叶域与强化学习的结合为低光增强提供了新思路，零频分量可作为光照分布的强有效指标。

Abstract: Low-light image enhancement presents two primary challenges: 1) Significant variations in low-light images across different conditions, and 2) Enhancement levels influenced by subjective preferences and user intent. To address these issues, we propose ReF-LLE, a novel personalized low-light image enhancement method that operates in the Fourier frequency domain and incorporates deep reinforcement learning. ReF-LLE is the first to integrate deep reinforcement learning into this domain. During training, a zero-reference image evaluation strategy is introduced to score enhanced images, providing reward signals that guide the model to handle varying degrees of low-light conditions effectively. In the inference phase, ReF-LLE employs a personalized adaptive iterative strategy, guided by the zero-frequency component in the Fourier domain, which represents the overall illumination level. This strategy enables the model to adaptively adjust low-light images to align with the illumination distribution of a user-provided reference image, ensuring personalized enhancement results. Extensive experiments on benchmark datasets demonstrate that ReF-LLE outperforms state-of-the-art methods, achieving superior perceptual quality and adaptability in personalized low-light image enhancement.

[101] Boosting Classification with Quantum-Inspired Augmentations

Matthias Tschöpe,Vitor Fortes Rey,Sogo Pierre Sanon,Paul Lukowicz,Nikolaos Palaiodimopoulos,Maximilian Kiefer-Emmanouilidis

Main category: cs.CV

TL;DR: 该论文研究了量子启发的数据增强技术，通过模拟量子扰动（如随机Bloch球旋转）提升图像分类性能，在经典机器学习中取得了显著的准确性提升。

Details

Motivation: 量子计算中的小扰动通常被视为不利因素，但作者发现这些扰动可以作为数据增强的来源，从而提升机器学习性能。此外，这些扰动可以在经典硬件上高效模拟，为经典方法提供量子启发的改进途径。

Contribution: 1. 提出一种简单有效的量子启发数据增强方法（随机Bloch球旋转）。2. 在ImageNet数据集上验证了该方法优于传统增强方法，显著提升了分类准确性。3. 探讨了强酉变换的潜在隐私应用及其局限性。

Method: 采用随机Bloch球旋转（SU(2)变换）作为量子启发的数据增强技术，直接应用于经典数据。通过与传统的翻转、旋转、裁剪等增强方法对比，验证其效果。

Result: 在ImageNet上，该方法使Top-1准确率提升3%，Top-5准确率提升2.5%，F1分数从8%增至12%。但强酉变换未显示出差分隐私的潜力。

Insight: 量子扰动可作为增强数据多样性的有效工具，无需复杂量子硬件即可实现性能提升；然而，其隐私应用仍需进一步研究。

Abstract: Understanding the impact of small quantum gate perturbations, which are common in quantum digital devices but absent in classical computers, is crucial for identifying potential advantages in quantum machine learning. While these perturbations are typically seen as detrimental to quantum computation, they can actually enhance performance by serving as a natural source of data augmentation. Additionally, they can often be efficiently simulated on classical hardware, enabling quantum-inspired approaches to improve classical machine learning methods. In this paper, we investigate random Bloch sphere rotations, which are fundamental SU(2) transformations, as a simple yet effective quantum-inspired data augmentation technique. Unlike conventional augmentations such as flipping, rotating, or cropping, quantum transformations lack intuitive spatial interpretations, making their application to tasks like image classification less straightforward. While common quantum augmentation methods rely on applying quantum models or trainable quanvolutional layers to classical datasets, we focus on the direct application of small-angle Bloch rotations and their effect on classical data. Using the large-scale ImageNet dataset, we demonstrate that our quantum-inspired augmentation method improves image classification performance, increasing Top-1 accuracy by 3%, Top-5 accuracy by 2.5%, and the F$_1$ score from 8% to 12% compared to standard classical augmentation methods. Finally, we examine the use of stronger unitary augmentations. Although these transformations preserve information in principle, they result in visually unrecognizable images with potential applications for privacy computations. However, we show that our augmentation approach and simple SU(2) transformations do not enhance differential privacy and discuss the implications of this limitation.

[102] 4D-VLA: Spatiotemporal Vision-Language-Action Pretraining with Cross-Scene Calibration

Jiahui Zhang,Yurui Chen,Yueming Xu,Ze Huang,Yanpeng Zhou,Yu-Jie Yuan,Xinyue Cai,Guowei Huang,Xingyue Quan,Hang Xu,Li Zhang

Main category: cs.CV

TL;DR: 4D-VLA通过在输入中整合4D信息（深度和时序信息），解决了机器人数据预训练中的坐标系混乱和状态混乱问题，同时引入了内存银行采样策略，显著提升了模型的时空推理能力和训练效率。

Details

Motivation: 现有方法通常使用简单观察作为输入，导致条件动作分布分散（称为坐标系混乱和状态混乱），严重影响了预训练效率。为解决这一问题，研究者提出了4D-VLA。

Contribution: 1. 提出4D-VLA模型，通过整合深度和时序信息，解决坐标系混乱问题；2. 引入内存银行采样策略，优化帧采样；3. 提出MV-Bench基准测试，评估空间感知和新视角泛化能力。

Method: 1. 利用连续的RGB-D输入引入深度和时序信息，对齐机器人与场景的坐标系；2. 采用内存银行采样策略提取信息丰富的帧；3. 设计了MV-Bench多视角仿真基准。

Result: 模型在仿真和真实实验中均显著优于OpenVLA，成功率和空间理解能力显著提升。MV-Bench上的测试结果也表明模型具有更强的适应性和空间感知能力。

Insight: 整合4D信息和有效的帧采样策略可以显著提升机器人预训练模型的时空推理能力和训练效率，同时多视角基准测试有助于评估模型的泛化能力。

Abstract: Leveraging diverse robotic data for pretraining remains a critical challenge. Existing methods typically model the dataset’s action distribution using simple observations as inputs. However, these inputs are often incomplete, resulting in a dispersed conditional action distribution-an issue we refer to as coordinate system chaos and state chaos. This inconsistency significantly hampers pretraining efficiency. To address this, we propose 4D-VLA, a novel approach that effectively integrates 4D information into the input to mitigate these sources of chaos. Our model introduces depth and temporal information into visual features with sequential RGB-D inputs, aligning the coordinate systems of the robot and the scene. This alignment endows the model with strong spatiotemporal reasoning capabilities while minimizing training overhead. Additionally, we introduce memory bank sampling, a frame sampling strategy designed to extract informative frames from historical images, further improving effectiveness and efficiency. Experimental results demonstrate that our pretraining method and architectural components substantially enhance model performance. In both simulated and real-world experiments, our model achieves a significant increase in success rate over OpenVLA. To further assess spatial perception and generalization to novel views, we introduce MV-Bench, a multi-view simulation benchmark. Our model consistently outperforms existing methods, demonstrating stronger spatial understanding and adaptability.

[103] EAMamba: Efficient All-Around Vision State Space Model for Image Restoration

Yu-Cheng Lin,Yu-Syuan Xu,Hao-Wei Chen,Hsien-Kai Kuo,Chun-Yi Lee

Main category: cs.CV

TL;DR: 论文提出了EAMamba（高效全方位视觉状态空间模型），用于解决Vision Mamba在图像复原任务中的计算复杂性和局部像素遗忘问题，通过多头选择性扫描模块和全方位扫描机制显著提升了效率。

Details

Motivation: Vision Mamba虽然在建模长程依赖方面表现出色，但在低层视觉任务中面临计算复杂性和局部像素遗忘的挑战，因此需要一种更高效的改进方法。

Contribution: 提出了EAMamba框架，引入了多头选择性扫描模块（MHSSM）和全方位扫描机制，显著降低了FLOPs（31-89%）并解决了局部像素遗忘问题。

Method: MHSSM通过聚合多扫描序列避免了计算复杂度增加，全方位扫描机制采用多模式捕捉全局信息。

Result: 实验表明，EAMamba在超分辨率、去噪、去模糊和去雾等任务中保持了良好性能的同时，显著降低了计算开销。

Insight: EAMamba通过高效的多头扫描和全局信息捕捉机制，为图像复原任务提供了一种更高效的解决方案，同时为其他低层视觉任务提供了改进方向。

Abstract: Image restoration is a key task in low-level computer vision that aims to reconstruct high-quality images from degraded inputs. The emergence of Vision Mamba, which draws inspiration from the advanced state space model Mamba, marks a significant advancement in this field. Vision Mamba demonstrates excellence in modeling long-range dependencies with linear complexity, a crucial advantage for image restoration tasks. Despite its strengths, Vision Mamba encounters challenges in low-level vision tasks, including computational complexity that scales with the number of scanning sequences and local pixel forgetting. To address these limitations, this study introduces Efficient All-Around Mamba (EAMamba), an enhanced framework that incorporates a Multi-Head Selective Scan Module (MHSSM) with an all-around scanning mechanism. MHSSM efficiently aggregates multiple scanning sequences, which avoids increases in computational complexity and parameter count. The all-around scanning strategy implements multiple patterns to capture holistic information and resolves the local pixel forgetting issue. Our experimental evaluations validate these innovations across several restoration tasks, including super resolution, denoising, deblurring, and dehazing. The results validate that EAMamba achieves a significant 31-89% reduction in FLOPs while maintaining favorable performance compared to existing low-level Vision Mamba methods.

[104] COOCO – Common Objects Out-of-Context – Semantic Violation in Scenes: Investigating Multimodal Context in Referential Communication

Filippo Merlo,Ece Takmaz,Wenkai Chen,Albert Gatt

Main category: cs.CV

TL;DR: 该论文提出了COOCO数据集，用于研究视觉-语言模型（VLMs）在场景-对象一致性不同时的参考生成行为，发现模型会动态平衡局部和上下文信息。

Details

Motivation: 研究VLMs是否依赖场景上下文生成对象参考，尤其是在场景-对象语义不一致或噪声干扰时。

Contribution: 引入了COOCO数据集，并揭示了VLMs如何自适应地利用场景上下文进行对象参考生成。

Method: 通过COOCO数据集测试VLMs在不同场景-对象一致性和噪声水平下的表现，并结合注意力机制分析。

Result: 模型在高场景-对象一致性或对象退化时更依赖上下文，并通过中层注意力聚焦目标对象。

Insight: VLMs能够动态平衡局部和全局信息，尤其在噪声环境下表现出适应性。

Abstract: Natural scenes provide us with rich contexts for object recognition and reference. In particular, knowing what type of scene one is looking at generates expectations about which objects will occur, and what their spatial configuration should be. Do Vision-Language Models (VLMs) learn to rely on scene contexts in a similar way, when generating references to objects? To address this question, we introduce the \textit{Common Objects Out-of-Context (COOCO)} dataset and test to what extent VLMs rely on scene context to refer to objects under different degrees of scene-object congruency, and different perturbations. Our findings show that models leverage scene context adaptively, depending on both the semantic relatedness between object and scene and the level of noise. In particular, models rely more on context under high target-scene congruence or when objects are degraded. Attention analysis reveals that successful object categorisation involves increased focus on the target in mid-level layers, especially under moderate noise, suggesting that VLMs dynamically balance local and contextual information for reference generation. We make our dataset, code and models available at \href{https://github.com/cs-nlp-uu/scenereg}{https://github.com/cs-nlp-uu/scenereg}.

[105] Rethinking Visual Token Reduction in LVLMs under Cross-modal Misalignment

Rui Xu,Yunke Wang,Yong Luo,Bo Du

Main category: cs.CV

TL;DR: 该论文提出VisionDrop，一种无需训练、仅基于视觉信息的修剪框架，通过视觉-视觉注意力选择信息丰富的视觉令牌，解决了跨模态错位问题，显著提升了视觉令牌减少的效率。

Details

Motivation: 现有视觉令牌减少方法依赖文本条件交互，假设文本令牌能可靠捕捉视觉令牌的重要性，但研究发现跨模态错位（因果、语义和空间）会削弱这种方法的有效性。

Contribution: 揭示了跨模态错位的三种形式；提出了VisionDrop框架，通过视觉-视觉注意力实现无需训练的视觉令牌修剪；设计了一种渐进式修剪流程，统一视觉编码器和LLM的处理。

Method: 基于视觉-视觉注意力选择重要视觉令牌；采用渐进式修剪流程，结合令牌选择和轻量级上下文合并；无需额外训练或复杂修改。

Result: 在多个基准测试中，VisionDrop相比现有方法表现更优，保留了细粒度视觉信息的同时实现了高效推理。

Insight: 视觉令牌减少应避免依赖文本信号，而是利用视觉模态内部的信息相关性；跨模态错位是影响现有方法的关键因素。

Abstract: Large Vision-Language Models (LVLMs) encode visual inputs as dense sequences of patch-level tokens to capture fine-grained semantics. These visual tokens often outnumber their textual counterparts by a large margin, leading to substantial computational overhead and limiting the scalability of LVLMs in practice. Previous efforts have explored visual token reduction either prior to or within the large language models (LLM). However, most in-LLM reduction approaches rely on text-conditioned interactions, implicitly assuming that textual tokens can reliably capture the importance of visual tokens. In this work, we revisit this assumption and reveal causal, semantic, and spatial forms of cross-modal misalignment. These misalignments undermine the effectiveness of text-guided visual token reduction. To address this, we introduce VisionDrop, a training-free, visual-only pruning framework that selects informative visual tokens based on intra-modal (visual-to-visual) attention, without relying on textual signals. To further suppress redundancy throughout the model hierarchy, we treat the visual encoder and the LLM as a unified system and design a progressive pruning pipeline. Our method performs dominant token selection and lightweight contextual merging at multiple stages, enabling fine-grained visual information to be retained even under aggressive token budgets. Extensive experiments across diverse benchmarks show that VisionDrop achieves consistent improvements over existing methods, despite requiring no additional training or complex modifications. Its simple yet effective design enables efficient inference while preserving strong performance across tasks.

[106] RoomCraft: Controllable and Complete 3D Indoor Scene Generation

Mengqi Zhou,Xipeng Wang,Yuxi Wang,Zhaoxiang Zhang

Main category: cs.CV

TL;DR: RoomCraft是一个多阶段生成3D室内场景的框架，通过约束驱动优化和冲突感知策略解决现有方法在全局一致性、多约束场景和布局完整性上的问题。

Details

Motivation: 现有方法在生成3D室内场景时存在全局空间推理不足、多约束下对象冲突频繁等问题，RoomCraft旨在通过可控生成和优化策略解决这些局限性。

Contribution: 1. 提出多阶段生成框架；2. 统一约束表示支持多种输入形式；3. 冲突感知策略确保布局完整性；4. 结合优化算法实现可控生成。

Method: 1. 提取高层次场景信息并结构化；2. 构建空间关系网络生成布局序列；3. 统一约束表示与优化；4. CAPS动态调整权重减少冲突。

Result: 实验表明，RoomCraft在多种输入模态下均能生成更真实、语义一致且视觉吸引的室内场景布局。

Insight: RoomCraft通过结构化约束和动态优化策略，为复杂场景生成提供了新的解决方案。

Abstract: Generating realistic 3D indoor scenes from user inputs remains a challenging problem in computer vision and graphics, requiring careful balance of geometric consistency, spatial relationships, and visual realism. While neural generation methods often produce repetitive elements due to limited global spatial reasoning, procedural approaches can leverage constraints for controllable generation but struggle with multi-constraint scenarios. When constraints become numerous, object collisions frequently occur, forcing the removal of furniture items and compromising layout completeness. To address these limitations, we propose RoomCraft, a multi-stage pipeline that converts real images, sketches, or text descriptions into coherent 3D indoor scenes. Our approach combines a scene generation pipeline with a constraint-driven optimization framework. The pipeline first extracts high-level scene information from user inputs and organizes it into a structured format containing room type, furniture items, and spatial relations. It then constructs a spatial relationship network to represent furniture arrangements and generates an optimized placement sequence using a heuristic-based depth-first search (HDFS) algorithm to ensure layout coherence. To handle complex multi-constraint scenarios, we introduce a unified constraint representation that processes both formal specifications and natural language inputs, enabling flexible constraint-oriented adjustments through a comprehensive action space design. Additionally, we propose a Conflict-Aware Positioning Strategy (CAPS) that dynamically adjusts placement weights to minimize furniture collisions and ensure layout completeness. Extensive experiments demonstrate that RoomCraft significantly outperforms existing methods in generating realistic, semantically coherent, and visually appealing room layouts across diverse input modalities.

[107] OutDreamer: Video Outpainting with a Diffusion Transformer

Linhao Zhong,Fan Li,Yi Huang,Jianzhuang Liu,Renjing Pei,Fenglong Song

Main category: cs.CV

TL;DR: OutDreamer提出了一种基于扩散变换器（DiT）的视频外绘框架，通过高效视频控制分支和条件外绘分支，结合掩码驱动的自注意力层和潜在对齐损失，实现了高质量且时间一致的视频扩展。

Details

Motivation: 视频外绘任务需要同时保证时间和空间一致性，现有基于U-Net的潜在扩散模型难以兼顾质量和适应性。扩散变换器（DiT）因其优异性能成为新选择。

Contribution: 1. 基于DiT的框架OutDreamer；2. 掩码驱动的自注意力层；3. 潜在对齐损失和跨视频片段细化器，用于长视频一致性。

Method: 1. 高效视频控制分支提取信息；2. 条件外绘分支生成缺失内容；3. 掩码驱动自注意力层动态整合掩码信息；4. 潜在对齐损失保证帧内外一致性；5. 跨视频片段细化器迭代生成内容。

Result: OutDreamer在零样本设置下超越现有方法，基准测试表现优异。

Insight: 扩散变换器在视频生成任务中潜力巨大，掩码驱动机制和长视频细化策略对提升时空一致性至关重要。

Abstract: Video outpainting is a challenging task that generates new video content by extending beyond the boundaries of an original input video, requiring both temporal and spatial consistency. Many state-of-the-art methods utilize latent diffusion models with U-Net backbones but still struggle to achieve high quality and adaptability in generated content. Diffusion transformers (DiTs) have emerged as a promising alternative because of their superior performance. We introduce OutDreamer, a DiT-based video outpainting framework comprising two main components: an efficient video control branch and a conditional outpainting branch. The efficient video control branch effectively extracts masked video information, while the conditional outpainting branch generates missing content based on these extracted conditions. Additionally, we propose a mask-driven self-attention layer that dynamically integrates the given mask information, further enhancing the model’s adaptability to outpainting tasks. Furthermore, we introduce a latent alignment loss to maintain overall consistency both within and between frames. For long video outpainting, we employ a cross-video-clip refiner to iteratively generate missing content, ensuring temporal consistency across video clips. Extensive evaluations demonstrate that our zero-shot OutDreamer outperforms state-of-the-art zero-shot methods on widely recognized benchmarks.

[108] A Deep Learning framework for building damage assessment using VHR SAR and geospatial data: demonstration on the 2023 Turkiye Earthquake

Luigi Russo,Deodato Tapete,Silvia Liberata Ullo,Paolo Gamba

Main category: cs.CV

TL;DR: 该论文提出了一种多模态深度学习框架，利用单日超高分辨率SAR图像和辅助地理空间数据，快速评估建筑物损毁情况，并展示了在2023年土耳其地震中的有效性。

Details

Motivation: 传统的光学卫星图像在灾害后的建筑物损毁评估中受限于云层覆盖或缺乏灾前数据，亟需一种不依赖灾前图像的快速评估方法。

Contribution: 1) 提出了一种仅依赖灾后数据的多模态深度学习框架；2) 结合SAR图像和地理空间数据（如OSM、DSM和GEM属性）提升检测精度；3) 在2023年土耳其地震中验证了框架的有效性。

Method: 框架整合了SAR图像块、OSM建筑物轮廓、DSM数据以及GEM的结构和暴露属性，通过深度学习模型实现建筑物损毁检测，无需灾前图像。

Result: 结果表明，整合地理空间特征显著提升了检测性能和泛化能力，能够高效评估建筑物损毁情况。

Insight: 该方法解决了灾后快速响应的关键问题，通过自动化和可扩展的数据处理，为灾害管理提供了实用工具。

Abstract: Building damage identification shortly after a disaster is crucial for guiding emergency response and recovery efforts. Although optical satellite imagery is commonly used for disaster mapping, its effectiveness is often hampered by cloud cover or the absence of pre-event acquisitions. To overcome these challenges, we introduce a novel multimodal deep learning (DL) framework for detecting building damage using single-date very high resolution (VHR) Synthetic Aperture Radar (SAR) imagery from the Italian Space Agency (ASI) COSMO SkyMed (CSK) constellation, complemented by auxiliary geospatial data. Our method integrates SAR image patches, OpenStreetMap (OSM) building footprints, digital surface model (DSM) data, and structural and exposure attributes from the Global Earthquake Model (GEM) to improve detection accuracy and contextual interpretation. Unlike existing approaches that depend on pre and post event imagery, our model utilizes only post event data, facilitating rapid deployment in critical scenarios. The framework effectiveness is demonstrated using a new dataset from the 2023 earthquake in Turkey, covering multiple cities with diverse urban settings. Results highlight that incorporating geospatial features significantly enhances detection performance and generalizability to previously unseen areas. By combining SAR imagery with detailed vulnerability and exposure information, our approach provides reliable and rapid building damage assessments without the dependency from available pre-event data. Moreover, the automated and scalable data generation process ensures the framework’s applicability across diverse disaster-affected regions, underscoring its potential to support effective disaster management and recovery efforts. Code and data will be made available upon acceptance of the paper.

[109] From Ground to Air: Noise Robustness in Vision Transformers and CNNs for Event-Based Vehicle Classification with Potential UAV Applications

Nouf Almesafri,Hector Figueiredo,Miguel Arana-Catania

Main category: cs.CV

TL;DR: 本文研究了CNN（ResNet34）和ViT（ViT B16）基于事件的相机数据在车辆分类任务中的表现，比较了它们在干净和有噪声数据下的性能，并探讨了其在无人机应用中的潜力。

Details

Motivation: 事件相机因其动态捕捉能力适用于无人机和自动驾驶等动态环境，但现有研究较少比较CNN和ViT在此领域的性能差异和噪声鲁棒性。

Contribution: 比较了ResNet34和ViT B16在事件相机数据集（GEN1）上的分类性能，发现ResNet34略优，但ViT在小数据集预训练下表现更强的鲁棒性。

Method: 在GEN1数据集上微调ResNet34和ViT B16，并在标准和有噪声条件下评估其分类准确率。

Result: ResNet34和ViT B16在干净数据上分别达到88%和86%的准确率；ViT在噪声环境下表现更稳定。

Insight: ViT在小数据集和噪声环境下的鲁棒性为其在无人机等动态场景中的应用提供了潜力。

Abstract: This study investigates the performance of the two most relevant computer vision deep learning architectures, Convolutional Neural Network and Vision Transformer, for event-based cameras. These cameras capture scene changes, unlike traditional frame-based cameras with capture static images, and are particularly suited for dynamic environments such as UAVs and autonomous vehicles. The deep learning models studied in this work are ResNet34 and ViT B16, fine-tuned on the GEN1 event-based dataset. The research evaluates and compares these models under both standard conditions and in the presence of simulated noise. Initial evaluations on the clean GEN1 dataset reveal that ResNet34 and ViT B16 achieve accuracies of 88% and 86%, respectively, with ResNet34 showing a slight advantage in classification accuracy. However, the ViT B16 model demonstrates notable robustness, particularly given its pre-training on a smaller dataset. Although this study focuses on ground-based vehicle classification, the methodologies and findings hold significant promise for adaptation to UAV contexts, including aerial object classification and event-based vision systems for aviation-related tasks.

[110] Exploiting Vision Language Model for Training-Free 3D Point Cloud OOD Detection via Graph Score Propagation

Tiankai Chen,Yushu Li,Adam Goodge,Fei Teng,Xulei Yang,Tianrui Li,Xun Xu

Main category: cs.CV

TL;DR: 本文提出了一种无需训练的框架，利用视觉语言模型（VLM）进行3D点云的离群检测（OOD），通过图分数传播（GSP）方法结合提示聚类和自训练负提示，显著提升了OOD检测效果。

Details

Motivation: 3D点云数据中的离群检测（OOD）在实际应用中至关重要，但目前的方法主要针对2D图像，扩展到3D环境面临挑战。本文旨在填补这一空白。

Contribution: 1. 提出了一种无需训练的3D点云OOD检测框架；2. 设计了图分数传播（GSP）方法，结合VLM提升了检测效果；3. 方法支持少样本场景，具有实际应用价值。

Method: 通过构建基于类别原型和测试数据的图，利用数据流形结构增强VLM的效果，提出GSP方法，结合提示聚类和自训练负提示优化OOD评分。

Result: 实验表明，GSP在合成和真实3D点云数据集上的OOD检测效果显著优于现有方法。

Insight: VLM在3D点云OOD检测中具有潜力，图结构可以有效利用数据流形信息；提示聚类和负提示技术能进一步提升性能。

Abstract: Out-of-distribution (OOD) detection in 3D point cloud data remains a challenge, particularly in applications where safe and robust perception is critical. While existing OOD detection methods have shown progress for 2D image data, extending these to 3D environments involves unique obstacles. This paper introduces a training-free framework that leverages Vision-Language Models (VLMs) for effective OOD detection in 3D point clouds. By constructing a graph based on class prototypes and testing data, we exploit the data manifold structure to enhancing the effectiveness of VLMs for 3D OOD detection. We propose a novel Graph Score Propagation (GSP) method that incorporates prompt clustering and self-training negative prompting to improve OOD scoring with VLM. Our method is also adaptable to few-shot scenarios, providing options for practical applications. We demonstrate that GSP consistently outperforms state-of-the-art methods across synthetic and real-world datasets 3D point cloud OOD detection.

[111] Can Video Large Multimodal Models Think Like Doubters-or Double-Down: A Study on Defeasible Video Entailment

Yue Zhang,Jilei Sun,Yunhui Guo,Vibhav Gogate

Main category: cs.CV

TL;DR: 该论文提出了一个名为Defeasible Video Entailment (DVidE)的新任务，挑战视频大模型（VLMMs）动态更新推理能力，并提出了两种框架：基于反事实推理的分类任务框架和结合ASR与LLM的生成任务框架，同时引入新数据集和评估指标。实验表明该方法显著提升了VLMMs的动态推理能力。

Details

Motivation: 现有视频大模型（VLMMs）在抽象和自适应推理方面表现不足，无法根据新信息动态更新推理结果。DVidE任务旨在模拟现实世界中结论可被强化的场景，提升模型的动态推理能力。

Contribution: 1. 提出DVidE任务，挑战模型动态更新推理；2. 为分类任务设计Chain of Counterfactual Thought框架；3. 为生成任务结合ASR和LLM提出新框架；4. 引入新数据集和LLM-based评估指标。

Method: 1. 分类任务：基于反事实推理（Counterfactual Reasoning）、ASR增强视频内容、理性细化；2. 生成任务：结合ASR输出与LLM生成上下文相关的更新。

Result: 实验结果表明，所提方法显著提升了VLMMs在动态推理任务上的性能。

Insight: 1. 反事实推理有助于减少推理偏差；2. 结合ASR与LLM能够生成更连贯的更新；3. 动态推理能力是VLMMs未来发展的关键方向。

Abstract: Video Large Multimodal Models (VLMMs) have made impressive strides in understanding video content, but they often struggle with abstract and adaptive reasoning-the ability to revise their interpretations when new information emerges. In reality, conclusions are rarely set in stone; additional context can strengthen or weaken an initial inference. To address this, we introduce Defeasible Video Entailment (DVidE), a new task that challenges models to think like doubters, constantly updating their reasoning based on evolving evidence. In DVidE, given a video premise and a textual hypothesis, models must determine whether a new update strengthens or weakens the hypothesis (classification version) or generate a coherent update that modifies the entailment relationship (generation version). For solving the classification task, we propose the Chain of Counterfactual Thought framework, utilizing counterfactual reasoning, ASR-enhanced video content, and rationale refinement to reduce inference bias. For the generation task, we develop a framework that combines ASR output with a Large Language Model (LLM) to produce coherent, contextually relevant updates aligned with the intended strengthener or weakener goals. Additionally, we introduce a novel benchmark dataset, with strengthener/weakener annotations and an LLM-based evaluation metric specifically designed for assessing generative performance. Experimental results demonstrate significant improvements, highlighting our proposed method in enhancing dynamic reasoning capabilities of VLMMs.

[112] Test-Time Consistency in Vision Language Models

Shih-Han Chou,Shivam Chandhok,James J. Little,Leonid Sigal

Main category: cs.CV

TL;DR: 该论文提出了一种无需监督重新训练的测试时一致性框架，通过两种互补的目标函数增强视觉语言模型（VLM）的语义一致性，显著提升了模型在MM-R3基准上的表现。

Details

Motivation: 现有的视觉语言模型在语义等效输入下表现不一致，尽管平均准确率高，但影响了其可靠性和鲁棒性。论文旨在解决这一问题，而不依赖模型架构修改或大规模监督微调。

Contribution: 提出了一个简单有效的测试时一致性框架，通过交叉熵一致性损失和伪标签一致性损失，增强模型在测试时的语义一致性。该方法无需重新训练，适用于任何VLM。

Method: 框架包含两个目标函数：(1)交叉熵一致性损失，用于对齐语义等效输入的预测分布；(2)伪标签一致性损失，推动输出向自平均共识靠近。框架无需监督，仅利用测试输入本身的信息。

Result: 在MM-R3基准上，该方法显著提升了现有最先进模型的语义一致性表现，为多模态学习的推断时适应提供了新方向。

Insight: 测试时自适应方法可以显著提升模型的一致性，而无需修改模型架构或依赖于大规模监督数据，展示了在推断阶段优化的潜力。

Abstract: Vision-Language Models (VLMs) have achieved impressive performance across a wide range of multimodal tasks, yet they often exhibit inconsistent behavior when faced with semantically equivalent inputs, undermining their reliability and robustness. Recent benchmarks, such as MM-R3, highlight that even state-of-the-art VLMs can produce divergent predictions across semantically equivalent inputs, despite maintaining high average accuracy. Prior work addresses this issue by modifying model architectures or conducting large-scale fine-tuning on curated datasets. In contrast, we propose a simple and effective test-time consistency framework that enhances semantic consistency without supervised re-training. Our method is entirely post-hoc, model-agnostic, and applicable to any VLM with access to its weights. Given a single test point, we enforce consistent predictions via two complementary objectives: (i) a Cross-Entropy Agreement Loss that aligns predictive distributions across semantically equivalent inputs, and (ii) a Pseudo-Label Consistency Loss that draws outputs toward a self-averaged consensus. Our method is plug-and-play and leverages information from a single test input itself to improve consistency. Experiments on the MM-R3 benchmark show that our framework yields substantial gains in consistency across state-of-the-art models, establishing a new direction for inference-time adaptation in multimodal learning.

[113] Shape-for-Motion: Precise and Consistent Video Editing with 3D Proxy

Yuhao Liu,Tengfei Wang,Fang Liu,Zhenwei Wang,Rynson W. H. Lau

Main category: cs.CV

TL;DR: 该论文提出了Shape-for-Motion框架，通过将视频中的目标对象转换为时间一致的3D代理网格，实现了精确且一致的视频编辑。其关键创新包括双传播策略和3D到2D的投影机制，支持多种精确操作。

Details

Motivation: 现有的视频生成方法难以精确满足用户的编辑意图，尤其是在复杂操作（如姿态、纹理修改）中保持一致性。本研究旨在通过引入3D代理网格，提供更可控的视频编辑工具。

Contribution: 1. 提出了Shape-for-Motion框架，首次将3D代理网格用于视频编辑；2. 设计了双传播策略，简化多帧编辑流程；3. 实现了多样化的精确编辑操作（如姿态、缩放、纹理修改）。

Method: 1. 将视频中的目标对象转换为时间一致的3D网格；2. 通过双传播策略将单帧编辑自动传播至多帧；3. 结合解耦的视频扩散模型，从3D投影生成编辑后的2D渲染。

Result: 实验证明了该框架在姿态编辑、纹理修改等多种操作中的优越性，实现了高质量且一致的效果。

Insight: 3D代理网格为视频编辑提供了更结构化的控制手段，结合扩散模型的生成能力，展示了未来可控内容创作的潜力。

Abstract: Recent advances in deep generative modeling have unlocked unprecedented opportunities for video synthesis. In real-world applications, however, users often seek tools to faithfully realize their creative editing intentions with precise and consistent control. Despite the progress achieved by existing methods, ensuring fine-grained alignment with user intentions remains an open and challenging problem. In this work, we present Shape-for-Motion, a novel framework that incorporates a 3D proxy for precise and consistent video editing. Shape-for-Motion achieves this by converting the target object in the input video to a time-consistent mesh, i.e., a 3D proxy, allowing edits to be performed directly on the proxy and then inferred back to the video frames. To simplify the editing process, we design a novel Dual-Propagation Strategy that allows users to perform edits on the 3D mesh of a single frame, and the edits are then automatically propagated to the 3D meshes of the other frames. The 3D meshes for different frames are further projected onto the 2D space to produce the edited geometry and texture renderings, which serve as inputs to a decoupled video diffusion model for generating edited results. Our framework supports various precise and physically-consistent manipulations across the video frames, including pose editing, rotation, scaling, translation, texture modification, and object composition. Our approach marks a key step toward high-quality, controllable video editing workflows. Extensive experiments demonstrate the superiority and effectiveness of our approach. Project page: https://shapeformotion.github.io/

[114] WarpRF: Multi-View Consistency for Training-Free Uncertainty Quantification and Applications in Radiance Fields

Sadra Safadoust,Fabio Tosi,Fatma Güney,Matteo Poggi

Main category: cs.CV

TL;DR: WarpRF 是一个无需训练的多视角一致性框架，用于量化辐射场的不确定性，通过反向映射和一致性测量实现高效的不确定性评估，并在下游任务中表现优异。

Details

Motivation: 当前辐射场方法在不确定性量化方面缺乏高效且通用的解决方案，WarpRF 通过多视角一致性假设填补了这一空白。

Contribution: 提出了无需训练的通用框架 WarpRF，利用反向映射和多视角一致性量化辐射场的不确定性，并验证其在下游任务中的优越性。

Method: 通过反向映射将可靠渲染投影到新视角，测量与目标视角渲染的一致性，从而量化不确定性。

Result: WarpRF 在不确定性量化和下游任务（如主动视角选择和主动建图）中优于现有方法。

Insight: 多视角一致性是量化辐射场不确定性的有效指标，无需额外训练即可实现高效评估。

Abstract: We introduce WarpRF, a training-free general-purpose framework for quantifying the uncertainty of radiance fields. Built upon the assumption that photometric and geometric consistency should hold among images rendered by an accurate model, WarpRF quantifies its underlying uncertainty from an unseen point of view by leveraging backward warping across viewpoints, projecting reliable renderings to the unseen viewpoint and measuring the consistency with images rendered there. WarpRF is simple and inexpensive, does not require any training, and can be applied to any radiance field implementation for free. WarpRF excels at both uncertainty quantification and downstream tasks, e.g., active view selection and active mapping, outperforming any existing method tailored to specific frameworks.

[115] MiCo: Multi-image Contrast for Reinforcement Visual Reasoning

Xi Chen,Mingkang Zhu,Shaoteng Liu,Xiaoyang Wu,Xiaogang Xu,Yu Liu,Xiang Bai,Hengshuang Zhao

Main category: cs.CV

TL;DR: 该论文提出了一种多图像对比方法（MiCo），通过自监督学习构建图像三元组，结合规则化的强化学习训练模型进行跨图像推理，无需人工标注，显著提升了多图像推理任务的性能。

Details

Motivation: 现有方法依赖人工标注的问题-答案对，难以处理细粒度视觉细节和跨图像的复杂逻辑推理。受自监督视觉表征学习启发，作者发现图像本身的内在约束可以作为监督信号。

Contribution: 1. 提出了一种基于图像三元组的自监督方法，避免了人工标注；2. 通过规则化强化学习训练模型进行跨图像推理；3. 在多项任务中验证了方法的通用性和有效性。

Method: 构建包含同一图像两个增强视图和一张相似但不同图像的三元组，通过强化学习优化模型生成推理过程（比较图像异同），迫使模型关注细微视觉变化和逻辑推理。

Result: 在未使用人工标注的情况下，方法在多图像推理任务中表现显著优于基线，并在一般视觉任务中展现了强大性能。

Insight: 图像的内在约束可以作为自监督信号，通过强化学习训练模型进行复杂推理任务，减少对人工标注的依赖。

Abstract: This work explores enabling Chain-of-Thought (CoT) reasoning to link visual cues across multiple images. A straightforward solution is to adapt rule-based reinforcement learning for Vision-Language Models (VLMs). However, such methods typically rely on manually curated question-answer pairs, which can be particularly challenging when dealing with fine grained visual details and complex logic across images. Inspired by self-supervised visual representation learning, we observe that images contain inherent constraints that can serve as supervision. Based on this insight, we construct image triplets comprising two augmented views of the same image and a third, similar but distinct image. During training, the model is prompted to generate a reasoning process to compare these images (i.e., determine same or different). Then we optimize the model with rule-based reinforcement learning. Due to the high visual similarity and the presence of augmentations, the model must attend to subtle visual changes and perform logical reasoning to succeed. Experiments show that, although trained solely on visual comparison tasks, the learned reasoning ability generalizes effectively to a wide range of questions. Without relying on any human-annotated question-answer pairs, our method achieves significant improvements on multi-image reasoning benchmarks and shows strong performance on general vision tasks.

stat.ME [Back]

[116] Using Large Language Models to Suggest Informative Prior Distributions in Bayesian Statistics

Michael A. Riegler,Kristoffer Herland Hellton,Vajira Thambawita,Hugo L. Hammer

Main category: stat.ME

TL;DR: 论文探讨了使用大语言模型（LLMs）为贝叶斯统计中的先验分布提供建议，结果表明LLMs能够正确识别关联性，但在校准先验宽度以避免过度自信或不足方面仍存挑战。

Details

Motivation: 贝叶斯统计中选择合适的先验分布通常具有挑战性且主观性强，论文旨在利用LLMs提供高效、客观的先验建议。

Contribution: 提出了利用LLMs生成基于知识的先验分布的方法，并通过实验验证了Claude和Gemini的表现优于ChatGPT。

Method: 设计了一种提示方法，要求LLMs不仅建议先验分布，还需验证和反思其选择，并通过Kullback-Leibler散度评估先验质量。

Result: LLMs能够正确识别变量间的关联性，但生成的适度信息先验常过于自信，而Claude在生成弱信息先验时表现更优。

Insight: LLMs在高效生成先验分布方面潜力巨大，但需进一步解决先验宽度的校准问题以提高准确性。

Abstract: Selecting prior distributions in Bayesian statistics is challenging, resource-intensive, and subjective. We analyze using large-language models (LLMs) to suggest suitable, knowledge-based informative priors. We developed an extensive prompt asking LLMs not only to suggest priors but also to verify and reflect on their choices. We evaluated Claude Opus, Gemini 2.5 Pro, and ChatGPT-4o-mini on two real datasets: heart disease risk and concrete strength. All LLMs correctly identified the direction for all associations (e.g., that heart disease risk is higher for males). The quality of suggested priors was measured by their Kullback-Leibler divergence from the maximum likelihood estimator’s distribution. The LLMs suggested both moderately and weakly informative priors. The moderate priors were often overconfident, resulting in distributions misaligned with the data. In our experiments, Claude and Gemini provided better priors than ChatGPT. For weakly informative priors, a key performance difference emerged: ChatGPT and Gemini defaulted to an “unnecessarily vague” mean of 0, while Claude did not, demonstrating a significant advantage. The ability of LLMs to identify correct associations shows their great potential as an efficient, objective method for developing informative priors. However, the primary challenge remains in calibrating the width of these priors to avoid over- and under-confidence.

quant-ph [Back]

[117] QuKAN: A Quantum Circuit Born Machine approach to Quantum Kolmogorov Arnold Networks

Yannick Werner,Akash Malemath,Mengxi Liu,Vitor Fortes Rey,Nikolaos Palaiodimopoulos,Paul Lukowicz,Maximilian Kiefer-Emmanouilidis

Main category: quant-ph

TL;DR: 该论文提出了QuKAN，一种基于量子电路生成模型（QCBM）的量子Kolmogorov Arnold网络，探索了KAN在量子机器学习中的潜力。

Details

Motivation: Kolmogorov Arnold网络（KANs）因其通过学习边参数而非节点参数来高效表达复杂函数的能力而受到关注，但其在量子机器学习中的应用尚未深入研究。

Contribution: 论文的主要贡献是提出了混合和全量子形式的KAN架构，通过QCBM实现，并展示了其在可行性和性能方面的优势。

Method: 方法包括：1）将KAN的残差函数转换为量子模型；2）在混合模型中结合经典KAN组件与量子子程序；3）利用预训练残差函数优化量子电路参数。

Result: 实验证明了QuKAN架构的可行性和解释性，并展示了其性能优于传统方法。

Insight: 研究揭示了量子电路在增强KAN表达能力和效率方面的潜力，为量子机器学习提供了新思路。

Abstract: Kolmogorov Arnold Networks (KANs), built upon the Kolmogorov Arnold representation theorem (KAR), have demonstrated promising capabilities in expressing complex functions with fewer neurons. This is achieved by implementing learnable parameters on the edges instead of on the nodes, unlike traditional networks such as Multi-Layer Perceptrons (MLPs). However, KANs potential in quantum machine learning has not yet been well explored. In this work, we present an implementation of these KAN architectures in both hybrid and fully quantum forms using a Quantum Circuit Born Machine (QCBM). We adapt the KAN transfer using pre-trained residual functions, thereby exploiting the representational power of parametrized quantum circuits. In the hybrid model we combine classical KAN components with quantum subroutines, while the fully quantum version the entire architecture of the residual function is translated to a quantum model. We demonstrate the feasibility, interpretability and performance of the proposed Quantum KAN (QuKAN) architecture.

cs.RO [Back]

[118] TOMD: A Trail-based Off-road Multimodal Dataset for Traversable Pathway Segmentation under Challenging Illumination Conditions

Yixin Sun,Li Li,Wenke E,Amir Atapour-Abarghouei,Toby P. Breckon

Main category: cs.RO

TL;DR: 论文提出了TOMD数据集和动态多尺度数据融合模型，专注于解决非结构化户外环境中的可通行路径分割问题，特别是在具有挑战性光照条件下的窄道场景。

Details

Motivation: 现有数据集和模型主要关注城市环境或宽阔的可通行道路，忽视了窄道等复杂户外场景的需求。搜索救援和森林火灾等关键应用需要更精准的可通行路径分割方法。

Contribution: 1. 推出了TOMD数据集，包含多模态传感器数据；2. 提出动态多尺度数据融合模型；3. 分析了不同光照条件对分割性能的影响。

Method: 采用了高保真多模态传感器数据（如128通道LiDAR、立体图像等），并设计了动态多尺度数据融合模型。通过早期、交叉和混合融合策略分析光照对性能的影响。

Result: 实验验证了模型的有效性，并揭示了光照对分割性能的重要性。数据集和模型已开源。

Insight: 光照是户外可通行路径分割的关键因素，动态多尺度融合能显著提升性能。窄道场景的研究填补了现有技术的空白。

Abstract: Detecting traversable pathways in unstructured outdoor environments remains a significant challenge for autonomous robots, especially in critical applications such as wide-area search and rescue, as well as incident management scenarios like forest fires. Existing datasets and models primarily target urban settings or wide, vehicle-traversable off-road tracks, leaving a substantial gap in addressing the complexity of narrow, trail-like off-road scenarios. To address this, we introduce the Trail-based Off-road Multimodal Dataset (TOMD), a comprehensive dataset specifically designed for such environments. TOMD features high-fidelity multimodal sensor data – including 128-channel LiDAR, stereo imagery, GNSS, IMU, and illumination measurements – collected through repeated traversals under diverse conditions. We also propose a dynamic multiscale data fusion model for accurate traversable pathway prediction. The study analyzes the performance of early, cross, and mixed fusion strategies under varying illumination levels. Results demonstrate the effectiveness of our approach and the relevance of illumination in segmentation performance. We publicly release TOMD at https://github.com/yyyxs1125/TMOD to support future research in trail-based off-road navigation.

[119] AeroLite-MDNet: Lightweight Multi-task Deviation Detection Network for UAV Landing

Haiping Yang,Huaxing Liu,Wei Wu,Zuohui Chen,Ning Wu

Main category: cs.RO

TL;DR: 论文提出了一个轻量级的视觉模型AeroLite-MDNet，用于检测无人机（UAV）着陆时的偏差，通过多尺度融合模块和分割分支提升了偏差检测和方向估计的性能，并引入了新的评价指标AWD和数据集UAVLandData。实验结果表明模型具有高准确性和低延迟。

Details

Motivation: 无人机着陆过程中的偏差检测是一个具有挑战性的任务，尤其是GPS信号干扰等问题可能影响着陆的准确性。论文旨在通过视觉方法解决这一问题，提升无人机着陆的安全性和可靠性。

Contribution: 1. 提出了轻量级多任务偏差检测模型AeroLite-MDNet，结合了多尺度融合和目标检测模块。2. 引入了新的评价指标Average Warning Delay（AWD）。3. 贡献了一个新数据集UAVLandData，用于训练和评估。

Method: AeroLite-MDNet集成了多尺度融合模块以提升跨尺度目标检测能力，并加入了分割分支用于高效的着陆方向估计。

Result: 实验结果显示模型在偏差检测上达到98.6%的准确率，AWD为0.7秒，验证了其有效性。

Insight: 视觉方法可以弥补GPS信号的局限性，多任务学习和新的评价指标有助于提升无人机着陆系统的鲁棒性和实时性。

Abstract: Unmanned aerial vehicles (UAVs) are increasingly employed in diverse applications such as land surveying, material transport, and environmental monitoring. Following missions like data collection or inspection, UAVs must land safely at docking stations for storage or recharging, which is an essential requirement for ensuring operational continuity. However, accurate landing remains challenging due to factors like GPS signal interference. To address this issue, we propose a deviation warning system for UAV landings, powered by a novel vision-based model called AeroLite-MDNet. This model integrates a multiscale fusion module for robust cross-scale object detection and incorporates a segmentation branch for efficient orientation estimation. We introduce a new evaluation metric, Average Warning Delay (AWD), to quantify the system’s sensitivity to landing deviations. Furthermore, we contribute a new dataset, UAVLandData, which captures real-world landing deviation scenarios to support training and evaluation. Experimental results show that our system achieves an AWD of 0.7 seconds with a deviation detection accuracy of 98.6%, demonstrating its effectiveness in enhancing UAV landing reliability. Code will be available at https://github.com/ITTTTTI/Maskyolo.git

[120] Experimental investigation of pose informed reinforcement learning for skid-steered visual navigation

Ameya Salvi,Venkat Krovi

Main category: cs.RO

TL;DR: 该论文提出了一种基于姿态信息的强化学习方法，用于解决滑移转向车辆在视觉导航中的挑战，并通过仿真和硬件实验验证了其优越性能。

Details

Motivation: 滑移转向车辆的模型化（尤其是在越野环境下）存在困难，而端到端学习方法（如模仿学习和强化学习）为解决这一问题提供了可能，但其在动态操作区域的系统化验证仍需完善。

Contribution: 提出了一种新的结构化方法，用于学习滑移转向车辆的视觉导航，显著提升了性能。

Method: 采用姿态信息辅助的强化学习方法，结合仿真和硬件实验进行验证，并通过消融研究分析效果。

Result: 实验结果表明，该方法在视觉导航任务中的表现优于现有文献中的方法。

Insight: 姿态信息的引入可以显著提升滑移转向车辆在动态环境中的导航能力，为端到端学习方法的实际应用提供了新思路。

Abstract: Vision-based lane keeping is a topic of significant interest in the robotics and autonomous ground vehicles communities in various on-road and off-road applications. The skid-steered vehicle architecture has served as a useful vehicle platform for human controlled operations. However, systematic modeling, especially of the skid-slip wheel terrain interactions (primarily in off-road settings) has created bottlenecks for automation deployment. End-to-end learning based methods such as imitation learning and deep reinforcement learning, have gained prominence as a viable deployment option to counter the lack of accurate analytical models. However, the systematic formulation and subsequent verification/validation in dynamic operation regimes (particularly for skid-steered vehicles) remains a work in progress. To this end, a novel approach for structured formulation for learning visual navigation is proposed and investigated in this work. Extensive software simulations, hardware evaluations and ablation studies now highlight the significantly improved performance of the proposed approach against contemporary literature.

[121] Embodied Domain Adaptation for Object Detection

Xiangyu Shi,Yanyuan Qiao,Lingqiao Liu,Feras Dayoub

Main category: cs.RO

TL;DR: 论文提出了一种源数据无关的域自适应方法（SFDA），用于改进室内环境中物体检测的性能，特别是在动态和多样化条件下的适应性。

Details

Motivation: 移动机器人在室内环境中依赖物体检测器进行感知和定位，但传统的封闭集方法和开放词汇检测（OVOD）方法在应对室内环境的多样性和动态变化时表现不佳。

Contribution: 1. 提出了一种无需访问源数据的域自适应方法（SFDA）。2. 结合时序聚类和多尺度阈值融合优化伪标签。3. 引入Mean Teacher框架和对比学习提升自适应性能。

Method: 1. 通过时序聚类优化伪标签。2. 采用多尺度阈值融合策略。3. 结合Mean Teacher框架与对比学习。

Result: 在EDAOD基准测试中表现出显著的零样本检测性能提升，并能灵活适应动态室内条件。

Insight: 该研究表明，结合时序信息和多尺度特征可以显著提升模型在动态环境中的适应性，同时无需依赖源数据。

Abstract: Mobile robots rely on object detectors for perception and object localization in indoor environments. However, standard closed-set methods struggle to handle the diverse objects and dynamic conditions encountered in real homes and labs. Open-vocabulary object detection (OVOD), driven by Vision Language Models (VLMs), extends beyond fixed labels but still struggles with domain shifts in indoor environments. We introduce a Source-Free Domain Adaptation (SFDA) approach that adapts a pre-trained model without accessing source data. We refine pseudo labels via temporal clustering, employ multi-scale threshold fusion, and apply a Mean Teacher framework with contrastive learning. Our Embodied Domain Adaptation for Object Detection (EDAOD) benchmark evaluates adaptation under sequential changes in lighting, layout, and object diversity. Our experiments show significant gains in zero-shot detection performance and flexible adaptation to dynamic indoor conditions.

[122] Evaluating Pointing Gestures for Target Selection in Human-Robot Collaboration

Noora Sassali,Roel Pieters

Main category: cs.RO

TL;DR: 论文提出了一种用于人机协作中目标选择的指向手势评估方法，结合姿态估计和几何模型，并在多模态机器人系统中验证其效果。

Details

Motivation: 指向手势是人机协作中的常见交互方式，但目前缺乏对其准确性和实用性的系统评估方法。

Contribution: 1. 提出基于姿态估计和几何模型的指向手势目标选择方法；2. 提供全面评估方法与多模态系统集成演示；3. 开源工具与性能分析。

Method: 采用RGB-D数据流，通过姿态估计和肩-腕扩展的几何模型提取手势数据，并结合目标检测与语音技术实现多模态集成。

Result: 方法在典型机器人任务中验证了指向手势的准确性，并通过多模态系统展示了实用性。

Insight: 研究揭示了指向手势在复杂协作任务中的潜力，同时也指出了其局限性，为未来的多模态交互设计提供了参考。

Abstract: Pointing gestures are a common interaction method used in Human-Robot Collaboration for various tasks, ranging from selecting targets to guiding industrial processes. This study introduces a method for localizing pointed targets within a planar workspace. The approach employs pose estimation, and a simple geometric model based on shoulder-wrist extension to extract gesturing data from an RGB-D stream. The study proposes a rigorous methodology and comprehensive analysis for evaluating pointing gestures and target selection in typical robotic tasks. In addition to evaluating tool accuracy, the tool is integrated into a proof-of-concept robotic system, which includes object detection, speech transcription, and speech synthesis to demonstrate the integration of multiple modalities in a collaborative application. Finally, a discussion over tool limitations and performance is provided to understand its role in multimodal robotic systems. All developments are available at: https://github.com/NMKsas/gesture_pointer.git.

[123] KnotDLO: Toward Interpretable Knot Tying

Holly Dinkel,Raghavendra Navaratna,Jingyi Xiang,Brian Coltin,Trey Smith,Timothy Bretl

Main category: cs.RO

TL;DR: KnotDLO是一种用于单手打结的Deformable Linear Object（DLO）方法，具有鲁棒性、可重复性和可解释性，无需人工演示或训练。

Details

Motivation: 现有的打结方法通常在遮挡或绳子初始配置变化时表现不佳，且往往依赖人工演示或训练数据。KnotDLO旨在解决这些问题，提供一种更鲁棒和自适应的解决方案。

Contribution: KnotDLO的主要贡献包括：（1）提出了一种无需人工演示或训练的打结方法；（2）通过解耦视觉推理与控制，提高了系统的鲁棒性和可解释性；（3）在遮挡和不同初始配置下仍能实现50%的成功率。

Method: KnotDLO通过跟踪DLO的分段线性曲线形状来规划抓取和目标路径点，并基于当前DLO形状和期望状态计算中间路径点。系统将视觉推理与控制解耦，从而提升了鲁棒性。

Result: 在16次打结实验中，KnotDLO从未见过的配置中实现了50%的成功率。

Insight: KnotDLO展示了无需依赖人工数据的方法在复杂任务（如打结）中的潜力，同时突出了视觉推理与控制解耦的重要性。

Abstract: This work presents KnotDLO, a method for one-handed Deformable Linear Object (DLO) knot tying that is robust to occlusion, repeatable for varying rope initial configurations, interpretable for generating motion policies, and requires no human demonstrations or training. Grasp and target waypoints for future DLO states are planned from the current DLO shape. Grasp poses are computed from indexing the tracked piecewise linear curve representing the DLO state based on the current curve shape and are piecewise continuous. KnotDLO computes intermediate waypoints from the geometry of the current DLO state and the desired next state. The system decouples visual reasoning from control. In 16 trials of knot tying, KnotDLO achieves a 50% success rate in tying an overhand knot from previously unseen configurations.

cs.IR [Back]

[124] Hierarchical Patch Compression for ColPali: Efficient Multi-Vector Document Retrieval with Dynamic Pruning and Quantization

Duong Bach

Main category: cs.IR

TL;DR: HPC-ColPali 是一个用于多向量文档检索的层次化压缩框架，通过 K-Means 量化、动态剪枝和二进制编码技术显著提升了效率和存储利用率，同时保持了检索精度。

Details

Motivation: 现有的多向量文档检索系统（如 ColPali）因其依赖高维补丁嵌入和延迟交互评分，存储和计算成本高昂。作者提出 HPC-ColPali 以解决这些问题。

Contribution: 提出了 HPC-ColPali 框架，包括 K-Means 量化、注意力引导的动态剪枝和二进制编码技术，显著提升了效率和存储利用率。

Method: 1. K-Means 量化将补丁嵌入压缩为 1 字节的质心索引；2. 基于注意力权重的动态剪枝保留最显著补丁；3. 二进制编码支持快速汉明距离搜索。

Result: 在 ViDoRe 和 SEC-Filings 数据集上，HPC-ColPali 降低查询延迟 30-50%，同时保持高精度；在 RAG 管道中减少幻觉率 30%，延迟减半。

Insight: 层次化压缩技术（量化、剪枝、编码）可高效优化多向量检索系统，适用于资源受限环境，同时保持性能。

Abstract: Multi-vector document retrieval systems, such as ColPali, excel in fine-grained matching for complex queries but incur significant storage and computational costs due to their reliance on high-dimensional patch embeddings and late-interaction scoring. To address these challenges, we propose HPC-ColPali, a Hierarchical Patch Compression framework that enhances the efficiency of ColPali while preserving its retrieval accuracy. Our approach integrates three innovative techniques: (1) K-Means quantization, which compresses patch embeddings into 1-byte centroid indices, achieving up to 32$\times$ storage reduction; (2) attention-guided dynamic pruning, utilizing Vision-Language Model attention weights to retain only the top-$p%$ most salient patches, reducing late-interaction computation by up to 60% with less than 2% nDCG@10 loss; and (3) optional binary encoding of centroid indices into $b$-bit strings ($b=\lceil\log_2 K\rceil$), enabling rapid Hamming distance-based similarity search for resource-constrained environments. Evaluated on the ViDoRe and SEC-Filings datasets, HPC-ColPali achieves 30–50% lower query latency under HNSW indexing while maintaining high retrieval precision. When integrated into a Retrieval-Augmented Generation pipeline for legal summarization, it reduces hallucination rates by 30% and halves end-to-end latency. These advancements establish HPC-ColPali as a scalable and efficient solution for multi-vector document retrieval across diverse applications. Code is available at https://github.com/DngBack/HPC-ColPali.

[125] Evaluating VisualRAG: Quantifying Cross-Modal Performance in Enterprise Document Understanding

Varun Mannam,Fang Wang,Xin Chen

Main category: cs.IR

TL;DR: 该论文提出了一个系统化的量化评估框架VisualRAG，用于衡量企业文档理解中多模态输入的可信度，优化模态权重后性能提升57.3%。

Details

Motivation: 当前多模态生成AI的评估框架难以建立可信度，阻碍了企业对可靠AI的采用，需开发定量评估方法。

Contribution: 1. 提出量化多模态RAG可信度的系统化框架；2. 通过实验确定最优模态权重；3. 提供基础模型的比较分析。

Method: 采用多模态输入（文本、图像、标题、OCR），建立技术指标与用户信任度之间的定量关系，优化权重分配。

Result: 最优权重（文本30%、图像15%、标题25%、OCR30%）性能提升57.3%，并保持计算效率。

Insight: 多模态权重优化可显著提升性能，且不同基础模型对标题生成和OCR提取的可信度影响显著。

Abstract: Current evaluation frameworks for multimodal generative AI struggle to establish trustworthiness, hindering enterprise adoption where reliability is paramount. We introduce a systematic, quantitative benchmarking framework to measure the trustworthiness of progressively integrating cross-modal inputs such as text, images, captions, and OCR within VisualRAG systems for enterprise document intelligence. Our approach establishes quantitative relationships between technical metrics and user-centric trust measures. Evaluation reveals that optimal modality weighting with weights of 30% text, 15% image, 25% caption, and 30% OCR improves performance by 57.3% over text-only baselines while maintaining computational efficiency. We provide comparative assessments of foundation models, demonstrating their differential impact on trustworthiness in caption generation and OCR extraction-a vital consideration for reliable enterprise AI. This work advances responsible AI deployment by providing a rigorous framework for quantifying and enhancing trustworthiness in multimodal RAG for critical enterprise applications.

[126] CAL-RAG: Retrieval-Augmented Multi-Agent Generation for Content-Aware Layout Design

Najmeh Forouzandehmehr,Reza Yousefi Maragheh,Sriram Kollipara,Kai Zhao,Topojoy Biswas,Evren Korpeoglu,Kannan Achan

Main category: cs.IR

TL;DR: CAL-RAG是一个基于检索增强的多智能体框架，用于内容感知的布局生成，结合了多模态检索、大型语言模型和协作式智能体推理，显著优于现有方法。

Details

Motivation: 自动化的内容感知布局生成在智能设计系统中是一个基础但未被充分探索的问题，现有方法缺乏上下文设计示例的参考，且难以处理语义对齐和视觉一致性。

Contribution: 提出了CAL-RAG框架，通过多模态检索和基于LLM的智能体协作实现内容感知布局生成，并在PKU PosterLayout数据集上取得了最优性能。

Method: 通过检索相关布局示例，调用基于LLM的布局推荐器，利用视觉-语言评分智能体和反馈智能体迭代优化布局。

Result: 在PKU PosterLayout数据集上，CAL-RAG在多项布局指标（如底层效果、元素对齐和重叠）上优于现有基线，如LayoutPrompter。

Insight: 结合检索增强和多步智能体推理能够显著提升自动化布局生成的性能，同时提供了可扩展和可解释的解决方案。

Abstract: Automated content-aware layout generation – the task of arranging visual elements such as text, logos, and underlays on a background canvas – remains a fundamental yet under-explored problem in intelligent design systems. While recent advances in deep generative models and large language models (LLMs) have shown promise in structured content generation, most existing approaches lack grounding in contextual design exemplars and fall short in handling semantic alignment and visual coherence. In this work we introduce CAL-RAG, a retrieval-augmented, agentic framework for content-aware layout generation that integrates multimodal retrieval, large language models, and collaborative agentic reasoning. Our system retrieves relevant layout examples from a structured knowledge base and invokes an LLM-based layout recommender to propose structured element placements. A vision-language grader agent evaluates the layout with visual metrics, and a feedback agent provides targeted refinements, enabling iterative improvement. We implement our framework using LangGraph and evaluate it on the PKU PosterLayout dataset, a benchmark rich in semantic and structural variability. CAL-RAG achieves state-of-the-art performance across multiple layout metrics – including underlay effectiveness, element alignment, and overlap – substantially outperforming strong baselines such as LayoutPrompter. These results demonstrate that combining retrieval augmentation with agentic multi-step reasoning yields a scalable, interpretable, and high-fidelity solution for automated layout generation.

[127] ARAG: Agentic Retrieval Augmented Generation for Personalized Recommendation

Reza Yousefi Maragheh,Pratheek Vadla,Priyank Gupta,Kai Zhao,Aysenur Inan,Kehui Yao,Jianpeng Xu,Praveen Kanumala,Jason Cho,Sushant Kumar

Main category: cs.IR

TL;DR: 该论文提出了一种名为ARAG的新型框架，通过多智能体协作机制改进检索增强生成（RAG）方法，以更好地捕获动态推荐场景中的用户偏好。

Details

Motivation: 现有的RAG方法依赖静态检索启发式方法，难以捕捉动态推荐场景中的用户偏好。ARAG的动机是通过引入多智能体协作机制，提升推荐系统的个性化能力。

Contribution: 主要贡献是提出ARAG框架，集成四种专门化LLM智能体（用户理解、自然语言推理、上下文总结和项目排序），动态匹配用户意图与候选项目，显著提升推荐效果。

Method: ARAG的核心方法是通过多智能体协作优化RAG流程：用户理解智能体分析长期和会话行为，NLI智能体评估语义对齐，上下文总结智能体整合结果，项目排序智能体生成个性化推荐列表。

Result: 实验表明，ARAG在NDCG@5和Hit@5指标上分别比标准RAG和基于时效的基线方法提升了42.1%和35.5%。消融研究验证了各组件的重要性。

Insight: 该研究表明，在基于LLM的推荐系统中引入智能体协作机制，可以更有效地建模用户偏好和动态意图，为个性化推荐提供了新方向。

Abstract: Retrieval-Augmented Generation (RAG) has shown promise in enhancing recommendation systems by incorporating external context into large language model prompts. However, existing RAG-based approaches often rely on static retrieval heuristics and fail to capture nuanced user preferences in dynamic recommendation scenarios. In this work, we introduce ARAG, an Agentic Retrieval-Augmented Generation framework for Personalized Recommendation, which integrates a multi-agent collaboration mechanism into the RAG pipeline. To better understand the long-term and session behavior of the user, ARAG leverages four specialized LLM-based agents: a User Understanding Agent that summarizes user preferences from long-term and session contexts, a Natural Language Inference (NLI) Agent that evaluates semantic alignment between candidate items retrieved by RAG and inferred intent, a context summary agent that summarizes the findings of NLI agent, and an Item Ranker Agent that generates a ranked list of recommendations based on contextual fit. We evaluate ARAG accross three datasets. Experimental results demonstrate that ARAG significantly outperforms standard RAG and recency-based baselines, achieving up to 42.1% improvement in NDCG@5 and 35.5% in Hit@5. We also, conduct an ablation study to analyse the effect by different components of ARAG. Our findings highlight the effectiveness of integrating agentic reasoning into retrieval-augmented recommendation and provide new directions for LLM-based personalization.

cs.HC [Back]

[128] 3Description: An Intuitive Human-AI Collaborative 3D Modeling Approach

Zhuodi Cai

Main category: cs.HC

TL;DR:

Details

Motivation: 3D建模传统上需要专业技能，存在较高的学习门槛，限制了非专业人士的参与。3Description旨在通过结合人类自然语言和手势输入与AI技术（如NLP和计算机视觉），降低3D建模的复杂度，使其更易用和普及。

Contribution: 1. 提出了一种人机协作的直观3D建模方法3Description；2. 结合自然语言和手势输入，使非专业人士也能参与3D建模；3. 基于Web平台实现跨平台可用性，提升可访问性。

Method: 结合定性研究、产品分析和用户测试，整合OpenAI的NLP技术和MediaPipe的计算机视觉技术，通过Web平台实现用户通过语音和手势描述来生成和调整3D模型。

Result: 3Description实现了非专业人士通过自然交互方式参与3D建模的目标，同时保留了人类创造力，避免了过度依赖技术。

Insight: 该研究展示了人机协作在降低专业工具门槛方面的潜力，同时强调了人类在创意过程中的核心地位。

Abstract: This paper presents 3Description, an experimental human-AI collaborative approach for intuitive 3D modeling. 3Description aims to address accessibility and usability challenges in traditional 3D modeling by enabling non-professional individuals to co-create 3D models using verbal and gesture descriptions. Through a combination of qualitative research, product analysis, and user testing, 3Description integrates AI technologies such as Natural Language Processing and Computer Vision, powered by OpenAI and MediaPipe. Recognizing the web has wide cross-platform capabilities, 3Description is web-based, allowing users to describe the desired model and subsequently adjust its components using verbal and gestural inputs. In the era of AI and emerging media, 3Description not only contributes to a more inclusive and user-friendly design process, empowering more people to participate in the construction of the future 3D world, but also strives to increase human engagement in co-creation with AI, thereby avoiding undue surrender to technology and preserving human creativity.

cs.AI [Back]

[129] CitySim: Modeling Urban Behaviors and City Dynamics with Large-Scale LLM-Driven Agent Simulation

Nicolas Bougie,Narimasa Watanabe

Main category: cs.AI

TL;DR: CitySim是一个基于大语言模型（LLM）驱动的城市模拟器，通过智能代理模拟人类行为和城市动态，比传统基于规则的方法更贴近真实人类行为。

Details

Motivation: 传统城市模拟方法依赖手工规则，难以捕捉复杂的人类行为和意图。CitySim利用大语言模型展示的人类级智能，提供更灵活、真实的模拟。

Contribution: 1. 提出CitySim，利用LLM驱动的代理实现城市行为的细粒度模拟；2. 引入递归值驱动的方法生成日程；3. 代理具备信念、长期目标和空间记忆能力。

Method: 采用递归值驱动方法生成代理日程，结合信念、目标和记忆能力，支持长周期模拟和复杂行为生成。

Result: CitySim在微观和宏观层面均比传统方法更贴近真实人类行为，并能模拟大规模代理的集体行为。

Insight: LLM为城市模拟提供了强大的泛化能力，未来的研究方向包括多模态输入和更动态的城市环境建模。

Abstract: Modeling human behavior in urban environments is fundamental for social science, behavioral studies, and urban planning. Prior work often rely on rigid, hand-crafted rules, limiting their ability to simulate nuanced intentions, plans, and adaptive behaviors. Addressing these challenges, we envision an urban simulator (CitySim), capitalizing on breakthroughs in human-level intelligence exhibited by large language models. In CitySim, agents generate realistic daily schedules using a recursive value-driven approach that balances mandatory activities, personal habits, and situational factors. To enable long-term, lifelike simulations, we endow agents with beliefs, long-term goals, and spatial memory for navigation. CitySim exhibits closer alignment with real humans than prior work, both at micro and macro levels. Additionally, we conduct insightful experiments by modeling tens of thousands of agents and evaluating their collective behaviors under various real-world scenarios, including estimating crowd density, predicting place popularity, and assessing well-being. Our results highlight CitySim as a scalable, flexible testbed for understanding and forecasting urban phenomena.

[130] Conceptual Topic Aggregation

Klara M. Gutekunst,Dominik Dürrschnabel,Johannes Hirth,Gerd Stumme

Main category: cs.AI

TL;DR: 本文提出了一种名为FAT-CAT的新方法，基于形式概念分析（FCA），用于提升主题聚合的可解释性和可视化效果，解决了传统主题模型在语义表达上的不足。

Details

Motivation: 随着数据规模的急剧增长，传统的手动检查方法已不可行，需要更高效的计算方法进行数据探索。然而，现有主题模型在提供可解释性和深度语义结构分析方面存在不足。

Contribution: 主要贡献是提出了FAT-CAT方法，利用FCA技术实现更直观、层次化的主题表示，适用于多样化的主题和文件类型。

Method: 基于形式概念分析（FCA），通过构建概念格（concept lattice）来实现主题的层次化聚合和可视化。

Result: 在ETYNTKE数据集上的实验表明，FAT-CAT比其他主题建模方法更能提供有意义的、可解释的数据集结构洞察。

Insight: FCA不仅提升了主题模型的可解释性，还为数据的层次化分析提供了新的视角，特别适用于多类型、多主题的数据集。

Abstract: The vast growth of data has rendered traditional manual inspection infeasible, necessitating the adoption of computational methods for efficient data exploration. Topic modeling has emerged as a powerful tool for analyzing large-scale textual datasets, enabling the extraction of latent semantic structures. However, existing methods for topic modeling often struggle to provide interpretable representations that facilitate deeper insights into data structure and content. In this paper, we propose FAT-CAT, an approach based on Formal Concept Analysis (FCA) to enhance meaningful topic aggregation and visualization of discovered topics. Our approach can handle diverse topics and file types – grouped by directories – to construct a concept lattice that offers a structured, hierarchical representation of their topic distribution. In a case study on the ETYNTKE dataset, we evaluate the effectiveness of our approach against other representation methods to demonstrate that FCA-based aggregation provides more meaningful and interpretable insights into dataset composition than existing topic modeling techniques.

[131] The Automated LLM Speedrunning Benchmark: Reproducing NanoGPT Improvements

Bingchen Zhao,Despoina Magka,Minqi Jiang,Xian Li,Roberta Raileanu,Tatiana Shavrina,Jean-Christophe Gagnon-Audet,Kelvin Niu,Shagun Sodhani,Michael Shvartsman,Andrei Lupu,Alisia Lupidi,Edan Toledo,Karen Hambardzumyan,Martin Josifoski,Thomas Foster,Lucia Cipolina-Kun,Abhishek Charnalia,Derek Dunfield,Alexander H. Miller,Oisin Mac Aodha,Jakob Foerster,Yoram Bachrach

Main category: cs.AI

TL;DR: 该论文提出了一个自动化LLM速度跑分基准，用于评估AI代理在重现NanoGPT改进方面的能力，结果显示即使提供详细提示，当前LLM仍难以重现已知的创新。

Details

Motivation: 研究如何利用AI自动重现科学研究成果，以推动科学进步。

Contribution: 提出了一个专门用于评估LLM重现能力的自动化基准，涵盖了从算法改进到硬件优化的多样化任务。

Method: 设计了19个基于NanoGPT速度跑分的任务，提供不同粒度的提示（从伪代码到论文式描述），并记录代理的表现。

Result: 研究发现，即便是结合了先进推理能力的LLM，也无法有效重现已知的改进，表明其自动科学研究再现能力有限。

Insight: 自动化科学研究再现是一个复杂且尚未解决的问题，当前LLM的能力仍有很大提升空间。

Abstract: Rapid advancements in large language models (LLMs) have the potential to assist in scientific progress. A critical capability toward this endeavor is the ability to reproduce existing work. To evaluate the ability of AI agents to reproduce results in an active research area, we introduce the Automated LLM Speedrunning Benchmark, leveraging the research community contributions on the NanoGPT speedrun, a competition to train a GPT-2 model in the shortest time. Each of the 19 speedrun tasks provides the agent with the previous records training script, optionally paired with one of three hint formats, ranging from pseudocode to paper-like descriptions of the new records improvements. Records execute quickly by design and speedrun improvements encompass diverse code-level changes, ranging from high-level algorithmic advancements to hardware-aware optimizations. These features make the benchmark both accessible and realistic for the frontier problem of improving LLM training. We find that recent reasoning LLMs combined with SoTA scaffolds struggle to reimplement already-known innovations in our benchmark, even when given detailed hints. Our benchmark thus provides a simple, non-saturated measure of an LLMs ability to automate scientific reproduction, a necessary (but not sufficient) skill for an autonomous research agent.

cs.LG [Back]

[132] APO: Enhancing Reasoning Ability of MLLMs via Asymmetric Policy Optimization

Minjie Hong,Zirun Guo,Yan Xia,Zehan Wang,Ziang Zhang,Tao Jin,Zhou Zhao

Main category: cs.LG

TL;DR: 该论文提出了Asymmetric Policy Optimization (APO)方法，通过Difficulty-Adaptive Divergence Shaping (DADS)和Suboptimal Trajectory Complexity Regularization (STCR)技术，增强多模态大语言模型(MLLMs)的推理能力，同时避免通用任务性能下降和过度推理的问题。

Details

Motivation: 多模态大语言模型(MLLMs)在复杂推理任务上表现不佳，而传统的强化学习(RL)方法容易导致通用任务性能下降或生成过度冗长的推理。

Contribution: 1. 提出APO方法，通过不对称策略优化提升MLLMs的推理能力；2. 引入DADS动态调整KL散度权重，避免策略熵急剧下降；3. 提出STCR惩罚冗长响应，缓解过度推理问题。

Method: 1. 将采样响应分为正负样本组；2. 对正样本使用DADS动态调整KL散度权重；3. 对负样本使用STCR惩罚冗长响应。

Result: View-R1-3B模型在推理基准上比基准模型平均提升7%，且不牺牲通用任务性能，甚至优于更大的MLLMs模型(7-11B)。

Insight: 1. 动态调整KL散度权重有助于保持模型知识并提升训练稳定性；2. 惩罚冗长响应能够有效抑制过度推理，同时保留模型的探索能力。

Abstract: Multimodal Large Language Models (MLLMs) are powerful at integrating diverse data, but they often struggle with complex reasoning. While Reinforcement learning (RL) can boost reasoning in LLMs, applying it to MLLMs is tricky. Common issues include a drop in performance on general tasks and the generation of overly detailed or “overthinking” reasoning. Our work investigates how the KL penalty and overthinking affect RL training in MLLMs. We propose Asymmetric Policy Optimization (APO) to address these issues, which divides the sampled responses into positive and negative groups. For positive samples, Difficulty-Adaptive Divergence Shaping (DADS) is introduced to dynamically adjust the KL divergence weight based on their difficulty. This method prevents policy entropy from dropping sharply, improves training stability, utilizes samples better, and preserves the model’s existing knowledge. For negative samples, Suboptimal Trajectory Complexity Regularization (STCR) is proposed to penalize overly long responses. This helps mitigate overthinking and encourages more concise reasoning while preserving the model’s explorative capacity. We apply our method to Qwen2.5-VL-3B, creating View-R1-3B. View-R1-3B significantly enhances reasoning capabilities, showing an average 7% gain over the base model and outperforming larger MLLMs (7-11B) on various reasoning benchmarks. Importantly, unlike other reasoning-tuned MLLMs that often degrade on general tasks, View-R1-3B maintains consistent improvement, demonstrating superior generalization. These results highlight the effectiveness and broad applicability of our DADS and STCR techniques for advancing complex multimodal reasoning in MLLMs. The code will be made available at https://github.com/Indolent-Kawhi/View-R1.

[133] SceneDiffuser++: City-Scale Traffic Simulation via a Generative World Model

Shuhan Tan,John Lambert,Hong Jeon,Sakshum Kulshrestha,Yijing Bai,Jing Luo,Dragomir Anguelov,Mingxing Tan,Chiyu Max Jiang

Main category: cs.LG

TL;DR: SceneDiffuser++是首个端到端的生成式世界模型，通过单一损失函数训练，实现城市规模的A点到B点交通模拟，整合了场景生成、动态代理行为建模和环境模拟等功能。

Details

Motivation: 目标是通过生成式模拟技术扩展有限的人工驾驶数据，实现城市规模的交通模拟，从而支持自动驾驶软件的测试与验证。

Contribution: 提出了首个端到端的生成式世界模型SceneDiffuser++，整合了场景生成、动态代理行为建模、遮挡推理、动态场景生成和环境模拟等多种功能。

Method: 采用单一损失函数训练的生成式世界模型，利用增强版的Waymo Open Motion Dataset进行训练，并支持长时模拟条件下的高真实感表现。

Result: 在城市规模的交通模拟中展现出卓越的真实性，尤其是在长时模拟条件下。

Insight: 通过整合多技术模块的端到端模型，能够更高效地实现复杂的城市交通模拟，为自动驾驶测试提供了更强大的工具。

Abstract: The goal of traffic simulation is to augment a potentially limited amount of manually-driven miles that is available for testing and validation, with a much larger amount of simulated synthetic miles. The culmination of this vision would be a generative simulated city, where given a map of the city and an autonomous vehicle (AV) software stack, the simulator can seamlessly simulate the trip from point A to point B by populating the city around the AV and controlling all aspects of the scene, from animating the dynamic agents (e.g., vehicles, pedestrians) to controlling the traffic light states. We refer to this vision as CitySim, which requires an agglomeration of simulation technologies: scene generation to populate the initial scene, agent behavior modeling to animate the scene, occlusion reasoning, dynamic scene generation to seamlessly spawn and remove agents, and environment simulation for factors such as traffic lights. While some key technologies have been separately studied in various works, others such as dynamic scene generation and environment simulation have received less attention in the research community. We propose SceneDiffuser++, the first end-to-end generative world model trained on a single loss function capable of point A-to-B simulation on a city scale integrating all the requirements above. We demonstrate the city-scale traffic simulation capability of SceneDiffuser++ and study its superior realism under long simulation conditions. We evaluate the simulation quality on an augmented version of the Waymo Open Motion Dataset (WOMD) with larger map regions to support trip-level simulation.

[134] Unfolding Generative Flows with Koopman Operators: Fast and Interpretable Sampling

Erkan Turan,Aristotelis Siozopoulos,Maks Ovsjanikov

Main category: cs.LG

TL;DR: 该论文提出了一种基于Koopman算子的生成流模型（Koopman-CFM），通过将非线性流建模为可学习观测空间中的线性演化，实现了快速且可解释的采样。相比传统条件流匹配（CFM），该方法显著提升了采样效率，并提供了生成过程的结构化分析工具。

Details

Motivation: 传统条件流匹配（CFM）依赖于数值求解非线性常微分方程（ODE），计算成本高且难以解释。现有的加速方法（如轨迹拉直或蒸馏）未揭示生成过程的结构。论文旨在解决这些问题，提出一种兼具高效采样和可解释性的方法。

Contribution: 1. 提出Koopman-CFM架构，通过Koopman算子理论将生成流线性化，实现一步闭式采样。2. 展示了该方法在2D数据集（MNIST、F-MNIST等）上的高效性。3. 通过谱分析提供生成行为的结构化解释（如时间缩放、模式稳定性等）。

Method: 1. 利用Koopman算子理论，在可学习观测空间中将非线性流建模为线性演化。2. 设计了无解码器的Koopman-CFM架构，通过矩阵指数实现单步采样。3. 分析了生成器的谱特性（如特征值和特征函数）以理解生成行为。

Result: 在2D合成数据集和真实数据集（MNIST、F-MNIST、TFD）上，Koopman-CFM显著优于传统CFM的采样速度，同时提供了对生成过程的谱分析能力。

Insight: Koopman算子理论为生成模型提供了一种将非线性动力学线性化的有效工具，不仅加速了采样，还增强了模型的可解释性。这种结合了高效性和结构分析的方法，为生成建模提供了新的思路。

Abstract: Conditional Flow Matching (CFM) offers a simulation-free framework for training continuous-time generative models, bridging diffusion and flow-based approaches. However, sampling from CFM still relies on numerically solving non-linear ODEs which can be computationally expensive and difficult to interpret. Recent alternatives address sampling speed via trajectory straightening, mini-batch coupling or distillation. However, these methods typically do not shed light on the underlying \textit{structure} of the generative process. In this work, we propose to accelerate CFM and introduce an interpretable representation of its dynamics by integrating Koopman operator theory, which models non-linear flows as linear evolution in a learned space of observables. We introduce a decoder-free Koopman-CFM architecture that learns an embedding where the generative dynamics become linear, enabling closed-form, one-step sampling via matrix exponentiation. This results in significant speedups over traditional CFM as demonstrated on controlled 2D datasets and real-world benchmarks, MNIST, Fashion-MNIST (F-MNIST), and the Toronto Face Dataset (TFD). Unlike previous methods, our approach leads to a well-structured Koopman generator, whose spectral properties, eigenvalues, and eigenfunctions offer principled tools for analyzing generative behavior such as temporal scaling, mode stability, and decomposition in Koopman latent space. By combining sampling efficiency with analytical structure, Koopman-enhanced flow matching offers a potential step toward fast and interpretable generative modeling.

[135] Probabilistic Optimality for Inference-time Scaling

Youkang Wang,Jian Wang,Rubing Chen,Xiao-Yong Wei,Qing Li

Main category: cs.LG

TL;DR: 论文提出了一种概率框架，用于推理时扩展的最优性，设计了一种动态确定最优样本数的算法OptScale，显著减少采样开销，同时在数学推理任务上保持或优于现有性能。

Details

Motivation: 现有推理时扩展方法多基于启发式策略，缺乏理论基础。论文旨在填补这一空白，为推理时扩展提供理论指导。

Contribution: 1. 提出了一个概率框架，形式化了推理时扩展的最优性。2. 推导了达到目标性能所需样本数的理论下界。3. 开发了动态确定最优样本数的算法OptScale。

Method: 1. 假设并行样本独立同分布，Best-of-N策略服从可估计的概率分布。2. 使用语言模型预测先验参数，动态决定满足性能阈值和置信水平的最小样本数。

Result: OptScale在数学推理基准（MATH-500等）上显著减少采样开销，性能与现有方法相当或更好。

Insight: 1. 推理时扩展需要理论支撑。2. 动态调整样本数量能提高计算效率。

Abstract: Inference-time scaling has emerged as a powerful technique for enhancing the reasoning performance of Large Language Models (LLMs). However, existing approaches often rely on heuristic strategies for parallel sampling, lacking a principled foundation. To address this gap, we propose a probabilistic framework that formalizes the optimality of inference-time scaling under the assumption that parallel samples are independently and identically distributed (i.i.d.), and where the Best-of-N selection strategy follows a probability distribution that can be estimated. Within this framework, we derive a theoretical lower bound on the required number of samples to achieve a target performance level, providing the first principled guidance for compute-efficient scaling. Leveraging this insight, we develop \textsc{OptScale}, a practical algorithm that dynamically determines the optimal number of sampled responses. \textsc{OptScale} employs a language model-based predictor to estimate probabilistic prior parameters, enabling the decision of the minimal number of samples needed that satisfy predefined performance thresholds and confidence levels. Extensive experiments on mathematical reasoning benchmarks (including MATH-500, GSM8K, AIME, and AMC) demonstrate that \textsc{OptScale} significantly reduces sampling overhead while remaining better or on par with state-of-the-art reasoning performance. Our work offers both a theoretical foundation and a practical solution for principled inference-time scaling, addressing a critical gap in the efficient deployment of LLMs for complex reasoning.

cs.MM [Back]

[136] RiverEcho: Real-Time Interactive Digital System for Ancient Yellow River Culture

Haofeng Wang,Yilin Guo,Zehao Li,Tong Yue,Yizong Wang,Enci Zhang,Rongqun Lin,Feng Gao,Shiqi Wang,Siwei Ma

Main category: cs.MM

TL;DR: 本文提出了RiverEcho，一个基于大语言模型和文化知识数据集的实时交互系统，用于保护和传承古代黄河文化。系统通过语音查询和数字人讲解提供专业信息。

Details

Motivation: 黄河文化作为人类文明的重要组成部分，需要现代技术手段加以保护和传承。通过构建实时交互系统，可以多样化传播方式并提供更深层次的文化洞察。

Contribution: 1. 构建了专注于古代黄河文化的知识数据库；2. 设计了结合RAG技术和大语言模型的实时交互系统；3. 通过数字人讲解提升了用户体验和文化传播效果。

Method: 1. 收集并处理古代黄河文化的历史文本数据；2. 应用RAG技术增强大语言模型的响应质量；3. 通过语音交互和数字人讲解实现实时互动。

Result: 实验表明，RAG技术显著提升了大语言模型在黄河文化知识上的响应质量，系统生成的回答更专业、信息更丰富。

Insight: 结合现代技术与传统文化数据库可以有效提升文化传播的深度和广度，同时为用户提供更具吸引力的互动体验。

Abstract: The Yellow River is China’s mother river and a cradle of human civilization. The ancient Yellow River culture is, moreover, an indispensable part of human art history. To conserve and inherit the ancient Yellow River culture, we designed RiverEcho, a real-time interactive system that responds to voice queries using a large language model and a cultural knowledge dataset, delivering explanations through a talking-head digital human. Specifically, we built a knowledge database focused on the ancient Yellow River culture, including the collection of historical texts and the processing pipeline. Experimental results demonstrate that leveraging Retrieval-Augmented Generation (RAG) on the proposed dataset enhances the response quality of the Large Language Model(LLM), enabling the system to generate more professional and informative responses. Our work not only diversifies the means of promoting Yellow River culture but also provides users with deeper cultural insights.

physics.optics [Back]

[137] Inverse Design of Diffractive Metasurfaces Using Diffusion Models

Liav Hen,Erez Yosef,Dan Raviv,Raja Giryes,Jacob Scheuer

Main category: physics.optics

TL;DR: 该论文提出了一种基于扩散模型的衍射超表面逆向设计方法，解决了传统设计中的局部极小值和计算开销大的问题，实现了快速且低误差的超表面设计。

Details

Motivation: 传统的衍射超表面逆向设计方法由于结构和光学性能之间的复杂非线性关系，通常需要专家调整，容易陷入局部极小值，且计算开销大。论文旨在通过引入扩散模型的生成能力，解决这些问题。

Contribution: 主要贡献包括：1) 提出了一种基于扩散模型的超表面逆向设计方法；2) 展示了该方法在均匀分束器和偏振分束器设计中的有效性；3) 公开了代码和数据集以支持进一步研究。

Method: 论文使用RCWA模拟器生成训练数据（包括超表面几何形状和对应的远场散射模式），并训练一个条件扩散模型，根据目标空间功率分布预测元原子几何形状和高度。训练完成后，模型可直接生成低误差超表面，或为传统优化方法提供初始化。

Result: 实验结果表明，该方法可以在30分钟内设计出低误差的空间均匀分束器和偏振分束器，证明了其高效性。

Insight: 扩散模型在超表面逆向设计中展现出了强大的生成能力，能够有效替代或辅助传统优化方法，显著降低计算开销和设计时间。

Abstract: Metasurfaces are ultra-thin optical elements composed of engineered sub-wavelength structures that enable precise control of light. Their inverse design - determining a geometry that yields a desired optical response - is challenging due to the complex, nonlinear relationship between structure and optical properties. This often requires expert tuning, is prone to local minima, and involves significant computational overhead. In this work, we address these challenges by integrating the generative capabilities of diffusion models into computational design workflows. Using an RCWA simulator, we generate training data consisting of metasurface geometries and their corresponding far-field scattering patterns. We then train a conditional diffusion model to predict meta-atom geometry and height from a target spatial power distribution at a specified wavelength, sampled from a continuous supported band. Once trained, the model can generate metasurfaces with low error, either directly using RCWA-guided posterior sampling or by serving as an initializer for traditional optimization methods. We demonstrate our approach on the design of a spatially uniform intensity splitter and a polarization beam splitter, both produced with low error in under 30 minutes. To support further research in data-driven metasurface design, we publicly release our code and datasets.

eess.IV [Back]

[138] TUS-REC2024: A Challenge to Reconstruct 3D Freehand Ultrasound Without External Tracker

Qi Li,Shaheer U. Saeed,Yuliang Huang,Mingyuan Luo,Zhongnuo Yan,Jiongquan Chen,Xin Yang,Dong Ni,Nektarios Winter,Phuc Nguyen,Lucas Steinberger,Caelan Haney,Yuan Zhao,Mingjie Jiang,Bowen Ren,SiYeoul Lee,Seonho Kim,MinKyung Seo,MinWoo Kim,Yimeng Dou,Zhiwei Zhang,Yin Li,Tomy Varghese,Dean C. Barratt,Matthew J. Clarkson,Tom Vercauteren,Yipeng Hu

Main category: eess.IV

TL;DR: 该论文介绍了TUS-REC2024挑战赛，旨在通过公开数据集和评估框架推动无追踪器的自由手超声3D重建技术发展，吸引了多支团队参与，并总结了当前方法的进展与局限。

Details

Motivation: 无追踪器的自由手超声重建技术具有低成本、便携性和广泛部署的优势，但在帧间运动估计和漂移累积等方面面临挑战。挑战赛旨在通过公开基准数据集加速技术发展。

Contribution: 1. 首次提供公开的无追踪器自由手超声数据集；2. 建立基线模型和评估框架；3. 总结多种算法方法的比较分析。

Method: 挑战赛采用了多种算法方法，包括循环模型、配准驱动的体积优化、注意力机制和物理启发模型，并在公开数据集上进行了评估。

Result: 结果表明，现有方法在3D重建方面取得了一定进展，但仍存在漂移累积和通用性等局限。

Insight: 无追踪器自由手超声重建技术仍需进一步研究以解决漂移问题和提升跨扫描协议的通用性。公开数据集和持续挑战赛将推动该领域发展。

Abstract: Trackerless freehand ultrasound reconstruction aims to reconstruct 3D volumes from sequences of 2D ultrasound images without relying on external tracking systems, offering a low-cost, portable, and widely deployable alternative for volumetric imaging. However, it presents significant challenges, including accurate inter-frame motion estimation, minimisation of drift accumulation over long sequences, and generalisability across scanning protocols. The TUS-REC2024 Challenge was established to benchmark and accelerate progress in trackerless 3D ultrasound reconstruction by providing a publicly available dataset for the first time, along with a baseline model and evaluation framework. The Challenge attracted over 43 registered teams, of which 6 teams submitted 21 valid dockerized solutions. Submitted methods spanned a wide range of algorithmic approaches, including recurrent models, registration-driven volume refinement, attention, and physics-informed models. This paper presents an overview of the Challenge design, summarises the key characteristics of the dataset, provides a concise literature review, introduces the technical details of the underlying methodology working with tracked freehand ultrasound data, and offers a comparative analysis of submitted methods across multiple evaluation metrics. The results highlight both the progress and current limitations of state-of-the-art approaches in this domain, and inform directions for future research. The data, evaluation code, and baseline are publicly available to facilitate ongoing development and reproducibility. As a live and evolving benchmark, this Challenge is designed to be continuously developed and improved. The Challenge was held at MICCAI 2024 and will be organised again at MICCAI 2025, reflecting its growing impact and the sustained commitment to advancing this field.

[139] UnMix-NeRF: Spectral Unmixing Meets Neural Radiance Fields

Fabian Perez,Sara Rojas,Carlos Hinojosa,Hoover Rueda-Chacón,Bernard Ghanem

Main category: eess.IV

TL;DR: UnMix-NeRF 结合了光谱分解与神经辐射场，实现了高光谱新视图合成和无监督材料分割，解决了传统 NeRF 方法缺乏材料感知的问题。

Details

Motivation: 传统 NeRF 主要依赖 RGB 数据，无法准确感知材质特性，限制了其在机器人、增强现实等应用中的潜力。需要一种方法同时实现高质量渲染和材料属性理解。

Contribution: 1. 首次将光谱分解引入 NeRF，支持高光谱新视图合成和材料分割；2. 提出无监督材料聚类方法；3. 支持基于材质的场景编辑。

Method: 通过建模漫反射和镜面反射的光谱反射率，学习全局端元字典表示纯材料特征，并利用点级丰度分布。使用端元预测光谱特征，实现无监督材料分割。

Result: 实验表明，UnMix-NeRF 在光谱重建和材料分割方面优于现有方法，并支持灵活的材质编辑。

Insight: 将光谱特性与 NeRF 结合，不仅提升了渲染质量，还为场景理解和编辑提供了新工具。

Abstract: Neural Radiance Field (NeRF)-based segmentation methods focus on object semantics and rely solely on RGB data, lacking intrinsic material properties. This limitation restricts accurate material perception, which is crucial for robotics, augmented reality, simulation, and other applications. We introduce UnMix-NeRF, a framework that integrates spectral unmixing into NeRF, enabling joint hyperspectral novel view synthesis and unsupervised material segmentation. Our method models spectral reflectance via diffuse and specular components, where a learned dictionary of global endmembers represents pure material signatures, and per-point abundances capture their distribution. For material segmentation, we use spectral signature predictions along learned endmembers, allowing unsupervised material clustering. Additionally, UnMix-NeRF enables scene editing by modifying learned endmember dictionaries for flexible material-based appearance manipulation. Extensive experiments validate our approach, demonstrating superior spectral reconstruction and material segmentation to existing methods. Project page: https://www.factral.co/UnMix-NeRF.

[140] Towards Scalable and Robust White Matter Lesion Localization via Multimodal Deep Learning

Julia Machnio,Sebastian Nørgaard Llambias,Mads Nielsen,Mostafa Mehdipour Ghazi

Main category: eess.IV

TL;DR: 该论文提出了一种基于多模态深度学习的白质病变定位方法，通过多任务学习联合预测病变和解剖区域掩模，实验表明多模态输入显著提升了分割性能，但联合学习效果不如单独任务模型。

Details

Motivation: 白质高信号（WMH）是小血管疾病和神经退行性疾病的影像学标志，其准确分割和空间定位对诊断和监测至关重要。现有方法在处理缺失模态和高效整合解剖定位方面表现不足，因此需要一种更灵活且鲁棒的框架。

Contribution: 1. 提出一种直接处理单模态和多模态MRI输入的白质病变分割与定位框架；2. 评估了四种输入配置（FLAIR-only、T1-only、多模态等）；3. 引入多任务模型联合预测病变和解剖区域掩模。

Method: 1. 设计深度学习框架，支持单模态和多模态输入；2. 测试了四种输入配置；3. 引入多任务学习联合预测病变和解剖区域掩模；4. 使用MICCAI WMH分割挑战数据集进行实验验证。

Result: 多模态输入显著优于单模态模型，但模态可互换设置以部分准确性为代价提升了鲁棒性。多任务学习在联合预测中效果不如单独任务模型。

Insight: 多模态融合对WMH分析至关重要，而任务间的表征冲突可能导致联合学习效果不佳，未来研究需进一步优化多任务整合策略。

Abstract: White matter hyperintensities (WMH) are radiological markers of small vessel disease and neurodegeneration, whose accurate segmentation and spatial localization are crucial for diagnosis and monitoring. While multimodal MRI offers complementary contrasts for detecting and contextualizing WM lesions, existing approaches often lack flexibility in handling missing modalities and fail to integrate anatomical localization efficiently. We propose a deep learning framework for WM lesion segmentation and localization that operates directly in native space using single- and multi-modal MRI inputs. Our study evaluates four input configurations: FLAIR-only, T1-only, concatenated FLAIR and T1, and a modality-interchangeable setup. It further introduces a multi-task model for jointly predicting lesion and anatomical region masks to estimate region-wise lesion burden. Experiments conducted on the MICCAI WMH Segmentation Challenge dataset demonstrate that multimodal input significantly improves the segmentation performance, outperforming unimodal models. While the modality-interchangeable setting trades accuracy for robustness, it enables inference in cases with missing modalities. Joint lesion-region segmentation using multi-task learning was less effective than separate models, suggesting representational conflict between tasks. Our findings highlight the utility of multimodal fusion for accurate and robust WMH analysis, and the potential of joint modeling for integrated predictions.

[141] Advanced Deep Learning Techniques for Automated Segmentation of Type B Aortic Dissections

Hao Xu,Ruth Lim,Brian E. Chapman

Main category: eess.IV

TL;DR: 论文提出了四种基于深度学习的自动化分割管道，用于B型主动脉夹层的精确分割，性能优于现有方法。

Details

Motivation: 主动脉夹层是一种危及生命的心血管疾病，手动分割耗时且不一致，需要自动化解决方案以提高效率和准确性。

Contribution: 提出了四种深度学习框架（单步模型、序列模型、序列多任务模型和集成模型），并验证了其在B型主动脉夹层分割中的优越性。

Method: 采用了3D U-Net和Swin-UnetR架构，在包含100例CTA图像的数据集上评估了不同管道的性能。

Result: Dice系数分别为TL 0.91、FL 0.88、FLT 0.47，性能显著优于对比方法。

Insight: 深度学习模型能够有效分割复杂的心血管结构，尤其在自动提取形态参数方面具有潜力，可用于临床监测和治疗规划。

Abstract: Purpose: Aortic dissections are life-threatening cardiovascular conditions requiring accurate segmentation of true lumen (TL), false lumen (FL), and false lumen thrombosis (FLT) from CTA images for effective management. Manual segmentation is time-consuming and variable, necessitating automated solutions. Materials and Methods: We developed four deep learning-based pipelines for Type B aortic dissection segmentation: a single-step model, a sequential model, a sequential multi-task model, and an ensemble model, utilizing 3D U-Net and Swin-UnetR architectures. A dataset of 100 retrospective CTA images was split into training (n=80), validation (n=10), and testing (n=10). Performance was assessed using the Dice Coefficient and Hausdorff Distance. Results: Our approach achieved superior segmentation accuracy, with Dice Coefficients of 0.91 $\pm$ 0.07 for TL, 0.88 $\pm$ 0.18 for FL, and 0.47 $\pm$ 0.25 for FLT, outperforming Yao et al. (1), who reported 0.78 $\pm$ 0.20, 0.68 $\pm$ 0.18, and 0.25 $\pm$ 0.31, respectively. Conclusion: The proposed pipelines provide accurate segmentation of TBAD features, enabling derivation of morphological parameters for surveillance and treatment planning

[142] Cardiovascular disease classification using radiomics and geometric features from cardiac CT

Ajay Mittal,Raghav Mehta,Omar Todd,Philipp Seeböck,Georg Langs,Ben Glocker

Main category: eess.IV

TL;DR: 该论文提出了一种基于放射组学和几何特征的心脏CT心血管疾病分类方法，通过图像分割、配准和分类三个步骤，提升了分类准确率和临床可解释性。

Details

Motivation: 目前大多数基于深度学习的CVD分类方法直接使用原始CT数据或结合解剖结构分割，缺乏临床可解释性。论文旨在通过引入放射组学和几何特征，提高分类准确率的同时增强方法的可解释性。

Contribution: 1. 将CVD分类流程分解为图像分割、配准和分类三个步骤；2. 利用Atlas-ISTN框架和基础分割模型提取放射组学和几何特征；3. 在公开数据集ASOCA上实现了优于直接使用原始CT数据的分类准确率（87.50% vs 67.50%）。

Method: 1. 使用Atlas-ISTN和基础分割模型生成解剖结构分割和健康图谱；2. 提取放射组学特征和基于配准的几何特征；3. 利用这些特征训练CVD分类模型。

Result: 在ASOCA数据集上，所提方法的分类准确率达到87.50%，显著高于直接使用原始CT数据的67.50%。

Insight: 放射组学和几何特征的引入不仅提高了CVD分类的准确率，还增强了模型在临床中的可解释性，为医学图像分析提供了更透明的方法。

Abstract: Automatic detection and classification of Cardiovascular disease (CVD) from Computed Tomography (CT) images play an important part in facilitating better-informed clinical decisions. However, most of the recent deep learning based methods either directly work on raw CT data or utilize it in pair with anatomical cardiac structure segmentation by training an end-to-end classifier. As such, these approaches become much more difficult to interpret from a clinical perspective. To address this challenge, in this work, we break down the CVD classification pipeline into three components: (i) image segmentation, (ii) image registration, and (iii) downstream CVD classification. Specifically, we utilize the Atlas-ISTN framework and recent segmentation foundational models to generate anatomical structure segmentation and a normative healthy atlas. These are further utilized to extract clinically interpretable radiomic features as well as deformation field based geometric features (through atlas registration) for CVD classification. Our experiments on the publicly available ASOCA dataset show that utilizing these features leads to better CVD classification accuracy (87.50%) when compared against classification model trained directly on raw CT images (67.50%). Our code is publicly available: https://github.com/biomedia-mira/grc-net

[143] DIGS: Dynamic CBCT Reconstruction using Deformation-Informed 4D Gaussian Splatting and a Low-Rank Free-Form Deformation Model

Yuliang Huang,Imraj Singh,Thomas Joyce,Kris Thielemans,Jamie R. McClelland

Main category: eess.IV

TL;DR: 这篇论文提出了一种动态CBCT重建方法，结合了变形感知的4D高斯泼溅和低秩自由形变模型，以解决呼吸运动引起的伪影问题，并通过6倍加速和更优的图像质量验证了方法的有效性。

Details

Motivation: CBCT在放疗中广泛应用，但呼吸运动会导致伪影。现有的基于呼吸相的分段重建方法无法捕捉呼吸的变异性，而动态CBCT虽然能捕捉连续运动，但现有4D高斯泼溅方法的计算成本高，且缺乏空间一致性。

Contribution: 1) 提出了一种基于自由形变的低秩空间基函数；2) 开发了变形感知框架，统一高斯均值、尺度和旋转的变化；3) 在6个CBCT数据集上验证了方法的图像质量和计算效率优势。

Method: 结合自由形变（FFD）模型和4D高斯泼溅（4DGS），通过低秩表示和空间正则化实现动态CBCT重建。

Result: 相比HexPlane方法，图像质量更优，且计算速度提升了6倍。

Insight: 变形感知的4DGS能够有效平衡动态场景建模的计算效率和空间一致性，为动态CBCT重建提供了新思路。

Abstract: 3D Cone-Beam CT (CBCT) is widely used in radiotherapy but suffers from motion artifacts due to breathing. A common clinical approach mitigates this by sorting projections into respiratory phases and reconstructing images per phase, but this does not account for breathing variability. Dynamic CBCT instead reconstructs images at each projection, capturing continuous motion without phase sorting. Recent advancements in 4D Gaussian Splatting (4DGS) offer powerful tools for modeling dynamic scenes, yet their application to dynamic CBCT remains underexplored. Existing 4DGS methods, such as HexPlane, use implicit motion representations, which are computationally expensive. While explicit low-rank motion models have been proposed, they lack spatial regularization, leading to inconsistencies in Gaussian motion. To address these limitations, we introduce a free-form deformation (FFD)-based spatial basis function and a deformation-informed framework that enforces consistency by coupling the temporal evolution of Gaussian’s mean position, scale, and rotation under a unified deformation field. We evaluate our approach on six CBCT datasets, demonstrating superior image quality with a 6x speedup over HexPlane. These results highlight the potential of deformation-informed 4DGS for efficient, motion-compensated CBCT reconstruction. The code is available at https://github.com/Yuliang-Huang/DIGS.

stat.ML [Back]

[144] Optimal Estimation of Watermark Proportions in Hybrid AI-Human Texts

Xiang Li,Garrett Wen,Weiqing He,Jiayuan Wu,Qi Long,Weijie J. Su

Main category: stat.ML

TL;DR: 本文研究了混合AI-人类文本中水印比例的最优估计问题，首次提出了基于关键统计的混合模型参数估计方法，并证明了在水印方法满足条件下比例参数的可识别性。

Details

Motivation: 随着大语言模型的广泛应用，识别合成文本与水印内容成为重要课题。然而，现有研究多集中于纯文本水印检测，而实际场景中常出现混合来源的文本。本文旨在解决混合文本中水印比例的最优估计问题。

Contribution: 1）首次提出混合文本中水印比例的最优估计问题；2）证明了在水印方法使用连续关键统计时比例参数的可识别性；3）提出了高效估计器，并推导了极小极大下界，展示其最优性。

Method: 将问题建模为基于关键统计的混合模型参数估计，提出高效的估计器，并通过理论分析和实验验证其性能。

Result: 在合成数据和开源模型生成的混合文本上，所提估计器表现出高精度。

Insight: 水印方法的可识别性取决于关键统计的性质，连续统计方法在实际应用中更具优势。

Abstract: Text watermarks in large language models (LLMs) are an increasingly important tool for detecting synthetic text and distinguishing human-written content from LLM-generated text. While most existing studies focus on determining whether entire texts are watermarked, many real-world scenarios involve mixed-source texts, which blend human-written and watermarked content. In this paper, we address the problem of optimally estimating the watermark proportion in mixed-source texts. We cast this problem as estimating the proportion parameter in a mixture model based on \emph{pivotal statistics}. First, we show that this parameter is not even identifiable in certain watermarking schemes, let alone consistently estimable. In stark contrast, for watermarking methods that employ continuous pivotal statistics for detection, we demonstrate that the proportion parameter is identifiable under mild conditions. We propose efficient estimators for this class of methods, which include several popular unbiased watermarks as examples, and derive minimax lower bounds for any measurable estimator based on pivotal statistics, showing that our estimators achieve these lower bounds. Through evaluations on both synthetic data and mixed-source text generated by open-source models, we demonstrate that our proposed estimators consistently achieve high estimation accuracy.