Table of Contents

cs.AI [Back]

[1] Self-Play Pretraining with Zero Data cs.AI | cs.CLPDF

Aditya Cowsik, Kfir Dolev, Michael Y. Li, G. Bruno De Luca, Nourya Cohen

TL;DR: 以零人工语料自博弈预训练，让生成器搜索可计算程序产生训练数据并为语言模型自适应安排课程。

Details

Motivation: 语言模型预训练依赖人类整理数据，作者探索能否由模型自己生成有助于学习的、理论上由算力限制的训练来源。

Result: 从随机初始化训练时，多个自然数据集上的零样本损失随自博弈算力可预测扩展；模型还表现出上下文学习并发现可辨识数学序列。

Insight: 生成器提出程序交由通用图灵机执行产生字节序列，学习器用交叉熵预测，生成器则通过RL寻找接近学习器能力前沿的样本；两者均未用自然语料训练，使迁移检验更直接。

Abstract: Advances in language modeling have been driven by scaling pretraining on ever more data. Yet, the training data is still largely curated on the model’s behalf. A more general approach to pretraining would let the model learn to generate the data most useful for its own improvement. This would provide an effectively unbounded source of training data, limited by compute rather than human knowledge. We introduce Self-Play Pretraining with Zero Data, an initial proof-of-concept towards realizing this vision. Our procedure casts synthetic data generation as a search over the space of all computable structure, taking inspiration from Solomonoff induction. Starting from random initialization, two models learn in tandem: a generator proposes programs interpreted by a universal Turing machine, generating byte sequences, while a learner autoregressively predicts these byte sequences. The learner is trained with standard cross-entropy, while the generator is trained with reinforcement learning to produce sequences at the frontier of the learner’s capabilities, yielding an adaptive curriculum. A universal Turing machine gives us a search space over all computable data-generating processes, imposing little domain-specific structure, and self-play searches over this space for useful training data. We test whether zero-shot performance on natural data improves predictably with self-play compute; this is a clean test of transfer since neither generator nor learner is trained on natural data. Across several natural datasets, zero-shot loss exhibits predictable scaling in compute. The models also exhibit in-context learning, and discover recognizable mathematical sequences during training.

[2] Training Object Permanence in World Models cs.AI | cs.CVPDF

Haotian Zhang, Fengyuan Yu, Dezhi Luo, Haoran Sun, Zehong Zhao

TL;DR: 以认知科学启发任务数据训练世界模型的物体恒存能力，评测其在遮挡和物体消失情境中的物理推理。

Details

Motivation: 视频生成世界模型可能出现涌现推理，但是否具备人类认知中的物体恒存和实体性尚未明确。

Result: WROP含150项任务、六类认知能力、每项超过1万样例，并发布150万样本训练语料和300题测验；16B的PWM-WROP在盲测Elo比较中续写模型排名第一、总体第三。

Insight: Blender生成器随机化速度、光照、视角等干扰因素但保留任务认知结构；项目公开数据、测验、模型输出、权重及原生PyTorch训练栈。

Abstract: Object permanence and solidity are hallmarks of human cognitive priors. Recent studies show that video generation models, a paradigmatic class of current world models, have begun to show emerged reasoning abilities, making them ideal candidates for building human-like physical intelligence. Do video models have emerged object permanence in them? If not, could we train them with a core-cognition inspired dataset? We introduce WROP (World Reasoning with Object Permanence), a data infrastructure of 150 hand-designed cognitive science inspired tasks, divided into six cognitive categories. We build Blender generators that randomize speed, lighting, camera angle, and other nuisance parameters while preserving each task’s cognitive structure, yielding 10,000+ samples per task. We release a 1.5M-sample training corpus and a 300-question exam. On this exam we evaluate 14 video models: 3 reference-to-video, 7 edit, and 4 continuation, among which PWM-WROP, our 16B world model. In a blind pairwise Elo study, PWM-WROP ranks first among continuation models and third overall, behind only a statistical tie between two reference-to-video models. We release the data, exam, model answers, scores, weights, and PWM, our native-PyTorch training stack on AWS Trainium2.