facebookresearch/context-language-models
Official repository for "Context Language Models". Contribute to facebookresearch/context-language-models development by creating an account on GitHub.
105 stars · 15 forks
README
Context Language Models (CLMs)
Rulin Shao1,2, Shannon Zejiang Shen3, Junjie Oscar Yin1,2, Yuetai Li1, Minheng Wang1, Hamish Ivison1, Radha Poovendran1, Nathan Lambert4, Teng Xiao1, Mike Lewis2, Wen-tau Yih2, Luke Zettlemoyer1,2, Pang Wei Koh1
1University of Washington 2Meta Superintelligence Labs 3MIT 4Trillium Labs
We introduce Context Language Models (CLMs), language models that natively manage their own context. We implement this by treating the context as a file and allowing the model to make unrestricted updates to this file. This allows the model to learn what is most important to maintain in context, and naturally extends to multi-agent systems where multiple agent contexts coexist as files.
Zero-shot. Building CLMs zero-shot with existing models outperforms SOTA context-management strategies across a variety of tasks: 11.4% higher accuracy with 21.5% fewer FLOPs on BrowseComp-Plus, 5% higher scores with 59% fewer FLOPs on 12-hour EdgeBench, and 65% greater improvement with the same compute on a 24-hour multi-repository agent-swarm task. In-context learning. We show that CLMs can be steered with natural-language instructions evolved through a standard skill-optimization loop, improving held-out accuracy by up to 35.9 points on a context-management task while reducing compute. Reinforcement learning. We also introduce an online reinforcement learning method for CLMs, improving Qwen3.5-9B performance on BrowseComp-Plus by 47.6% while using 12% fewer FLOPs.
Getting started Run the minimal CLM agent on any Harbor task: pip install -e . clm-harbor run -p <harbor-task> -a clm-minimal -m openai/<model> \ --agent-kwarg api_base=http://localhost:8000/v1 clm-harbor is the Harbor CLI with CLM available as -a clm-minimal. See clm/clm_harness for configuration and serving. Day 1 support: pi-clm for Pi agent pi install npm:@lolipopshock/pi-clm Repository
clm/clm_harness CLM implemented in Harbor
clm/clm_icl skill evolution
clm/clm_rl reinforcement learning
suffix_cache_reuse Suffix Cache Reuse: KV-cache reuse for CLM serving, as a patch to SGLang
Coming soon
ContextBench
Citation If you find our work helpful, we would appreciate it if you could cite our paper: @article{shao2026context, title = {Context Language Models}, author = {Shao, Rulin and Shen, Shannon Zejiang and Yin, Junjie Oscar and Li, Yuetai and Wang, Minheng and Ivison, Hamish and Poovendran, Radha and Lambert, Nathan and Xiao, Teng and Lewis, Mike and Yih, Wen-tau and Zettlemoyer, Luke and Koh, Pang Wei}, journal = {arXiv preprint arXiv:2609.37725}, year = {2026} } License This project is licensed under CC BY-NC 4.0. See also NOTICE.