soofi-project/Soofi-Pretraining
Scripts, configs and documentation for the pre- and midtraining activities of the Soofi Project. - soofi-project/Soofi-Pretraining
24 stars · 3 forks · Shell
README
Soofi Pretraining This repository contains scripts, documentation, debugging, etc. of the pretraining for the Soofi project. The current code release contains the complete codebase used for training Soofi S a 30B model based on Nvidia's Nemotron 3 Nano architecture.c The relevant training code can be found in: Preprocessing
src/preprocessing — corpus construction, filtering, tokenization, and mixture assembly
Training
pretraining/08_nemotron_3_nano_on_20T midtraining/02_nemotron_3_nano_annealing midtraining/03_nemotron_3_nano_more_annealing midtraining/04_nemotron_3_nano_final_annealing midtraining/01_nemotron_3_nano_long_context_training
Open model weights coming soon.