VISReg: Variance-Invariance-Sketching Regularization for JEPA training
arxiv:2606.02572
Copy markdown
VISReg: Variance-Invariance-Sketching Regularization for JEPA training
Published on Jun 1
Authors:
Haiyu Wu ,
Randall Balestriero ,
Morgan Levine
Abstract
VISReg combines variance and sketching regularization to improve self-supervised learning by enforcing full distributional shape while maintaining training stability.
Generated by Qwen/Qwen2.5-Coder-32B-Instruct
Self-supervised learning methods prevent embedding collapse via modeling heuristics or explicit regularization of the embedding space. Among the latter, VICReg decomposes regularization into variance and covariance objectives, offering flexibility and interpretability. However, covariance captures only second-order statistics -- encouraging decorrelation but failing to enforce the full distributional shape needed for stable training. Sketching-based methods such as SIGReg address this by aligning embeddings to an isotropic Gaussian, but lack flexibility and suffer from vanishing gradients under collapse. We propose Variance-Invariance-Sketching Regularization (VISReg), which replaces covariance with a Sliced-Wasserstein\-based sketching objective that enforces full distributional shape, while retaining a variance term for scale control. By decoupling scale and shape, VISReg combines VICReg's flexibility with the distributional rigor of sketching methods, providing robust gradients even under collapse. We show that VISReg scales linearly, outperforms existing regularization on low-quality datasets, and is resilient to long-tailed and low-rank regimes. Pre-trained on ImageNet-1K, VISReg achieves state-of-the-art performance on out-of-distribution datasets. Pre-trained on ImageNet-22K, it matches DINOv2's OOD performance despite the latter using 10x more data (LVD-142M). Project and code: https://haiyuwu.github.io/visreg.
View arXiv page View PDF Add to collection
Community
EditPreview
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
Comment
· Sign up or log in to comment
Get this paper in your agent:
`hf papers read 2606.02572`
Don't have the latest CLI?
`curl -LsSf https://hf.co/cli/install.sh | bash`
Models citing this paper 1
Datasets citing this paper 0
No dataset linking this paper
Cite arxiv.org/abs/2606.02572 in a dataset README.md to link it from this page.
Spaces citing this paper 0
No Space linking this paper
Cite arxiv.org/abs/2606.02572 in a Space README.md to link it from this page.
Collections including this paper 0
No Collection including this paper
Add this paper to a collection to link it from this page.