Weight-space geometry encodes traces of training data. Can we use it to reverse-engineer data recipes? Introducing WARP: a new strategy to estimate domain mixtures from model weights alone!</p>\n","updatedAt":"2026-07-03T15:59:25.850Z","author":{"_id":"6363edf3de4bc2f294accb16","avatarUrl":"/avatars/9baddfec7170ec662e974116f561ab2c.svg","fullname":"Tzu-Heng Huang","name":"zihengh1","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":4,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.8421685099601746},"editors":["zihengh1"],"editorAvatarUrls":["/avatars/9baddfec7170ec662e974116f561ab2c.svg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2607.01686","authors":[{"_id":"6a47dc363daa34221c7c1bf5","name":"Tzu-Heng Huang","hidden":false},{"_id":"6a47dc363daa34221c7c1bf6","name":"Aditya Goyal","hidden":false},{"_id":"6a47dc363daa34221c7c1bf7","name":"John Cooper","hidden":false},{"_id":"6a47dc363daa34221c7c1bf8","name":"Frederic Sala","hidden":false}],"publishedAt":"2026-07-02T00:00:00.000Z","submittedOnDailyAt":"2026-07-03T00:00:00.000Z","title":"WARP: Weight-Space Analysis for Recovering Training Data Portfolios","submittedOnDailyBy":{"_id":"6363edf3de4bc2f294accb16","avatarUrl":"/avatars/9baddfec7170ec662e974116f561ab2c.svg","isPro":false,"fullname":"Tzu-Heng Huang","user":"zihengh1","type":"user","name":"zihengh1"},"summary":"Foundation models are routinely released to the public, yet the data recipes used to train them -- such as domain mixture weights that determine how different sources are sampled -- are rarely disclosed. This creates an access asymmetry: researchers study the resulting models but lack visibility into the training distribution that produces them. Prior works for inferring training data, such as membership inference, detect at the level of individual samples and thus cannot characterize the global composition of the training corpus. We introduce WARP, a framework that recovers a fine-tuned model's training mixtures directly from its released weights. WARP interpolates between the base and fine-tuned models using model merging, generating pseudo-checkpoints that approximate the missing training trajectory and expose a geometric footprint of the training data in the weight space. From these simulated footprints, WARP extracts geometric features and maps them to domain proportions using either a parameter-free softmax readout or an MLP projector trained on synthetic mixtures. In controlled experiments with BERT and GPT-2, WARP recovers domain mixtures with an average MAE as low as 0.046 and 0.104 respectively, outperforming membership inference and a variant with access to the true training trajectory.","upvotes":3,"discussionId":"6a47dc373daa34221c7c1bf9","githubRepo":"https://github.com/SprocketLab/WARP","githubRepoAddedBy":"user","ai_summary":"WARP is a framework that infers training data compositions from released model weights by analyzing geometric footprints in weight space through model merging and feature extraction.","ai_keywords":["foundation models","model merging","pseudo-checkpoints","weight space","geometric footprint","geometric features","softmax readout","MLP projector","membership inference","training trajectory"],"ai_summary_model":"Qwen/Qwen2.5-Coder-32B-Instruct","githubStars":2,"organization":{"_id":"63aaa3e7b7f3e1c60728194a","name":"sprocket-lab","fullname":"sprocket-lab","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/629436f4fa1501cdf1a1086e/kpkvUxJsH_R9LHSOk6WdP.jpeg"}},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"6363edf3de4bc2f294accb16","avatarUrl":"/avatars/9baddfec7170ec662e974116f561ab2c.svg","isPro":false,"fullname":"Tzu-Heng Huang","user":"zihengh1","type":"user"},{"_id":"68416ddcafc844c48a921225","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/CkamXjlcuth0abSWY2EGu.png","isPro":false,"fullname":"Aditya Goyal","user":"AdityaGtheOg","type":"user"},{"_id":"698e15b1f2dc6c20c89e3d19","avatarUrl":"/avatars/12e456599cb74c43ded4d97da855527e.svg","isPro":false,"fullname":"Shengqi Qiu","user":"abeQ213","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"organization":{"_id":"63aaa3e7b7f3e1c60728194a","name":"sprocket-lab","fullname":"sprocket-lab","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/629436f4fa1501cdf1a1086e/kpkvUxJsH_R9LHSOk6WdP.jpeg"},"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2607/2607.01686.md","query":{}}">
WARP: Weight-Space Analysis for Recovering Training Data Portfolios
Abstract
WARP is a framework that infers training data compositions from released model weights by analyzing geometric footprints in weight space through model merging and feature extraction.
Foundation models are routinely released to the public, yet the data recipes used to train them -- such as domain mixture weights that determine how different sources are sampled -- are rarely disclosed. This creates an access asymmetry: researchers study the resulting models but lack visibility into the training distribution that produces them. Prior works for inferring training data, such as membership inference, detect at the level of individual samples and thus cannot characterize the global composition of the training corpus. We introduce WARP, a framework that recovers a fine-tuned model's training mixtures directly from its released weights. WARP interpolates between the base and fine-tuned models using model merging, generating pseudo-checkpoints that approximate the missing training trajectory and expose a geometric footprint of the training data in the weight space. From these simulated footprints, WARP extracts geometric features and maps them to domain proportions using either a parameter-free softmax readout or an MLP projector trained on synthetic mixtures. In controlled experiments with BERT and GPT-2, WARP recovers domain mixtures with an average MAE as low as 0.046 and 0.104 respectively, outperforming membership inference and a variant with access to the true training trajectory.
Community
Weight-space geometry encodes traces of training data. Can we use it to reverse-engineer data recipes? Introducing WARP: a new strategy to estimate domain mixtures from model weights alone!
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
Cite arxiv.org/abs/2607.01686 in a model README.md to link it from this page.
Cite arxiv.org/abs/2607.01686 in a dataset README.md to link it from this page.
Cite arxiv.org/abs/2607.01686 in a Space README.md to link it from this page.
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.