Tweet: <a href=\"https://x.com/HuggingPapers/status/2089887693799084315\" rel=\"nofollow\">https://x.com/HuggingPapers/status/2089887693799084315</a></p>\n","updatedAt":"2026-08-19T08:00:02.884Z","author":{"_id":"5f1158120c833276f61f1a84","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1608042047613-5f1158120c833276f61f1a84.jpeg","fullname":"Niels Rogge","name":"nielsr","type":"user","isPro":false,"isHf":true,"isHfAdmin":false,"isMod":false,"followerCount":1278,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.627574622631073},"editors":["nielsr"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/1608042047613-5f1158120c833276f61f1a84.jpeg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2608.17402","authors":[{"_id":"6a84feec536bdd3bdd48f6d0","name":"Bonan Zhang","hidden":false},{"_id":"6a84feec536bdd3bdd48f6d1","name":"Shiyu Dong","hidden":false},{"_id":"6a84feec536bdd3bdd48f6d2","name":"Quan Hung Tran","hidden":false},{"_id":"6a84feec536bdd3bdd48f6d3","name":"Katharina Gschwind","hidden":false},{"_id":"6a84feec536bdd3bdd48f6d4","name":"Shuqi Yang","hidden":false},{"_id":"6a84feec536bdd3bdd48f6d5","name":"Sijia Chen","hidden":false},{"_id":"6a84feec536bdd3bdd48f6d6","name":"Adel Ahmadyan","hidden":false},{"_id":"6a84feec536bdd3bdd48f6d7","name":"Seungwhan Moon","hidden":false},{"_id":"6a84feec536bdd3bdd48f6d8","name":"Lu Zhang","hidden":false},{"_id":"6a84feec536bdd3bdd48f6d9","name":"Ahmed Kirmani","hidden":false},{"_id":"6a84feec536bdd3bdd48f6da","name":"Babak Damavandi","hidden":false},{"_id":"6a84feec536bdd3bdd48f6db","name":"Anuj Kumar","hidden":false}],"publishedAt":"2026-08-18T00:00:00.000Z","submittedOnDailyAt":"2026-08-19T00:00:00.000Z","title":"MoE-ViE: Mixture of Experts Vision Encoder for Efficient Image and Video Understanding","submittedOnDailyBy":{"_id":"5f1158120c833276f61f1a84","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1608042047613-5f1158120c833276f61f1a84.jpeg","isPro":false,"fullname":"Niels Rogge","user":"nielsr","type":"user","name":"nielsr"},"summary":"Vision encoders are a critical component of vision-language models, and scaling their capacity effectively improves performance. However, dense scaling increases compute cost and inference latency. Mixture-of-Experts (MoE) architectures offer a compelling alternative, having enabled efficient scaling in LLMs, yet the MoE design space for CLIP-style vision encoders remains underexplored at State-of-the-Art (SOTA) levels. In this work, we systematically study MoE designs for vision encoder scaling and find that fine-grained MoE topologies yield substantial gains over both dense and standard MoE counterparts. We further propose an auxiliary-loss-free balancing variant for better expert utilization, and design a specialized MoE kernel to mitigate inference latency overhead. To enhance video capabilities while preserving image knowledge, we introduce frame-level distillation paired with a novel freezing mechanism. We pretrain a series of Mixture-of-Experts Vision Encoders (MoE-ViE) across a range of sizes, all consistently outperforming their dense counterparts. Our largest model matches the zero-shot performance of a SOTA encoder 1.7x its size at 76% of its latency. When aligned with an LLM, MoE-ViE surpasses all compared encoders on image and video benchmarks, including those with up to 5x more activated parameters. Code is available at https://github.com/facebookresearch/moe_vie.","upvotes":0,"discussionId":"6a84feec536bdd3bdd48f6dc","githubRepo":"https://github.com/facebookresearch/moe_vie","githubRepoAddedBy":"admin","ai_summary":"Mixture-of-Experts vision encoders with fine-grained topologies, auxiliary-loss-free balancing, and specialized kernels scale efficiently while outperforming larger dense models on image and video tasks.","ai_keywords":["Mixture-of-Experts","MoE","vision encoder","CLIP","fine-grained MoE","auxiliary-loss-free balancing","expert utilization","MoE kernel","inference latency","frame-level distillation","freezing mechanism","MoE-ViE"],"ai_summary_model":"thinkingmachines/Inkling-Small","githubStars":0,"organization":{"_id":"5e63d8713071d5be688861b8","name":"facebook","fullname":"AI at Meta","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/1592839207516-noauth.png"}},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[],"acceptLanguages":["en"],"organization":{"_id":"5e63d8713071d5be688861b8","name":"facebook","fullname":"AI at Meta","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/1592839207516-noauth.png"},"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2608/2608.17402.md","query":{}}">
MoE-ViE: Mixture of Experts Vision Encoder for Efficient Image and Video Understanding
Abstract
Mixture-of-Experts vision encoders with fine-grained topologies, auxiliary-loss-free balancing, and specialized kernels scale efficiently while outperforming larger dense models on image and video tasks.
Vision encoders are a critical component of vision-language models, and scaling their capacity effectively improves performance. However, dense scaling increases compute cost and inference latency. Mixture-of-Experts (MoE) architectures offer a compelling alternative, having enabled efficient scaling in LLMs, yet the MoE design space for CLIP-style vision encoders remains underexplored at State-of-the-Art (SOTA) levels. In this work, we systematically study MoE designs for vision encoder scaling and find that fine-grained MoE topologies yield substantial gains over both dense and standard MoE counterparts. We further propose an auxiliary-loss-free balancing variant for better expert utilization, and design a specialized MoE kernel to mitigate inference latency overhead. To enhance video capabilities while preserving image knowledge, we introduce frame-level distillation paired with a novel freezing mechanism. We pretrain a series of Mixture-of-Experts Vision Encoders (MoE-ViE) across a range of sizes, all consistently outperforming their dense counterparts. Our largest model matches the zero-shot performance of a SOTA encoder 1.7x its size at 76% of its latency. When aligned with an LLM, MoE-ViE surpasses all compared encoders on image and video benchmarks, including those with up to 5x more activated parameters. Code is available at https://github.com/facebookresearch/moe_vie.
Community
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
Cite arxiv.org/abs/2608.17402 in a dataset README.md to link it from this page.
Cite arxiv.org/abs/2608.17402 in a Space README.md to link it from this page.
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.