Can a video generator actually finish the task—not merely make a convincing video? SemComp-Bench evaluates outcome achievement together with task-relevant semantic grounding.</p>\n","updatedAt":"2026-08-20T03:21:25.332Z","author":{"_id":"630636bcd37ce67e0e4d1d42","avatarUrl":"/avatars/c3ba695d339eaaf9e810bfa0d9a7689a.svg","fullname":"Mengqi Huang","name":"CoreloneH","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":12,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.783562958240509},"editors":["CoreloneH"],"editorAvatarUrls":["/avatars/c3ba695d339eaaf9e810bfa0d9a7689a.svg"],"reactions":[{"reaction":"🔥","users":["ZhuoweiChen"],"count":1}],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2608.17426","authors":[{"_id":"6a867283db13816030683ecd","name":"Keyu Tu","hidden":false},{"_id":"6a867283db13816030683ece","name":"Zhuowei Chen","hidden":false},{"_id":"6a867283db13816030683ecf","name":"Mengqi Huang","hidden":false},{"_id":"6a867283db13816030683ed0","name":"Yuxin Wang","hidden":false},{"_id":"6a867283db13816030683ed1","name":"Jiahao Zhu","hidden":false},{"_id":"6a867283db13816030683ed2","name":"Zhendong Mao","hidden":false},{"_id":"6a867283db13816030683ed3","name":"Yongdong Zhang","hidden":false}],"publishedAt":"2026-08-18T00:00:00.000Z","submittedOnDailyAt":"2026-08-20T00:00:00.000Z","title":"SemComp-Bench: Benchmarking Semantic Task Completion in Video Generation","submittedOnDailyBy":{"_id":"630636bcd37ce67e0e4d1d42","avatarUrl":"/avatars/c3ba695d339eaaf9e810bfa0d9a7689a.svg","isPro":false,"fullname":"Mengqi Huang","user":"CoreloneH","type":"user","name":"CoreloneH"},"summary":"We introduce Semantic Task Completion Video Generation, an outcome-oriented video generation task. Under this formulation, success requires both achievement of the intended outcome and semantic grounding. Semantic grounding characterizes the correspondence between the reference image and the generated outcome in terms of high-level semantics relevant to the task. Evaluation focuses on the generated outcome and requires neither the presentation of a complete sequence of intermediate task steps nor conventional appearance consistency with the reference image. To support systematic evaluation, we construct SemComp-Data, an evaluation dataset covering six domains. Each instance comprises a reference image, a detailed instruction, a brief instruction, and an outcome-centric video clip. A scalable four-stage curation pipeline converts raw videos into standardized SemComp-Data instances. We further introduce SemComp-Bench, an evaluation protocol that uses a vision-language model (VLM) to answer structured binary questions. SemComp-Bench reports the OA Score and the GR Score for Outcome Achievement and Generation Reliability, respectively. Experiments on representative video generation models show that achieving intended outcomes while maintaining task-relevant semantic grounding in reference images remains challenging.","upvotes":53,"discussionId":"6a867283db13816030683ed4","projectPage":"https://semcomp-bench.github.io/","githubRepo":"https://github.com/Kelly372/SemComp-Bench","githubRepoAddedBy":"user","ai_summary":"Semantic task completion video generation evaluates whether generated videos achieve intended outcomes with semantic grounding, supported by a curated dataset and vision-language model-based benchmark.","ai_keywords":["video generation","vision-language model","semantic grounding","outcome achievement","generation reliability"],"ai_summary_model":"thinkingmachines/Inkling-Small","githubStars":2,"organization":{"_id":"69fc10a2619a0cc5d22a1b64","name":"FrameXAI","fullname":"FrameX-AI","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/69f6c6749a57b7a8097ce1b1/jjmD84WN_w-5wgCEuh0f6.jpeg"}},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"630636bcd37ce67e0e4d1d42","avatarUrl":"/avatars/c3ba695d339eaaf9e810bfa0d9a7689a.svg","isPro":false,"fullname":"Mengqi Huang","user":"CoreloneH","type":"user"},{"_id":"64b7609c88b86014d7e9bc77","avatarUrl":"/avatars/6ebf0b20edec26ba76338d74006c98a6.svg","isPro":false,"fullname":"metingxing","user":"colorful101","type":"user"},{"_id":"692855eabdadcb130bd0fb35","avatarUrl":"/avatars/dd9a608d6ac196e349eb3b48ca8ffaea.svg","isPro":false,"fullname":"XMY","user":"showmeon","type":"user"},{"_id":"687474530485cd4b3786b88a","avatarUrl":"/avatars/986283120c0333d72f67edc64b20ef56.svg","isPro":false,"fullname":"shifshufen","user":"shufenshi","type":"user"},{"_id":"6a471c3c01fef16ba4a602cc","avatarUrl":"/avatars/5d2ea9405996c16421794aec41a94e54.svg","isPro":false,"fullname":"quelin huang","user":"pellucid0521","type":"user"},{"_id":"69fb3a2f8a72101c1fe39b36","avatarUrl":"/avatars/7ff8a33d0edac9d43122fc55ebaa0af2.svg","isPro":false,"fullname":"Hao","user":"Tuixiu","type":"user"},{"_id":"6683ab3b9e8f3018845c51f0","avatarUrl":"/avatars/0d873bb3744cf2f4e11feb456e35d5ed.svg","isPro":false,"fullname":"wangbei","user":"wangbei1","type":"user"},{"_id":"6a27c365ba0906408e40eee8","avatarUrl":"/avatars/86522b23d5781a6007e474aedc1728b6.svg","isPro":false,"fullname":"TIAN,SIYU","user":"Tshowy","type":"user"},{"_id":"660cce99e6569b99a20aa9ef","avatarUrl":"/avatars/1f3452d5643af290415e29822fc8e59c.svg","isPro":false,"fullname":"Lw","user":"Longwangmaomao","type":"user"},{"_id":"640de2dcc364a086c632903e","avatarUrl":"/avatars/d8a24cfe7eb2c4a83c86683dacbb4142.svg","isPro":false,"fullname":"Auxotaku","user":"Auxotaku","type":"user"},{"_id":"6a83d6b9b4f93ce50cbc140f","avatarUrl":"/avatars/704a98fc87bda043d1e8cea91ca39723.svg","isPro":false,"fullname":"wangttt","user":"wangttt2323","type":"user"},{"_id":"68a32b6f814d0858f57af748","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/3gOls4ljGs2aNjrIAid8p.png","isPro":false,"fullname":"Zijian Liu","user":"entropy123123","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"organization":{"_id":"69fc10a2619a0cc5d22a1b64","name":"FrameXAI","fullname":"FrameX-AI","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/69f6c6749a57b7a8097ce1b1/jjmD84WN_w-5wgCEuh0f6.jpeg"},"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2608/2608.17426.md","query":{}}">

Get this paper in your agent:

Don't have the latest CLI?

Cite arxiv.org/abs/2608.17426 in a model README.md to link it from this page.

Cite arxiv.org/abs/2608.17426 in a dataset README.md to link it from this page.

Cite arxiv.org/abs/2608.17426 in a Space README.md to link it from this page.