<a href=\"https://youtu.be/bLLtwXVcqEc\" rel=\"nofollow\">https://youtu.be/bLLtwXVcqEc</a></p>\n","updatedAt":"2026-08-17T02:50:51.670Z","author":{"_id":"68d601e807861246de23ca88","avatarUrl":"/avatars/80868d1bb158505baa165da430504a80.svg","fullname":"Zian","name":"SII-Sarfflow","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"ru","probability":0.4003506600856781},"editors":["SII-Sarfflow"],"editorAvatarUrls":["/avatars/80868d1bb158505baa165da430504a80.svg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2608.14530","authors":[{"_id":"6a8267eab601d59c6528135e","name":"Zian Meng","hidden":false},{"_id":"6a8267eab601d59c6528135f","name":"Zhen Li","hidden":false},{"_id":"6a8267eab601d59c65281360","name":"Chuanhao Li","hidden":false},{"_id":"6a8267eab601d59c65281361","name":"Qiang Li","hidden":false},{"_id":"6a8267eab601d59c65281362","name":"Kaipeng Zhang","hidden":false}],"publishedAt":"2026-08-14T00:00:00.000Z","submittedOnDailyAt":"2026-08-17T00:00:00.000Z","title":"Marionette: Predicting World States, Rendering Geometry, Painting Appearance","submittedOnDailyBy":{"_id":"6039478ab3ecf716b1a5fd4d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6039478ab3ecf716b1a5fd4d/_Thy4E7taiSYBLKxEKJbT.jpeg","isPro":true,"fullname":"taesiri","user":"taesiri","type":"user","name":"taesiri"},"summary":"Interactive game world models typically autoregress visual observations directly in pixel or latent space, forcing structured properties such as pose, geometry, and occlusion to be implicitly maintained by the same generative sequence. Over long horizons, errors in these latent world properties accumulate, making consistency and controllability fragile. We explicitly model the evolving world state, delegate exact geometric computation to a fixed, zero-parameter renderer, and leave the neural model to synthesize appearance. We instantiate this idea as Marionette, a world model for interactive games with articulated characters. First, a two-stage autoregressive dynamics model predicts an explicit and interpretable 276-dimensional 3D world state comprising multi-entity articulated skeletons, metric root trajectories, and rotations. Second, a zero-parameter graphics bridge converts the predicted state into pose-control videos, computing world-space geometry and occlusion in closed form. Third, a control-conditioned video-diffusion observation model synthesizes photorealistic RGB observations from the resulting structured controls. Our experiments establish two properties of Marionette. First, the predicted world state is directly controllable. Forcing a mismatched action stream changes root-aligned joint error by 31% across 48 held-out segments. Second, long-horizon behaviour is determined in the state, and can be repaired there. Left free, the two generated characters drift to 21.2 m apart (recorded sessions stay near 5 m) and a third of frames show ground penetration. Two rules imposed on the explicit state, a terrain collider and a separation cap, cut penetration by 66% and keep the pair engaged, with no change to the observation model. Routing appearance through the predicted state costs no fidelity we can detect, at an FVD of 831 against 799 for recorded pose.","upvotes":6,"discussionId":"6a8267ebb601d59c65281363","projectPage":"https://alayalab.github.io/Marionette/","ai_summary":"Marionette predicts explicit 3D articulated world states for interactive games, uses a fixed renderer for geometry, and synthesizes video via diffusion, enabling direct state-level control and long-horizon consistency repair.","ai_keywords":["autoregressive dynamics model","3D world state","articulated skeletons","zero-parameter renderer","video-diffusion observation model","control-conditioned","FVD"],"ai_summary_model":"thinkingmachines/Inkling-Small"},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"6039478ab3ecf716b1a5fd4d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6039478ab3ecf716b1a5fd4d/_Thy4E7taiSYBLKxEKJbT.jpeg","isPro":true,"fullname":"taesiri","user":"taesiri","type":"user"},{"_id":"63342778d92c5842ae728aef","avatarUrl":"/avatars/888eb265643633c5fdd7048be9bfe98f.svg","isPro":false,"fullname":"Fengbo Lan","user":"fblan","type":"user"},{"_id":"68d601e807861246de23ca88","avatarUrl":"/avatars/80868d1bb158505baa165da430504a80.svg","isPro":false,"fullname":"Zian","user":"SII-Sarfflow","type":"user"},{"_id":"685269199f20805782107cfb","avatarUrl":"/avatars/3a11c7658ccd7ad5b7cdf30afe3d3b31.svg","isPro":false,"fullname":"Zhen Li","user":"Lixsp11","type":"user"},{"_id":"620783f24e28382272337ba4","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/620783f24e28382272337ba4/zkUveQPNiDfYjgGhuFErj.jpeg","isPro":false,"fullname":"GuoLiangTang","user":"Tommy930","type":"user"},{"_id":"643bd9695fcffe09fb664ae2","avatarUrl":"/avatars/b81e27c8a5aaf527abe82b0ea6f11eed.svg","isPro":false,"fullname":"Viktor Cerny","user":"Nazzaroth2","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2608/2608.14530.md","query":{}}">
Marionette: Predicting World States, Rendering Geometry, Painting Appearance
Abstract
Marionette predicts explicit 3D articulated world states for interactive games, uses a fixed renderer for geometry, and synthesizes video via diffusion, enabling direct state-level control and long-horizon consistency repair.
Interactive game world models typically autoregress visual observations directly in pixel or latent space, forcing structured properties such as pose, geometry, and occlusion to be implicitly maintained by the same generative sequence. Over long horizons, errors in these latent world properties accumulate, making consistency and controllability fragile. We explicitly model the evolving world state, delegate exact geometric computation to a fixed, zero-parameter renderer, and leave the neural model to synthesize appearance. We instantiate this idea as Marionette, a world model for interactive games with articulated characters. First, a two-stage autoregressive dynamics model predicts an explicit and interpretable 276-dimensional 3D world state comprising multi-entity articulated skeletons, metric root trajectories, and rotations. Second, a zero-parameter graphics bridge converts the predicted state into pose-control videos, computing world-space geometry and occlusion in closed form. Third, a control-conditioned video-diffusion observation model synthesizes photorealistic RGB observations from the resulting structured controls. Our experiments establish two properties of Marionette. First, the predicted world state is directly controllable. Forcing a mismatched action stream changes root-aligned joint error by 31% across 48 held-out segments. Second, long-horizon behaviour is determined in the state, and can be repaired there. Left free, the two generated characters drift to 21.2 m apart (recorded sessions stay near 5 m) and a third of frames show ground penetration. Two rules imposed on the explicit state, a terrain collider and a separation cap, cut penetration by 66% and keep the pair engaged, with no change to the observation model. Routing appearance through the predicted state costs no fidelity we can detect, at an FVD of 831 against 799 for recorded pose.
Community
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
Cite arxiv.org/abs/2608.14530 in a dataset README.md to link it from this page.
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.