| Rank | Model | Composite | PSNR (dB) | SSIM | Flicker deviation | Object permanence | State error k10 (px) | Energy drift (diagnostic) |
|---|---|---|---|---|---|---|---|---|
| 1 | linear_extrapolation | 0.8000 | 11.5165 | 0.7218 | 0.0092 | 0.6875 | 5.0258 | 0.7411 |
| 2 | frame_persistence | 0.4052 | 9.8755 | 0.6738 | 0.0440 | 0.8646 | 11.3266 | 0.0000 |
| 3 | noised_dynamics | 0.0971 | 9.7151 | 0.6688 | 0.0951 | 0.7734 | 13.3963 | 30.5465 |
| Key | Value |
|---|---|
| suite | toy |
| num_episodes | 16 |
| frames_per_episode | 60 |
| context_frames | 12 |
| horizon | 48 |
| world.n_objects | 3 |
| world.gravity_px_s2 | 60.0 |
| frame.height | 64 |
| frame.width | 64 |
PSNR (higher, dB). Peak signal-to-noise ratio on greyscale frames, capped at 100 dB on exact matches. Discriminates pixel-level fidelity; cannot distinguish plausible-but-shifted content from noise.
SSIM (higher, [-1, 1]). Structural similarity (skimage default 7x7 window, data_range=255). More tolerant of small photometric offsets than PSNR; still purely appearance-based.
Flicker deviation (lower). Absolute difference between the predicted sequence's mean-abs second temporal difference and the ground truth's. Uses deviation from GT rather than raw flicker so that a frozen model (flicker 0) does not trivially "win" on temporal smoothness.
Object permanence rate (higher). Fraction of frames where the predicted count of connected components (binarize at 128, area ≥ 5 px) matches the ground-truth count of that frame index. Uses per-frame GT, not N, so legitimate overlap in GT is credited to the model too.
State error at horizon k (lower, px). Mean L2 position error between predicted and GT positions at prediction-window index k (1-based). The cheapest honest proxy for downstream utility: if a planner consumed these state forecasts, this bounds its planning error at horizon k.
Energy drift (diagnostic only). Max relative deviation of total mechanical energy over the prediction window. Deliberately excluded from the composite: a frozen model conserves energy trivially and would invert the ranking. Reported as a column so you can see when a model's dynamics are inconsistent, and used to validate the GT simulator itself.
Deferred (v2). Action-conditioning is meaningless without action-conditioned models: none of the v1 baselines consume actions, so we skip it rather than fake a score. Diversity is meaningless for deterministic models: all three baselines produce exactly one rollout per context, so coverage/novelty are trivially degenerate. Both dimensions plug into extensions of the WorldModel Protocol in v2.