BRKMYR/ AI Builder Portfolio/ World Model Benchmark
A harness that scores predictive-video "world" models on six distinct failure modes. Toy dynamics for now, so the harness itself can be trusted before any real model plugs in.
Problem
Frontier labs are shipping video-generation models and calling them "world models." Product teams evaluating one for downstream use (robotics simulation, VLM grounding, synthetic data augmentation) have no shared scorecard. Existing video-quality metrics measure pixel fidelity, which is the wrong thing: a model that produces beautiful-looking video but forgets that objects persist between frames is useless for downstream physical reasoning.
Building an eval harness against a real frontier model first is backwards. You can't tell whether a low score means the model is bad or the harness is broken.
Solution
Ground truth is analytic (closed-form trajectories, so the correct answer is known). Three baselines with expected orderings are used to validate the harness: linear_extrapolation (should win overall), frame_persistence (should win object-permanence, lose state-error), noised_dynamics (should lose everything but be worst at energy drift).
Six metrics score six different failure modes. The composite is a blend; no single metric can hide a specific failure. Every episode gets a filmstrip so failures are debuggable, not just numeric.
Result
| Model | Composite | PSNR (dB) | SSIM | Flicker | Obj. permanence | State err. k10 (px) |
|---|---|---|---|---|---|---|
| 1linear_extrapolation | 0.800 | 11.52 | 0.72 | 0.009 | 0.69 | 5.03 |
| 2frame_persistence | 0.405 | 9.88 | 0.67 | 0.044 | 0.86 | 11.33 |
| 3noised_dynamics | 0.097 | 9.72 | 0.67 | 0.095 | 0.77 | 13.40 |
Reading the table. Composite ordering matches the predicted ordering, which is what says the harness is calibrated. Note that frame_persistence wins object permanence (0.86) even though it's second overall: holding a frame preserves objects trivially. This is why the composite is a blend, not a single number.
Success criteria
Metrics, KPIs, evals
| Metric | Category | Failure mode it catches |
|---|---|---|
| PSNR | reconstruction | Basic pixel fidelity. If a model can't get pixels roughly right, it fails here first. Included as a sanity check, not as the interesting metric. |
| SSIM | reconstruction | Structural similarity: does the output "look like" the target, allowing for small shifts. More forgiving than PSNR. |
| Flicker deviation | temporal | Frame-to-frame smoothness. Models that produce jittery output score badly here even if pixels are individually fine. |
| Object permanence | structural | Do objects persist across frames when they should? A model that spawns and destroys objects randomly fails this even if every frame looks great. |
| State error at k=10 | predictive | Where will the object be 10 steps out? This is the "does it actually predict" test. Pixel-perfect single-frame extrapolation is easy; getting the trajectory right at k=10 is hard. |
| Energy drift | diagnostic | Physical plausibility. High drift means the model is inventing or destroying energy. Diagnostic-only; not composited. |