BRKMYR/ AI Builder Portfolio/ AV Safety Benchmark

AV Safety Benchmark

A vendor-neutral safety scorecard for autonomous-vehicle planners. Emit a trajectory log; get back a composite score, a per-family breakdown, and a paper trail traceable to a regulatory clause.

Problem

Nobody's safety numbers are comparable.

Every AV company publishes a safety claim. Vendor A ran their planner through their scenarios. Vendor B did something different. Nobody agrees on the metric. Regulators, procurement teams, and press coverage have no vendor-neutral leaderboard to reach for. Meanwhile the field-standard evals (nuScenes, Waymo Open Motion) are prediction benchmarks, not safety benchmarks.

For a product team choosing a planner, this means every claim has to be re-verified in-house before it can be trusted. That's expensive and slow, and it doesn't scale.

Solution

One contract. One suite. One score.

Any planner that emits a JSONL trajectory log in the documented schema gets scored on the same 60 scenarios across four families (cut-in, occluded pedestrian, weather ramp, hard brake). The score is a composite of surrogate safety metrics: minimum time-to-collision, collision flag, off-road excursion, jerk, harshness. Each metric traces to a specific ISO/UNECE clause in a shipped YAML.

Three reference planners ship in the box so you can see the discrimination range: cautious_idm (competent baseline), idm (textbook), constant_velocity (broken control).

Result

Leaderboard from 180 runs.

Scenarios

Families

Planners

Runs

180

Rank	Planner	Composite	cut_in	ped_occluded	weather_ramp	hard_brake	Collisions
1	cautious_idm	74.17	88.66	24.75	99.55	83.73	11
2	idm	64.88	82.91	14.91	95.99	65.71	14
3	constant_velocity	12.50	36.44	12.86	0.00	0.71	52

Reading the table. constant_velocity earns its 52 collisions honestly: it's the null baseline. The 62-point composite gap to cautious_idm says the pipeline actually discriminates. The bigger story is in the ped_occluded column: even the leader scores 24.75. That's the family a product team should fix first.

Open full report →

Success criteria

When we'd call this "working."

The score separates a broken baseline from a competent one by at least 50 composite points. Currently: 62.
Every metric traces to a specific regulatory clause in the shipped YAML.
Reruns are byte-identical. Fixed seed, deterministic simulator, deterministic scoring.
The JSONL contract is documented well enough that a vendor can integrate without touching the benchmark repo.
A worst-scenario ID is reported per family so failure investigation has a starting point.

Metrics, KPIs, evals

What we measure, and why.

Metric	Type	What it tells the PM
Composite score	0-100	The one number that lets a stakeholder compare planners. Weighted blend of family scores. Higher is safer.
Per-family score	4 families	Where a planner fails is more useful than whether it fails. The occluded-pedestrian family is the regulator's first question.
Collision count	integer	The hard-safety KPI. A planner that scores 90 but has 30 collisions is not a safe planner.
Worst scenario per family	scenario_id	Directly actionable: rerun this ID, watch the failure, fix it, rerun. No hunting.
Regulatory mapping	YAML	Every metric points at the ISO/UNECE clause it's a proxy for. Compliance conversations become audit trails, not arguments.
Determinism	byte-identical	Regression testing works. Bisecting a score change to a code commit is possible.

What we haven't shipped yet.

Third-party planner integration. The contract is documented; we've only run our own baselines through it.
OpenSCENARIO export is complete for the cut_in family; other families need per-family handlers.
More scenario families. Roundabout, unprotected left, and construction-zone are next on the list.