BRKMYR/Prototypes I built/AV Safety Benchmark
A deterministic safety scorecard for the planning software of an autonomous vehicle (AV). Sixty scenarios, three reference planners, one composite score per planner, and a mapping to safety standards that is marked informative on every line.
Shipped v1 · 60 scenarios · 3 planners · 180 runs · 33 testsAt a glance
What
A scoring harness for the software that drives an autonomous vehicle. It generates 60 short driving scenarios in four families, runs three simple reference planners through a 2D simulator, and scores every run on surrogate safety metrics such as time to collision (TTC, the seconds until two vehicles would touch). One table ranks the planners.
Why it matters
AV companies publish safety claims on scenarios and metrics they chose themselves, so no two numbers compare. This benchmark fixes the scenarios, the metrics and the log format, and points each scored metric at a clause in SOTIF (ISO 21448, safety of the intended functionality) and UL 4600 (the safety case standard for autonomous products). Anyone can rerun it.
How it was built
Python with five libraries. A 10 Hz kinematic simulator replaces the CARLA game engine simulator the original plan named, so the whole suite runs on a laptop with no GPU and no network. Fixed seeds make every rerun identical on one machine. 33 tests pin hand computed values, and a second, pre-registered holdout suite feeds the Assurance Gate.
Problem
An AV company that publishes a safety figure chose the scenarios, the metric and the pass line itself. A second company chose differently. In my experience on the carmaker side, a procurement team has to rerun every claim in house before it can trust it. That is slow and does not scale.
The public benchmarks the field cites most, nuScenes and the Waymo Open Motion Dataset, score how well a model predicts where other road users will go. They say nothing about whether the planner kept a safe distance, and they carry no link to the standards a regulator reads.
What it does
The benchmark, packaged as avsb, turns a seed into 60 scenario files, runs each planner against them in a fixed step simulator, and scores the resulting trajectory logs. Scenarios come in four families: cut in, occluded pedestrian, lead vehicle hard brake, and a weather ramp that pushes visibility and grip past the conditions the vehicle is designed for. Metrics read only the logs, so any planner that writes the same log format can be scored.
Solution
Any planner that emits a JSONL trajectory log (one JSON record per line) in the documented schema gets scored on the same 60 scenarios. Six surrogate safety metrics come out of each log: minimum TTC, minimum PET, minimum distance margin, a collision flag with a delta-v severity index, where delta-v is the sudden change in speed at impact and a standard measure of crash severity, hard brake fraction and mean speed ratio. Each scored metric points at a clause in ISO 26262, ISO 21448 (SOTIF), UL 4600 or the EU AI Act in a shipped YAML file, and every entry is tagged informative.
Three reference planners ship in the box so the discrimination range is visible: cautious_idm (wider headway, lower target speed), idm (the textbook Intelligent Driver Model, IDM) and constant_velocity (a null baseline that never brakes). The spec demanded that they land in that order, and they do.
Result
The composite is the plain mean of the four family scores. cautious_idm leads at 74.17, idm follows at 64.88, and constant_velocity earns its 52 collisions as the null baseline at 12.50. The gap of 61.7 points between the leader and the null baseline shows the pipeline discriminates. The occluded pedestrian family is where every planner is weak: the leader scores 24.75 there with 11 collisions in 16 scenarios.
Scenarios
60
Families
4
Planners
3
Runs
180
| Rank | Planner | Composite | cut_in | ped_occluded | weather_ramp | hard_brake | Collisions |
|---|---|---|---|---|---|---|---|
| 1 | cautious_idm | 74.17 | 88.66 | 24.75 | 99.55 | 83.73 | 11 |
| 2 | idm | 64.88 | 82.91 | 14.91 | 95.99 | 65.71 | 14 |
| 3 | constant_velocity | 12.50 | 36.44 | 12.86 | 0.00 | 0.71 | 52 |
Core suite, seed 42, scores 0 to 100 from summary.json. On the separate holdout suite the Assurance Gate reports cautious_idm failing 2 of 9 gates, idm 3 of 9 and constant_velocity 6 of 9. The full report with 28 plots is served from this site, since generated outputs are not committed to the repository.
Solution architecture
Sources: numpy is BSD 3, pydantic, PyYAML and Jinja2 are MIT, matplotlib carries its PSF style licence. The IDM planner follows Treiber, Hennecke and Helbing (2000). The mapping cites ISO 26262-3:2018, ISO 21448:2022, UL 4600:2023 and EU Regulation 2024/1689 by clause and reproduces no text. OpenSCENARIO is an ASAM format, free to read. Everything else is this project, MIT as declared in pyproject.toml with no licence file committed yet.
Why it matters
The three baselines land in the order the spec demanded, and the occluded pedestrian family exposes the leader at 24.75 with 11 collisions. Every scored metric carries an informative clause reference and a disclaimer that no ASIL (automotive safety integrity level) or conformity claim follows from it. A second suite drawn from a different seed now feeds the Assurance Gate, where all three planners fail at least two of nine gates. The limits are stated: one straight road, analytic planners, no sensor models, no real logs.
In five years
Type approval of automated driving will run on scenario libraries and surrogate metrics that regulators accept, and vendors will submit logs in place of claims. Learned planners will sit behind the same log interface as the analytic ones here. This benchmark is a small early version of that submission contract.
How it was built
Python
4,433lines
Tests
33
Commits
18
Build span
210days
Coding days
3
Runtime deps
5no GPU
Spec, data contracts, metric formulas, acceptance criteria and review by hand. Implementation with Claude Code agents. Measured from the repository.
The repo began as a README charter for a framework on CARLA, an open driving simulator, with the public nuScenes and Waymo datasets, and that plan never got code. A later 651 line spec replaced CARLA with a 10 Hz kinematic engine and fixed the data contracts, the metric formulas and eleven acceptance criteria. Eleven commits implemented the spec in order, schemas first, tests last. The spec flagged the leaderboard ordering as the main risk and gave a tuning ladder for it. A second round added the speed metric, holdout suites and per pair plots for the Assurance Gate. The README still describes the original plan.
Success criteria
Metrics, KPIs, evals
| Metric | Type | What it tells the PM |
|---|---|---|
| Composite score | 0 to 100 | The one number that lets a stakeholder compare planners. Unweighted mean of the four family scores. Higher is safer. Leader 74.17. |
| Per family score | 4 families | Where a planner fails is more useful than whether it fails. Each scenario blends TTC (0.40), distance (0.25), PET (0.20) and comfort (0.15) before the family mean. |
| Collision count | integer | The hard safety KPI. Measured 11, 14 and 52 across the three planners. A planner that scores 90 with 30 collisions is still unsafe. |
| Worst scenario per family | scenario_id | Directly actionable: rerun this ID, watch the failure, fix it, rerun. The report plots exactly these twelve cases. |
| Regulatory mapping | YAML, informative | Nine entries, 24 clause references to ISO 26262, ISO 21448, UL 4600 and the EU AI Act. It gives a compliance conversation a starting point and asserts no conformity. |
| Determinism | identical on one machine | Three tests compare generation, simulation and plots byte for byte in one process. Regression testing works, and a score change can be bisected to a commit. |