Fleet Safety Monitor · Prototypes I built

At a glance

What

A monitoring screen for remote operators of driverless taxis. It replays short driving scenes, scores every vehicle and pedestrian for risk ten times a second, and raises an AMBER or RED alert when a person should watch or take over. Every alert names its reason in one line.

Why it matters

Robotaxi operators keep remote staff on standby. One person will soon watch many vehicles, so software has to say which vehicle needs a human now, and why. This project shows that a small, tested rule engine can make that call the same way every run, with a reason an operator can act on.

How it was built

Python, with numpy for the maths and pygame for the drawing. Scenes follow the shape of the Waymo Open Motion Dataset (WOMD, public driving logs recorded by Waymo cars), so the same code can later read real data. Five synthetic scenes ship in the repo. 84 tests and a continuous integration job, meaning checks that run automatically on every code change, confirm that the hard brake alert fires at exactly the right frame.

Live capture of the safety monitor running the intersection_conflict scenario. The map view shows the ego vehicle and surrounding agents; the right panel lists trigger events as they fire.

Direct capture of the pygame dashboard, headless. 46 frames at 5 fps (stride 2 over the 10 Hz sim). Every element on screen (map, agents, playback bar, trigger panel, fleet panel) is drawn by the shipping app itself. Nothing on screen is a mock.

Problem

Operators cannot watch every screen.

Commercial robotaxi services keep humans on standby to take over when a vehicle asks for help. As the number of vehicles per operator grows past a handful, nobody can keep eyes on every tile. The hard part is spotting which of the twenty tiles needs a human right now, and why.

Video and telemetry alone show what is happening. They do not say which vehicle is closest to trouble. They do not explain the reason in one glance. They do not let the operator rewind to see what led up to it.

What it does

Seven metrics, nine trigger kinds, one reason per alert.

The monitor takes a 9.1 second driving scene with a handful of vehicles, cyclists and pedestrians, sampled ten times a second. At every step it computes seven safety metrics per agent, including time to collision (TTC, the seconds until two agents would touch at their current speed) and distance to vulnerable road users (VRU, pedestrians and cyclists). A trigger engine turns those metrics into AMBER and RED events, and a dashboard draws the scene, the events and the playback controls.

Computes per agent safety metrics each timestep: overlap, TTC, off road, wrong way, lane compliance, acceleration and VRU distance, with pedestrians excluded from the lane based metrics. metrics/safety.py
Fires each alert once when a condition starts, once more when it escalates from AMBER to RED, and only re arms after a clear window, so a persisting condition cannot flood the operator. triggers/engine.py
Ships three threshold presets (conservative, default, permissive) and a run configuration that rejects unknown keys instead of ignoring them. triggers/config.py, dashboard/app.py
Produces the same event log byte for byte on two independent headless runs. tests/test_headless.py
Runs as an interactive window and headless, meaning with no screen for automated testing, from one code path, writing 91 frames and a sorted event log that CI checks for one hard brake RED at agent 1, timestep 31. dashboard/app.py, ci.yml

Solution

A per agent risk view with a reason for every alert.

The dashboard runs a trigger engine over the scene and emits severity tagged events: AMBER for "watch this" and RED for "take over now". Each event names the specific condition, for example time to collision, off road excursion, hard brake or a pedestrian too close. A scrubbable playback bar lets the operator rewind and audit what led to the event.

The trigger engine is deterministic and re arms only after a clear window, so an alert does not fire again and again while a condition persists. Thresholds ship in three policy presets so a fleet can set its own false positive budget.

Result

Every conflict scene fires RED first, at a frame the tests derive by hand.

In all three captured scenes the first RED event precedes the first physical overlap. In the hard brake scene the lead vehicle decelerates at 6 m/s² at timestep 31, and the hard brake RED fires at timestep 31. The stall detector completes at timestep 68 in the stalled ego scene. Two headless runs of any scene produce identical event logs.

Scenes bundled

5

Trigger kinds

9

RED events

8 / 9 / 12

Tests

84

Edge triggering made visible. Each mark is one event from the shipped log. Hollow is AMBER, filled is RED. The hard brake RED lands on the frame the lead vehicle starts braking, and no kind fires twice while its condition persists.

Why the operator is not flooded. An alert fires once on the way in and once on escalation. It cannot fire again until the condition has been clear for a set number of steps.

Solution architecture

What it stands on, and what is new.

Inputs and datascenes and settings

WOMD scene shape91 steps at 10 Hz, road graph, typed agents. Shape only, no data downloaded

5 synthetic scenesintersection, merge, crossing, hard brake, stalled ego

run config JSONscene, seed, policy, weights. Unknown keys raise

Waymax loaderoptional, maps real tfrecords, not yet run

one scene object per run

Models and algorithmswhat scores the risk

7 safety metricsTTC, overlap, off road, wrong way, lane, acceleration, VRU distance

composite riskweighted sum, GREEN AMBER RED bands

3 trigger policiesconservative, default, permissive

metrics per agent per timestep

Engineown code, Python

edge trigger enginefire once, escalate, re arm after clear window

dashboard stateall 91 steps precomputed once, playback reads the cache

numpyBSD 3

pygame-ceLGPL, window, drawing, headless SDL driver

event list and frames

Outputs and interfaceswhere a reader lands

ops dashboardmap, risk halos, trails, trigger panel, timeline, playback

frames + events.json91 PNGs and a sorted event log

GitHub Actions CIpytest on 3.11 and 3.12, asserts the hard brake event

Built in this projectThird party

Sources: the Waymo Open Motion Dataset sets the scene format and carries a non commercial research licence, and no dataset file is in the repo. Waymax is Waymo's open simulator. numpy is BSD, pygame-ce is LGPL. Everything else is this project, MIT as declared in pyproject.toml with no licence file committed yet. This is independent personal work with no affiliation to Waymo or to Waymax.

Why it matters

Takeover triggers can be specified and tested like any software.

Each alert fires once per condition edge, carries a kind from a closed set of nine, and appears at a timestep the test suite derives by hand. Two independent headless runs produce identical event logs. A remote operations team could review the threshold presets and the reason strings without reading the code. The trade off is stated plainly: the metrics are heuristics, the scenes are synthetic, and real Waymo data has not been loaded yet.

In five years

Fleet supervision software will rank hundreds of vehicles by learned risk scores and hand each operator a queue instead of a wall of tiles. Those systems will still need a deterministic, auditable layer that says why a vehicle was flagged, for the operator in the moment and for the investigator afterwards. This monitor is an early, small version of that layer.

How it was built

A pivot, a defect list, then one rewrite.

Python

5,141lines

Tests

84

Commits

25

Build span

182days

Coding days

3

Runtime deps

2+3 optional

Spec, defect verification, acceptance timesteps and review by hand. Implementation with Claude Code agents. Measured from the repository.

The repo started as a charter for a geo fenced operating domain map of London. That concept was dropped and the first dashboard landed the same day with 41 tests. A later specification then verified defects by running the code, among them a level triggered engine that emitted 1,082 events in one scene and wrong way false positives on oncoming traffic. The hardest change was the rewrite to edge triggered firing with re arm, which the next agent run shipped in eleven commits together with three new trigger kinds, two analytic scenes, headless mode and CI.

Algorithms and mathTime to collision from bounding boxes and velocities, direction aware lane matching, analytic speed profiles. Heuristic, no learning.

Systems integrationTwo runtime libraries, a headless SDL driver, and an optional Waymax adapter that has not been exercised.

Data and domainThe WOMD scene shape, teleoperation trigger semantics, vulnerable road user rules and the pedestrian exclusions in the metrics.

Evaluation rigor84 tests, hand derived exact timesteps, a double run determinism test, a closed kind set check and a CI acceptance gate. No baseline and no real data.

Interface and productFull pygame dashboard with map, halos, trails, trigger panel, timeline, fleet panel, pan, zoom, scrubbing and a headless frame dump.

Difficulty 16 of 25 · AdvancedRelative to one competent engineer building it alone

Success criteria

When this counts as working.

Every collision in a scene is preceded by at least one RED event, with lead time long enough to hand over.
False positives are low enough that operators do not tune out. Nothing in the repository measures this yet, so it stays an open criterion.
The reason string on every event is specific enough that an operator knows what to look at without opening the log.
The same scene re run produces the same event stream, byte for byte.
The system runs headless for regression tests and interactively for operations from the same code path.

Metrics, KPIs, evals

Numbers from the shipped logs.

KPI	Value	What it means
RED before conflict	3 / 3 scenes	In every captured scene that ends in an overlap, a RED event fires first. In the hard brake scene two of those early REDs come from oncoming traffic in the adjacent lane, which is a known weakness of the head on TTC geometry.
RED events per scene	8 / 9 / 12	intersection_conflict, pedestrian_crossing, hard_brake. This is the operator's workload budget. Higher is worse.
AMBER to RED ratio	4.8x / 1.6x / 1.1x	Warnings should outnumber takeover calls. Too low and the trigger is jumpy, too high and operators tune out. The two analytic scenes sit low because they are built to hit RED.
Trigger coverage	9 kinds	Time to collision, overlap, off road, wrong way, lane compliance, hard brake, stalled, VRU proximity, and a composite that blends five of them.
Acceptance timesteps	t=31, t=68	Hard brake RED at 3.1 s and stall completion at 6.8 s are derived by hand from the scene construction and asserted in tests and CI.
Reproducibility	deterministic	Fixed seed per scene, and the two analytic scenes are seed independent by construction. Two runs give identical event logs.

What this does not do yet.

Real Waymo Open Motion Dataset scenes end to end. The loader exists, has no test, and needs a JAX and TensorFlow install to run against tfrecords.
Rank a fleet. The dashboard shows one scene at a time with per agent risk halos and one aggregate fleet score.
Two of the five bundled scenes, highway merge and stalled ego, have no capture on this page yet.
Operator ergonomics testing. The KPIs above cover engineering behaviour only. No user research ran.

Other captures

pedestrian_crossing

9 RED events. TTC and overlap tie as the leading triggers on two pedestrians at a crosswalk. One VRU proximity RED.

hard_brake

12 RED events. TTC from the front, hard brake fires on the lead vehicle at t=3.1s.

Full event logs: intersection_conflict · pedestrian_crossing · hard_brake