BRKMYR/ AI Builder Portfolio/ Assurance Gate

Assurance Gate

Evaluation results in, release decision out. Gates are written down and frozen before the run. Each one reports a confidence bound next to its threshold, and the dashboard carries the evidence for every number it shows.

Problem

A score is not a release decision.

Every benchmark I have built ends in a number, and the number was never the hard part. The hard part is the release meeting: is this good enough, against which threshold, decided when, by whom. If the threshold is set after the run, the answer is worthless. This project makes the meeting a page and forces the threshold to exist first.

Two habits make an evaluation score unusable as a decision. Thresholds move after the numbers are known, and a rate from a small sample is read as though it were certain. Both are invisible in the final slide.

Solution

Freeze the gates, publish the hash, then run.

A gate file holds every threshold with its rationale and its source. The file is hashed, the hash is published, and the run manifest records both timestamps. A reader can see whether the gates existed before the result did.

Each rate gate passes on a Clopper Pearson bound rather than on the point estimate, so a small sample cannot buy a pass. Count gates on collisions stay absolute. Coverage checks that every parameter bin was drawn at all. A paired regression compares the candidate against its baseline scenario by scenario. The verdict is GO, NO GO or CONDITIONAL, and a waiver can only soften a soft gate, never a hard one.

Two tracks share the engine. Track A reads driving scenario runs from a simulation suite. Track B reads language model safety evals produced with Inspect.

Result

One verdict, with the evidence under it.

On the demo suite the cautious planner fails 3 of 9 gates. 11 of 16 occluded pedestrian scenarios end in a collision, so the hard gate fires before any margin matters. The cut in TTC gate passes on the point estimate (13 of 16) and fails on the 95 percent lower bound (0.54 against a floor of 0.60). That gap is why the bound is the rule.

Tracks

Gate classes

Pages

Taps to evidence

The live dashboard runs on a hosted Space. It loads only when you ask for it, so this page makes no outside request on its own.

Success criteria

When this counts as working.

The gates for a run were frozen and published before that run started, and the dashboard shows both times.
Every rate gate reports a bound next to its point estimate, and the bound is what decides.
Coverage names every empty bin, and an empty bin blocks the release on its own.
A regression against the baseline catches any scenario that got worse, even when the average improved.
Nothing private reaches a published file. No raw trajectory log, no local path, no address.
A reader reaches the evidence behind a failing gate in three taps on a phone.
The build is deterministic. The same inputs produce the same bytes.
Every judgement call in the build is written down where a reviewer can find it.

Metrics, KPIs, evals

What the dashboard reports.

Metric	Type	What it tells the PM
Verdict	GO, NO GO, CONDITIONAL	The release call for one candidate against one frozen gate file. CONDITIONAL means a soft gate failed and a signed waiver carries it.
Gates failed of n	count	How much of the gate set is red, and which classes. Hard gates rank first, then severity, so the first row is the one to fix.
Coverage empty bins	count	Parameter ranges the suite never drew. An empty bin means the claim was never tested there, and it blocks the release.
Regression drops	count of pairs	Scenarios where the candidate got worse than the baseline by tier or by time to collision. A better average still fails here.
Bound example	rate and bound	Fourteen passes out of sixteen give a point estimate of 0.875 and a lower bound near 0.62. A threshold of 0.70 fails on the bound.

What this does not do.

It does not replace a safety case. It produces the evidence layer one would sit on.
A gate can only see what the suite tested. The limitations page lists the hazards no gate covers.
Demo runs are labelled as demo. Their gates were written after the fact, and the dashboard says so.