Kartograf

At a glance

Recording · two questions in real time · qwen3:8b on a laptop

Responses captured from the running server with qwen3:8b, thinking off. Raw JSON: query_road_km.json · query_pharmacies.json · query_construction.json · meta.json · eval_summary.md

Problem

Map data answers real questions. Asking it takes a specialist.

Open map data is now good and free: Overture Maps publishes roads, places, buildings and boundaries for the whole world every month. The people with the questions (city planners, retail site scouts, logistics teams, journalists) mostly cannot write spatial SQL, and a GIS analyst has a queue.

A language model closes the gap only if it is grounded. Without tools it produces a confident number with no source. With a database tool it can still filter wrongly and present the result as fact. The product question is not whether a model can write SQL, but how a user can see when it did not.

What it does

One question in, an answer with its evidence out.

The web app loads a Frankfurt extract of Overture roads and places into an in-memory DuckDB database and shows it on a kepler.gl map. A question typed into the ask bar goes to a local model with one tool, run_sql. The answer arrives with a chart, a map layer when places are involved, and the list of SQL statements, each marked as written by the model or added by Kartograf. A second surface, a Streamlit workbench, holds the data tables, release statistics and the query log.

• Answers plain questions in English or German through a tool-calling loop with one read-only SQL tool, using a local model (Qwen3 8B with thinking off by default, Llama 3.1 8B selectable). llm.py, server.py
• Derives a chart from the model's own query: a simple total over roads or places is rerun grouped by road class or place category with the model's filter unchanged, and places matching a filter are listed for the map. server.py derived_queries, tests/test_server.py
• Highlights one chart bar on the map when clicked, through a whitelisted column and a bound SQL parameter, and flies the map to the result. server.py highlight_layer, web/index.html
• Fetches any area from the public Overture bucket in one pass with Parquet row-group pruning, with a footer-only cost estimate first: about 40 MB for Frankfurt roads and places. fetch_overture.py --places --estimate
• Scores models on 20 questions with reference SQL, in English and German, including four the data cannot answer. run_evals.py, evals/questions.yaml
• Renders the map in a theme built for this site: Inter Tight, a warm monochrome palette, base data in light greys and ink only for answers. web/kartograf-theme.js

Solution

Give the model one tool. Show everything it did with it.

The model sees the table schema, three sample rows per table and a short data note, and has one tool. Every statement passes a read-only guard (a single SELECT, no file or network readers) and runs in DuckDB with external access disabled. Spatial know-how sits in the prompt as recipes, for example that distances on longitude and latitude need ST_Distance_Sphere in metres, rather than in extra tools that would hide the logic.

One design choice was measured rather than assumed. Asking the 8B model to also write the chart query made its first query worse: Llama 3.1 started emitting failing SQL and invented a road total of 2,312 km. So the chart moved to the server. It reuses the model's filter verbatim, which means the chart explains the model's number, including when the number is wrong. In one test Llama counted 5,193 "pharmacies" near the station, and the derived chart showed at a glance that it had counted every business.

Result

Runnable SQL every time. Right answers 12 to 15 times in 20.

On the 20-question baseline, both models produced SQL that ran for all 16 answerable questions. Qwen3 8B answered 15 correctly and Llama 3.1 8B 12. The largest failure class is guessed category values: the models wrote 'permanently closed' or 'dauerhaft geschlossen' instead of 'permanently_closed' and reported 0 without noticing. German costs both models: each scored 6 of 10 there. Llama invented one answer, counting engineering firms as 346 construction sites. One failure was mine, not the models': a loader bug made is_named = TRUE match nothing, so both scores are one too low until the next run.

Solution architecture

What it stands on, and what is new.

Sources: Overture transportation data is ODbL and requires "© OpenStreetMap contributors"; the places theme combines sources under CDLA Permissive 2.0, Apache 2.0 and CC0 1.0 (docs.overturemaps.org/attribution). Extracts stay out of the repository and are rebuilt with the fetcher. Qwen3 is Apache 2.0. Llama 3.1 is under the Llama 3.1 Community License; the project calls a locally installed model and ships no weights. DuckDB, kepler.gl, React, Redux and styled-components are MIT, MapLibre is BSD 3, pandas and Altair are BSD 3, Streamlit and pydeck are Apache 2.0. Basemap © CARTO, © OpenStreetMap contributors. Inter Tight and IBM Plex Mono are SIL Open Font License. Everything else is this project, MIT. Independent project, not affiliated with Overture Maps Foundation, OpenStreetMap, Meta or Alibaba.

Why it matters

An agent over a data platform is only as good as what it lets you check.

Natural-language access to data platforms is becoming a standard feature, and most demos stop at the answer. This project puts the weight on the parts a product team has to own: one narrow tool with a guard, every statement visible, a chart that is derived from the model's own query so that wrong filters show, and an eval with reference answers that turns "it seems to work" into 12 and 15 out of 20. It runs entirely on a laptop, which makes the same pattern usable on data that may not leave the building.

How it was built

A data chat prototype, then two days to agent, map and eval.

Product direction, design decisions and review by hand. Implementation with Claude Code agents. The prototype was kept outside version control; git history starts with its import, followed by the agent, map, eval and web app. The web front end adds 464 lines of HTML and JavaScript. Python lines include 626 lines of tests, which run offline without a model. Measured from the repository.

The starting point was a small Streamlit app that let a model query CSV and GeoJSON files through one SQL tool, with an Overture fetcher for German road statistics. It then became Kartograf: a local Llama as the default, a Frankfurt preset with the places theme (whose category field had moved to taxonomy.primary since earlier releases), a kepler.gl map, and the eval harness. The first full eval run surfaced the loader bug that cost both models a point. The web app followed: the ask bar, server-side chart derivation after the prompt-based version made Llama worse, Qwen3 with thinking off as the default after it answered correctly in about a tenth of the time, and the paper-map styling.

Success criteria

When this counts as working.

• Every answer shows every SQL statement behind it, marked as written by the model or by Kartograf. No number without a visible query.
• No statement can write, read files or reach the network, whatever the model emits.
• Questions the data cannot answer get a refusal, not a number. Met by Qwen3 in the eval (4 of 4), not yet reliably in the web app.
• At least 18 of 20 eval questions correct with a local 8B model, in both languages.
• An answer in under 15 seconds on a 16 GB laptop, with no data leaving the machine.

Metrics, KPIs, evals

Numbers from the baseline eval and the captures above.