backend / hypatia-engine
Hypatia content engine
A Cloudflare Worker that turns raw exam files into a solved, blind-verified question bank draft, using durable workflows, queues, a Python container and typed model calls behind one audited writer.
2026 · live · private repository
- [*]LanguagesTypeScript · Python · SQL
- [*]FrameworksCloudflare Workflows · Cloudflare Containers · Vercel AI SDK · Zod · @cfworker/json-schema
- [*]StoresD1 (SQLite) · D1 FTS5 · R2 · Vectorize · Durable Object storage
- [*]AsyncCloudflare Workflows (5) · Cloudflare Queues (2 + 2 DLQs) · Durable Object alarms/schedules · Workflow waitForEvent · WebSocket hibernation
- [*]InfraCloudflare Workers · Cloudflare Containers · Cloudflare AI Gateway · Workers AI · Docker · Workers Builds
- [*]Third partiesDeepSeek · Z.ai (GLM) · TypeSafe (Jev) · Workers AI (Qwen, bge-m3, Clef) · Google Drive API · R2 S3 API (presigned uploads) · Claude CLI (founder-side lane)
Hypatia is MBPrep's content engine, a separate Worker (hypatia.mbprep.app) with its own D1 database, R2 bucket, Vectorize index, two work queues plus their dead-letter queues, five Workflows and four Durable Object classes. An intake lists a Drive folder or presigned uploads, matches each file against the programme's register, streams new files into content-addressed R2 while it hashes them, and starts one document run per unique file. From there: extraction in a Python container, near-duplicate clustering, classification, question recovery, picture masking, question clustering, topic mapping, solving, a blind second-model check, and finally a bank export built with the laptop pipeline's own Python code.
Models only do typed jobs (18 skill packs, each with a JSON output schema or a set of typed questions, plus 113 eval cases), and code applies the thresholds. Every D1 and R2 write goes through one writer that logs who decided it, which decision caused it, and a before-image, so a whole decision can be undone atomically. Every stage run is keyed by a hash of its input plus the pack's folder hash, so editing a pack reprocesses exactly the items it touched and a redelivered message costs nothing.
Spend is capped per intake by a Budget Durable Object that reserves each call's worst-case cost before it starts, so parallel calls cannot overrun the cap. A ModelGate Durable Object shares five Z.ai slots across every Worker and shrinks or grows that window the way TCP congestion control does; a call that finds no slot goes to its DeepSeek stand-in instead of retrying into a 429.
Built in 120 commits between 2026-10-03 and 2026-10-10: about 27,100 lines of TypeScript in src/, 11,100 lines of tests across 33 Bun test files (430 test cases), and 1,900 lines of Python in the container with 32 unittest cases. Export parity with the laptop pipeline is checked byte for byte: 6,536 of 6,536 MB2 records and 13,475 of 13,475 MB4 records through the finish op.
How it holds up
- Auth
- Every route but /health requires a bearer HYPATIA_TOKEN, compared by SHA-256 hash to avoid timing leaks; an x-hypatia-actor header (cli:, mcp:, admin:, founder:) names who acts for the audit log, and founder: is accepted only for addresses in FOUNDER_EMAILS. The container has no internet and reaches R2 only through a Worker-side proxy that allows writes to derived/, a run's exports/*/*/out/ folder, and served pictures whose bytes hash to their key.
- Data
- 13 D1 migrations create 39 tables plus an FTS5 page index: sources, blobs, documents, doc_clusters/doc_lsh, classifications, questions, q_clusters, q_occurrences, question_images, mappings, solutions, verifications, intel_versions, predictions, escalations, decisions, stage_runs, writes, lane_jobs and more. R2 is content-addressed (blobs/sha256/..., media/<sha256>.webp); SQLite triggers refuse any recovered question whose record carries an answer, key, correct or explanation field.
- Consistency
- Stage runs take a lease with one INSERT ... ON CONFLICT ... WHERE status='failed' OR lease expired ... RETURNING statement, so two deliveries of the same item never pay a model twice. The writer pairs each statement with its log row in one D1 batch, logs only when changes() > 0 for conditional writes, and makes undo all-or-nothing with an in-batch guard that inserts NULL into a NOT NULL table to roll the batch back if anything else wrote the row in the meantime. Clustering and mapping run under per-programme leases, and Claude-lane jobs are claimed with a 30-minute renewable lease.
- Failure
- Queue consumers classify failures: provider outages and busy stages retry with exponential backoff capped at 10 minutes, deterministic failures (bad schema output, export.py raising) are answered back to the waiting workflow at once, and dead letters from hy-extract-dlq and hy-llm-dlq are recorded as events and replied to the workflow, never retried. Model calls fall back GLM to DeepSeek to deepseek-v4-pro to Qwen; budget pauses wait up to 7 days for a resume event; a container watchdog on a Durable Object schedule sends SIGTERM to idle or stuck Extractors and destroys them if they ignore it.
- Testing
- 33 Bun test files (430 test/it cases, about 11,100 lines) cover the writer and undo, Budget, ModelGate, stage leases, queues policy, flows over a fake Steps, the API, contracts, clustering, intel parity with patterns.py, solving, verifying, rework and lanes; container/test_ops.py adds 32 Python unittest cases; 18 skill packs carry 113 eval cases run live with
hy eval; parity scripts compare exports with the laptop bank byte for byte.
Techniques
- One audited writer with atomic undoEvery D1/R2 write goes through a writer that limits tables to a fixed allow-list, logs actor, decision and before-image in the same D1 batch, and undoes a whole decision newest-first. Undo re-checks inside the batch by inserting NULL into a NOT NULL guard table when a row changed since, which rolls the whole batch back instead of overwriting someone else's write.
apps/hypatia/src/writer.ts:15apps/hypatia/src/writer.ts:177apps/hypatia/src/writer.ts:411apps/hypatia/src/writer.ts:461apps/hypatia/migrations/0001_init.sql:323 - Content-keyed stage runs with a single-statement leaseA stage's key is sha256(stage | version | digest(input)), and the pack version is a hash of its folder. The lease is taken with one upsert that succeeds only on a new, failed or expired row, so redelivered queue messages and replayed workflow steps return the cached output and never pay the model twice; large outputs spill to R2.
apps/hypatia/src/stages.ts:51apps/hypatia/src/stages.ts:81apps/hypatia/scripts/pack-skills.ts:4 - Reservation-based spend cap in a Durable ObjectOne Budget object per intake admits a model call only if spent + outstanding reservations + the call's worst-case estimate (across its whole fallback chain) fits under the cap. Reservations expire after 16 minutes so a dead call releases its headroom, and a paused intake's workflow waits on a budget-resume event rather than failing.
apps/hypatia/src/budget.ts:71apps/hypatia/src/objects.ts:12apps/hypatia/src/run-skill.ts:152apps/hypatia/src/workflows/steps.ts:38 - Character-shingle MinHash + LSH for same-paper detectionDocuments are reduced to letters and digits, shingled into 5-character pieces, signed with 128 seeded hash permutations, and bucketed into 32 bands of 4 rows. Character shingles survive OCR errors: a Tesseract scan scored 0.86 Jaccard against its PDF where word 5-grams gave 0.59. A document writes its own buckets before it reads candidates, so two copies clustered at the same moment still find each other and merge under the smallest id.
apps/hypatia/src/minhash.ts:12apps/hypatia/src/minhash.ts:57apps/hypatia/src/dedup.ts:30 - Failure-classifying queue consumers with dead-letter repliesonFailure separates retryable failures (provider outages, busy stages, reserved budget) from deterministic ones (schema failures after retries, export.py raising) so the queue does not pay a model again for the same wrong answer. Dead letters become events and are replied to the waiting workflow, so a workflow never waits forever on a lost job.
apps/hypatia/src/queues-policy.ts:18apps/hypatia/src/queues.ts:58apps/hypatia/wrangler.jsonc
Counted
- 120Commits touching apps/hypatia
git log --oneline -- apps/hypatia | wc -l - 904Monorepo commits
git rev-list --count HEAD - 2026-10-03 / 2026-10-10First / last commit
git log --format=%ad --date=short -- apps/hypatia - 5Workflows
wrangler.jsonc workflows[] - 4 (Workspace, Budget, Extractor, ModelGate)Durable Object classes
wrangler.jsonc durable_objects - 2 work + 2 dead-letterQueues
wrangler.jsonc queues.consumers - 18Skill packs
ls apps/hypatia/skills | wc -l - 113Skill eval cases
cat skills/*/evals/cases.jsonl | wc -l - 21JSON Schema contracts
ls apps/hypatia/contracts | wc -l - 13D1 migrations
ls apps/hypatia/migrations - 39 + 1 FTS5Tables created
grep CREATE TABLE migrations/*.sql - 77 (35 GET, 40 POST, 2 PUT)API routes
grep -oE 'method === "(GET|POST|PUT)"' src/api.ts | sort | uniq -c - 27,139TypeScript LOC in src/
git ls-files apps/hypatia/src | grep .ts$ | xargs wc -l - 33 / 430Test files / cases
ls test/*.test.ts; grep -cE '^\s*(test|it)\(' test/*.test.ts - 11,094Test LOC
git ls-files apps/hypatia/test | grep .ts$ | xargs wc -l - 1,938 / 32Container Python LOC / unittest cases
git ls-files apps/hypatia/container | grep .py$ | xargs wc -l; grep -c 'def test_' container/test_ops.py - HTTP 200, engine 6, packs 18; /status 401 without tokenLive health
curl -sI https://hypatia.mbprep.app/health (2026-10-12)