backend / coursified-backend
Coursified backend
A Hono API on Cloudflare Workers that turns a YouTube video into a structured, graded course through a ten-step pipeline run by a Durable Object, and offers the same catalogue and generation tools to AI agents over an OAuth-protected MCP endpoint.
2025–26 · live · private repository
- [*]LanguagesTypeScript · SQL
- [*]FrameworksHono · Drizzle ORM · better-auth · Vercel AI SDK · zod · MCP (Streamable HTTP) · Bun
- [*]StoresPostgres (Neon) · Cloudflare KV · R2 · Durable Object storage (SQLite) · Analytics Engine
- [*]AsyncCloudflare Queues (with DLQ) · Durable Object alarms · cron (every 5 min, daily, weekly)
- [*]InfraCloudflare Workers · Durable Objects · Docker · Traefik / Dokploy (self-hosted relay) · Vercel (web)
- [*]Third partiesZ.ai GLM · YouTube Innertube · YouTube Data API v3 · E2B sandboxes · Bachs payments · Google OAuth · residential proxy (DataImpulse) · Cloudflare Analytics Engine SQL API · self-hosted mail relay
The API is one Cloudflare Worker (coursified-api): Hono, 24 sub-routers and 131 route registrations, Neon Postgres through Drizzle's HTTP driver, KV as the cache, R2 for AI trace bodies, and four Analytics Engine datasets for request, product, AI and MCP telemetry. A POST to create a course writes the course and job rows and puts a course.requested event on a Cloudflare Queue. The queue consumer hands it to a JobOrchestrator Durable Object keyed course:<id>. That object holds a per-step ledger and runs ten steps, from resolve_source to finalize, one per alarm so each step starts with a fresh Worker subrequest budget.
Transcripts come down a fallback ladder. First a separate fetcher Worker that tries Innertube client identities in turn (ANDROID, then IOS). Then a Bun relay in Docker on a self-hosted box, which sends the same request through a residential proxy, since Workers cannot proxy their own fetch(). Then the transcript library, run through the proxy, inside the API Worker. Last, a transcript built from the creator's own chapter marks. Every tier's outcome goes into a KV-backed health counter that emails an alert after 5 consecutive failures, with a 6-hour cooldown. Caption text is stored with a 21-day TTL and purged by a daily cron, to stay inside YouTube's 30-day rule for stored data.
Every model call goes through one generate() function. It checks a KV artifact cache keyed by video, prompt version and transcript span. On a miss it calls GLM over Z.ai's OpenAI-compatible endpoint, validates the reply against a zod schema and allows one repair round. Every attempt's prompt and response goes to an R2 trace. Generated coding projects are run in an E2B sandbox before they are saved: the reference solution has to pass every test, and the starter code has to fail the hidden ones.
The same Worker serves an MCP endpoint at mcp.coursified.app. better-auth acts as the OAuth authorization server, with dynamic client registration (rate-limited per IP by a token-bucket Durable Object), CIMD, PKCE and audience-bound tokens. API keys are a second credential path. The tool registry defines 18 tools under 6 scopes; 11 have handlers, and a client only sees tools that have a handler and that its scopes allow. 543 commits between 2026-09-03 and 2026-09-17, rebuilt from an older Next.js + tRPC + Inngest version started in December 2025.
How it holds up
- Auth
- better-auth with email OTP, Google OAuth, cookie sessions on .coursified.app plus a bearer plugin for mobile, admin roles. It is also a full OAuth 2.1 authorization server for MCP: dynamic client registration (open, capped at 10 per IP by a Durable Object), CIMD, PKCE, JWKS-signed tokens bound to the MCP resource audience, and scoped API keys. /internal routes need a service token, and Bachs webhooks are checked by HMAC-SHA256 with timing-safe comparison and a 5-minute replay window.
- Data
- 49 Drizzle tables (13 for auth and OAuth, 36 for the app: courses, modules, jobs, job_steps, transcripts, pathways, certificates, billing_events, credit_grants, cost_entries, events_dlq and more) and 6 Postgres enums. Schema changes are 24 hand-written SQL migrations applied one at a time by a script, because the database was built with drizzle-kit push and has no migration history.
- Consistency
- Neon's HTTP driver has no interactive transactions (zero .transaction() calls), so correctness comes from a different place. The Durable Object serialises each course's pipeline and keeps an in-memory guard against overlapping runs. Steps already marked done are skipped when a job is replayed. Batched steps hand back a cursor that must move forward. Webhooks are deduplicated on the event id as primary key, and the idempotency row is deleted again if the handler throws. Conditional UPDATEs only demote a course that is still generating, and there are 26 onConflictDo upserts.
- Failure
- A failed step retries up to 4 times, scheduled by Durable Object alarm with exponential backoff and jitter (2^tries × 30 s × 0.5–1.5); transcript errors known to be permanent skip the retries. After the last try the job is marked failed, a row goes to events_dlq and the user gets a failure email with a resume link; the queue itself has max_retries 3 and its own DLQ. A 5-minute cron watchdog fails jobs that have made no progress for 15 minutes. Retry resumes from the failed step and keeps finished steps' output. The rate limiter returns tokens for work that was charged but never started.
- Testing
- 47 bun:test files in apps/server with 648 test/it cases, and 2 in packages/mcp with 73 (contract and transport), so 721 backend cases. 89 test files across the whole repo. CI (ci.yml, deploy-worker.yml) is written but has never run because of GitHub Actions billing, so CLAUDE.md requires a manual gate before every PR (typecheck, test, lint, build) with the output pasted on the Linear issue.
Techniques
- Durable Object step ledger with alarm-driven resumptionEach course gets one Durable Object that stores a record per step. A replay skips steps already done. A batched step hands back a cursor that must move forward, and progress resets its retry budget. The DO re-arms its alarm after every step so a long course never runs out of Worker subrequests. Failures back off exponentially with jitter, and after the last try the job goes to a dead-letter table.
apps/server/src/do/orchestrator.ts:55apps/server/src/do/orchestrator.ts:250apps/server/src/do/orchestrator.ts:404apps/server/src/do/orchestrator.ts:425 - Transcript fallback ladder with tier health alertingTranscripts come from an edge Worker that rotates Innertube client identities, then a Bun relay behind a residential proxy (it exists because Workers cannot proxy fetch()), then the library through the proxy, then the creator's chapter marks. Each tier's outcome feeds a pure decision function backed by KV that emails an alert after 5 consecutive failures, so a broken tier gets noticed before users report it.
apps/server/src/lib/transcripts.ts:96apps/server/src/lib/transcripts.ts:247apps/server/src/lib/tier-health.ts:52apps/fetcher/src/index.ts:40apps/relay/src/index.ts:1 - Schema-enforced LLM calls with a content-addressed cacheZ.ai ignores response_format, so the zod schema is serialised into the system prompt, JSON is pulled out of the reply and validated, and validation errors go back to the model for one repair round. A valid result is cached in KV under a key built from video, prompt version and transcript span, and the key includes a hash of the prompt contract, so changing the prompt wording invalidates the old entries.
apps/server/src/ai/client.ts:30apps/server/src/ai/client.ts:82apps/server/src/ai/pipeline.ts:761 - Proving generated exercises before saving themA model-written project exercise is graded in an E2B sandbox twice. It is rejected if the reference solution fails any test, or if the starter code already passes the hidden tests. A rejected exercise is also deleted from the KV cache, so the next run cannot serve it again.
apps/server/src/ai/pipeline.ts:1033apps/server/src/ai/pipeline.ts:1037apps/server/src/ai/validate.ts:270apps/server/src/ai/validate.ts:275 - Webhook idempotency that survives handler failureBachs webhooks are checked with HMAC-SHA256: every candidate signature is compared in constant time, and timestamps older than 5 minutes are rejected. The event id is inserted as a primary key before the handler runs, so a redelivery stops there. If the handler throws, the row is deleted and the endpoint returns 500, so a retry starts clean instead of being recorded as a duplicate with nothing applied.
apps/server/src/routes/billing.ts:382apps/server/src/routes/billing.ts:406apps/server/src/lib/bachs.ts:260apps/server/src/lib/bachs.ts:265
Counted
- 543Commits
git rev-list --count HEAD (2026-09-03 to 2026-09-17) - 131 across 24 sub-routers (+3 in index.ts)API route registrations
grep -E '^\s*[a-zA-Z]+Route\.(get|post|put|patch|delete|on|all)\(' apps/server/src/routes/*.ts apps/server/src/mcp/*.ts - 10Pipeline steps
apps/server/src/ai/pipeline-steps.ts PIPELINE_STEPS - 49 (13 auth + 36 core)Postgres tables
grep -c '= pgTable(' packages/db/src/schema/*.ts - 24SQL migrations
ls packages/db/migrations | wc -l - 18 / 11MCP tools registered / implemented
packages/mcp/src/schemas/index.ts vs handlers in apps/server/src/mcp/*.ts - 6OAuth scopes
packages/mcp/src/schemas/common.ts SCOPES - 49 (47 server + 2 mcp)Backend test files
git ls-files 'apps/server/*.test.ts' 'packages/mcp/*.test.ts' - 721Backend test cases
grep -cE '^\s*(it|test)\(' over those files - 21,124Server TypeScript LOC (non-test)
git ls-files 'apps/server/src/*.ts' | grep -v test | xargs wc -l - 8,481Server test LOC
git ls-files 'apps/server/src/*.test.ts' | xargs wc -l - 3Cron schedules
apps/server/wrangler.jsonc triggers.crons - 4 tries, 2^n x 30s backoffStep retry budget
apps/server/src/do/orchestrator.ts:404 - 21 daysTranscript retention
packages/db/src/schema/core.ts:31