backend / vectorless-control-plane
Vectorless control plane
The multi-tenant Go front door for the Vectorless engine: accounts, orgs and stores, API keys and OAuth 2.1 for MCP clients, scope and quota checks on every proxied request, usage metering, and billing through Polar and Bachs.
2026 · paused · private repository
- [*]LanguagesGo · SQL · HCL
- [*]Frameworkschi · pgx · argon2id (x/crypto) · Prometheus client
- [*]StoresPostgres (Cloud SQL) · In-memory plan cache and rate-limit buckets
- [*]AsyncGoroutine tickers (hourly usage rollups, 30 s webhook delivery worker, quota alerts) · Fire-and-forget usage writes with a 5 s bound · Postgres-backed webhook delivery queue with backoff and dead-letter flag
- [*]InfraGoogle Cloud Run · Cloud SQL Postgres 16 · Global HTTPS load balancer + Certificate Manager · Secret Manager · Artifact Registry / Cloud Build · Terraform · Docker · GitHub Actions with Workload Identity Federation · Lorx CI · AWS App Runner / ECS Fargate / Aurora Serverless v2 (earlier Terraform) · AWS Lightsail + Caddy (starter stack)
- [*]Third partiesPolar · Bachs · Plunk · Amazon SES · SMTP · Google OAuth · GitHub OAuth · Vectorless engine
A private Go service (about 18,700 lines of non-test Go, 56 commits since 23 April 2026) that sits between every customer and the engine. It owns identity and tenancy in its own Postgres database: 18 numbered SQL migrations create 38 tables covering users, orgs, memberships, stores, API keys, sessions, OAuth clients and tokens, SSO and SCIM config, usage events with daily and monthly rollups, plans, billing events, invoices, outbound webhooks, audit log and MCP tool-call stats.
The customer API is a reverse proxy at /v1/*. Each request is authenticated by API key, session cookie or OAuth access token, mapped to a required scope (documents:read, documents:write, queries:run; unclassified paths are refused), checked against the org's monthly plan limits, then forwarded with caller credentials and any X-Vectorless-* headers stripped and the org, store, key and profile headers injected. Token counts come back from the engine as response headers or HTTP trailers on streams and are written as usage events off the request path.
Around that sit 55 route registrations in the main router plus 43 org-scoped routes declared with a minimum role each, an OAuth 2.1 authorization server for MCP clients (PKCE S256 only, dynamic registration, Client ID Metadata Documents, rotating refresh tokens), the CRUD adapter the Next.js dashboard's Better Auth uses as its database, outbound customer webhooks, and two billing providers behind one interface.
Production ran on Google Cloud Run behind a global HTTPS load balancer at api.vectorless.store, with Cloud SQL Postgres, defined in Terraform in vectorless-deploy (44 GCP resources), after earlier AWS App Runner/Fargate/Aurora and Lightsail Terraform. The hosted service is being wound down; the API still answered on 2026-10-12.
How it holds up
- Auth
- Argon2id passwords and SHA-256-hashed session tokens and vls_live_ API keys (prefix stored for display, hash for lookup); org routes check membership (404 for non-members so ids cannot be probed) and a per-route minimum role; /v1 accepts API key, session cookie or HS256 OAuth access token and enforces a required scope per path; internal routes use a dashboard service token.
- Data
- 38 tables from 18 migrations: users/orgs/memberships/org_invites/stores for tenancy; api_keys, sessions, auth_* (Better Auth), oauth_clients, oauth_authorization_codes, oauth_refresh_tokens, oauth_consents, oauth_revoked_jtis, sso_configs and scim_tokens for identity; usage_events rolled into usage_daily and usage_monthly; plans, plan_changes, billing_events, invoices; outbound_webhooks, webhook_deliveries, audit_log, mcp_tool_calls.
- Consistency
- Migrations run under a Postgres advisory lock against a schema_migrations ledger with SHA-256 checksums, adopting rows written by the older runner and refusing a changed file. Billing webhooks are deduplicated by (provider, event_id) under a unique constraint, and usage rollups are idempotent upserts recomputed from events.
- Failure
- A circuit breaker (5 consecutive failures, 30 s cooldown, 3 half-open probes) fails fast with 503 and Retry-After when the engine is down; outbound webhooks retry at 30 s, 2 m, 8 m, 32 m then dead-letter, refuse redirects, and re-check the destination at send time with a dialer that blocks non-public addresses. Quota checks fail open on an unknown plan or a usage-query error, and a usage write that fails is logged, not retried.
- Testing
- 25 _test.go files with 90 func Test cases (about 3,900 lines): org access and role checks, proxy forwarding and header stripping, billing providers and the launch offer, the migration ledger, OAuth provider sign-in, SSRF guard, webhook dispatcher, usage metering. CI runs gofmt, go vet, golangci-lint v2.14, an AI review gate (bit review --fail-on high) and go test ./....
Techniques
- Scope-gated, header-sanitising tenant proxyEvery /v1 request is mapped to a required scope by method and cleaned path (paths with '..' or '//' are refused, as are unclassified routes), then forwarded with Authorization, Cookie and all client-supplied X-Vectorless-* headers stripped and the trusted org, store, key and profile headers injected, so a caller cannot impersonate another tenant to the engine.
internal/model/scope.go:25internal/handler/proxy.go:67internal/proxy/proxy.go:201internal/proxy/proxy.go:211 - Metering from response headers and HTTP trailersThe engine reports tokens in X-Vectorless-Tokens-In/Out; for streamed answers the proxy reads them from HTTP trailers after the body. Each request, including rejected ones with their error code, becomes a usage event written off the request path, rolled hourly into usage_daily and usage_monthly by idempotent upserts, and the quota read adds today's unrolled events on top.
internal/proxy/proxy.go:263internal/proxy/proxy.go:292internal/handler/proxy.go:130internal/store/usage.go:311internal/store/usage.go:41 - Membership-then-role routing for org resourcesAll 43 org-scoped routes are declared in one table with a minimum role (any member, manager, owner) and mounted under a subrouter that resolves membership first; non-members get the same 404 as a non-existent org, and the spec type has no way to register a handler without a role.
internal/handler/orgaccess.go:147internal/handler/orgaccess.go:216internal/handler/router.go:243 - OAuth 2.1 server for MCP clientsAuthorization-code flow with S256-only PKCE, dynamic client registration behind a rate limit, Client ID Metadata Documents fetched (size-capped, SSRF-guarded) so a client like Claude can present a URL as its client_id, HS256 access tokens with a revoked-JTI table, refresh tokens rotated on use (old one revoked, new one linked by parent hash), and RFC 8414 / protected-resource metadata endpoints.
internal/oauth/pkce.go:22internal/oauth/cimd.gointernal/oauth/jwt.go:33internal/handler/router.go:151 - Checksummed migration ledger under an advisory lockMigrations apply once each under pg_advisory_lock so concurrent Cloud Run instances booting together cannot race, with a SHA-256 per file; rows from the older runner are adopted on first run and an edited, already-applied migration is detected rather than silently skipped.
internal/store/migrate.go:127internal/store/migrate.go:134internal/store/migrate.go:193 - SSRF-safe outbound webhooksCustomer webhook URLs are re-validated at delivery time and sent through an http.Client whose dialer Control hook refuses non-public IPs after DNS resolution and which follows no redirects; payloads are HMAC-SHA256 signed and failed deliveries back off 30 s / 2 m / 8 m / 32 m before dead-lettering.
internal/netguard/netguard.go:40internal/netguard/netguard.go:53internal/webhook/dispatcher.go:147internal/store/webhook.go:166
Counted
- 56Commits on main
git rev-list --count origin/main - 18,671Go LOC, non-test
find . -name '*.go' ! -name '*_test.go' | xargs wc -l - 22,572Go LOC including tests
find . -name '*.go' | xargs wc -l - 18Migrations
ls migrations/*.sql - 38Tables
grep -h 'CREATE TABLE' migrations/*.sql | wc -l - 55Route registrations in router.go
grep -cE '(Get|Post|Put|Delete|Patch|HandleFunc|Handle)\("/' internal/handler/router.go - 43Org-scoped routes with role specs
grep -c http.Method internal/handler/orgaccess.go - 25Test files
find . -name '*_test.go' | wc -l - 90func Test cases
grep -rh '^func Test' --include=*_test.go | wc -l - 44GCP Terraform resources (vectorless-deploy/gcp)
grep -c '^resource "' gcp/*.tf - 70AWS Terraform resources (vectorless-deploy/terraform)
grep -c '^resource "' terraform/*.tf - 29Deploy repo commits
git -C vectorless-deploy rev-list --count origin/main