backend / godlogic

GodLogic

A Bun/Hono API that runs hybrid dense-plus-lexical search over one table of Quran, Hadith, Bible and Tafsir passages, builds cited arguments from what it retrieved, and turns live debate audio into claim-triggered evidence cards over a WebSocket.

2026 · paused · private repository

clientserviceworkerstorethird partymodelasync

[*]LanguagesTypeScript
[*]FrameworksHono · tRPC · Drizzle ORM · Bun · Turborepo · transformers.js
[*]StoresPostgres · pgvector · tsvector/GIN
[*]AsyncSSE streaming · WebSocket · resumable batch embed job
[*]InfraDocker Compose (pgvector/pgvector:pg16) · Xata Postgres
[*]Third partiesDeepgram · Google Gemini · Zhipu GLM · Hugging Face (local e5 model)

Every ayah, hadith, Bible verse and tafsir block is one row in a single Postgres passages table, keyed by a canonical reference string such as quran:4:34@sahih-intl. A pgvector embedding column sits next to a generated tsvector column, so one table serves both search legs.

Search runs a dense query (cosine distance on pgvector) and a lexical query (websearch_to_tsquery + ts_rank_cd on a GIN index) at the same time. Each leg pulls max(50, 5k) candidates with the same metadata filters, and Reciprocal Rank Fusion (k=60) merges them, breaking ties by id so the order is reproducible. A SEARCH_MODE=bm25 switch falls back to lexical-only.

The argument builder sends the model only the passages it retrieved, under a JSON schema. A pure guard function then drops any point that cites a passage id outside the retrieved set, and if no point survives it forces insufficientEvidence. The live mode streams 16 kHz PCM from the client over a WebSocket to Deepgram, debounces final transcript segments for 1.2 s, extracts claims with a fast model, dedupes them against the last 20, and pushes retrieved cards back down the same socket.

Ingest is a set of idempotent batch jobs: parsers for 4 corpora upsert in batches of 500 with ON CONFLICT on the canonical ref and never overwrite an existing embedding. A resumable embed job picks up only rows where the embedding is NULL and writes each chunk back in one UPDATE ... FROM unnest(). Arabic is stored twice: the untouched Uthmani text for display and a normalised form (diacritics stripped, alef/ya/ta-marbuta folded) that feeds the tsvector.

How it holds up

Auth: None on the API: every tRPC procedure is publicProcedure, CORS is *, and the tRPC context carries only the DB handle. Provider keys stay server-side, so clients talk only to the API's WebSocket and never to Deepgram directly.
Data: 3 Drizzle tables (passages, decks, deck_items) and 2 enums (source, hadith grade). There are no migration files; the schema is applied with drizzle-kit push after a setup script enables pgvector. passages has a unique canonical-ref index, a composite location index for verse-window context lookups, a collection+grade index and a GIN index on the generated tsvector.
Consistency: Ingest is idempotent: upserts use ON CONFLICT (ref_canonical) DO UPDATE and leave embeddings out of the SET list. The embed job is restartable because it only selects rows where the embedding is NULL and writes each chunk in a single bulk UPDATE. RRF breaks ties by id, so the same inputs always give the same order.
Failure: An LLM failure during /argue becomes an SSE error event instead of a dropped stream. A missing Deepgram or LLM key on /live is sent to the client as an error frame instead of crashing the socket. A retrieval failure for one live claim is logged and skipped. If the model cites nothing grounded, the guard forces insufficientEvidence instead of returning a confident answer with no evidence.
Testing: 3 bun:test files with 7 cases (RRF ordering, determinism and formula; hallucination-guard drop and force; live missing-key). There is also a DB-free corpus check script (packages/ingest/src/verify.ts, 16 assert/eq calls) covering per-source row counts, the Arabic store-twice rule and hadith narrator/grade extraction.

Techniques

Parallel hybrid retrieval with Reciprocal Rank FusionDense pgvector and lexical tsvector legs run at the same time, each pulling max(50, 5k) candidates under the same pushed-down filters. RRF (k=60, ties broken by id) merges them, and only the surviving ids are hydrated and re-sorted by fused score.packages/retrieval/src/index.ts:95 packages/retrieval/src/index.ts:141 packages/retrieval/src/rrf.ts:33
Deterministic hallucination guardAfter the schema-constrained model call, a pure function drops every argument point whose passageId was not among the retrieved passages. If none survive it forces insufficientEvidence, so the grounding rule is enforced in code whatever the model does.packages/argue/src/guard.ts:17 packages/argue/src/index.ts:84 packages/argue/src/guard.test.ts:17
Audio-to-evidence pipeline over one WebSocketBinary PCM frames go to a server-held Deepgram stream. Final segments feed a debounced (1.2 s), windowed (1200 chars) claim detector that dedupes against the last 20 claims, and each claim triggers a hybrid search whose cards return over the same socket.apps/api/src/index.ts:49 packages/live/src/index.ts:146 packages/live/src/index.ts:184
Idempotent, resumable corpus pipelineBatched upserts keyed on the canonical ref deliberately leave embeddings out of the SET list. The embed job processes only rows with a NULL embedding, fans out EMBED_CONCURRENCY x 64-text requests, and writes each chunk with one UPDATE ... FROM unnest(uuid[], text[]).packages/ingest/src/ingest.ts:28 packages/ingest/src/embed.ts:63 packages/ingest/src/embed.ts:81
Store-twice Arabic indexingThe faithful Uthmani text is kept for display, and a separate normalised form (Quranic marks and tatweel stripped, alef/hamza/ya/ta-marbuta folded, NFC) feeds a generated tsvector column, so lexical search matches spelling variants without changing the displayed text.packages/ingest/src/normalize/arabic.ts:37 packages/db/src/schema.ts:73
Pluggable embedder behind one importIngest and query encoding both import the same lazily loaded provider (Gemini, GLM, or local transformers.js e5-small with query:/passage: prefixes), so the corpus and the queries are always embedded by the same model.packages/embed/src/index.ts:12 packages/embed/src/local.ts:28

Counted

22commits
git rev-list --count HEAD
6655TypeScript LOC (all tracked .ts/.tsx)
git ls-files '*.ts' '*.tsx' | xargs wc -l | tail -1
2617backend LOC (packages/* + apps/api .ts, excl. tests)
git ls-files 'packages/*.ts' 'apps/api/*.ts' | grep -v test | xargs wc -l | tail -1
3DB tables
packages/db/src/schema.ts (passages, decks, deck_items)
4tRPC procedures
packages/api/src/router.ts (health, search, lookupRef, passageContext)
4HTTP/WS endpoints on Hono
apps/api/src/index.ts (/health, /trpc/*, POST /argue SSE, GET /live WS)
4corpora parsed
packages/ingest/src/parse/{quran,hadith,bible,tafsir}.ts
3 / 7test files / cases
grep -E '^\s*(it|test)\(' packages/*/src/*.test.ts
60RRF damping constant
packages/retrieval/src/rrf.ts:15