backend / voxtar

Voxtar backend

A Hono + tRPC API and a LiveKit voice-agent worker that collect patients' voice recordings, send them to an acoustic-analysis service, and file the results as de-identified clinical observations.

2026 · paused · private repository

mobile → caddy: HTTPS tRPC + bearerweb → caddy: HTTPS tRPC + cookieweb → livekit: WebRTC (wss)caddy → api: reverse proxy :8080caddy → livekit: wss :7880livekit → agent: room job dispatchagent → livekit: clinical-telemetry data channelagent → web: GET agent configcron → api: sendDueRemindersapi → pyz: multipart WAV + X-API-Keyagent → pyz: multipart WAV + X-API-Keyapi → pg: Drizzle (Neon WS)api → s3: PutObject SSEagent → speech: STT/TTS/LLM streamsapi → notify: OTP SMS, pushPatient app (Expo)Patient app (Expo) (client) phone-OTP, daily captureClinician web(Next.js)Clinician web (Next.js) (client) Vercel, own tRPC routersHost cronHost cron (worker) hourly remindersCaddyCaddy (edge) TLS for api + livekit hostsLiveKit serverLiveKit server (service) self-hosted WebRTC SFUHono + tRPC backendHono + tRPC backend (service) 33 procedures, BetterAuthLiveKit voice agentLiveKit voice agent (worker) Node, Silero VAD, turn detectorDeepgram / GLMDeepgram / GLM (third party) STT, TTS, LLMpyzheimer FastAPIpyzheimer FastAPI (service) localhost:8001, API keyNeon PostgresNeon Postgres (store) 48 tablesS3 audio bucketS3 audio bucket (store) SSE, consent-gatedTwilio / Termii / ExpoTwilio / Termii / Expo (third party) OTP SMS, pushHTTPS tRPC + bearerHTTPS tRPC + cookieWebRTC (wss)reverse proxy :8080wss :7880room job dispatchclinical-telemetry da…GET agent configsendDueRemindersmultipart WAV + X-API…multipart WAV + X-API…Drizzle (Neon WS)PutObject SSESTT/TTS/LLM streamsOTP SMS, push mobile → caddy: HTTPS tRPC + bearerweb → caddy: HTTPS tRPC + cookieweb → livekit: WebRTC (wss)caddy → api: reverse proxy :8080caddy → livekit: wss :7880livekit → agent: room job dispatchagent → livekit: clinical-telemetry data channelagent → web: GET agent configcron → api: sendDueRemindersapi → pyz: multipart WAV + X-API-Keyagent → pyz: multipart WAV + X-API-Keyapi → pg: Drizzle (Neon WS)api → s3: PutObject SSEagent → speech: STT/TTS/LLM streamsapi → notify: OTP SMS, pushPatient app (Expo)Patient app (Expo) (client) phone-OTP, daily captureClinician web(Next.js)Clinician web (Next.js) (client) Vercel, own tRPC routersHost cronHost cron (worker) hourly remindersCaddyCaddy (edge) TLS for api + livekit hostsLiveKit serverLiveKit server (service) self-hosted WebRTC SFUHono + tRPC backendHono + tRPC backend (service) 33 procedures, BetterAuthLiveKit voice agentLiveKit voice agent (worker) Node, Silero VAD, turn detectorDeepgram / GLMDeepgram / GLM (third party) STT, TTS, LLMpyzheimer FastAPIpyzheimer FastAPI (service) localhost:8001, API keyNeon PostgresNeon Postgres (store) 48 tablesS3 audio bucketS3 audio bucket (store) SSE, consent-gatedTwilio / Termii /ExpoTwilio / Termii / Expo (third party) OTP SMS, pushHTTPS tRPC + bearerHTTPS tRPC + cookieWebRTC (wss)reverse proxy :8080wss :7880room job dispatchclinical-telemetry da…GET agent configsendDueRemindersmultipart WAV + X-API…multipart WAV + X-API…Drizzle (Neon WS)PutObject SSESTT/TTS/LLM streamsOTP SMS, push
clientedgeserviceworkerstorethird partyasync
  • [*]LanguagesTypeScript · Python · SQL
  • [*]FrameworksHono · tRPC · Drizzle ORM · BetterAuth · LiveKit Agents (Node) · FastAPI · Next.js
  • [*]StoresPostgres (Neon serverless) · S3
  • [*]Asynccron (host crontab -> tRPC mutation) · LiveKit data channel (clinical-telemetry topic)
  • [*]InfraAWS EC2 · Docker Compose · Caddy · LiveKit server (self-hosted) · Vercel · GitHub Actions (OIDC deploy)
  • [*]Third partiesDeepgram (STT nova-3, TTS aura-2) · Zhipu GLM (OpenAI-compatible) · OpenAI · Google Gemini · ElevenLabs · Twilio · Termii · Expo Push · OpenWeather · Edge TTS

Voxtar is a clinical voice platform: patients record a daily set of short voice tasks (sustained vowel, /pataka/, cough) in a mobile app, or speak to a LiveKit voice agent, and the recordings are scored for acoustic biomarkers. The backend is a Bun/Turbo monorepo with apps/backend (Hono + tRPC, 33 procedures across 7 routers), apps/agent (a Node LiveKit worker), a shared Drizzle schema of 48 Postgres tables, and services/pyzheimer (the Python analysis service). 113 commits from 2026-07-09 to 2026-08-25.

The patient path runs startSession → submitRecording → completeSession. submitRecording takes a base64 WAV, posts it to pyzheimer, mirrors the result into the EMR tables as an encounter and a voice-biomarker vital_observation, writes a capture_recording row, and copies the raw audio to an SSE-encrypted S3 bucket only if the patient gave research consent.

Patient identity is split at write time: an HMAC-SHA256 pseudonym with a server-side pepper goes into the clinical patient row along with Safe-Harbor age bands and a region label, and names, phone numbers and MRNs go into a separate patient_identity table, both in one transaction. Phone-OTP sign-in (BetterAuth phoneNumber plugin, Twilio or Termii) is for patients; clinicians use org membership checks.

Deployment is one EC2 box running docker-compose with Caddy, LiveKit server, the agent, pyzheimer and the backend on host networking. pyzheimer is reachable only on localhost, and a GitHub Actions workflow ships a git archive over SSH using OIDC credentials. An hourly host cron calls a CRON_SECRET-guarded tRPC mutation to send Expo push reminders, matched against each patient's local timezone.

How it holds up

Auth
BetterAuth sessions shared across subdomains through a cookie on .hallelx2.com, with bearer and Expo plugins for the mobile app and phone-OTP sign-up for patients. tRPC protectedProcedure rejects requests without a session; assertProjectAccess checks org membership for clinicians and assertEnrollment checks for an active enrollment for patients. pyzheimer requires an X-API-Key, and the cron mutation takes a shared secret.
Data
48 Postgres tables in packages/db/src/schema.ts (BetterAuth user/session/org, project, agent and marketplace tables, FHIR-like encounter/vital_observation, patient + segregated patient_identity, cohort_invite/patient_enrollment, capture_session/capture_recording/recording_context, baselines). There is one Drizzle migration that creates all 48 tables plus one ALTER migration, and a hand-written idempotent DDL script for the patient-app additions.
Consistency
The two patient-creation paths use Postgres transactions, deduplicate on (projectId, pseudonymId), and joinCohort is idempotent per (user, project) with a unique constraint behind it. The invite useCount increment is read-then-write (useCount + 1 from the earlier read), so concurrent redemptions can undercount, and submitRecording's encounter, observation and recording inserts are not in one transaction.
Failure
pyzheimer non-2xx responses become UNPROCESSABLE_CONTENT errors that ask the patient to re-record. S3 retention is best-effort and never fails a capture. OpenWeather enrichment returns nulls if the key is missing. The agent falls back to a default prompt, Deepgram TTS and GLM if fetching its config fails. There are no timeouts, retries or queues on any outbound fetch.
Testing
1 test file (packages/shared/src/agent-config.test.ts, 4 cases) for the shared agent-config schema and prompt builder. No tests for backend routers, the agent or pyzheimer. apps/agent/src/test_simulation.ts is a manual script, not a test. CI runs security scans and deploys; nothing runs a test suite.

Techniques

  1. Pseudonymised patient splitEach incoming patient becomes a de-identified clinical row (an HMAC-SHA256 pseudonym with a server-side pepper, an age band that collapses to 90+, a region label) plus a separate identity row, written in one transaction. The pseudonym is deterministic, so re-imports deduplicate and longitudinal drift still links records without storing an MRN in clinical tables.apps/backend/src/services/deid.ts:25 apps/backend/src/services/deid.ts:103 apps/backend/src/routers/patient.ts:47
  2. Idempotent cohort enrollmentjoinCohort checks the invite's expiry and max uses, returns the existing enrollment on a repeat call, and otherwise creates the patient, identity and enrollment and bumps useCount in one transaction. A (user, project) unique constraint backs the idempotency. The useCount bump reuses the value read before the transaction, so concurrent joins can lose an increment.apps/backend/src/routers/patient-app.ts:210 apps/backend/src/routers/patient-app.ts:237 apps/backend/src/routers/patient-app.ts:288 packages/db/src/schema.ts:685
  3. Analysis result mirrored into the EMREach recording is analysed synchronously by pyzheimer. The backend lazily opens a 'virtual-voice' encounter for the session, writes a voice-biomarker vital_observation, and links it from the capture_recording row, so self-recorded data shows up on the clinician's patient record as ordinary clinical data.apps/backend/src/routers/patient-app.ts:553 apps/backend/src/routers/patient-app.ts:641 apps/backend/src/routers/patient-app.ts:656
  4. Low-latency voice turn-takingThe LiveKit worker preloads Silero VAD in prewarm and runs a local end-of-turn model with dynamic endpointing (300 to 2000 ms), preemptive LLM generation and adaptive barge-in. Tool calls record audio frames, build a 16 kHz WAV in memory, send it to pyzheimer, and publish biomarkers to the browser on a LiveKit data-channel topic.apps/agent/src/agent.ts:148 apps/agent/src/agent.ts:189 apps/agent/src/agent.ts:247
  5. Timezone-aware reminder sweepAn hourly host cron calls a secret-guarded mutation with the UTC hour. The backend selects profiles whose reminder hour matches, keeps those with an active enrollment, removes anyone who already completed today's session in their own timezone, and batches Expo push messages.apps/backend/src/routers/notifications.ts:14 apps/backend/src/routers/notifications.ts:36 deploy/cron/send-reminders.sh

Counted

  • 113commits
    git rev-list --count HEAD (2026-07-09 to 2026-08-25)
  • 33backend tRPC procedures
    grep '^\s+name: (protected|public)Procedure' apps/backend/src/routers/*.ts apps/backend/src/trpc/root.ts
  • 7backend routers
    ls apps/backend/src/routers
  • 71web (Next.js) tRPC procedures
    grep Procedure definitions in apps/web/src/server
  • 48Postgres tables
    grep -c 'pgTable(' packages/db/src/schema.ts; 48 CREATE TABLE in packages/db/drizzle/0000_ambitious_hemingway.sql
  • 2 (+1 manual idempotent DDL script)Drizzle migrations
    ls packages/db/drizzle/*.sql; packages/db/scripts/apply-patient-app.mjs
  • 1 (4 cases)test files
    git ls-files | grep .test.ts; grep -c 'test(\|it(' packages/shared/src/agent-config.test.ts
  • 4996backend + agent + packages TS/MJS LOC
    git ls-files apps/backend apps/agent packages | grep -E '\.(ts|mjs)$' | xargs wc -l
  • 802agent LOC
    wc -l apps/agent/src/*.ts
  • 57329TypeScript LOC (whole monorepo, incl. web/admin/patient)
    git ls-files '*.ts' '*.tsx' | xargs wc -l
  • 5docker-compose services
    deploy/docker-compose.yml (caddy, livekit, pyzheimer, agent, backend)
  • 7GitHub workflows
    ls .github/workflows