backend / pipelines

Pipelines

Three batch pipelines: a gym-lead sourcing and outreach tracker backed by SQLite, a resumable Telegram account archiver and organiser, and a narration job that runs on a short-lived GCP spot VM.

2026 · paused · private repository

cli → tinyfish: search / fetch / agent runcli → files: cached JSON → scored CSVfiles → sqlite: seed from tracker.csv (tx)dash → sqlite: better-sqlite3 sync queriestg → mtproto: iter_messages / DialogFiltersdeploy → gcs: upload inputs, poll sentineldeploy → vm: gcloud create / deletevm → gcs: _PROGRESS, MP3s, _DONENext.js dashboard +APINext.js dashboard + API (service) 5 route files, 7 handlersOperator shell (stagescripts)Operator shell (stage scripts) (client) 01–12 *.sh / *.pyTelethon scriptsTelethon scripts (worker) archive, organise, leavedeploy.ps1 driverdeploy.ps1 driver (client) poll every 30 sLead CSV / JSON cacheLead CSV / JSON cache (store) 49 auditsSpot VM (F5-TTS)Spot VM (F5-TTS) (worker) startup.sh, self-shutdownSQLite pipeline.dbSQLite pipeline.db (store) 3 tables, WALTinyFishsearch/fetch/agentTinyFish search/fetch/agent (third party) incl. Google MapsTelegram MTProtoTelegram MTProto (third party) user sessionGCS job prefixGCS job prefix (store) in/, out/, _DONE.txtsearch / fetch / agen…cached JSON → scored …seed from tracker.csv…better-sqlite3 sync q…iter_messages / Dialo…upload inputs, poll s…gcloud create / delete_PROGRESS, MP3s, _DONE cli → tinyfish: search / fetch / agent runcli → files: cached JSON → scored CSVfiles → sqlite: seed from tracker.csv (tx)dash → sqlite: better-sqlite3 sync queriestg → mtproto: iter_messages / DialogFiltersdeploy → gcs: upload inputs, poll sentineldeploy → vm: gcloud create / deletevm → gcs: _PROGRESS, MP3s, _DONENext.js dashboard +APINext.js dashboard + API (service) 5 route files, 7 handlersOperator shell(stage scripts)Operator shell (stage scripts) (client) 01–12 *.sh / *.pyTelethon scriptsTelethon scripts (worker) archive, organise, leavedeploy.ps1 driverdeploy.ps1 driver (client) poll every 30 sLead CSV / JSONcacheLead CSV / JSON cache (store) 49 auditsSpot VM (F5-TTS)Spot VM (F5-TTS) (worker) startup.sh, self-shutdownSQLite pipeline.dbSQLite pipeline.db (store) 3 tables, WALTinyFishsearch/fetch/agentTinyFish search/fetch/agent (third party) incl. Google MapsTelegram MTProtoTelegram MTProto (third party) user sessionGCS job prefixGCS job prefix (store) in/, out/, _DONE.txtsearch / fetch / agen…cached JSON → scored …seed from tracker.csv…better-sqlite3 sync q…iter_messages / Dialo…upload inputs, poll s…gcloud create / delete_PROGRESS, MP3s, _DONE
clientservicestorethird partyworkerasync
  • [*]LanguagesPython · TypeScript · Bash · PowerShell
  • [*]FrameworksNext.js 15 (route handlers) · Telethon · F5-TTS · Chatterbox TTS
  • [*]StoresSQLite (better-sqlite3, WAL) · CSV / JSONL files · GCS
  • [*]AsyncBash background-job fan-out · asyncio (Telethon) · GCS sentinel polling
  • [*]InfraGCP Compute Engine spot VM · Google Cloud Storage · local Node server (port 3030)
  • [*]Third partiesTinyFish (search, fetch, browser agent) · Google Maps (via TinyFish agent) · Telegram MTProto API · GCE metadata server · Hugging Face model download (F5-TTS)

Four one-commit private repositories, all pushed on 2026-08-25. Three of them have backend character and are covered here. The fourth, video-pipeline, is a local FFmpeg/Whisper post-production command and is left out.

contract-pipeline is a lead-generation pipeline for the spec-website outbound track. Twelve numbered stage scripts (1,143 lines of Bash and Python) run 113 search queries through the TinyFish CLI with concurrency 3, three retries, JSON validation and a per-query cache file. They then fetch and audit 49 gym websites (19 Nigerian, 30 US/UK), score each with an 8-signal heuristic capped at 5, and run a browser agent over Google Maps for contact details. A Next.js 15 dashboard reads the results from SQLite through 5 API route files with 7 handlers, and writes an append-only timeline event for every status change.

telegram-automation is 13 Telethon scripts (2,080 lines) that drive a personal account through the MTProto API, not the Bot API. archive.py checkpoints after each channel, so a run over 222 channels can stop at any FloodWait and resume the next day. Every destructive script is a dry run unless a flag is passed. delete_group.py goes further: it refuses to delete until it has re-checked three things live, namely the export size on disk, the message and attachment count in the destination lane, and the PDF.

tts-job is a single-shot GCP batch job (491 lines). A PowerShell driver uploads inputs to a per-job GCS prefix, starts a c2 spot VM whose startup script installs F5-TTS and narrates 17 slides, and polls GCS every 30 s for a _PROGRESS.txt stage marker and a _DONE.txt sentinel. It then downloads the MP3s and deletes the VM. None of the three repos has automated tests.

How it holds up

Auth
The dashboard API has no authentication and is meant to run locally on port 3030. Telegram uses a Telethon user session created by an interactive login, with the code sent to the phone, plus TG_API_ID/TG_API_HASH from a gitignored .env. The TTS VM gets GCS access from its cloud-platform service-account scope, not from keys.
Data
The dashboard SQLite schema has 3 tables (leads keyed by host, outreach_log with an ON DELETE CASCADE foreign key, timeline_events), 7 indexes and an AFTER UPDATE trigger that touches updated_at. On first boot it seeds itself from tracker.csv inside one transaction. Telegram state is one JSONL file per channel plus state.json; TTS inputs and outputs live under gs://<project>-tts-job/<jobId>/{in,out}.
Consistency
Pipeline stages are idempotent through cache files: any query, audit, Maps or screenshot output that already exists and is non-empty is skipped on re-run. updateLead accepts only allow-listed fields, coerces types, and diffs against the current row so that a no-op PATCH writes nothing. logOutreach, however, runs its insert, lead update and event write as three separate statements without a transaction. archive.py commits progress per channel to state.json, and Telethon is limited to 3 connection retries so a stuck run fails fast.
Failure
TinyFish calls retry 3 times with a 2 s sleep and reject output that is not valid JSON. Audits that come back without a text field are deleted so the next run retries them. Telethon FloodWaitError sleeps min(seconds+2, 300) and leaves the channel for the next run; any other per-channel error is recorded in state and skipped. deploy.ps1 confirms the VM exists after create, because gcloud can exit 0 on a quota failure, and times out after 90 minutes. A failed generate leaves no _DONE sentinel, and the startup script exits before its shutdown line.
Testing
No automated tests in any of the three repos. tts-job has two manual smoke scripts (test-f5-tts.py, test-one-slide.py) that synthesise one slide. The pipeline scripts print ✓/✗ per item, and delete_group.py's pre-flight checks act as a runtime guard.

Techniques

  1. Ephemeral spot-VM batch job with GCS sentinelInputs go under a per-job GCS prefix and the VM reads its paths from instance metadata. Stage markers are written to _PROGRESS.txt so the local poller can show which step the VM is on, and _DONE.txt marks completion. The driver also checks that the VM exists after create, because gcloud can return 0 on a quota failure.tts-job/deploy.ps1:67 tts-job/deploy.ps1:84 tts-job/deploy.ps1:98 tts-job/startup.sh:8 tts-job/startup.sh:12
  2. Resumable, rate-limit-tolerant archiverProgress is committed to state.json after each channel. On FloodWaitError the script saves state, sleeps for the wait Telegram asked for (capped at 300 s) and moves on. Connection retries are capped at 3, so a second process holding the SQLite session lock makes the run fail fast instead of stalling silently.telegram-automation/archive.py:98 telegram-automation/archive.py:151 telegram-automation/archive.py:165
  3. Pre-flight-gated irreversible deleteBefore calling DeleteChannelRequest, the script checks that the export file is larger than 5 KB, that the destination topic holds at least 3 messages and 2 attachments, and that the PDF is on disk. Without --delete it only reports. Afterwards it re-resolves the entity to confirm the group is gone.telegram-automation/delete_group.py:65 telegram-automation/delete_group.py:81 telegram-automation/delete_group.py:104
  4. Cached, retried CLI fan-outEach search query runs in a background subshell in batches of 3, with 3 attempts and a JSON-validity check, and writes to its own q-NNN.json cache. A re-run skips finished queries, and the per-query files are concatenated into JSONL at the end.contract-pipeline/scripts/01-source-nigeria.sh:16 contract-pipeline/scripts/01-source-nigeria.sh:21 contract-pipeline/scripts/01-source-nigeria.sh:56
  5. Diffing PATCH with an event-sourced timelineupdateLead allow-lists fields, coerces booleans and ints, compares each field with the stored row, and builds a dynamic UPDATE only from the fields that changed. Status, spec and outreach changes each add a typed timeline_events row, which the funnel and feed views read.contract-pipeline/dashboard/lib/tracker.ts:105 contract-pipeline/dashboard/lib/tracker.ts:135 contract-pipeline/dashboard/lib/tracker.ts:158
  6. Heuristic site-audit scorerTurns fetched page text into a redesign-opportunity score from 8 signals: HTTPS, a booking CTA, a schedule, visible pricing, payment integration, copyright-year freshness, an Instagram link and latency. The score is capped at 5, bucketed P1 to P4, and merged with Google Maps contact data into an 'actionable' ordering.contract-pipeline/scripts/05-score-audits.py:21 contract-pipeline/scripts/05-score-audits.py:108 contract-pipeline/scripts/12-merge-gmaps.py:65

Counted

  • 4 (1 per repo)commits
    git rev-list --count HEAD in each repo
  • 12 files, 1,143 LOCpipeline stage scripts
    git ls-files 'scripts/*.sh' 'scripts/*.py' | xargs wc -l (contract-pipeline)
  • 5 / 7dashboard API route files / handlers
    dashboard/app/api/*/route.ts
  • 753dashboard backend LOC (lib + api)
    git ls-files 'dashboard/lib/*' 'dashboard/app/api/*' | xargs wc -l
  • 3 / 7 / 1SQLite tables / indexes / triggers
    dashboard/lib/db.ts SCHEMA_SQL
  • 113 (63 NG + 50 US/UK)search queries
    wc -l scripts/*queries.txt
  • 49 (19 NG + 30 US/UK)websites audited
    git ls-files leads/raw/audits*/ | wc -l
  • 13 files, 2,080 LOCTelethon scripts
    git ls-files '*.py' | xargs wc -l (telegram-automation)
  • 491 (py + sh + ps1)TTS job LOC
    git ls-files '*.py' '*.sh' '*.ps1' | xargs wc -l (tts-job)
  • 17slides narrated per job
    script.json
  • 0automated test files
    all three repos