backend / lorx-actions
Lorx actions engine and runner fleet
Lorx's CI system: it plans .lorx workflows into dependency levels, queues jobs in the database, leases them to self-registering runners over HTTP, and starts stopped GCP machines only when work is waiting.
2026 · live · private repository
- [*]LanguagesRust · Shell
- [*]Frameworksaxum · SeaORM · tokio · reqwest · serde_yaml
- [*]StoresSQLite · Postgres · Runner-local disk (checkouts, content-addressed step cache)
- [*]AsyncDatabase-backed job queue with leases and heartbeats · Outbox consumer (push events) · Lease-expiry timer · Runner control stream (long-lived HTTP) · Pool scheduler (event wake + 30 s look)
- [*]InfraGCP Compute Engine (stopped-VM runner pool) · Docker / Podman (job images) · systemd
- [*]Third partiesGCE Compute API (start/stop instances, instance metadata)
lorx-actions parses .lorx/actions/*.yml, attaches every problem to its source line, and plans a trigger into levels. Matrices are expanded, and jobs excluded by paths, branches, if: expressions or schedules are set aside with the reason. Each step is keyed by a SHA-256 of everything that decides its outcome, so a step that already passed with those inputs is skipped. bit run uses this engine locally, and the runners use the same code remotely.
On the server, a push consumer reads the event outbox and creates runs. Jobs move from waiting to queued as their needs pass, and are leased to one runner at a time. Every state change is a single conditional UPDATE, under a Postgres advisory transaction lock per run. Runners heartbeat to extend the lease. Instead of polling, a reaper sleeps until the next lease expires, then requeues the job, up to 3 attempts, and fails it after that. Runs fail fast unless the workflow opts out.
lorx-runner is a separate binary, also bundled as bit runner. It trades a one-time registration token for its own credential, runs several jobs at once in per-repository slots that keep checkouts and cache: directories between jobs, evicts the least recently used idle slots when disk runs short, and holds GET /runner/stream open so a cancelled job stops at once. A runner pool keeps GCE VMs stopped (e2-standard-8 by default), starts one only when a job it could lease has waited long enough, hands it a registration token through instance metadata, and backs off failed boots from 5 minutes up to 6 hours.
Secrets use envelope encryption: XChaCha20-Poly1305 with a data key per value, associated data that binds each ciphertext to its scope, owner and name, and a key-encryption key that never touches the database and can be rotated. A lease carries only the secrets its job names. The CI code is about 12,900 lines and has 148 test cases. Lorx's own CI runs on it.
How it holds up
- Auth
- Single-use registration tokens (lorx_rr_) scoped to an account or repository are exchanged for a per-runner credential kept in a 0600 file. Each lease mints a job token whose hash is stored on the job row, and the job uses it to fetch its commit and report.
- Data
- runs, run_jobs (status, attempt, runner_id, lease_until, job_token_hash), run_log_lines (capped per attempt), runners, runner_registrations and secrets. These are 6 of the 35 entities, added by migrations 18, 20, 22, 23 and 24 (runs, runners, fail_fast, lease_index, secrets); migration 21 (merge_gate) adds require_checks and review_block_at to branch_rules.
- Consistency
- Every job transition is an UPDATE ... WHERE status IN (expected), with rows_affected == 1 as the guard, so two runners can never hold one job and a late heartbeat cannot revive a cancelled job. Postgres advisory transaction locks (pg_advisory_xact_lock on an FNV-1a key) serialise work on one run or log, and they are always taken before the outbox lock so the two cannot deadlock. A lease commits only if the runner still exists and is not draining.
- Failure
- Jobs whose runner stops answering are requeued when their lease expires, up to MAX_ATTEMPTS = 3, and the log records why. Fail-fast cancels running siblings, which learn of it through the stream or their next heartbeat. The pool starts a machine at most once per job attempt, doubles the wait after boots that never check in (5 minutes to 6 hours), and persists that memory to disk across restarts. Runner logs are capped by lines and bytes per attempt.
- Testing
- 7 test files: crates/lorx-server/tests/{runners,runs,run_events,runner_pools,secrets}.rs, crates/lorx-runner/tests/runner.rs and crates/bit/tests/run.rs. Together with unit tests in lorx-actions and the CI modules there are 148 cases (runners.rs alone has 34, runner_pools.rs 14).
Techniques
- Lease-based job queue on conditional updatesEach transition is a single guarded UPDATE that must affect exactly one row, so double leases and revivals of cancelled jobs cannot happen, on Postgres or on SQLite. Expired leases are requeued up to 3 times, by a reaper that sleeps until the next lease end rather than polling.
crates/lorx-db/src/runs.rs:1crates/lorx-db/src/runs.rs:24crates/lorx-db/src/runs.rs:216crates/lorx-db/src/runs.rs:742crates/lorx-db/src/runs.rs:966 - Advisory locks with a fixed lock orderpg_advisory_xact_lock on a stable FNV-1a hash of the run id serialises jobs that finish at the same time. Without it, each finishing job could settle the run without seeing the other's result, and dependent jobs would wait forever. The outbox lock is always taken after the run lock, so the two cannot deadlock.
crates/lorx-db/src/runs.rs:231crates/lorx-db/src/runs.rs:397crates/lorx-db/src/runs.rs:412 - Scale-to-zero runner poolStopped GCE machines are started only for a job their runner could actually lease: matching labels, serving that repository, not draining, and no capable runner already up. The machine gets a one-time registration token through instance metadata. Boots that never check in back off from 5 minutes to 6 hours, and that memory survives server restarts.
crates/lorx-server/src/runner_pools.rs:1crates/lorx-server/src/runner_pools.rs:116crates/lorx-server/src/runner_pools.rs:665deploy/gcp/runner/create.sh:12 - Content-addressed step cacheA step's key is a SHA-256 over its command, environment, image, matrix values and the content of the files it reads. Tracked files use git object ids, and dirty files are hashed from disk. A step whose key has passed before is skipped, locally and on runners.
crates/lorx-actions/src/cache.rs:1crates/lorx-actions/src/cache.rs:51crates/lorx-actions/src/cache.rs:104 - Envelope-encrypted secrets bound to their identityEach value is sealed with XChaCha20-Poly1305 under its own data key, with associated data naming scope, owner and name, so a ciphertext copied to another row will not open. Data keys are wrapped by a key-encryption key held in a file outside the database, with rotation through previous key files. A runner receives only the secrets its job names.
crates/lorx-server/src/secrets.rs:1crates/lorx-server/src/secrets.rs:147crates/lorx-server/src/secrets.rs:262crates/lorx-runner/src/lib.rs:440 - Multi-slot runner with instant cancel and disk-bounded workspacesA runner runs jobs from the same level side by side, one per slot. Each slot keeps its checkout and cache: directories so build tools stay incremental, and the idle slots used longest ago are evicted when disk runs short. A long-lived control stream stops cancelled jobs at once, and heartbeats cover the case where the stream is down.
crates/lorx-runner/src/lib.rs:1crates/lorx-runner/src/lib.rs:657crates/lorx-runner/src/space.rs:1
Counted
- 12,911CI code LOC
wc -l over lorx-actions/src, lorx-runner/src, server actions/runner_pools/secrets + api runners/runs/secrets + db runs/runners/secrets - 4,688lorx-actions LOC
git ls-files crates/lorx-actions | grep .rs$ | xargs cat | wc -l - 2,662lorx-runner LOC
same, crates/lorx-runner - 25CI REST operations
openapi.json paths matching runner|/runs|secrets - 148CI test cases
grep -cE '#\[(tokio::)?test' over the CI files listed above - 3Max lease attempts
crates/lorx-db/src/runs.rs:24 - 6CI tables
crates/lorx-db/src/entity: runs, run_jobs, run_log_lines, runners, runner_registrations, secrets