backend / llmgate

llmgate

A provider-agnostic Go LLM client with composable retry, an adaptive concurrency limiter, budget caps, a response cache, provider fallback and per-call cost, plus a separate Judge interface for typed-decision models.

Source Vectorless, the product The working drawing

2026 · paused · open source

clientservicestorethird partymodelasync

[*]LanguagesGo
[*]Frameworkslangchaingo · tiktoken-go
[*]Storesin-memory LRU
[*]Asyncgoroutines (parallel token counting) · background price refresh ticker · GitHub Actions cron (weekly price drift)
[*]InfraGo module (pkg.go.dev) · GitHub Actions · Lorx CI (.lorx/actions)
[*]Third partiesAnthropic · OpenAI · Google Gemini · TypeSafe (Jev) · LiteLLM price feed · OpenRouter models API

llmgate is the LLM layer under Vectorless, extracted as a public Go module (Apache 2.0, tags v0.1.0 to v0.5.0). One small Client interface covers Anthropic, OpenAI and Gemini. Each provider is a file of about 60 lines over one internal adapter built on langchaingo, and everything else is a func(Client) Client middleware: retry, limit, budget, cache and router.

The adaptive limiter is the main piece. Instead of a fixed concurrency number it runs AIMD: a 429 or transport failure halves the limit and honours Retry-After as a pause, and 20 consecutive successes widen it by one. It is built on a sync.Cond with context-aware wakeups and reports every change through a callback. A doc comment in the code records that a fixed concurrency of 6 had rate-limited 13 of 21 documents into empty results that reported success.

Cost is computed on every response from a three-layer price table: explicit registrations, a remote feed (LiteLLM, then OpenRouter, refreshed every 24 h), then an embedded table generated by go generate. A weekly scheduled workflow regenerates that table and fails when a vendor price has drifted. Usage is split into input, cache-write, cache-read and reasoning tokens.

Beside Client sits Judge, for System One models that return typed decisions (TypeSafe's Jev). Its context guard was the subject of a measured fix on main: build the tokenizer once, accept on byte length first, and count long text in parallel chunks at pre-tokenizer boundaries. Per the README, Vectorless's query median went from 50.5 s to 2.2 s with only llmgate swapped. The README now marks the project open source and not maintained.

How it holds up

Auth: A library, so there is no end-user auth; it passes provider API keys from config or the environment, and classifies 401/403 responses as ErrClassAuth so retry and router never retry or fall over on them by default.
Data: No persistent store. The state is in memory: a per-provider Limiter, a shared budget Ledger (daily and total USD with UTC rollover), an LRU response cache keyed on a versioned SHA-256 of the request shape, and an immutable snapshot of remote prices swapped on refresh.
Consistency: Limiter and Ledger are mutex-guarded, and release funcs are wrapped in sync.Once so a slot cannot be released twice. The budget is check-then-debit: concurrent calls can each pass the check and overshoot the cap by the calls in flight. Cache entries are cloned on put so callers cannot mutate the cached value.
Failure: Errors are classified (structured LLMError, then context sentinels, then string heuristics) into rate-limited, transient, timeout, auth, context-length and gateway classes. Retry uses exponential backoff with 0..d/2 jitter and prefers Retry-After. The router falls over by policy and returns a RouteError carrying every provider's error, with the primary one exposed. The limiter pauses new calls for the provider-specified Retry-After.
Testing: 34 _test.go files with 208 Test functions on main (32 files, 197 on the local branch). Tests run under go test -race in a 3-OS matrix in GitHub Actions and as a Lorx container pipeline, with a property test holding parallel token counting equal to serial counting. A weekly scheduled job fails when the embedded price table drifts from the live feed.

Techniques

AIMD adaptive concurrency limiterA cond-variable semaphore whose limit halves on a 429 or transient error, pauses for Retry-After and grows by one per 20 consecutive successes, with context-cancel wakeups and OnChange events. It sits inside retry, so the limit has already narrowed before the retry sleeps. Shared per provider across all callers.middleware/limit/limit.go:134 middleware/limit/limit.go:176 middleware/limit/limit.go:249 middleware/limit/limit.go:281
Error taxonomy driving retry and fallbackClassify maps any error to a class through three priorities: typed LLMError, context sentinels, then string heuristics checked in a deliberate order with rate-limit before generic 4xx. RetryAfter extraction feeds both backoff and the limiter's pause, and router policies (OnRateLimit, OnTransient) are just predicates over the class.errors.go:148 errors.go:227 middleware/retry/retry.go:135 router/router.go:65
Three-layer price resolution with drift detectionCost lookup goes override, then remote snapshot (LiteLLM, then OpenRouter, refreshed every 24 h and swapped immutably), then the embedded table generated by go generate. Model IDs are canonicalised first. A weekly cron regenerates the table and fails on diff, after gemini-2.5-flash output had sat at 0.60 against a real 2.50.pricing/remote.go:81 pricing/canonical.go:52 pricing/gen/main.go .github/workflows/price-drift.yml
Context guard: byte bound first, parallel exact countBecause every cl100k token covers at least one byte, a request whose byte length fits the limit is accepted without tokenising. Above that bound, text is cut at pre-tokenizer piece boundaries and counted in goroutines with a once-built encoder, and a property test holds the result equal to the serial count.judge/typesafe/typesafe.go:954 judge/typesafe/typesafe.go:999 judge/typesafe/typesafe.go:1029 judge/typesafe/count_internal_test.go
Middleware composition with ordering semanticsEvery concern is a func(Client) Client, and the order carries meaning: cache below budget so hits cost nothing, limit inside retry. The Judge has its own parallel middleware type (retry.NewJudge, limit.Judge), so typed-decision calls get the same resilience without pretending to be chat.middleware/retry/judge.go:25 middleware/cache/cache.go:92 middleware/budget/budget.go:127

Counted

40Commits (origin/main)
git rev-list --count origin/main
32Commits (local HEAD, feature branch)
git rev-list --count HEAD
2026-04-19 / 2026-10-12First / last commit
git log origin/main --format=%ad
12,096Go LOC, main
find . -name '*.go' | xargs wc -l (git archive origin/main)
6,348Go LOC excluding tests, main
find . -name '*.go' ! -name '*_test.go' | xargs wc -l
34 / 208Test files / Test functions, main
find -name '*_test.go'; grep -cE '^func (Test|Fuzz)'
6 (v0.1.0 to v0.5.0)Released tags
git ls-remote --tags origin
3 (179 lines total)Providers
wc -l provider/*/*.go
5 (retry, limit, budget, cache, router)Middlewares
ls middleware/ router/
ubuntu, macos, windows; go test -raceCI OS matrix
.github/workflows/ci.yml
HTTP 200 on 2026-10-12pkg.go.dev page
curl -sI https://pkg.go.dev/github.com/hallelx2/llmgate