backend / pdfgrab

pdfgrab

A pure-Go port of pdfplumber: a PDF content-stream interpreter that returns every character, word, line and table cell with its coordinates, for line-level citations in Vectorless.

Source Vectorless, the product The working drawing

2026 · paused · open source

clientthird partyserviceworkerasync

[*]LanguagesGo · Python
[*]Frameworkspdfcpu
[*]InfraGo module (pkg.go.dev) · GitHub Actions · Lorx CI (.lorx/actions)

pdfgrab exists so a Go retrieval service can cite the exact line of a PDF page in-process, with no CGo and no Python. pdfcpu handles the object layer. On top of it pdfgrab runs its own content-stream interpreter: a dispatch table of 57 operators covering graphics state, paths, text state, text showing and Form XObjects. The interpreter emits Char, Line, Rect and Curve records, which are rebuilt into words, lines and tables.

Most of the work went into fonts and geometry. It bundles the Adobe Core-14 AFM widths and vertical metrics, decodes ToUnicode CMaps, base encodings with /Differences and the Symbol and ZapfDingbats built-in encodings, and keeps every box in PDF user space with a separate ViewRect type for viewer coordinates. Per the 2026-08-02 report, horizontal and vertical drift against pdfplumber fell from 11.99 pt and 4.97 pt to 0.0000 pt, and the golden test now asserts 0.01 pt on both axes.

Table extraction ports pdfplumber's TableFinder: snap and join edges, find intersections, build cells and assign glyphs. It supports the lines, lines_strict, text and explicit strategies per axis, plus an opt-in auto strategy and Nurminen text-edge region detection. One edge case is fixed beyond the original: a glyph straddling a table's outer edge by 0.008 pt was dropped, which flipped the sign of 20 of 103 negative numbers in a 10-K.

A separate benchmark module scores ten systems on ICDAR 2013 (125 PDFs). The README reports pdfgrab at 0.442 F1 per document against pdfplumber's 0.458, at 81 ms per document against 794 ms. Seven dated evaluation reports, four of them negative results, record what was tried and rejected. The README now marks the project open source and not maintained.

How it holds up

Auth: None: an in-process library with no network calls; PDFs encrypted with only an owner password open normally, and a user-password PDF returns ErrEncrypted (there is no password parameter).
Data: No store. The in-memory model is Document, then Page, then Char/Line/Rect/Curve, then Word, then Table, with Rows [][]string and CellsBBox. Golden fixtures are 14 files in testdata/golden, generated from pdfplumber by scripts/gen_golden.py.
Consistency: Single-threaded per document by design; the README and code both show that concurrent Open/parse is a data race (the operator table in internal/pdf/ops.go is lazily built without synchronisation, plus pdfcpu's shared config), tracked as HAL-118.
Failure: Pathological pages are skipped with a logged warning when merged edges exceed MaxEdgesPerAxis (1000) or crossings exceed MaxIntersections (50,000). Unknown glyphs become (cid:N) placeholders that keep their boxes. Width lookup falls back from /Widths to Core-14 AFM to the default width to 500. Normalized returns a zero rect instead of NaN on a zero-size page.
Testing: 25 _test.go files with 151 Test functions and 2 benchmarks on main. Golden tests compare word text, order and boxes against pdfplumber output at 0.01 pt. Regression tests cover the outer-edge glyph fix, auto-not-default and encrypted documents. Go runs in a 3-OS matrix in GitHub Actions, and a separate bench/ module scores ICDAR 2013 against 9 other systems.

Techniques

PDF content-stream interpreterA lexer and a dispatch table cover the PDF 1.7 §A.2 operators so the operand stack stays balanced even for ignored ones. The CTM is pre-set for MediaBox origin and /Rotate, and Form XObjects run in a sub-interpreter under their own /Matrix and resources.internal/pdf/ops.go:49 internal/pdf/content.go:584 internal/pdf/content.go:622 internal/pdf/state.go
Font metrics and text decodingBundled Core-14 AFM widths and ascent/descent tables, ToUnicode CMap parsing (bfchar and bfrange), base encodings with /Differences, and Symbol/ZapfDingbats tables. Drift against pdfplumber went from 11.99 pt to 0.0000 pt, and Symbol now decodes 0x61 as alpha, where pdfplumber 0.11.9 still gives 'a'.internal/pdf/afm_widths.go internal/pdf/afm_vmetrics.go internal/pdf/cmap.go internal/pdf/font.go:152 golden_test.go:370
Outer-edge glyph retention in table cellsCentre-point assignment drops a glyph that straddles a table's outer boundary. charsInCellEdges keeps glyphs that overlap the outer left or right edge, bounded by the glyph itself, which restored 103 of 103 accounting-notation minus signs on a 10-K where pdfplumber also fails.page.go:975 cell_edges_test.go
Edge snapping and joining with pathological-page guardsRaw edges are prefiltered, snapped onto their mean within a tolerance, joined when collinear, then intersected into cells. Pages over 1000 edges per axis or 50,000 crossings skip table finding instead of blowing up.internal/layout/lines.go:287 internal/layout/lines.go:384 page.go:616 page.go:636
Reproducible benchmark with negative resultsA separate Go module and Python scorers run ten systems on ICDAR 2013 with per-document adjacency F1 and timing. It was run twice in independent processes with identical per-document F1, and it caught and corrected a pooled-versus-per-document metric error in earlier reports.bench/icdar2013/extract.go bench/icdar2013/score.py docs/evaluations/2026-09-17-field-comparison-and-metric-correction.md strategy_auto_test.go:113
Typed coordinate spacesEvery box is a BBox in PDF user space (bottom-left origin). Viewer output is a distinct ViewRect type, produced only by Viewport (pixels at a scale) or Normalized (0..1 fractions), so a top-left/bottom-left mix-up cannot compile silently at a call site.geometry.go:143 geometry.go:162

Counted

50Commits (origin/main)
git rev-list --count origin/main
43Commits (local HEAD, 7 behind)
git rev-list --count HEAD
2026-05-26 / 2026-10-12First / last commit
git log origin/main --format=%ad
19,220Go LOC, main
find . -name '*.go' | xargs wc -l (git archive origin/main)
13,922 (3,194 of it the generated AFM width table)Go LOC excluding tests and bench
find . -name '*.go' ! -name '*_test.go' ! -path './bench/*' | xargs wc -l
1,585Python LOC (bench scorers + fixture scripts)
find . -name '*.py' | xargs wc -l
25 / 151 / 2Test files / Test functions / benchmarks
find -name '*_test.go'; grep -cE '^func (Test|Fuzz)' / '^func Benchmark'
57PDF operators in dispatch table
grep -c entries in buildOperatorTable, internal/pdf/ops.go
14Golden fixture files
ls testdata/golden
7Dated evaluation reports
ls docs/evaluations
6 (v0.0.1 to v0.4.0), none valid for github.com/hallelx2/pdfgrabTags under any module path
git ls-remote --tags origin; README HAL-2690
HTTP 200 on 2026-10-12pkg.go.dev page
curl -sI https://pkg.go.dev/github.com/hallelx2/pdfgrab