5.3 KiB

Raw Blame History

Spike B — glyph→Unicode + line/paragraph reconstruction: M0 result

Spike: B — glyph→Unicode + line/paragraph reconstruction (engineering plan §14 step 5)
Date: 2025-07-25
Status: Complete. Gate MET on the synthetic corpus.
Gate: paragraph-boundary F1 ≥ 0.85 (M0 exit criterion); ≥ 0.93 is the release gate (§8.2) and requires the real labelled corpus.

What was built

spike/B_reconstruction/ implements the reconstruction pipeline (§4.1 steps 2–5) on a synthetic in-memory glyph model:

GlyphRun.h — the input model (Glyph, GlyphRun, Line, Paragraph, GroundTruthParagraph), mirroring §3.2/§3.3.
Reconstruct.cpp — step 2 (flatten runs to placed glyphs), step 3 (line detection by baseline clustering, tolerance 0.25 × font size per §4.1 step 3), step 4 (reading order: descending-y sort for the single-column LTR case — the recursive XY-cut reduces to this), step 5 (paragraph grouping), plus a boundary-F1 scorer.
Corpus.cpp — a deterministic synthetic corpus generator (seeded PRNG) producing 3–8 paragraphs of 1–5 lines each, with realistic leading, paragraph gaps, short last lines, and occasional list items.
main.cpp — runs the pipeline over the corpus and emits the contract JSON.

Step 1 (glyph→Unicode via font cmap / ToUnicode) is not exercised by this spike: the synthetic generator produces glyphs whose unicode is known by construction. Step 1 is exercised separately once the production L3 interpreter (M2) wires FreeType + the ToUnicode CMap path.

Result

Metric	Value	Target
paragraph-boundary F1	0.963	≥ 0.85
precision	0.929	—
recall	1.000	—
perfect documents	334 / 500	—

Tested on 500 synthetic documents, single-column LTR, 12pt font, 14.4pt leading, 23pt paragraph gap. Recall is perfect (no missed boundaries); the remaining imperfection is over-segmentation (precision 0.93) on the ~33% of documents where the synthetic generator creates edge cases — single-line paragraphs adjacent to multi-line ones, and list items.

Findings

The leading-gap signal is primary; the short-last-line signal is confirming, not primary. The first implementation treated "previous line ended short of the right margin" as a sufficient paragraph-break condition and over-segmented badly (F1 ≈ 0.37): a short first line of a paragraph is common, and the heuristic split paragraphs at every short line. The correct hierarchy (per §4.1 step 5) is: leading consistency is the merge condition; a short last line only forces a break when combined with above-normal leading. This recovered F1 to 0.963.
Coordinate-system orientation matters and is easy to get backwards. The synthetic corpus uses PDF coordinates (y increases upward), so reading order is descending y. An ascending-y sort inverted the reading order and broke both the leading sign and the "previous line" semantics. This is a concrete instance of the §11 "real PDFs are pathological" risk and the reason the plan keeps the corpus in PDF coordinates throughout.
Column-width inference must be robust to outliers. Inferring the column right edge as the max x_end across lines let a single overshooting line push the edge out and make every other line look "not wrapped". Using the 90th percentile instead fixed the wrap-tolerance. This is the kind of detail the real §4.1 step 4 XY-cut handles by computing column geometry rather than inferring it from line ends.
The 80/20 boundary is real and visible in the failure cases. The 166 imperfect documents are exactly the §11 "last 20%" cases: single-line paragraphs, list items, paragraphs whose first line is short. The synthetic corpus makes these diagnosable; the real labelled corpus (§8.1) is needed to score against ground truth on actual layout pathology and to validate the 0.93 release gate.

What this means for the project

The reconstruction pipeline shape works and scores above the M0 bar on a controlled corpus. The M0 exit criterion (F1 ≥ 0.85) is met; the project is not blocked on reconstruction feasibility.
The release gate (F1 ≥ 0.93, §8.2) must be scored on the real labelled corpus, not the synthetic one. The synthetic corpus validates the pipeline; it does not validate that the pipeline handles real PDF pathology. Acquiring/constructing the labelled corpus (§8.1) is the next prerequisite for the M4 text-editing milestone.
Step 1 (glyph→Unicode) remains the highest-risk step and is not covered by this spike. It is where the §4.1 priority ladder (ToUnicode → Encoding → built-in → cmap → OCR) lives, and where low-confidence mappings drive the UI's confidence-underlining. It should be the next reconstruction work.

Reproducing

cmake -S . -B build/manual -G Ninja -DCMAKE_BUILD_TYPE=Release
cmake --build build/manual --target spike_b_reconstruction
build/manual/bin/spike_b_reconstruction 500 0.85    # <n_docs> <target_F1>

The single-line JSON report on stdout is the CI contract. Exit 0 if the gate is met, 1 otherwise. Also verified clean under ASan+UBSan.

5.3 KiB Raw Blame History Unescape Escape