# Spike B — glyph→Unicode + line/paragraph reconstruction: M0 result * **Spike**: B — glyph→Unicode + line/paragraph reconstruction (engineering plan §14 step 5) * **Date**: 2025-07-25 * **Status**: Complete. **Gate MET** on the synthetic corpus. * **Gate**: paragraph-boundary F1 ≥ 0.85 (M0 exit criterion); ≥ 0.93 is the release gate (§8.2) and requires the real labelled corpus. ## What was built `spike/B_reconstruction/` implements the reconstruction pipeline (§4.1 steps 2–5) on a synthetic in-memory glyph model: - `GlyphRun.h` — the input model (`Glyph`, `GlyphRun`, `Line`, `Paragraph`, `GroundTruthParagraph`), mirroring §3.2/§3.3. - `Reconstruct.cpp` — step 2 (flatten runs to placed glyphs), step 3 (line detection by baseline clustering, tolerance 0.25 × font size per §4.1 step 3), step 4 (reading order: descending-y sort for the single-column LTR case — the recursive XY-cut reduces to this), step 5 (paragraph grouping), plus a boundary-F1 scorer. - `Corpus.cpp` — a deterministic synthetic corpus generator (seeded PRNG) producing 3–8 paragraphs of 1–5 lines each, with realistic leading, paragraph gaps, short last lines, and occasional list items. - `main.cpp` — runs the pipeline over the corpus and emits the contract JSON. Step 1 (glyph→Unicode via font cmap / ToUnicode) is not exercised by this spike: the synthetic generator produces glyphs whose `unicode` is known by construction. Step 1 is exercised separately once the production L3 interpreter (M2) wires FreeType + the ToUnicode CMap path. ## Result | Metric | Value | Target | |---|---|---| | paragraph-boundary F1 | **0.963** | ≥ 0.85 | | precision | 0.929 | — | | recall | **1.000** | — | | perfect documents | 334 / 500 | — | Tested on 500 synthetic documents, single-column LTR, 12pt font, 14.4pt leading, 23pt paragraph gap. Recall is perfect (no missed boundaries); the remaining imperfection is over-segmentation (precision 0.93) on the ~33% of documents where the synthetic generator creates edge cases — single-line paragraphs adjacent to multi-line ones, and list items. ## Findings 1. **The leading-gap signal is primary; the short-last-line signal is confirming, not primary.** The first implementation treated "previous line ended short of the right margin" as a sufficient paragraph-break condition and over-segmented badly (F1 ≈ 0.37): a short *first* line of a paragraph is common, and the heuristic split paragraphs at every short line. The correct hierarchy (per §4.1 step 5) is: leading consistency is the merge condition; a short last line only forces a break when *combined* with above-normal leading. This recovered F1 to 0.963. 2. **Coordinate-system orientation matters and is easy to get backwards.** The synthetic corpus uses PDF coordinates (y increases upward), so reading order is *descending* y. An ascending-y sort inverted the reading order and broke both the leading sign and the "previous line" semantics. This is a concrete instance of the §11 "real PDFs are pathological" risk and the reason the plan keeps the corpus in PDF coordinates throughout. 3. **Column-width inference must be robust to outliers.** Inferring the column right edge as the max `x_end` across lines let a single overshooting line push the edge out and make every other line look "not wrapped". Using the 90th percentile instead fixed the wrap-tolerance. This is the kind of detail the real §4.1 step 4 XY-cut handles by computing column geometry rather than inferring it from line ends. 4. **The 80/20 boundary is real and visible in the failure cases.** The 166 imperfect documents are exactly the §11 "last 20%" cases: single-line paragraphs, list items, paragraphs whose first line is short. The synthetic corpus makes these diagnosable; the real labelled corpus (§8.1) is needed to score against ground truth on actual layout pathology and to validate the 0.93 release gate. ## What this means for the project - **The reconstruction pipeline shape works and scores above the M0 bar on a controlled corpus.** The M0 exit criterion (F1 ≥ 0.85) is met; the project is not blocked on reconstruction feasibility. - **The release gate (F1 ≥ 0.93, §8.2) must be scored on the real labelled corpus**, not the synthetic one. The synthetic corpus validates the pipeline; it does not validate that the pipeline handles real PDF pathology. Acquiring/constructing the labelled corpus (§8.1) is the next prerequisite for the M4 text-editing milestone. - **Step 1 (glyph→Unicode) remains the highest-risk step** and is not covered by this spike. It is where the §4.1 priority ladder (ToUnicode → Encoding → built-in → cmap → OCR) lives, and where low-confidence mappings drive the UI's confidence-underlining. It should be the next reconstruction work. ## Reproducing ```bash cmake -S . -B build/manual -G Ninja -DCMAKE_BUILD_TYPE=Release cmake --build build/manual --target spike_b_reconstruction build/manual/bin/spike_b_reconstruction 500 0.85 # ``` The single-line JSON report on stdout is the CI contract. Exit 0 if the gate is met, 1 otherwise. Also verified clean under ASan+UBSan.