α) ≤ δ | 0.171 | 0.068 | 0.03 |\n| — cluster-corrected | same, n_eff = n/deff | 0.140 | 0.051 | 0.00 |\n| 4. rigorous doc-iid PAC | P(macro doc risk > α) ≤ δ | 0.060 | 0.020 | 0.00 |\n\nThe vocabulary matters: **tiers 1–2 control expected risk; only tiers 3–4 certify.**\nTier 1's 47.5% violation rate isn't a bug — that's what a mean-controlled bound\nsitting near its boundary looks like. If you need P(violation) ≤ δ, you buy tier 3\nand pay 0.318 → 0.171.\n\n**Tier 4 is the only tier whose assumptions actually match documents, and it is\nnear-vacuous today** — 0.060 coverage, certifying nothing in 19/40 splits. We lead\nwith that rather than bury it. Powered document-level PAC procedures are the named\nopen problem.\n\n## When conditioning pays\n\nA two-regime law, with mechanism and scope:\n\n- **With a learned score, covariates belong in the score.** A tree fusion splits on\n the covariate internally and equalizes per-group scales, so external Mondrian\n conditioning only fragments the threshold sample. It's *actively harmful* here\n (−0.062 / −0.036 coverage, p < 10⁻⁴).\n- **With a weak or frozen score, covariates belong in the taxonomy** — and pay\n precisely where a pooled threshold *cannot certify* at the target α. On CORD,\n support-bin 0.171 vs pooled 0.091–0.098 at the rigorous tier.\n\nFormalized at the ρ=0 boundary (Props. 1–3); the sign prediction is verified on all\nthree corpora, though n=3 corpora is n=3.\n\n## What we're careful *not* to claim\n\nProvenance conditioning wins every rigor tier on the sonnet CORD capture (p < 10⁻⁴,\nBonferroni-corrected over taxonomies) — and **the same win collapses on the identical\ndocuments under haiku or Qwen2.5-14B**, where the practical-tier winners are\nfield-type taxonomies instead. That counterexample is printed in §7, not hidden. It's\nthe boundary of the claim.\n\nWhat *does* transfer: a frozen configuration, executed once with no tuning on\nselection-untouched captures, held at both risk levels on `claude-haiku-4-5`\n(0.167 at 0.093) and on open-weights Qwen (0.149 at 0.099). **Across both untouched\ncaptures the risk contract never failed** — what varies is coverage, which tracks\nsignal quality.\n\nAnd the guarantee survives contact with humans: three blind annotators (Fleiss'\nκ=0.83) re-judged 149 fields from the accepted set. **Human-verified selective risk:\n2/149 = 1.3%, against a 10% budget.** The automatic calibration labels err\none-sidedly pessimistic (21% of auto-flagged errors are actually correct, 0%\nfalse-optimism), so thresholds fit against them are conservative.\n\n## The price of rigor\n\nCertified coverage retained vs the invalid folklore baseline: **0% at α=0.05, 28% at\n0.10, 42% at 0.15, 84% at 0.20.** Rigor is a knob, and its cost collapses as the risk\nbudget grows. That's the operating guidance, not a footnote.\n\n## Artifact\n\nApache-2.0, seed-pinned, regression-gated: **https://github.com/bhaskargurram-ai/verifydoc**\n\n```bash\npip install verifydoc\nmake results # regenerates every number in the paper from configs/\n```\n\n40 fixed document-level splits (seed 7), bit-exact sanity gates chained across\nexperiments, a 463/624-cells-bit-identical regression check — and **full forensics on\na capture-stage data defect** in Appendix B, including how it was found and what it\ninvalidated. Try it with no install: 🤗 https://huggingface.co/spaces/bhaskar1225/verifydoc\n\nCompanion benchmark paper (**VerifyDocBench**) owns the datasets, labeling protocol,\nreliability audit, and the per-model measurement study. This paper owns the\nprocedures, guarantees, diagnoses, and characterization.\n\nWe claim no new conformal theory — the machinery is classical (Vovk 2003;\nAngelopoulos 2021, 2024). The contribution is the diagnoses, the protocol, the\ncertified application, and the two-regime characterization.\n\n---\n\n**Open question I'd genuinely like input on:** tier 4 is the assumption-honest tier\nand it's nearly vacuous at 800 documents. Has anyone had luck with\nvariance-adaptive or clustered document-level bounds that stay powered at this scale?\nThat's the bottleneck between \"honest\" and \"useful,\" and I don't think it's a data\nproblem.","html":"<p><strong>Author here</strong> (Bhaskar Gurram, Zasti AI). The abstract is dense, so here's what<br>the paper actually argues.</p>\n<h2 class=\"relative group flex items-baseline\">\n\t<a id=\"the-claim\" class=\"block pr-1.5 text-lg md:absolute md:p-1.5 md:opacity-0 md:group-hover:opacity-100 md:right-full\" href=\"#the-claim\" rel=\"nofollow\">\n\t\t<span class=\"header-link\"><svg class=\"text-gray-500 hover:text-black dark:hover:text-gray-200 w-4\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\" aria-hidden=\"true\" role=\"img\" width=\"1em\" height=\"1em\" preserveAspectRatio=\"xMidYMid meet\" viewBox=\"0 0 256 256\"><path d=\"M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z\" fill=\"currentColor\"></path></svg></span>\n\t</a>\n\t<span>\n\t\tThe claim\n\t</span>\n</h2>\n<p>\"Accept a field only if the error rate among accepted fields stays under α\" is the<br>trust contract every document-extraction system implicitly sells. The standard way<br>to implement it — fit a confidence score, hold out a calibration split, take the<br>smallest add-one-smoothed threshold with empirical selective risk ≤ α — <strong>silently<br>violates that contract on real documents.</strong> Not visibly. Not in a way your<br>dashboard catches.</p>\n<p>Testbed: <strong>13,859 genuine <code>claude-sonnet-5</code> fields from 800 CORD receipts, of<br>which only 49.0% are correct.</strong> Deliberately hard, deliberately real.</p>\n<h2 class=\"relative group flex items-baseline\">\n\t<a id=\"three-failure-modes-each-pinned-by-a-counterfactual\" class=\"block pr-1.5 text-lg md:absolute md:p-1.5 md:opacity-0 md:group-hover:opacity-100 md:right-full\" href=\"#three-failure-modes-each-pinned-by-a-counterfactual\" rel=\"nofollow\">\n\t\t<span class=\"header-link\"><svg class=\"text-gray-500 hover:text-black dark:hover:text-gray-200 w-4\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\" aria-hidden=\"true\" role=\"img\" width=\"1em\" height=\"1em\" preserveAspectRatio=\"xMidYMid meet\" viewBox=\"0 0 256 256\"><path d=\"M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z\" fill=\"currentColor\"></path></svg></span>\n\t</a>\n\t<span>\n\t\tThree failure modes, each pinned by a counterfactual\n\t</span>\n</h2>\n<p><strong>1. Document clustering.</strong> Fields aren't exchangeable — errors arrive in<br>document-shaped clumps. Measured design effect <strong>1.84–2.45</strong>, so your effective<br>calibration sample is roughly half its nominal size. A clean-fit control (score fit<br>on held-out docs) <em>still</em> overshoots at risk 0.105 in 50% of splits, which isolates<br>clustering from the next failure.</p>\n<p><strong>2. Score-refit leakage.</strong> Fit a high-capacity score and its threshold on the same<br>fields and the score's optimism transfers straight into the threshold: <strong>coverage<br>0.416 at risk 0.127, violating nominal α=0.10 in 95% of splits.</strong> It looks<br>spectacular. It's invalid. A 5-parameter logistic barely overfits (0.105) — which is<br>exactly why nobody notices until score capacity grows.</p>\n<p><strong>3. Tie-mass pathology.</strong> A degenerate score distribution collapses the threshold<br>grid. Our first capture silently shipped an all-zero entailment column; the fused<br>score went from 1,702 distinct values to 257, with tie masses of 221 and 183 fields<br>at the acceptance head. A threshold takes a tie mass whole or not at all, so no<br>certificate existed at any confidence level. Counterfactually reproduced by zeroing<br>that one signal on the intact dump: <strong>certified coverage 0.030 → 0.001.</strong></p>\n<h2 class=\"relative group flex items-baseline\">\n\t<a id=\"the-validity-ladder\" class=\"block pr-1.5 text-lg md:absolute md:p-1.5 md:opacity-0 md:group-hover:opacity-100 md:right-full\" href=\"#the-validity-ladder\" rel=\"nofollow\">\n\t\t<span class=\"header-link\"><svg class=\"text-gray-500 hover:text-black dark:hover:text-gray-200 w-4\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\" aria-hidden=\"true\" role=\"img\" width=\"1em\" height=\"1em\" preserveAspectRatio=\"xMidYMid meet\" viewBox=\"0 0 256 256\"><path d=\"M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z\" fill=\"currentColor\"></path></svg></span>\n\t</a>\n\t<span>\n\t\tThe validity ladder\n\t</span>\n</h2>\n<p>The fixes are organized as tiers, each row stating its own estimand, assumption, and<br>violation fraction — because these bound <em>different functionals under different<br>assumptions</em> and shouldn't be read as one price curve.</p>\n<div class=\"max-w-full overflow-auto\">\n\t<table>\n\t\t<thead><tr>\n<th>tier</th>\n<th>guarantee</th>\n<th>coverage</th>\n<th>risk</th>\n<th>viol</th>\n</tr>\n\n\t\t</thead><tbody><tr>\n<td>1. practical (learned score)</td>\n<td>E[sel. risk] ≤ α, field</td>\n<td>0.318</td>\n<td>0.096</td>\n<td>0.475</td>\n</tr>\n<tr>\n<td>2. shared low-capacity fusion</td>\n<td>E[sel. risk] ≤ α, field</td>\n<td>0.212</td>\n<td>0.095</td>\n<td>0.35</td>\n</tr>\n<tr>\n<td>3. rigorous field-iid PAC</td>\n<td>P(group risk &gt; α) ≤ δ</td>\n<td>0.171</td>\n<td>0.068</td>\n<td>0.03</td>\n</tr>\n<tr>\n<td>— cluster-corrected</td>\n<td>same, n_eff = n/deff</td>\n<td>0.140</td>\n<td>0.051</td>\n<td>0.00</td>\n</tr>\n<tr>\n<td>4. rigorous doc-iid PAC</td>\n<td>P(macro doc risk &gt; α) ≤ δ</td>\n<td>0.060</td>\n<td>0.020</td>\n<td>0.00</td>\n</tr>\n</tbody>\n\t</table>\n</div>\n<p>The vocabulary matters: <strong>tiers 1–2 control expected risk; only tiers 3–4 certify.</strong><br>Tier 1's 47.5% violation rate isn't a bug — that's what a mean-controlled bound<br>sitting near its boundary looks like. If you need P(violation) ≤ δ, you buy tier 3<br>and pay 0.318 → 0.171.</p>\n<p><strong>Tier 4 is the only tier whose assumptions actually match documents, and it is<br>near-vacuous today</strong> — 0.060 coverage, certifying nothing in 19/40 splits. We lead<br>with that rather than bury it. Powered document-level PAC procedures are the named<br>open problem.</p>\n<h2 class=\"relative group flex items-baseline\">\n\t<a id=\"when-conditioning-pays\" class=\"block pr-1.5 text-lg md:absolute md:p-1.5 md:opacity-0 md:group-hover:opacity-100 md:right-full\" href=\"#when-conditioning-pays\" rel=\"nofollow\">\n\t\t<span class=\"header-link\"><svg class=\"text-gray-500 hover:text-black dark:hover:text-gray-200 w-4\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\" aria-hidden=\"true\" role=\"img\" width=\"1em\" height=\"1em\" preserveAspectRatio=\"xMidYMid meet\" viewBox=\"0 0 256 256\"><path d=\"M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z\" fill=\"currentColor\"></path></svg></span>\n\t</a>\n\t<span>\n\t\tWhen conditioning pays\n\t</span>\n</h2>\n<p>A two-regime law, with mechanism and scope:</p>\n<ul>\n<li><strong>With a learned score, covariates belong in the score.</strong> A tree fusion splits on<br>the covariate internally and equalizes per-group scales, so external Mondrian<br>conditioning only fragments the threshold sample. It's <em>actively harmful</em> here<br>(−0.062 / −0.036 coverage, p &lt; 10⁻⁴).</li>\n<li><strong>With a weak or frozen score, covariates belong in the taxonomy</strong> — and pay<br>precisely where a pooled threshold <em>cannot certify</em> at the target α. On CORD,<br>support-bin 0.171 vs pooled 0.091–0.098 at the rigorous tier.</li>\n</ul>\n<p>Formalized at the ρ=0 boundary (Props. 1–3); the sign prediction is verified on all<br>three corpora, though n=3 corpora is n=3.</p>\n<h2 class=\"relative group flex items-baseline\">\n\t<a id=\"what-were-careful-not-to-claim\" class=\"block pr-1.5 text-lg md:absolute md:p-1.5 md:opacity-0 md:group-hover:opacity-100 md:right-full\" href=\"#what-were-careful-not-to-claim\" rel=\"nofollow\">\n\t\t<span class=\"header-link\"><svg class=\"text-gray-500 hover:text-black dark:hover:text-gray-200 w-4\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\" aria-hidden=\"true\" role=\"img\" width=\"1em\" height=\"1em\" preserveAspectRatio=\"xMidYMid meet\" viewBox=\"0 0 256 256\"><path d=\"M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z\" fill=\"currentColor\"></path></svg></span>\n\t</a>\n\t<span>\n\t\tWhat we're careful <em>not</em> to claim\n\t</span>\n</h2>\n<p>Provenance conditioning wins every rigor tier on the sonnet CORD capture (p &lt; 10⁻⁴,<br>Bonferroni-corrected over taxonomies) — and <strong>the same win collapses on the identical<br>documents under haiku or Qwen2.5-14B</strong>, where the practical-tier winners are<br>field-type taxonomies instead. That counterexample is printed in §7, not hidden. It's<br>the boundary of the claim.</p>\n<p>What <em>does</em> transfer: a frozen configuration, executed once with no tuning on<br>selection-untouched captures, held at both risk levels on <code>claude-haiku-4-5</code><br>(0.167 at 0.093) and on open-weights Qwen (0.149 at 0.099). <strong>Across both untouched<br>captures the risk contract never failed</strong> — what varies is coverage, which tracks<br>signal quality.</p>\n<p>And the guarantee survives contact with humans: three blind annotators (Fleiss'<br>κ=0.83) re-judged 149 fields from the accepted set. <strong>Human-verified selective risk:<br>2/149 = 1.3%, against a 10% budget.</strong> The automatic calibration labels err<br>one-sidedly pessimistic (21% of auto-flagged errors are actually correct, 0%<br>false-optimism), so thresholds fit against them are conservative.</p>\n<h2 class=\"relative group flex items-baseline\">\n\t<a id=\"the-price-of-rigor\" class=\"block pr-1.5 text-lg md:absolute md:p-1.5 md:opacity-0 md:group-hover:opacity-100 md:right-full\" href=\"#the-price-of-rigor\" rel=\"nofollow\">\n\t\t<span class=\"header-link\"><svg class=\"text-gray-500 hover:text-black dark:hover:text-gray-200 w-4\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\" aria-hidden=\"true\" role=\"img\" width=\"1em\" height=\"1em\" preserveAspectRatio=\"xMidYMid meet\" viewBox=\"0 0 256 256\"><path d=\"M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z\" fill=\"currentColor\"></path></svg></span>\n\t</a>\n\t<span>\n\t\tThe price of rigor\n\t</span>\n</h2>\n<p>Certified coverage retained vs the invalid folklore baseline: <strong>0% at α=0.05, 28% at<br>0.10, 42% at 0.15, 84% at 0.20.</strong> Rigor is a knob, and its cost collapses as the risk<br>budget grows. That's the operating guidance, not a footnote.</p>\n<h2 class=\"relative group flex items-baseline\">\n\t<a id=\"artifact\" class=\"block pr-1.5 text-lg md:absolute md:p-1.5 md:opacity-0 md:group-hover:opacity-100 md:right-full\" href=\"#artifact\" rel=\"nofollow\">\n\t\t<span class=\"header-link\"><svg class=\"text-gray-500 hover:text-black dark:hover:text-gray-200 w-4\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\" aria-hidden=\"true\" role=\"img\" width=\"1em\" height=\"1em\" preserveAspectRatio=\"xMidYMid meet\" viewBox=\"0 0 256 256\"><path d=\"M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z\" fill=\"currentColor\"></path></svg></span>\n\t</a>\n\t<span>\n\t\tArtifact\n\t</span>\n</h2>\n<p>Apache-2.0, seed-pinned, regression-gated: <strong><a href=\"https://github.com/bhaskargurram-ai/verifydoc\" rel=\"nofollow\">https://github.com/bhaskargurram-ai/verifydoc</a></strong></p>\n<pre><code class=\"language-bash\">pip install verifydoc\nmake results <span class=\"hljs-comment\"># regenerates every number in the paper from configs/</span>\n</code></pre>\n<p>40 fixed document-level splits (seed 7), bit-exact sanity gates chained across<br>experiments, a 463/624-cells-bit-identical regression check — and <strong>full forensics on<br>a capture-stage data defect</strong> in Appendix B, including how it was found and what it<br>invalidated. Try it with no install: 🤗 <a href=\"https://huggingface.co/spaces/bhaskar1225/verifydoc\">https://huggingface.co/spaces/bhaskar1225/verifydoc</a></p>\n<p>Companion benchmark paper (<strong>VerifyDocBench</strong>) owns the datasets, labeling protocol,<br>reliability audit, and the per-model measurement study. This paper owns the<br>procedures, guarantees, diagnoses, and characterization.</p>\n<p>We claim no new conformal theory — the machinery is classical (Vovk 2003;<br>Angelopoulos 2021, 2024). The contribution is the diagnoses, the protocol, the<br>certified application, and the two-regime characterization.</p>\n<hr>\n<p><strong>Open question I'd genuinely like input on:</strong> tier 4 is the assumption-honest tier<br>and it's nearly vacuous at 800 documents. Has anyone had luck with<br>variance-adaptive or clustered document-level bounds that stay powered at this scale?<br>That's the bottleneck between \"honest\" and \"useful,\" and I don't think it's a data<br>problem.</p>\n","updatedAt":"2026-08-18T14:53:28.986Z","author":{"_id":"679b924d8985c50c019f27f9","avatarUrl":"/avatars/39ed56ef774066a2bb34fe2b55abe0f3.svg","fullname":"Bhaskar Gurram","name":"bhaskar1225","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.880997896194458},"editors":["bhaskar1225"],"editorAvatarUrls":["/avatars/39ed56ef774066a2bb34fe2b55abe0f3.svg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2608.14639","authors":[{"_id":"6a8470d2236d8fccf7c418b2","name":"Bhaskar Gurram","hidden":false}],"publishedAt":"2026-07-28T00:00:00.000Z","submittedOnDailyAt":"2026-08-18T00:00:00.000Z","title":"Valid Per-Field Selective Risk Control for Document Extraction: Three Failure Modes, a Validity Ladder, and When Conditioning Pays","submittedOnDailyBy":{"_id":"679b924d8985c50c019f27f9","avatarUrl":"/avatars/39ed56ef774066a2bb34fe2b55abe0f3.svg","isPro":false,"fullname":"Bhaskar Gurram","user":"bhaskar1225","type":"user","name":"bhaskar1225"},"summary":"Per-field accept/review with selective risk at most alpha -- accept a field only if the error rate among accepted fields is controlled -- is the trust contract document-extraction systems need, and the natural procedure silently violates it on real documents. On 13,859 genuine claude-sonnet-5 fields from 800 CORD receipts (49.0% correct) we diagnose three failure modes: document clustering (design effect 1.84-2.45), score-refit leakage (coverage 0.416 at risk 0.127, violating alpha=0.10 in 95% of splits), and a tie-mass pathology (a degenerate score collapses the threshold grid, 0.030 to 0.001). We organize the fixes as a validity ladder, guarantee form stated per tier. A fit/val split protocol restores expected-selective-risk control for a learned fusion: coverage 0.318 at risk 0.096 at nominal alpha=0.10, no tolerance band (production variant 0.326) -- an on-average point whose realized risk exceeds alpha in 47.5% of resplits, not a certificate. Mondrian Learn-then-Test with exact binomial tails yields per-group PAC certificates: field-iid 0.171 at risk 0.068, cluster-corrected 0.140, doc-iid 0.060 -- the only tier matching documents, honestly near-vacuous today. Support-bin, the pre-specified provenance taxonomy, wins every rigor tier on the sonnet CORD capture (p<1e-4, Bonferroni-corrected) -- a win that does not replicate on the same documents under haiku or qwen -- while on higher-accuracy corpora pooled thresholds win: conditioning helps exactly where pooled cannot certify, subsumed by a learned score elsewhere. A frozen-configuration confirmation on selection-untouched claude-haiku-4-5 held at both risk levels, and a blind three-annotator human-gold audit verifies the practical tier's accepted-set risk at 1.3% against its 10% budget (Fleiss' kappa=0.83; labels err one-sidedly pessimistic). Released Apache-2.0 with seed-pinned, regression-gated procedures.","upvotes":0,"discussionId":"6a8470d3236d8fccf7c418b3","ai_summary":"Per-field selective risk control for document extraction requires a validity ladder with fit/val splits and Mondrian PAC certificates, revealing that support-bin provenance outperforms learned fusion only under specific model conditions.","ai_keywords":["Mondrian Learn-then-Test","PAC certificates","selective risk","document clustering","score-refit leakage","tie-mass pathology","validity ladder","support-bin provenance"],"ai_summary_model":"thinkingmachines/Inkling-Small"},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[],"acceptLanguages":["en"],"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2608/2608.14639.md","query":{}}">

Get this paper in your agent:

Don't have the latest CLI?

Cite arxiv.org/abs/2608.14639 in a model README.md to link it from this page.

Cite arxiv.org/abs/2608.14639 in a dataset README.md to link it from this page.

Cite arxiv.org/abs/2608.14639 in a Space README.md to link it from this page.

Add this paper to a collection to link it from this page.