Conformance credibility

Who this is for: Integrators, auditors, and contributors who need to read the PDOC agreement rate without mistaking harness noise for engine truth.

When you finish: You know what 8905/9002 (and 8987/9002 with rounding_compat: pdoc) counts, what it excludes, and how to spot a defect versus a published disagreement.

The live numbers live in CONFORMANCE.md. That file is generated by tools/conformance. Do not hand-edit rates or counts there. This page explains what those numbers mean.

Agreement rate: meaning

Overall agreement rate (PDOC corpora): 8905/9002. With rounding_compat: pdoc: 8987/9002. Before the 22 October 2026 audit (finding 003, dated 2026-10-22): 8838/9002.

The numerator counts measured PDOC corpora cells where the engine and live CRA PDOC (https://apps.cra-arc.gc.ca/ebci/rhpd/beta/) match on identically specified inputs. The denominator counts every measured cell in those corpora. Partial corpora do not publish a rate until every defined vector in that corpus is measured.

A disagreement is a remaining delta after both sides received the input the operator intended. We publish the full request, both outputs, the cent delta, and our reading. Some disagreements conclude PDOC diverges from T4127 (Payroll Deductions Formulas). They still count in the rate when the inputs match.

The rate is not “every test that ever failed.” Harness bugs, field mis-mapping, or vectors specified differently on the two sides are defects. We fix them, add a regression test, and they leave the disagreement record. Inflating the disagreement count with our own mistakes would reduce trust, not increase honesty.

Agreement rate: what it does not mean

The rate does not cover the full 1,263,636-cell grid as per-cell PDOC checks. Most grid cells use other oracle classes (see below). The rate does not include pdoc-bonus-2026 (20 bonus vectors, own class). It does not inherit weaker evidence from t4127-option2-2026 (T4127 Chapter 5 worked examples; live PDOC has no Option 2 control). It does not treat 780 PDOC step-2 non-advance forms as failures: those forms never completed capture and sit outside the denominator. It says nothing about claim codes 2 to 10 (TD1 personal amount codes): the PDOC corpora use claim codes 0, 1 and E only. Codes 2 to 10 rest on the CRA claim-code tables and an independent oracle (tools/claim-codes/oracle.py, 384 vectors in crates/takehome-core/tests/vectors/claim_codes_2026.json).

Oracle classes and counts

Each grid cell carries one oracle class. Boundary forms that share a PDOC-visible fingerprint collapse to one queue item.

pdoc: Live PDOC capture on capturable boundary forms. 9,762 distinct queue forms; corpora total 9,002 measured in the published rate.
invariants: Engine properties and API year-projection checks. 1,126,620 interior ladder cells plus named invariant corpora.
uncapturable: Salary UI cannot enter the form. 5,502 boundary cells (see below).
engine_error: Date before coverage, invalid keys. 140,354 cells.
t4127_worked_examples: Option 2 Chapter 5 examples. 3 vectors; weaker than PDOC.

Corpus breakdown in the generated report: M1 Ontario twenty-twenty (20/20), grid PDOC boundary (8818/8982 within that corpus), plus bonus and invariant corpora named in CONFORMANCE.md.

Conformance pipeline — The grid splits cases by oracle class before capture. Only capturable PDOC boundary forms go to live PDOC. Uncapturable forms and the interior ladder are validated by invariants. Honesty of that split is the point of the published rate.

Why the interior ladder skips PDOC

The spec’s “250,000 cases” target is a case list, not a finishable PDOC queue. At three seconds per uncached form, a queue that size exceeds overnight capacity. The adopted split sends every distinct capturable live-edition boundary form to PDOC. The interior log-ladder (1,123,344 cells) and uncapturable class do not.

Interior cells still matter. They run under oracle class invariants plus the differential oracle described in the conformance methodology (same engine checks as uncapturable forms). Skipping PDOC there is a sampling decision, not a waiver of correctness.

Four PDOC-uncapturable pay periods

PDOC’s salary UI accepts ten of fourteen legal pay periods (P values). It cannot capture forms for:

P=1: Annual
P=2: Semi-annual
P=4: Quarterly
P=2000: Hourly (2000 hours)

Those periods, plus $0 and sub-dollar gross on the boundary grid, form the uncapturable class (5,502 July boundary forms). We validate them with invariants and the differential oracle. They are named in the census, not silently dropped from a queue that still claimed 15,264 all-fourteen-P forms.

Option 2 and weaker evidence

T4127 Option 2 (cumulative averaging) appears in corpus t4127-option2-2026. Live PDOC (observed 2026-06-11) offers Salary, Commission, Pension, and CPP/EI verification only. No Option 2 control exists on the public calculator, so no PDOC vectors were captured for that path. The three worked examples plus invariants match the engine (3/3). That corpus does not inherit the Option 1 PDOC agreement rate.

Methodology findings (M-001 through M-004)

M-001: Published whole-dollar K (ADR-002). Accepted discontinuity at bracket thresholds.
M-002: PDOC K2 maximum in reaching period. Documented; default k2_method is pdoc_observed.
M-003: PDOC midpoint direction. Mostly closed by finding 003; 15 one-cent tax lines remain open and count in both rates.
M-004: PDOC's CPP line matches binary64 arithmetic (inferred from its outputs). Closed; selectable as rounding_compat: pdoc (ADR-007).

Details: docs/findings/README.md, 001-k2-maximum-in-reaching-period.md, 002-pdoc-midpoint-direction.md, 003-pdoc-cpp-binary64-and-split-credits.md.

Defect versus disagreement

Identical intended inputs, outputs differ: Disagreement. Publish in CONFORMANCE.md; may become a finding.
Harness mapped the wrong field or reused engine output as expected: Defect. Fix harness; never publish as disagreement.
Vector inputs differ between engine JSON and PDOC form: Defect. Align specification; regression test.

Credibility depends on keeping defects out of the disagreement table.

Reproduce one case against PDOC (five minutes)

Use vector on-biweekly-midyear-k2-max (M-002) or any M-003 grid id such as NL-P26-1718.39-F0-P0 from the disagreement table in CONFORMANCE.md.

K2 example (M-002):

Open https://apps.cra-arc.gc.ca/ebci/rhpd/beta/
Province: Ontario. Pay date: 2026-01-15. Pay period: biweekly (26).
Salary: 2500.00.
Use Claim codes 1 federal and 1 provincial (TD1 claim code selects the CRA personal amount; do not type a fixed TD1 dollar that skips BPAF (Basic Personal Amount Factor) phaseout).
Year-to-date: CPP 4200.00, EI 1000.00, pensionable earnings 71000.00, insurable earnings 50000.00, periods elapsed 20.
Compare Federal 243.95, Provincial 132.29, Net 2052.56 to PDOC. Run the same JSON through /v1/deductions with default k2_method.

Midpoint example (M-003, open): Newfoundland, semi-monthly P=26, gross 1718.39, July live edition, federal and provincial claim code 0. Engine provincial period tax is one cent above PDOC (138.87 vs 138.86). Neighbour gross 1718.40 matches PDOC. Full neighbour table: 002-pdoc-midpoint-direction.md.

For API reproduction without PDOC, follow FIVE-MINUTE-TEST.md for signup and the canonical Ontario weekly call.

Last reviewed: 2026-10-24
Engine: 0.1.0