Tour — How the quality gates decide¶

Same run as the provenance tour: knovaryn demo, the deterministic offline pipeline on two bundled sample documents. All content is synthetic demo text. Ten candidates went through validation; 3 passed, 6 were quarantined, 1 was routed to human review — all three outcomes in one small run, with the reason codes the gates emitted.

The twelve gate dimensions¶

What each dimension checks, and what happened in this run:

Dimension	What it checks	In this run
Schema	The example matches its topology's shape (messages, roles, fields).	Scored on all 10 candidates; all passed.
Grounding	The answer is supported by the cited spans only — scored against quoted evidence, not the whole document.	Scored; one candidate rejected with `low_grounding`, `grounding<0.9`, `critical_failed:grounding`.
Answerability	The prompt is actually answerable from the cited material.	Scored on all candidates; all passed.
Completeness	The answer covers what the prompt asks, per the cited spans.	Scored on all candidates; all passed.
Semantic consistency	The answer preserves the source's meaning (subject/object roles, negations, added claims).	One rejection: `subject_object_reversal` → `critical_failed:semantic_consistency`.
Duplicate detection	Content-hash dedup removes repeated rows before they ship.	Not exercised in the demo path (each candidate was unique); runs as a pipeline stage in full projects.
Contamination	Known eval-set overlap is flagged before export.	Not exercised in the demo (no eval corpus configured); runs in full projects.
Information gain	Preference pairs must separate the chosen answer meaningfully, not trivially.	The review-routed pair scored `trivial_separation` — the two answers differ by one appended sentence.
Preference signal	The chosen/rejected gap must clear a floor for preference topologies.	The pair scored `preference_signal<0.7` → routed to review, not exported.
Privacy	PII scanning at intake and before publication; findings block or flag.	Intake recorded a privacy classification for both demo sources (`unknown` — the synthetic texts carry no classification signal); the publication gate re-checks at publish time.
License	License status is recorded at intake and gates publication.	Both demo sources recorded `license_status: unknown` — and the demo bundle's license summary reports `status: review, count: 0` rather than pretending clearance.
Provenance integrity	Export re-resolves every citation; an example whose lineage cannot be walked is not exported.	Held for all 3 exported examples (see the provenance tour).

The run also scored format, refusal, instruction fulfillment, and artifact resistance — the full per-candidate dimension set recorded on each example as quality_dimensions with per-dimension verify states.

The four outcomes¶

Pass Quarantine Review required Unverified

Pass — exported with its evidence

The accepted SFT example scored 1.0 across every scored dimension and every dimension's verify state is verified (one exception noted under Unverified). It became a member of dataset version 0.1.0 and shipped in release.zip with its span references, content hash, and quality record attached.

Example

ex_01a03d42-29ab-7b0f-9c3d-a45cac20c1c1

Score

1.0 — schema, grounding, answerability, completeness, semantic consistency, format, refusal, instruction fulfillment, artifact resistance

Destination

data/validation.jsonl in release.zip, split validation

Quarantine — rejected with reason codes, never exported

Six candidates were rejected. Two rejection signatures appeared:

Reason codes (verbatim)	Score	What happened
`subject_object_reversal`, `critical_failed:semantic_consistency`	0.89	The answer swapped the source's subject and object. Semantic consistency is a critical dimension; a critical failure is a hard reject.
`low_grounding`, `grounding<0.9`, `critical_failed:grounding`	0.89	The answer was not sufficiently supported by the cited span. Grounding below the floor is a critical failure.

Quarantined rows keep their full lineage and reason codes in the workspace quarantine store — they are auditable, they just never enter a version or an export.

Review required — a human decides

The preference pair scored below the preference-signal floor with four reason codes: length_band_violation, trivial_separation, preference_signal<0.7, artifact_resistance<0.75. None is a hard failure on its own — the separation is weak, which is exactly the judgment a human reviewer is better at than a heuristic. The pair waits in the review queue; knovaryn review approve/reject (or knovaryn_review_example over MCP) records the decision as an immutable revision with evidence.

Unverified — recorded honestly, not passed silently

Even the accepted example carries one honest gap: its artifact_resistance dimension has verify state unverified (the demo's deterministic checks cannot certify that dimension without a configured judge). The example ships — artifact resistance is not a critical floor in the demo profile — but the gap is recorded on the example, visible in its quality metadata, instead of being silently treated as a pass. That is what fail-closed means in practice: dimensions that are critical and unverified block the row; dimensions that are not critical are exported with the gap stated.

What the gates do not claim¶

Deterministic heuristics do not prove general entailment. The grounding and semantic-consistency checks are string- and role-level heuristics over the cited spans. They catch mechanical failures — reversals, unsupported additions, low overlap — not subtle untruths.
certified-semantic requires a configured judge. Stricter semantic verification needs a judge model configured in your profile; the demo runs without one.
Unavailable judges fail closed. If a profile demands judge certification and no judge is reachable, validation fails those dimensions rather than waving rows through.
License handling is not legal advice. Knovaryn records license status and gates publication on your policy; it does not tell you what you may legally do with a source.

Reproduce the run behind this tour:

pip install knovaryn
knovaryn demo --examples 6 --json