Skip to content

Tour — Follow one example back to its source

This page walks the provenance chain on real output from the offline demo (knovaryn demo): a deterministic, synthetic run on two bundled sample documents with the fake provider — no network, no keys, no external model. Every identifier, hash, quote, and quality result below is copied from that run. The content itself is synthetic demo text ("MLOps Lifecycle", "Incident response runbook") — it exists to demonstrate mechanics, not to teach MLOps.

The run produced 10 candidates from 10 chunks: 3 accepted, 6 rejected, 1 held for review. Select a row to see everything Knovaryn recorded about it:

An accepted example, fully traced

Prompt (user message)
"Based only on the provided material, summarize the key point about: MLOps Lifecycle"
Answer (assistant message)
"According to the provided material: MLOps Lifecycle MLOps Lifecycle"
Source document
"MLOps lifecycle overview" — text/markdown, 957 bytes
Source quote (the span)
"MLOps Lifecycle"
Location precision
section — chars 0–15, section path MLOps Lifecycle
Page / bounding box
Not available — markdown has no pages. A Docling-parsed PDF records page number and bounding box; this source honestly records section precision.
Content hash
085f04348bb5a3b4c7f45fb5a30ca3674d64319d741d3b32484a5df2e2027f1e
Quality result
accepted score 1.0 — all deterministic dimensions verified
Review state
No human review required (auto-accepted under demo policy floors)
Dataset version
0.1.0 — immutable snapshot, content-hash of all members
Export format
OpenAI-chat JSONL at data/validation.jsonl inside release.zip, with detached checksums + manifest

The full walk for this row, hop by hop:

Chain hopRecord (from the run)
Exported exampleex_01a03d42-29ab-7b0f-9c3d-a45cac20c1c1 — carries source_span_ids, source_document_ids, generation_candidate_ids, content_hash
Generation candidatecand_01a03d42-29a1-755b-863d-e02a7dcd176d — task family factual_explanation, topology sft, status accepted
Chunkck_7f68289ce0b8faeb2d231e81 — heading path MLOps Lifecycle, ordinal 0, chunker structure_aware, sha256 fb19b9f2…9f37
Source spansp_7f68289ce0b8faeb2d231e81 — quotes "MLOps Lifecycle", characters 0–15, precision section, same sha256 as the chunk text
Parsed documentpar_01a03d42-298a-725d-955e-5b04a3f96948 — parser fallback-text v1, config hash demo, extraction status parsed
Source documentsrc_01a03d42-2986-7e13-a70f-4423c932cb06 — 957 bytes, group key MLOps lifecycle overview, license/privacy recorded at intake

A rejected example — quarantined with its reasons

Prompt (user message)
"Based only on the provided material, summarize the key point about: Data preparation
Data preparation is the first step of any machine learning project."
Answer (assistant message)
"According to the provided material: Data preparation
Data preparation is the first step of any machine learning project. It involves collecting raw data, cleaning it, and transforming it into a usable format."
Source quote (the span)
"Data preparation
Data preparation is the first step of any machine learning project. It involves collecting raw data, cleaning it, and transforming it into a usable format. Practitioners must document the provenance of every data source to keep the dataset auditable."
Location precision
section — chars 0–267, section path MLOps Lifecycle/Data preparation
Page / bounding box
Not available — markdown source; section precision is recorded honestly
Content hash
108468b395092ad4cac21ec28338edadf1e943e3f80040852dcfe8ff7115ae7f
Quality result
rejected score 0.89 — subject_object_reversal, critical_failed:semantic_consistency
Review state
Quarantined — never exported, reason codes kept for audit
Dataset version / export
Not a member of version 0.1.0 — absent from release.zip by construction

What the gate caught: the answer reorders the source's subject and object ("collecting raw data" becomes the project's definition rather than its first step). The deterministic semantic-consistency check flags the reversal as subject_object_reversal, it is a critical dimension, and a critical failure is a hard reject — the row is quarantined with its reason codes and never reaches the export.

A preference pair held for human review

Prompt (user message)
"Based only on the provided material, summarize the key point about: MLOps Lifecycle"
Chosen answer
"According to the provided material: MLOps Lifecycle MLOps Lifecycle"
Rejected answer
"According to the provided material: MLOps Lifecycle MLOps Lifecycle (this additional claim was not supported)."
Source quote (the span)
"MLOps Lifecycle" — same span as the accepted row above
Location precision
section — chars 0–15
Page / bounding box
Not available — markdown source
Content hash
51db07d68f0840795ebdf7a6bc0cc6cb1a2cfb3a8471ac87907e4617262d8ae9
Quality result
review score 0.85 — length_band_violation, trivial_separation, preference_signal<0.7, artifact_resistance<0.75
Review state
Waiting in the review queue — a human approves or rejects with knovaryn review (immutable revisions, evidence attached)
Dataset version / export
Not exported while unreviewed — preference pairs ship only after a decision

Why review, not reject? The pair's separation is weak (the chosen and rejected answers differ by one appended sentence), so the deterministic preference-signal check scores it below the floor. That is a judgment call a human should make, so the pair is routed to review instead of silently shipping or silently vanishing.

Reading the chain

  • Every hop is a persisted record with a stable ID — the walk above is a real query path, not a narrative: the knovaryn_lineage MCP tool, the GET /v1/projects/{id}/examples/{id}/lineage REST endpoint, and the web console's lineage view all traverse it.
  • The span quotes its text and hashes it. Grounding is checkable: the validator scores the answer against the quoted span only.
  • Precision is reported, never exaggerated. A Docling-parsed PDF yields page numbers and bounding boxes; a markdown file honestly yields section precision and no bounding box. The demo uses the fallback text parser because the Docling extra is not installed in a bare pip install knovaryn — the parsed-document record says so explicitly.
  • Quarantine is a first-class outcome. Rejected rows keep their reason codes and lineage; they just never enter a version.

Reproduce this tour

pip install knovaryn
knovaryn demo --examples 6 --json

The run is deterministic (fake provider, seeded splits), so your identifiers will differ (they are time-derived) but the shape, counts, and outcome types match what you see above. Inspect any row yourself:

unzip -p knovaryn-demo/release.zip data/validation.jsonl | python3 -m json.tool

Related: provenance concepts · quality-gates tour · CLI reference