Knovaryn
Every training example, traced to its source.
Turn permitted PDFs and documents into quality-gated SFT, preference, KTO, and evaluation datasets — with provenance enforced from source span through exported row.
pip install knovaryn
Knovaryn is an open-source, self-hosted training-data foundry: a CLI, an MCP server, a REST control plane with a local web console, and a Python SDK over the same application core. It runs local-first by default (SQLite + filesystem) and scales to a team deployment (PostgreSQL + S3-compatible storage) as configuration, not a fork.
How it works¶
Select a stage to see what it does, what it records, how it fails, and the exact command or tool that drives it. The full information on this page is static HTML — it works with JavaScript disabled.
Ingest documents you are permitted to use — as untrusted input.
Input: local files or paths (PDF, DOCX, Markdown, plain text).
Output: source records with SHA-256 digest, byte size, media type.
Provenance created: document IDs, content digests, license and privacy classification, group key.
Failure behavior: unknown or blocked licenses fail closed on the public path; preflight failures refuse ingestion rather than guessing.
knovaryn source add proj_… ./handbook.md --license CC-BY-4.0MCP tool: knovaryn_add_source · report with knovaryn_license_report.
Convert documents into canonical, structured representations.
Input: raw source bytes from intake.
Output: canonical DoclingDocument JSON for PDFs/office documents (Docling optional extra), built-in parsers for Markdown/plain text; derived markdown/text/tables.
Provenance created: parsed-document records and source spans with a machine-reported location precision (exact_bbox, exact_page, page_range, section, chunk) that reflects what the parser actually recorded.
Failure behavior: parsing runs behind a resource guard (memory limits, worker recycling); if the optional Docling extra is missing, PDF parsing is unavailable and knovaryn doctor reports it before you spend time.
knovaryn run --project proj_… # parsing runs inside the durable jobMCP tool: knovaryn_start_pipeline.
Split documents into generation-sized units without tearing structure apart.
Input: normalized documents.
Output: structure-aware chunks — headings, tables, and lists stay together, with neighbor context; train/validation splits grouped by source so one document never crosses both.
Provenance created: chunk IDs carrying their source_span_ids.
Failure behavior: a source group too small to split is kept whole rather than leaking across boundaries.
This stage runs inside the same durable job as parsing — checkpointed per stage, resumable without repeating earlier work.
MCP tools: knovaryn_start_pipeline · inspect results with knovaryn_preview_examples.
Decide what to generate — and what it will cost — before anything is spent.
Input: your plan parameters: task-family mix, difficulty distribution, target size, spend cap.
Output: a generation profile across topologies (SFT, preference, KTO, evaluation) plus an estimated cost and expected yield checked against budget.maximum_cost_usd.
Provenance created: the plan itself, stored with the run.
Failure behavior: planning is a dry run — no model calls — so there is nothing to fail expensively.
knovaryn run --project proj_… \
--family factual_explanation:1.0 --target 200 --budget-usd 5MCP tool: knovaryn_estimate_run (dry-run cost estimate).
Generate typed candidates through a provider-agnostic gateway.
Input: chunks + plan.
Output: SFT, preference, KTO, and evaluation candidates from OpenAI-compatible, Anthropic-compatible, or DeepSeek endpoints — or the deterministic fake provider for fully offline runs.
Provenance created: generation candidate IDs on every example.
Failure behavior: --budget-usd is a hard cap; jobs checkpoint per stage, so a resumed run continues from the last completed stage instead of repeating already-paid model calls.
knovaryn job status job_…MCP tools: knovaryn_run_job · watch with knovaryn_get_job.
Score every candidate against a dated acceptance policy — quarantine weak examples so they never export.
Input: generated candidates.
Output: accepted examples and quarantined ones, each with per-dimension scores and reason codes.
Provenance created: quality assessments tied to candidate IDs.
Failure behavior: failures route to quarantine with their cause recorded — never silently dropped, never exported.
knovaryn dataset validate proj_…MCP tool: knovaryn_validate_dataset.
Put a human decision on top of machine judgment.
Input: validated examples with their evidence.
Output: approve/reject decisions recorded as immutable revisions, with reviewer identity and note.
Provenance created: review revisions — who decided what, when, against which evidence.
Failure behavior: revisions are append-only; nothing is edited away.
knovaryn review ex_… approve --reviewer alice --note "grounded in page 3"MCP tool: knovaryn_review_example.
Freeze one canonical snapshot of the dataset.
Input: accepted, reviewed examples.
Output: a semantic-versioned snapshot with manifest, quality report, dataset card, source manifest, license report, and privacy report.
Provenance created: the version record binding every exported row back to its evidence.
Failure behavior: versioning requires validation to have passed; you cannot freeze an ungoverned state.
knovaryn dataset version proj_… --set 1.0.0MCP tool: knovaryn_create_dataset_version.
Write trainer-ready formats from the one canonical version.
Input: a frozen dataset version.
Output: JSONL, Parquet, TRL, ShareGPT, Alpaca, OpenAI chat, Hugging Face layout, and evaluation formats — repeat the command with another --format; the generated table in the exporter reference is authoritative.
Provenance created: export artifacts referencing the version they came from.
Failure behavior: optional publication is gated by license/privacy reports and requires an explicit confirmation token.
knovaryn dataset export proj_… --format openai_chat --out ./exportMCP tool: knovaryn_export_dataset.
Run it offline¶
No API keys, no network, no account:
pip install knovaryn
knovaryn demo --examples 20 --json
The demo ingests bundled sample documents, generates on the deterministic fake provider, validates, versions, and exports — the whole loop in about a minute, reproducible byte-for-byte. The ten-minute walkthrough then drives the same loop command by command: quickstart.
Why Knovaryn¶
Provenance by design
Every exported row carries source_document_ids, source_span_ids, a content_hash, and candidate IDs — walk any row back to the exact source spans it came from.
Scope: location precision reflects what the parser recorded — not every format yields bounding boxes.
Fail-closed quality
A dated acceptance policy scores each candidate; weak ones are quarantined with reason codes and never reach export.
Limitation: deterministic heuristics measure policy compliance, not general entailment; certified-semantic checks need a configured judge, and unavailable judges fail closed.
Durable jobs
Leased workers, checkpoints, budgets, and retries: a crashed run resumes from the last completed stage without repeating paid model calls.
Scope: multi-worker operation wants PostgreSQL; SQLite suits single-machine use.
MCP-native control
Drive the entire pipeline — sources, plans, runs, lineage, reviews, exports — from any MCP client. The catalogue is generated from the app.
Scope: compatibility is tested against MCP SDK 1.x and 2.x releases; upstream APIs still move.
Offline-first workflow
Install, demo, and develop with zero credentials using the deterministic fake provider — the full pipeline, reproducible byte-for-byte.
Limitation: fake-provider output proves plumbing, not training signal — real data needs a real endpoint.
Provider & trainer flexibility
Bring OpenAI-compatible, Anthropic-compatible, or DeepSeek endpoints; export one canonical dataset to the major trainer layouts.
Limitation: other providers mean speaking one of those protocols; no managed hosting is offered.
License & privacy gates
Sources are classified at intake; blocked or unknown licenses fail closed on the public path, and publication is gated by license and privacy reports.
Limitation: these gates are safety rails, not legal clearance or legal advice.
Reproducible releases
Frozen dataset versions bundle manifest, quality/license/privacy reports, and checksums — verify any release artifact after the fact.
Scope: regeneration reproduces given the same inputs; generated text itself depends on provider behavior.
Interfaces¶
All four sit on the same application-services core:
- CLI —
knovaryn(generated command table): lifecycle groupsproject,source,run,job,review,dataset, plus operational commandsdemo,init,doctor,repair,backup,restore,server,worker,mcp,verify-release,version. - MCP server —
knovaryn-mcp(orknovaryn mcp); catalogue in the generated tool reference, env prefixKNOVARYN_. - REST control plane + local web console —
knovaryn server(endpoint table).
knovaryn server. The linked 24-second animation (deliberately not autoplayed) walks source → run → quality result → provenance → export against the deterministic demo workspace; a full transcript is in record_demo.md.- Python SDK — the
Workspaceapplication core.
Trust and evidence¶
Verifiable engineering facts, linked at the source — no numbers here that rot:
Tested across supported Python majors in CI; MCP SDK 1.x and 2.x verified cell-by-cell.
Lint, strict typing, tests with a coverage floor, pip-audit, Trivy container/IaC scanning on every change.
Every GitHub release ships SBOM (CycloneDX) + SHA256SUMS; knovaryn verify-release checks a downloaded bundle; deployments are tested against the published tag before publish completes.
Every public claim — capability, benchmark, compatibility — is mapped to its executable evidence in the claim matrix.
Zero third-party requests on this site; no analytics in the product; your documents never leave your machine unless you point Knovaryn at a provider.
Start from your task¶
- Get started
- Build an SFT dataset
- Build preference data
- Use Knovaryn over MCP
- Inspect provenance
- See the gates decide
- Deploy for a team
- Export and publish
- Contribute
- Troubleshoot
New to the ecosystem? The Knovaryn MCP overview explains how the pieces work together as a product.
Honest limitations¶
- Quality gates measure compliance with the configured policy; they do not guarantee the absence of bias or hallucination, nor that a dataset improves any particular model.
- License handling gates blocked/unknown sources on the public path — it is not legal clearance.
- Deterministic heuristics do not prove general entailment;
certified-semanticrequires a configured judge, and unavailable judges fail closed. - This is alpha software (
0.2.2): APIs are not stabilized, and there is no hosted service — you self-host.
License¶
Apache-2.0. See the license and privacy notes for what that means for generated data.