stele

Open Source

A quality system for AI-assisted content production: automated consistency checks against your source material, staged human approval, and a correction loop that compounds.

$ make demo
GREEN -- stele catalog demo
  [gate/PASS] batch_derivation: every derivable field in all 30 batch items matches what the rules actually produce.
  [gate/PASS] fingerprint_change_rate: 100% of this batch differs from its source, at or above the floor of 60%.
  [gate/PASS] fingerprint_diversity: this batch has 100% unique values on this dimension, at or above the floor of 60%.
  [gate/PASS] seal_scan_shipped: scanned 4 file(s); no sealed terms found.
  [gate/PASS] seal_canary_self_test: all 1 canary(s) were found: the scanner works.
SUITE: GREEN

the honest run passes

$ make catch
RED -- stele catalog demo
  [gate/RED] batch_derivation: 60 derived field(s) in this batch don't match what the rules produce -- hand-written history wearing derived clothes: GR900.price_last_year: batch says 84.36, the rules produce 79.58 (+56 more)
  [gate/RED] fingerprint_change_rate: only 0% of this batch actually differs from its source, below the floor of 60%. That's the signature of a shortcut: values that look derived but were really just history copied from the present.
  [gate/RED] fingerprint_diversity: this batch only has 17% unique values on this dimension, below the floor of 60%. That's the signature of a recycled template: the same handful of values stamped across many items with just the nouns swapped.
SUITE: RED

the cheating run is caught, and the report names why

Real output from the repository, trimmed for display. Offline, no API key, plain Python. make catch exits nonzero on purpose: a caught batch is the tool's failure signal, so CI can act on it.

How it works

One unit of content, from idea to published.

How the checks work: each one recomputes the source, measures the batch against known-good work, or scans for terms that must never ship. It doesn't just block on a keyword match. When something is off, the report names which field, which item, and why.
SOURCE MATERIAL The record specs · canon · approved facts Never-ship list embargoes · keys · spoilers Known-good work the shape honest output has Rules learned from every past correction 1 · BRIEF a person writes what's wanted + which checks it must pass 2 · GENERATE Drafter writes at volume, cheaply Critic · isolated checks a written failure list Reviser fixes findings, nothing else 3 · CHECK machines verify everything provable Recompute & compare derived fields match the rules Measure the batch vs the shape of honest work Search for sealed terms planted decoys prove it works Confirm status unapproved facts can't ship every run writes a readable verdict page 4 · REVIEW a person decides: packets of ≤5 items, each with a recommendation PUBLISHED part of the record now the relevant slice only work batch passed → proposed approved, with a name what to search for · the shape to measure against flagged: sent back with the finding named rejected THE CORRECTION LOOP every human catch is written down as one durable rule the same mistake shouldn't cost review twice a reviewer catches something anyway rules load into every future brief caught twice → a new detector into the critic's list
Source material feeds a human-written brief; a drafter, an isolated critic, and a reviser produce the work; detectors verify everything provable; what passes enters review as small packets with recommendations; an approval carries a name. Anything a human still catches becomes a written rule, loaded into future briefs and the critic's checklist, and promoted into a new detector when it repeats.
the forward path sent back, with the finding the loop that compounds

What it checks

  • Did the numbers that were supposed to come from the source actually come from the source? Anything that should be computed from source data gets recomputed and compared against what shipped, so a hand-typed value can't pass itself off as the real thing.
  • Does this batch have the shape of honest work, or the too-clean shape of a shortcut? A batch is measured against known-good work, so a recycled template or a skipped step shows up even when every item looks fine on its own.
  • Is anything that must never ship in what's about to ship, and is the thing checking for it still working? Embargoed names, answer keys, and spoilers are searched for on every build, and a planted test secret proves the search still catches things.
  • Did a person actually approve this, by name, before it went out? New material enters as proposed and stays that way until someone signs off on it. Shipping anything unapproved is itself a caught failure.

Where it came from

For a while I'd been building a large fictional world with AI: four constructed languages, hundreds of canon entries, dozens of characters, all of it growing faster than one person could possibly read and check by hand. Generation was never the bottleneck. AI could produce material faster than I could keep up with it, and that was exactly the problem. The real limit wasn't how much could be generated; it was how much a single reviewer could actually verify.

The system: a fictional-world project with four constructed languages, where every modern word is derived from an authored ancient root by that language's sound-change laws. The corpus at the time: roughly 22,000 entries across the four languages, grown in AI-generated batches of a few hundred words at a time, all gated.

One growth pass, 500 words across two batches for one language, came back looking perfect. Phonotactics passed. No collisions. No duplicate glosses. Counts exact. Every individual word was, on its own, a well-formed entry.

Then someone looked at the derivations. In that language, roughly 96.5% of real words change between their ancient and modern forms; the sound laws almost always fire. In the new batch, almost nothing changed: the generator had quietly built ancient roots that dodged the laws: no triggering sounds, so ancient equaled modern, so no historical depth had to be invented. Less work. Every validity check green. The work was still dishonest.

  1. The batch was reverted: all 500 words removed.
  2. The shortcut became a gate. A distribution check was written: measure each language's natural change rate, set a floor safely below it (85% for this language), and turn the suite RED on any batch that comes in under the floor. The lazy batch had measured near zero.
  3. The words were re-authored the honest way, with ancient roots built so the laws fire by construction. The re-authored batches came back at 100% (800 of 800 words on the full allocation), and the language's whole corpus measured 98.3% after.
  4. The gate stayed. Every growth batch since has had to beat the floor, and the check was later hardened against the bypasses a lazy generator would find next (batch-sizing under the minimum, glosses missing from the comparison set).

That check, generalized, is kit/fingerprint.py. The one-sentence lesson is the whole product: the batch passed every check that looked at words one at a time, and failed the only check that asked whether the work had the shape of honest work.

Limits, stated plainly

stele catches lazy generators taking shortcuts. It does not catch a determined adversary deliberately tuning output to beat the checks. Seals are disclosure checks, not a confidentiality proof: GREEN means the enabled gates passed, not that prose is good or that a clever leak is impossible.

Repository: code, demos, and the full method →

Plain Python, no dependencies, 157 tests, CI on every push.