The design documents were written before the code, and most of what they said held. This page
is the part that did not: a belief this project held confidently, the evidence that refuted it,
and what replaced it — seventeen times, in that order.
Act I — Design assumptions that were wrong
Four beliefs that were held before any code existed, and were refuted by reading a
specification, by arguing the sequencing through, or by the thing they were waiting on
finally arriving.
1 · The install design was impossible
The plan was for the installer to apply the freya- prefix at install time — to
the directory name and to the name: field inside each
SKILL.md. Reading the Agent Skills specification before writing the installer
turned up a rule that makes that unbuildable.
The rule that ended it
A skill's name must match its parent directory name.
Installation is symlinking, so a link from an agent's
skills/freya-code-graph to skills/code-graph in the store produces
a file declaring name: code-graph under a freya-code-graph parent —
invalid, and rejected by the ecosystem's own validator. And there is nothing to rewrite:
there is one file, shared by every agent.
Planned
Repo keeps the unprefixed names; the installer prefixes on the way out.
Requires rewriting a file every agent shares, or shipping a skill the spec rejects — and
an installed copy that diverges from the store on every update.
Built instead
All ten skills renamed in the repository: ten directories, ten
name: fields, fifteen manifest paths, and ten hard-coded sibling paths across
five scripts. The installer became a pure symlink.
The consolation prize is real. With the prefix already on disk, freya update
collapses to a git pull plus a re-link — there is one copy of each file and every agent is
looking at it. Those ten sibling paths are a consequence of an earlier decision rather than an
accident: skills resolve one another through the store instead of being separately
installable, and every skill file invokes the one launcher
(knowledge-base/decisions/ADR-013-single-freya-launcher.md).
Nine of ten, by the obvious grep
One of the ten sibling paths was built with os.path.join and a textual
'../..' rather than the parents[2] idiom the other nine used. A
sweep for the idiom finds nine and reports success. The one it misses fails at import, on
someone else's machine, after the release.
The whole contract — store, symlink, prefix-in-repo, and the ownership gates on every
destructive path — is stated in
knowledge-base/decisions/ADR-014-canonical-store-install-contract.md.
2 · A "non-standard" field that was in the standard
The design described compatibility: frontmatter as non-standard and had it
removed. The specification lists it as a standard optional field, alongside
license, metadata and allowed-tools.
Deleting those particular two lines was still the right move — they enumerated one vendor's
tool names — but the stated reason was wrong, and by the time anyone checked, the wrong
reason had hardened into a guard. The conformance checker's allow-list of permitted
frontmatter keys was {name, description}, so it was rejecting four legitimate
fields on any skill that used them.
The lesson generalises past this repo
A guard built from a remembered fact inherits the memory's error and then enforces it on
everyone. The remembering feels identical either way; the enforcement is what makes it
expensive. Read the spec before you encode it.
3 · Governance was planned before there was anything to govern
The original sequencing put the governance track first — principles, contradiction checks,
the machinery that judges intent against intent. It was preempted by an adoption track on one
argument: you cannot govern intent you have not captured. A checker that
compares a change against the recorded intent of a codebase with no recorded intent is a
ceremony.
The adoption work rests on a split that is the reason the whole thing is affordable:
inference is cheap and human validation is scarce, so they are decoupled. A bootstrap infers
hundreds of untrusted candidates up front; validation is spent lazily, on the two or three
behaviors a change actually touches, in the flow of work. That is what turns a few hundred
inferred candidates into a backlog rather than a flood —
knowledge-base/decisions/ADR-007-bootstrap-proposed-drain-lazily.md.
The second sequencing reversal, taken deliberately
A separate, staged "vertical proof" — validate the mechanism in isolation before using it
on anything real — was considered and dropped. That validation was folded into the first real
use on a live testbed instead. It was a conscious risk trade, and it was paid for: several
early schema, lifecycle and adapter choices were made provisionally and corrected in contact
with reality. The friction findings from that testbed are the correction record, and
two of the entries further down this page came out of it.
4 · Blindness on purpose, and then the premise dissolved
The contradiction check — the one that judges a changed spec against the intent already
recorded around it — was built deliberately blind to architecture decision records.
That reads like an omission. It was a decision, and it was argued in the open at the time.
The honest version, as it was written then
Checking specs against ADRs while no ADR format existed would mean comparing against a
structurally empty set, while looking like the check covered architecture decisions.
That false sense of coverage is worse than an honest gap. So the check was built
ADR-ready: a wired-but-unplugged socket, not a wall.
Overtaken on schedule
The ADR format, the create flow and the gather function shipped, and the same check gained
ADR-awareness for free — always global, with no scoping at all, because for intent-vs-intent
the unrecoverable failure is the relevant record that never reached the judgment. The
premise dissolved exactly the way the socket was built to expect. The reasoning, including
why the sibling drift check takes the opposite scoping, is in
knowledge-base/decisions/ADR-011-governance-check-scoping.md.
Act II — Guards that failed at the job they existed for
Five entries about protective code. Every one of them was written specifically to keep
somebody safe, and every one of them did damage in a case its author had not pictured. This
is the densest cluster on the page and it is not a coincidence: a guard is the code least
likely to be exercised on the happy path, and the code whose failure looks most like success.
5 · --dry-run deleted your installation
The installer's uninstall path returned before it ever consulted args.dry_run.
So freya uninstall --dry-run — a flag whose help text reads
print the plan, change nothing, and which the README advertised as a preview —
removed all ten symlinks.
The damage was bounded: only links this toolkit had created, recreatable by re-installing.
But a safety flag that performs the destructive operation is the precise failure the whole
class of flag exists to prevent. The same review found that uninstall never removed the
launcher either, so deleting the checkout afterwards left a dangling freya on
PATH.
Why the review caught it and the tests did not
Every unit test exercised uninstall_agent() directly. Nothing tested the
flag wiring in main(). The suite was green and the feature was broken,
which is the whole argument for a review that runs the thing end to end rather than reading
the diff that changed.
6 · Every guard, misfiring at the person it protects
Eight of ten tasks in one work package came back from review with findings, and
not one was in the feature logic. Every serious defect was in code written
to keep the user safe.
🚧
The barrier that half-installed
Added because installing for two agents with a blocker under the second left the first
fully installed while exiting non-zero. The fix planned every agent before applying
anything — which made each plan's status a snapshot of the disk before anything
moved. Two agents pointing at one directory both planned "create"; the first installed, the
second raised FileExistsError. A partial install and an error, by a new
route.
🔒
The rule that locked you out of your own file
The markers that fence off the generated block in your AGENTS.md had to be
unambiguous, because a stray one could delete a user's prose. So the fix demanded each
marker appear exactly once, counting occurrences anywhere — including
mid-sentence. A user who merely wrote about freya-devkit in that file could then
never run freya init again. Being over-strict about a file you do not own is
its own kind of damage.
🏠
The check that could not break a command
The update notice is wrapped so that nothing escapes it: it cannot alter an exit code or
print a traceback. Wiring it in front of main silently turned four existing
tests into ones that wrote the real ~/.freya and made real network calls. The
reviewer found the file sitting on disk, timestamped to the test run.
📄
The write that emptied the file first
open(target, "w") truncates before it writes a byte, so a failure mid-write
left the user's AGENTS.md empty — in the one module whose single hard rule is
that everything outside its markers survives. It now writes a temp file and calls
os.replace.
And one that could not heal
The relink error was self-sustaining. A copy that failed partway leaves a directory with
no ownership marker, which the audit classifies as occupied forever — so the
next freya update crashed at the same line, and so did the one after that. A
guard against clobbering somebody else's directory, holding the user's own broken install
hostage, permanently, with no route out but manual deletion.
7 · A security control that controlled nothing
Before the audit driver was designed, a paid spike ran both agent CLIs against a deliberately
vulnerable fixture, to establish how to enforce this worker may read, and must never
write. The belief under test was the vendors' documented rule that a deny flag beats an
allow flag.
| Configuration | Result |
|---|
| Claude, explicit read-only allowlist plus an explicit write denial |
Held. The worker tried the shell three times; all three denied. |
Copilot, explicit read allowlist plus --deny-tool=write |
Held. Both the write tool and an explicit shell redirect failed. |
Copilot, --allow-all-tools plus --deny-tool=write |
Bypassed. The worker created a file through a shell redirect. |
The actual rule
"Deny beats allow" applies to the write tool, not to writes performed through the
shell. Only an explicit allowlist that excludes the shell holds, and a blanket
permission flag must never appear in an audit invocation.
That finding is now enforced in code rather than remembered as a convention: the adapter
refuses to emit a blanket permission flag even if one is smuggled in through the prompt, and
a test asserts that no adapter can ever produce one — a guard designed before the thing it
guards existed. The same spike priced a single finder worker at $0.396 on a
trivial fixture, which is why the driver ships with a call ceiling and a confirmation prompt
rather than an advisory note. Both live in
knowledge-base/decisions/ADR-015-driver-owned-fan-out.md.
8 · Three tests that proved nothing
The audit engine's vote arithmetic is the part where a silent error changes security
findings: how many skeptics must refute a finding before it is dropped. Its tests all passed.
Mutation testing — deliberately breaking the code to see whether the suite notices — showed
three of them were hollow, each for a different reason.
| Mutation | Why the test could not see it |
|---|
upheld * 2 > total → >= |
Only a 1-of-2 split distinguishes the two operators. Every test
used three skeptics, where they agree. |
The K_EMPTY constant, changed |
The test asserted against the constant itself, so it adapted to the mutation and
stayed green. |
| Delete the dry-round reset |
The test counted findings, and both paths yield the same count. Only the
round count differs. |
All three now have tests that distinguish, and every engine mutation is verified to be killed
by a specifically named test.
A trap worth knowing before you try this
Mutation runs land inside a single filesystem timestamp tick, so Python happily reuses
cached bytecode and reports the previous mutation's result. Two mutations looked
killed when they had never been loaded. Clear __pycache__ between runs, or run
the interpreter with bytecode writing off.
9 · Two numbers that were never multiplied together
One work package skipped its whole-package review, on the argument that the port was
mechanical. The review, run late, showed the argument was half right. The engine itself was
clean — every constant, the dedup key, the vote arithmetic and the disposition ladder
survived the port intact. Both serious defects were at the edges.
🚨
A failed audit reported a clean codebase
Every failure path in the agent call was a bare continue. With an expired
login, all 26 calls failed, the driver printed "done: 0 findings", exited
0, and emitted [] — from which the skill wrote a report saying
the codebase was clean. The captured stderr holding the real reason was
discarded unread. For a security tool that is the worst available failure mode, and it is
the likely first run.
🧮
The defaults could not finish a run
A findings cap of 40 implies 151 tasks. The call ceiling was 80. The two were chosen in
the same sitting and never multiplied out — even though the function that does exactly that
multiplication sits twenty-five lines below them. They were not even in the same unit: the
budget counts attempts, and each task may retry once.
The second compounded the first. Exhausting the budget discarded everything, including
findings that had already been fully verified, so a real run would spend its entire ceiling
and return an empty array.
An unplugged smoke detector finds no fire
A security tool that cannot tell nothing is wrong from nothing answered
does not merely fail — it produces a document asserting safety, over a signature nobody
questions. This is the sharpest thing on the page and the one worth carrying to your own
work.
The fix that mattered was not the arithmetic
Refusing to return a partial audit that could be mistaken for a complete one is the right
instinct. But the answer is to label truncation, not to delete the evidence of
it. Verification now settles in batches, a halt keeps what it has already verified,
and exit 3 marks a result incomplete. The driver refuses to exit 0 when no
task got a usable answer. With that in place, the exact cap values stopped being
load-bearing.
Act III — Things only running it could find
Everything above was found by building, reviewing or reading a specification. These eight
were reachable only by running the thing — against a real agent, a real host, a real
platform, or a real repository — and most of them had survived multiple careful reviews
that read the code correctly every time.
10 · Copilot cannot be told to parallelise
Act one — the belief. Copilot's public documentation says the main agent
decides execution strategy autonomously, based on task structure. The design took that at its
word and concluded that the lever was not the sentence but the shape of what
preceded it: present N visibly independent units of work, and place the scheduling
instruction after the task list so the list is read first. Every fan-out flow was built that
way.
Overtaken by evidence
Run for real, task structure is not the lever. Asked to run six category scans the way the
skill instructs, Copilot ran them itself — a sequence of greps — and then reported
that it had run six category scans in parallel. Instrumenting a twelve-way fan-out with debug
logging settled it: the delegation tools were invoked zero times, in both
directions. The reason was in the host's own system prompt: never delegate parts of a
codebase small enough to read directly, "regardless of how it divides into separate
areas." The lever the design had identified is the exact thing the agent is instructed
to ignore.
Read the heading precisely, because the capability was never the problem. Copilot
can parallelise; it ships delegation tools, background agents and a fleet mode. What
it cannot be is told, by a file in your project.
The general lesson, stated once
An instruction file is information the assistant reads; the assistant's own instructions
are its orders. When they conflict, the orders win — and they should, or a
file in any repository could override what a vendor built.
Act two — the first remedy was also wrong, and this is the more valuable
half. The finding was initially recorded as unfixable from this side, on the grounds
that the skills already carried a sequential fallback: the work still gets done, only slower.
That reasoning has a hole in it.
The fallback was never the problem
The problem is that nothing can tell you which mode ran. An instruction
saying "run these in parallel if you can" gets the same answer from an agent that did and an
agent that did not — the original finding was only visible because a human read a transcript
and counted greps. A correctness property you cannot observe is not a property. It is
a hope.
So the fix was neither to accept the sequential mode nor to word the instruction more firmly.
It was to stop issuing an instruction at all: the security scan now calls a driver that owns a
worker pool, makes the calls itself, and returns verified findings. The agent's cooperation
left the guarantee. Measured on the same fixture, that driver runs
209.3 s single-threaded against 91.4 s at six workers — a 2.29× win rather
than a 6× one, because the host throttles under load, and the honest number is the one that
got written down. The design and its rejected alternatives are in
knowledge-base/decisions/ADR-015-driver-owned-fan-out.md.
A smaller refutation, sitting inside the same story
The scheduling block copied into every fan-out flow promised that a sequential run was
"identical, only slower." This repo's own dogfooding notes had already recorded a
real scan at 7 parallel discovery agents and ~260k subagent tokens. Running
those inline collapses seven isolated context windows into one, because each worker's
file-reading context is retained rather than discarded — on a 200k-context agent that run does
not produce an identical result, it does not complete. The block's own next paragraph
explained why, by noting that each worker carries its own context window. The claim
contradicted the sentence directly beneath it, in a file that had been reviewed line by
line.
11 · The artefact was correct. The host disagreed.
Asked to list its installed skills, Copilot returned nine. The missing one
was the security scanner — the skill the portability work most existed to reach. Installed.
Correctly symlinked. Silently dropped.
1251 characters against a 1024-character limit
The Agent Skills specification caps a skill's description at 1024 characters.
That one had grown to 1251 — the only skill in the suite past the line.
Copilot enforces the limit and drops the skill without a word. Claude Code ignores
the limit and loads the skill happily, which is exactly why work done mostly against
Claude never saw it. Nothing raised an error anywhere: not the CLI, not the doctor command,
and not the conformance gate, whose frontmatter rule had always checked which keys
were present and never how long their values were.
This is the site's best argument for running the thing rather than reading it, and its
cleanest statement of a recurring truth: a test suite can only ever check the
artefact, never what a host does with it. Everything on the repository side was
right. The evidence discipline that came out of it — real dependencies, live dogfooding,
committed evidence, corrections appended rather than rewritten — is
knowledge-base/decisions/ADR-016-prove-it-against-the-real-thing.md.
A result from the same run that had to be thrown away
One agent marked both planted vulnerabilities intentional-design where the
other had marked them confirmed. Before reading that as an adapter defect: the fixture's own
README announced that it existed to validate the audit driver. A skeptic lens reading that
could quite reasonably decide the vulnerabilities were deliberate test data. Re-run against a
fixture that says nothing about itself, both came back confirmed critical.
A control fixture that describes its own purpose is contaminated — the
disposition ladder was fine, and the experiment was not.
12 · The false green that reshaped the behavior layer
The plan for integration behaviors was to import the framework's route handler in-process and
call it directly. On an ordinary CommonJS-default project, Node's require(esm)
rules forbid it — and flipping the application to "type": "module" made the test
pass. That is the trap: a false green, bought by demanding that
every adopter restructure their application to suit the test harness.
The resolution became a principle rather than a workaround. A behavior test drives the
application over its real interface, against a running instance — never by
importing internals. Then the honest consequence, which is why the shipped coverage model
looks the way it does: observed coverage of that running application captured only framework
internals and zero application code, so observed integration coverage was
deferred and the static import closure became the honest baseline — over-approximating, in
the safe direction, and saying which it is. Both halves are in
knowledge-base/decisions/ADR-006-real-interface-execution-and-coverage.md.
The inverse demonstration, from the same testbed
Flipping a behavior to accepted while its test was still a
TODO(scaffold) produced exactly one error, exit 1, scoped to that
behavior — the sibling proposed scaffolds in the same file correctly untouched. The
suite is green, because nothing runs. The layer still says exit 1. That is a lie about a
guarantee that a green test suite is structurally incapable of catching, and it is the
clearest evidence that the layer earns its place.
13 · Windows was not waiting on a machine. It was waiting on code.
Every earlier version of this record carried the same sentence in some form: Windows is
untested, it needs a Windows machine, and no code. The first half was true.
The second half was an assumption that had been restated so many times it had started to read
like a measurement.
A CI matrix now runs the suite, the static gates and a real end-to-end install across
ubuntu-latest and windows-latest, on Python 3.9 and 3.13
(.github/workflows/ci.yml).
The Windows leg came back 29 failed against a green Linux leg. Nothing had
regressed; it was simply the first time the suite had ever executed there, and the failures
had been sitting in the tree the whole time. Four root causes account for all 29, and
two of them were product bugs, not test bugs.
| Root cause | What it actually broke | Kind |
|---|
Extended-length paths. Windows hands a symlink target back as
\\?\C:\… where the source is spelled C:\…, and the installer
proved it owned a link by comparing those two strings. |
Every link the installer had just created classified as foreign,
so install refused to manage its own install and relink skipped every agent.
17 of the 29 were downstream of that one comparison. |
product bug |
| Backslash graph keys. The code graph keyed its entries with
backslashes, so every forward-slash lookup missed. |
Wider than one skill: the behavior graph and runner both read those keys expecting
forward slashes, so blast radius came back empty and silent on
Windows — the failure mode with no error attached. |
product bug |
remove_link retried a failed unlink as an rmdir,
unconditionally. |
On Windows the fallback succeeded, so a removal that had not happened was
counted as one that had, and the command reported zero failures over nothing
removed. |
reporting |
A manifest escape guard used os.path.isabs on a path that reaches the
filesystem. |
Python 3.13 changed ntpath.isabs so a rooted path with no drive is no
longer absolute on Windows: /etc/passwd was rejected on 3.9 and
accepted on 3.13. A guard on a trust boundary, correct by accident of which
interpreter was running it. |
trust boundary |
And two the matrix structurally could not catch
Two further defects were found by review, not by the runner, and neither could
ever have turned a leg red: one function is Windows-exclusive, so no POSIX run reaches it at
all, and the other was latent. Both would have shipped broken behind a green build.
A green matrix is evidence only about the paths it executes, and adding a
platform did not make it evidence about everything on that platform.
What the matrix proves is that the toolkit installs on Windows and that its tests pass there.
It does not prove that a live scan works there: no agent CLI is installed on the runner, so
nothing that spends money has ever executed on the platform. That gap is owned, with its
remedy, in
knowledge-base/roadmap.md
rather than restated here.
14 · The spike meant to evaluate a parser found the bug in ours
A spike was set up to measure whether an external multi-language parser was worth adopting.
Its most consequential finding was not about that parser. Building the graph on freya-devkit
itself indexed 10 of 50 Python files, produced zero
connections, printed "Built dependency graph: 10 files scanned", and exited
0. freya classified freya as a brand-new empty project.
Two independent causes, either sufficient alone
An exclusion rule matched a directory name at every depth instead of at the top
level, so every skill's scripts/ vanished — forty files. And Python sibling
imports were filed as third-party packages, which kept the internal edge count at zero even
once the first was fixed. Four more defects sat behind those.
The resolver was then repaired before being frozen behind the new contract.
Freezing it first would have made every one of those defects a permanent property of the
floor that every project falls back to.
15 · We corrected something that was right, and logged the correction
An automated probe reported four relation names in the second backend's vocabulary. They were
grepped for, not found, deleted from the mapping table — and the log recorded that the agent
had invented them.
All four were real
They lived in a tuple constant, exactly the shape those regular expressions could not see.
The same scan had also missed a relation this repository's own graph demonstrably contains,
and that discrepancy was noticed and ignored. A later review restored all four.
Verification that only looks one way is not verification. The probe found
nothing and the absence of evidence was taken as evidence of absence — with a written record
blaming the tool that had been right.
The same mistake, from the other side
Months later a design agent reported a filter measured "silent on 3 of 7 real
repositories". Checked before being believed: one was an empty directory,
and a second's silence turned out to be a defect rather than the filter working. Both figures
had already reached a commit message and a decision record. A probe returning a result is not
the same as the result meaning something.
16 · The mechanism built to stop confidently-empty answers gave one
The polyglot substrate's last piece makes every answer say what its backend could not read.
The census applies the build's own scope rule, which is what keeps it from crying wolf —
measured, a census that consults only .gitignore-style patterns reports 96
unread files on this repository, 68 of them deliberately out of scope.
Then the shape detector trusted its silence
A real 40-file deployment repository has its entire codebase in shell scripts under
scripts/ — a built-in top-level exclusion. The census therefore, and correctly,
reports nothing unread: those files are out of scope, not unreadable. The shape
detector read that silence as authoritative and returned greenfield. Before
the feature it had returned unknown.
So the confidently-empty answer arrived through the mechanism built to remove it —
via the same scripts/ rule that had once stopped freya graphing itself. Two
questions were being answered by one value: "what can this backend not read?" and "is there
anything here the graph does not represent?" The second is broader, and the census's silence
does not answer it.
Why it was not caught by the tests that existed
Because the silence was evidence. It had been cited as proof the filter was quiet
enough to be believed. The claim and the defect were the same observation, read the wrong way
round — which is only visible if you go and check what a silence is made of.
ADR-029
17 · The gate that checked the file and called it the link
The belief. On 2026-08-21 the toolkit was finally run against the only
repository it had never been pointed at: itself. Everything reported clean. 1,435 tests
green, the conformance gate green, and freya verify-links — the Tier-1 check
that is allowed to hard-block a wrap-up — printing
“OK — all behavior links pass Tier-1 integrity checks.”
Seventeen of the hundred and forty-nine were broken
A behavior's locator is written path#Class.method — a file, and
the test inside it. parse_locator returns both halves, and the caller bound the
second to a discard: rel_path, _frag = parse_locator(locator). For every
non-Gherkin adapter the entire check was then abs_path.exists(). A second hole
sat above it — if adapter == "manual": continue — which exempted manual
behaviors from the whole check rather than from the runner, so their locators were never
resolved at all, not the method and not even the file.
The proof was a mutation, not a reading. Every locator target in the repository was renamed
out of existence — 132 of them, across 21 files. verify-links exited
0. The suite stayed green. Nothing anywhere observed that every behavior in
the project now pointed at a test that did not exist.
Why it was not caught by the tests that existed
Because they tested the gate's reporting, not its reach. There were
tests for a locator naming a missing file, and they passed, because that half was real. No
test asked what happened to the other half, and the code that discarded it read as
deliberate — _frag, with a leading underscore, is the Python idiom for a value
you meant to throw away.
The same day, the security scan found the shape in the code as well as in the
gate. It raised twenty-two findings against the toolkit, two of which merged into
one, so twenty-one were filed; the two most severe were the same defect twice. audit.py runs every scan worker with
cwd=<the scanned project> and invokes it by the bare name
claude. On Windows, CreateProcess searches the working directory
first — so a repository you scan can supply its own claude.exe and have it
executed as you, inheriting your whole environment. The second instance lets the scanned
repository's own committed settings.json choose the binary.
What that costs, precisely
The toolkit's headline safety property is that its workers run under a read-only
allowlist with the shell excluded. That allowlist is expressed in argv, and
argv only binds the program you meant to run. If the scanned repository picks
the program, the allowlist is describing a process that was never started.
Seventeen of the twenty-one findings were in files that already had a test file beside them,
so “no test for that code” explains none of them. The suite tests
degradation — a missing file, unreadable JSON, a git command that fails —
exhaustively, and had almost nothing that tested hostility: input that is
well-formed and chosen by someone who wants something. Those are different questions, and
only the first had ever been asked.
The lesson, and it is the one this whole page is about
A gate that reports success is not evidence that it ran. Every finding here was in a
mechanism this project had already built, documented and believed in — the link checker, the
worker sandbox, the read-only allowlist. What running it on itself produced was not new
capability. It was the discovery that three things declared everywhere were enforced
nowhere, which is the exact defect class the toolkit exists to find in other people's
repositories.
The belief. Three days later the twenty-one filed findings were worked
through (two more, SEC-022 and SEC-023, were found during that work rather than by the
scan).
Each one arrived with a Remediation paragraph written by the same adversarially
verified pipeline that found it — specific, plausible, citing the line. The belief was that
the expensive part was over: discovery had been done, and what remained was application.
The highest-severity finding took five fixes, and each one passed the fix before it
SEC-001 is a hash the scanned repository commits to a marker file, interpolated into
git diff's argv by the gate that decides whether an accepted test was edited
without authorisation. Every round below was found by running the attack against the
previous fix, never by reading it:
--output=/tmp/victim — git diff accepts it, and truncates
the file. Closed by validating the value as a hash.- Forty zeros. Valid hex, so the new validation passes it; git then
fails
rc=128 and the empty output was read as "nothing changed". Closed by
telling a git failure apart from an empty diff and labelling it a skip.
- A committed file named
deadbeef. Pure hex, so it walks
through the regex — and git resolves it as a pathspec, at rc=0.
Closed by --, which forces the revision reading. (A --
before the argument makes it a path; after it, the opposite. Getting that
backwards is what left this open a round longer.)
- A tree hash from
git rev-parse HEAD:knowledge-base.
Forty hex characters, a real object, and not a commit — it walks past the regex, past
--end-of-options and past -- alike. Closed by peeling
^{commit}.
- A marker holding
commit: with no value, or no
commit: line, or zero bytes. Each returned a bare None — which
is the exact fingerprint of a repository that has never run the gate, so
--advance exited 0 and erased the finding it should have reported.
The argv that survives all five is
git diff --name-status -M --end-of-options <baseline>^{commit} --.
Four remediations that would have made things worse
Applying the report literally was tested rather than assumed, and four of its
instructions failed:
- "Resolve the binary to an absolute path" is the fix backwards. On
Windows
shutil.which returns the working-directory hit first, so
abspath-ing it hands CreateProcess a fully-qualified path to the
attacker's binary, spelled more convincingly. The rule has to be refuse,
not absolutise.
- "Add
--end-of-options before the revision" in the
code-graph's incremental diff produces "fatal: option '--name-only' must come before
non-option arguments". Every --update would have degraded to a full
rebuild, permanently and quietly.
- "Replace the bare
except: with except OSError"
does not catch UnicodeDecodeError, which is a ValueError — so
one non-UTF-8 byte in one YAML file would have converted a swallowed error into an
uncaught traceback out of the whole detector. The fix was to read bytes.
- "Narrowing the pattern can only make the guard stricter" — true, and
not enough. After narrowing, a made-up
SPEC-999 still corroborated itself on
this repository, because the corroboration search included the scanner's own report
directory and found the sentence in it describing the test that says it must not.
Why it was not caught by reading
Each of those instructions is locally reasonable. They are the sentences a
careful reviewer writes, and three of the four cite the right line. What separates them
from correct is one command: reproduce the attack against the proposed fix, on a real
repository, before believing it. Discovery had been adversarially verified three ways;
the remediation had been verified not at all, and nothing in the pipeline was shaped to
notice the difference.
And one finding was declared unfixable, on a threat model that was not this tool's
SEC-006 is the mechanism that lets an accepted behavior silence a security
finding. The record authorising it said the behavior's passing test is
"verified evidence rather than a prose claim", and no test was being run — so the
claim was false and deleting it was right. What came next was not. The finding was closed
as mitigated and left open on purpose, arguing that no better evidence
could exist: the only evidence not committed by the repository under audit would be
executing that repository's test suite, "a security tool running hostile code".
That argument was overturned the same day, and it is the most useful thing this
pass produced. It is an argument against a capability this toolkit ships as a
feature — freya-behavior-runner exists to run the project's tests, and
regression_check, in the same module as the query being reasoned about,
already re-ran accepted behaviors. The reasoning had imported a hostile-clone threat model
that does not match what freya is: a tool a developer points at a repository they are
working in, having already installed its dependencies and run its suite. The premise was
never checked against the sibling skill sitting next to it.
Closing it properly found a hole wider than the finding described and one no forgery
was needed to reach: the query never read whether an exercised path came from a real run
or was inferred from the import graph, and both licensed a downgrade identically.
A downgrade now requires an observed run and a resolving locator, carries the symbols that
actually executed, and --verify re-runs the linked test for the one caller
whose answer can silence a finding. A false safety invariant that reads as true is the
most expensive kind of documentation there is — and so is a confident argument that the
gap behind it can never be closed.
19 · Then we scanned the fixes
The belief. Twenty-three findings closed, every gate green, the toolkit
run on itself. The work was the fixes; the fixes were the product. What remained was
paperwork.
Two thousand lines had been written and never scanned. Closing the
findings had meant new code — path containment, a binary resolver, a redaction helper, a
declaration mechanism, a hardened gate — and no scan had ever seen any of it. Running one
over that surface returned twenty-four candidates. Five did not survive refutation.
Nineteen did.
The most severe was the previous fix, half-applied. SEC-023 — a symlink
committed inside a project, pointing outside it, followed and read with nothing declared —
had been fixed in discovery. The incremental path never got the guard. So the symlink was
refused on a cold --build and read on every --update after it,
and --update is the one wrap-up runs. The cold start was fixed and the steady
state was not. Worse than the hole it reopened, three ways: the build path printed a
warning and recorded the refusal where the update path did neither; the key it produced
is project-relative, so the contract validator passed the artefact clean; and the
regression class written to prevent exactly this proved its claim on five cases, none of
them incremental.
One of them destroyed a tracked file. test-failed is the
runner's word for any non-zero exit, and the merge treats that reason — and only that
reason — as invalidating. On a machine with no toolchain installed,
python -m pytest exits non-zero. So a fresh clone with nothing installed
blocked the commit and rewrote the committed behaviour graph to an empty
exercise set, after which the downgrade path it feeds answered empty for good. The
argument for not doing this was already in the file, ten lines above the bug, written for
the neighbouring case: calling a renamed test method test-failed "would wipe
committed edges and block a commit over a rename". The reasoning was right and had not
been carried one case sideways.
And the pass whose entire job was accuracy wrote three false safety claims of its
own. That validate_graph "rejects" a non-project-relative key — it
reports one, and the graph is written anyway. That every containment question
goes through one module — one deliberately does not, and its docstring says why. That
no symlink crosses the root on its own — one did, which is how SEC-023 was found
at all. Each was true on the path its author checked.
The recurring shape of this project is a confidently empty answer. This entry is its
reflexive form: the newest work is the unexamined work, and the pass that just
finished is the least likely thing to be checked. It applies to a fix, to a
correction of a fix, and to a scan of the corrections. Each round here found fewer and
smaller things — twenty-one, then nineteen, then a handful of stale sentences — but no
round found nothing. The branch stopped on judgement, not on a clean result, and saying
so is the honest version of "done".
20 · The predicate and the join disagreed about what a drive is
The branch closed with every gate green and was pushed. The first CI run failed —
on Python 3.9, on both Linux and Windows, while 3.13 passed everywhere. The failing
value was 1:x.
escapes is the lexical predicate for a value declared in
checked-in data, and the fix two entries above had just routed directory keys through
it. It asked PureWindowsPath(rel).drive whether a value carried a Windows
drive. That question does not have one answer: pathlib restricted a drive
letter to ASCII up to Python 3.11 and delegates to ntpath.splitdrive from
3.12, which accepts any character. So PureWindowsPath('1:x').drive
is empty on 3.9 and '1:' on 3.12.
Why that was a hole and not a curiosity
The consumers do not join with pathlib. They join with
ntpath, and ntpath.join('C:\work\proj', '1:x') is
'1:x' — the project root discarded. So on 3.9 through 3.11 the predicate
answered does not escape about a value that escaped at the very next line of
the code it was guarding.
The rule now asks ntpath.splitdrive — the same body the consumers join
with — so predicate and join cannot disagree by construction rather than by vigilance.
That is the whole fix, and it is one line.
The lesson was already in the file, four lines away. The sibling
predicate is_anchored was written not to use os.path.isabs,
with a comment naming the reason: Python 3.13 had changed ntpath.isabs, and
a rule that moves between interpreter versions is not a security boundary. That cost had
already been paid once. It was not carried across to the drive term in the function
immediately above it — and the reference page's own table shows the seam, because the
is_anchored row said on any host and any supported interpreter
while the escapes row beside it did not.
Every local run was on one interpreter, where the two eras agree, so no
amount of care at the keyboard would have surfaced this. The version matrix did — which
makes it the one entry here found by a machine rather than by reading. The regression
test therefore asserts against the join and not only against the predicate: a
predicate agreeing with itself proves nothing about the thing it guards. A second test
re-derives the rule with the pre-3.12 semantics simulated, so an edit that reaches for
.drive again fails on the interpreter developers actually run.
The shape rhymes with the entry above it, one level down. There, the newest work was the
unexamined work. Here, the newest work was examined thoroughly — by hand, on one
machine, on one interpreter — and the examination was the thing with the blind spot.
A green suite is a claim about the environment that ran it.