Extending it

Writing a skill, and the gate it has to pass

Everything here is a rule that exists because something real broke. The rule lists live in the repo, where an agent reads them and where they get updated; this page carries the arguments behind them — which is the part that does not fit in a checker.

Anatomy of a skill

A skill is a directory under skills/ containing a SKILL.md and, optionally, scripts/ for its deterministic engine, references/ for material the agent loads on demand, and evals/. Three of the ten skills are SKILL.md alone. Nothing else is required, and nothing else is read.

One frontmatter constraint shapes the whole repository. The Agent Skills specification requires name to equal the parent directory name — so the freya- prefix has to live in the checkout rather than be applied at install time. Installing is symlinking a directory straight into the agent's skills folder, and every agent shares the same file on disk; a single SKILL.md saying name: code-graph inside a directory called freya-code-graph cannot be rewritten per-install, because there is only one of it. That is why the directories were renamed instead of the installer doing the work.

Then two things that went wrong here, both of them the same shape: a guard built out of a remembered fact instead of a read one.

The "non-standard" field that was in the standard

The design described compatibility: frontmatter as non-standard and had it removed. The specification lists it as a standard optional field, alongside license, metadata and allowed-tools. Worse, the error had already propagated into the conformance checker, whose allow-list was {name, description} — so it was rejecting four legitimate fields. Removing those two particular lines was still right, because they enumerated Claude tool names. The stated reason was wrong, and the rule built on it was wrong. A guard built from a remembered fact inherits the memory's error and then enforces it on everyone.

Nine skills, not ten

Asked to list its installed skills, GitHub Copilot returned nine. The missing one was freya-codebase-security-scan — the security scanner, absent on the very agent the port existed to reach. Installed, correctly symlinked, silently dropped. Its description had grown to 1251 characters against the spec's 1024 limit. Copilot enforces the limit and says nothing; Claude Code ignores it and loads the skill happily, which is precisely why a long run of work done mostly against Claude never saw it. Nothing raised an error — not the CLI, not freya doctor, not the gate, whose rule R5 had only ever checked which frontmatter keys were present and never how long their values were.

What was added, and from where

R10 now measures every length the specification states — description 1024, compatibility 500, name 64 — and R11–R13 close the same class from the other side: a required description that is present and non-empty, a name inside the spec's grammar, and Claude-only locations no rule had ever looked at. All four came out of reading the specification rather than recalling it.

In the markdown

The full rule text a contributor has to satisfy — every convention, what the gate actually enforces versus what it merely hopes for, and the release mechanics for both consumer paths — is in the repo, and is the copy that gets updated.

Commands, not paths

No SKILL.md names a filesystem path. Scripts are invoked as freya <command> and nothing else. The line this replaced carried two independent failures at once:

  every SKILL.md, before
# a Claude-only variable, and an interpreter that may not exist
python "${CLAUDE_PLUGIN_ROOT}/skills/spec-manager/scripts/drift.py" gaps --project .

# and a host-specific way to name a sibling skill
/freya-devkit:code-graph

${CLAUDE_PLUGIN_ROOT} is set by Claude Code and by nothing else, so anywhere else it expands to nothing and the path is garbage. And python is not python3 on a great many machines — on some it is Python 2, on others it does not exist at all.

  every SKILL.md, after
freya drift gaps --project .

# a sibling skill is named, not pathed
freya-code-graph

One launcher owns both "where am I" and "which Python". Note the one-character distinction: freya code-graph (space) is a CLI command, freya-code-graph (hyphen) is a skill name. They are never interchangeable.

Behind that are three deliberately boring pieces. bin/freya is an executable shim, almost empty on purpose — it exists to put the real module on the import path. bin/freya_cli.py holds all the logic, so it is importable and unit-testable without spawning anything. bin/commands.json is a checked-in manifest mapping a friendly name to a script path. Those registered commands, plus six built-ins that main dispatches before it ever consults the manifest — help, doctor, init, install, uninstall and update — are the entire public surface.

Three mechanisms in there are worth stating, and none of them is worth transcribing.

One word that decides every install

The shim resolves the suite from its own __file__ via os.path.realpath, not os.path.abspath. That looks like a stylistic nit and is not. Every installation is a symlink, and abspath resolves to the symlink's own directory while realpath follows it to the real one. Ordinarily CPython papers over the difference by inserting a resolved sys.path[0]; under -P, PYTHONSAFEPATH or isolated mode it does not — and then import freya_cli fails from a directory that visibly contains freya_cli.py. The line was written before symlink installs existed, so the bug it avoids could not have surfaced for a long time after the mistake was made.

A latent breakage, fixed for free

Dispatch runs the target with sys.executable — the interpreter already running, which is by definition present and correct. Every one of the 80 original invocations called bare python, so routing them all through one launcher fixed a breakage nobody had reported yet.

The manifest is enforced in both directions

One test fails if a manifest entry points at a script that does not exist. A second fails if a script with a __main__ block is not registered. So a new CLI cannot be added and silently left unreachable, and a deleted script cannot leave a dangling command behind. If you add a script, add its manifest entry — the suite will tell you if you forget.

In the markdown

The command list itself is the manifest — read it there rather than from a page that goes stale the next time a command is added. The architecture file explains the resolution order, and the ADR records what this beat.

The gate: fourteen rules, no exemptions

bin/check_skill_conformance.py runs over skills/**/*.md and skills/**/*.py and exits non-zero on a single host-specific construct. Fourteen rules, each one added red, against a real violation — none speculative, none decorative. It runs on every push, separately from the test suite, because a shipped SKILL.md can violate most of the fourteen with the whole suite green.

One of the fourteen is not about portability. R14 arrived in the post-0.3.0 security pass and rides along on the same machinery for the same reason — it is a whole-tree property of prose that no unit test can express. It requires a skill that sends a worker at secret-bearing material to state the redaction rule, and to restate it inside the template slot the writer actually fills by copying source out of the scanned project. Stating it three sections away is what the rule exists to catch: that was the shape of SEC-009, where the report template's evidence block took a bare {code snippet} and, for a Secrets finding, the vulnerable code is the credential.

Zero exemptions, on purpose

R1 and R4 once carried exemptions for a Claude-only audit engine. Deleting that engine removed both exemptions in the same commit, and the checker has had none since — so one leftover host-specific reference anywhere fails the build. That is what makes "the port is finished" a claim you can check in one command rather than take on trust.

Red first, then green

The strongest thing on this page is the order the work happened in, and it appears in no markdown file. The checker was written before a single rewrite, and run against the tree on day one it failed loudly. Every rewrite task then existed to move it toward zero.

Why the order matters more than it sounds

A find-and-replace pass has no natural end. You stop when you stop seeing hits, which is exactly when the remaining hits are the ones your search missed. A checker inverts that: the job is done when a program says so, and every miss it catches becomes a permanent rule rather than a one-off correction. The rule set grew from seven to thirteen while the work ran (the fourteenth came later, from a different pass), and the checker was widened mid-flight to scan skills/**/*.py as well as markdown, after a Python file was caught writing a Claude-only command into its generated output — a regression class markdown-only scanning could never see.

What the gate cannot see

A gate that names its own blind spots is the point of having one. Three are known and written down rather than hidden:

  • R9 is file-scoped. If a file already carries the portability clause and you add a second, unrelated fan-out lower down in the same file, the rule will not catch it. That trade-off is recorded in CONTRIBUTING.md with the instruction to check by hand.
  • R4 was too narrow, and falsified a completeness proof. Retiring the audit engine claimed that R1 and R4 having no exemptions left meant a single leftover reference would fail the gate. R4's pattern required the literal word tool — so three "Workflow-powered" references shipped straight past it, two of them in the skill layer, describing an engine that had been deleted. The proof was sound in structure; the detector was too narrow. The definition-of-done grep made the identical mistake, which is how it survived twice.
  • The space-versus-hyphen hazard is structurally invisible to it. See the gotchas — that one is caught by review or not at all.
In the markdown

The rules themselves are not reproduced here on purpose: the checker is its own source of truth, it will grow, and a copy of R1–R14 on a web page is a copy that will be wrong. Read the RULES dict at the top of the file — it is written to be read.

Fan-out in a skill you write

If your skill presents N independent units of work, it has to schedule them portably. Two shapes are in the tree, and which one you need depends on who owns the loop.

Prose fan-out

Separate the N units of work from the scheduling, and place the canonical portability clause after the task list, so the structure is read first. The clause has to carry all three parts: run them in parallel if your agent supports subagents, a sequential fallback for when it does not, and the token-cost note — a sequential run accumulates every task's reading context into one window and may not fit. R9 fails the build wherever fan-out language appears without it. Copy the reference block rather than paraphrasing it; the gate checks for a sentinel phrase and a fallback, which is a floor, not the convention.

Driver-owned fan-out

Our own code owns the loop and calls whichever agent CLI is installed as a headless worker, through one injected ask callable. The engine never imports an agent, never spawns a process, and never knows its host. In production ask shells out; in tests it is three lines returning canned JSON, which is why the entire driver suite runs offline and free. The per-agent surface is one small module — audit_adapter.py, 224 lines — holding argv, envelope parsing, cost telemetry and CLI detection. Adding a third host means an argv builder and a stdout parser. (It was 116 until the post-0.3.0 security pass; the growth is the binary resolver and its reasoning, not a second host.)

The rule for choosing, stated once

Use the driver form when the workers only need to read. Keep the prose form when the workers write files — the driver's whole guarantee rests on workers that cannot write, so a writing worker cannot be put behind it. That is why freya-docs-manager keeps the prose block even though it fans out to twelve workers against the security scan's six. Scale was never the criterion.

If you build a driver, the allowlist is counter-intuitive

GitHub's "deny beats allow" applies to the write tool, not to writes performed through the shell: --allow-all-tools --deny-tool=write let a worker create a file with a shell redirect. Only an explicit allowlist that excludes the shell held. So every argv here is an allowlist rather than a blanket grant, and build_argv refuses to emit a blanket permission flag even if one is smuggled in through the prompt — both adapters return through that guard, and the one place a subprocess is spawned sits behind it, so there is no bypass path.

In the markdown

The canonical clause, which file holds the reference copy, and exactly what R9 does and does not enforce are all in the contributing guide; the reasoning for inverting the dependency is in its ADR.

Testing, and what a green suite does not prove

Four disciplines. The counts are deliberately not on this page — they are attached to every CI run, where they are current.

Nothing paid, nothing networked

The audit engine takes an injected ask; the updater takes an injected run. Not one test invokes a real agent or reaches a remote, so the whole suite is free and works offline. That constraint had to be re-enforced after review found the suite had escaped onto the developer's real home directory — a safety wrapper had been wired in front of main, quietly turning four existing tests into ones that wrote the real ~/.freya and called the network. Real dependencies are still preferred where one can be produced honestly: the updater's tests drive real git in temporary directories, an origin repo plus a clone.

The CI matrix, and its uncovered diagonal

Two jobs — the suite plus three static gates, and a real end-to-end install — across ubuntu-latest and windows-latest, on Python 3.9 and 3.13. Install is a separate job on purpose, so a failing test cannot hide the install's answer behind an early exit.

One of the three gates reads something the suite cannot; the other two overlap it on purpose. check_skill_conformance.py scans the shipped skill layer for host-specific constructs, and check_invariants.py reads the AST for two whole-tree properties: that every import is stdlib or a sibling, and that no subprocess call takes a bare-name argv[0] unless that site is on an explicit allowlist. Both of those also run from inside pytest — each has a ShippedTreeTest driving the same scan over the same tree — so they are kept for the report they print, naming the file, line and rule, not because the suite would miss the violation. check_doc_citations.py is the one that genuinely stands alone: its repo-level test pins only the rule that no document cites an untracked path, leaving the other two rules to the script. It resolves every path:line citation in the prose — a four-figure count the gate prints on each run, and one that moves with every commit, which is why the number belongs in that output and not on this page — because a citation that rots is a document quietly lying about the code. It catches three things and only three: a cited file that is not in the tracked tree, a line past the end of that file, and a citation landing on a blank line. A citation that has drifted onto the wrong non-blank line stays green, so a passing run is not evidence that a number is right — which is why the post-0.3.0 pass re-derived every moved citation by matching the cited text rather than by trusting the gate.

Why those two are AST checks over the whole tree

Both are properties of the whole tree, and a per-file test cannot see them — which is why the tests that do cover them are whole-tree scans calling the same code. The stdlib rule fails invisibly on the machine that breaks it — you write import yaml, it works for you because you happen to have it, and it breaks for everyone else. The argv rule is the shape of two high-severity findings this toolkit's own security scan raised against it: a worker invoked as bare claude with the scanned repository as its working directory is, on Windows, an invitation for that repository to supply its own claude.exe. Both of those two findings are fixed — the agent CLIs and graphify are resolved to an absolute path or refused — which is a narrower statement than "every external program", and the callout below says why.

What the argv rule does not say

It is not "no bare names remain". Eight sites still spawn a bare git, across seven files, and they are carried in an explicit allowlist that the gate reads. That list is a debt marker, not an approval: the gate goes red the moment a ninth appears or one of them names anything other than git, and a test pins the census at exactly eight. There is a second, subtler limit — a site that gets fixed stops being visible to the rule at all, because subprocess.run([exec_path.resolve(...).path, …]) is a call expression the AST check cannot evaluate. Fixed sites therefore leave the census rather than passing it, which is why a fixed site's allowlist entry is deleted in the same commit instead of being left behind as cover. The same blindness cuts the other way, and it is worth knowing before you read the census as a total: an argv built into a variable first is invisible too, so run_behaviors.py:228's ["pnpm", "vitest", …], spawned at :459 with the scanned project as its working directory, is a bare name the gate never counted — neither resolved nor allowlisted. Running a project's own test command means executing that project's code regardless, so it is left as it is; the point is that "eight" is eight the rule can see.

Read the green tick precisely

The install job runs symlink mode on Linux and --copy on Windows, because that mirrors how each platform is actually used — which leaves the opposite diagonal, Linux --copy and Windows with Developer Mode, unexercised on both. Touch either and test it by hand. And no agent CLI is installed on the runner: CI proves the toolkit installs and its own tests pass on Windows, never that a live scan runs there.

Mutation-tested guards

Safety-critical guards — the audit engine's vote arithmetic, the read-only allowlist, the installer's occupied classification — are broken on purpose to confirm a named test notices. The vote arithmetic's tests all passed before that; three of them turned out to be hollow, and each names a distinct way a test can be green without proving anything.

MutationWhy the test missed it
upheld * 2 > total → >= Only a 1-of-2 split distinguishes the two operators. Every test used three skeptics, where they agree.
K_EMPTY = 2 → 3 The test asserted against audit_engine.K_EMPTY itself, so it adapted to the mutation and stayed green.
delete the dry = 0 reset The test counted findings, and both paths yield one. Only the round count differs.
A trap worth knowing before you try this

Mutation runs that land inside a single filesystem timestamp tick make Python reuse cached bytecode and report the previous mutation's result. Two mutations looked killed when they had never been loaded. Clear __pycache__ between runs, or use python3 -B.

Review the plan, before the code

Three plan-mandated choices were sent to an adversarial reviewer before implementation, and two of them were wrong. The most useful one was not a style disagreement: the plan left a failed-fetch branch uncovered and forbade mocking it, on the grounds that an offline remote cannot be simulated honestly. But delete that guard and the flow reaches merge-base against the stale local ref, which succeeds — so an offline machine prints "already up to date" and exits 0 over a store that is not up to date. That is not a coverage gap; it is a wrong answer, and the one that command must never give.

The general form, which is the part to keep

"This error path can't be tested honestly" is a claim about the test. It says nothing at all about whether the branch's absence is visible to the user. Answer the second question separately, every time.

In the markdown

Current test counts, the exact matrix, and what each job asserts are in the workflow file, and every run's results are attached to the commit. The evidence discipline behind all of the above — real dependencies, mutation-tested guards, committed evidence, dated corrections — is one ADR.

The gotchas that cost us

Five things that have actually bitten, in the order you are likely to meet them.

One character, two meanings

freya code-graph with a space is a CLI invocation — the launcher runs a script. freya-code-graph with a hyphen is a skill name — the agent loads instructions. They are never interchangeable. Across 257 rewrites the confusion fired exactly once, a skill name carrying CLI flags, and it was caught by review, not by the checker, which structurally cannot see it.

Flag names are not uniform

Most scripts take --project. Some take --dir: graph_ops.py's --dir is a project root, while verify_links.py's is a specs directory. Same flag, different noun. It is a real inconsistency rather than a documentation error, and it is worth knowing before you write a wrapper around any of them.

A diagnostic must survive the install it diagnoses

freya doctor's first version had the same defect twice, in opposite directions. When the manifest failed to load it still printed "all scripts present" — reporting absence of evidence as evidence of absence. And given structurally valid but semantically wrong JSON, it dumped a traceback: the command whose entire job is diagnosing a broken manifest could not survive one.

A preview that lies is worse than no preview

The installer's "replaced" label was unreachable in --copy and --dry-run: copy always said "copied" even when it had just unlinked a foreign symlink, and dry-run picked its wording before it knew a replacement was involved. Separately, uninstall ignored --dry-run outright and deleted all ten symlinks — a flag whose help text reads "print the plan, change nothing". Every unit test called the uninstall function directly; nothing tested the flag wiring.

Keep the shell thin

install.sh, install.ps1 and freya install are all thin bootstraps over one Python module, and the payoff was measured on the first CI run that had ever executed on Windows: both install jobs went green while the two Windows test jobs failed 29 times. Nothing had broken — the problems had been sitting there the whole time. The part with no logic in it worked.

In the markdown

Two of the integration conventions are the ones a skill author trips over: scripts are invoked as freya <command> and no SKILL.md names a path, and any flow asking for parallel workers states what to do when the agent will not delegate. The rest — tracking files, artifact locations, dated-report idempotency, cross-reference naming — are in the conventions file.