Methodology

In every one of these the same property holds: the documentation IS the program. A line in a manifest that describes an attack and a line that performs one are the same bytes, and that ambiguity is the source of most of what this page reports as an accepted false positive.

A stated boundary is a strength. An unstated one is a bug: a directory that nobody had declared either in or out of scope was silently invisible to us until it was found and fixed, and an empty scan graded A while examining nothing.

Evidence class is derived from what a rule is gated to, not declared: 7 rules read a config value with defined semantics (parsed), and the rest match a pattern over prose or script text (heuristic), where the same bytes can be an instruction or a description of one.

The ASI and CWE columns are our own mapping onto OWASP's Top 10 for Agentic Applications and CWE. They are a navigation aid, not a certification by either body.

An A on an axis means nothing was counted against it. It is the absence of a detection, not a positive assessment.

"Counted" is the precise word. Where AI triage is enabled we ask a model to check each finding, and a finding it judges to be a benign example — documentation showing what an attack looks like, a test fixture — is set aside and does not affect the grade. Nothing is dropped: every set-aside finding is still shown, grouped under its own heading, so you can see exactly what was not counted.

Triage can only ever demote. It cannot create a finding, raise a severity, or turn a passing grade into a failing one. That is a deliberate limit, and the limits section below says what it costs us.

This is a property of the current ruleset, not of the grading algorithm. The cap table would issue B on security the day a rule emits a Low-severity finding there. We publish the scale as it is rather than renaming grades, because renaming would change grades already recorded on past scans and shown on badges already issued.

The corpus. MalSkillBench (arXiv 2606.07131), 3944 malicious and 4000 benign skills. Malice is verified by execution, not asserted: a candidate enters the benchmark only if its declared behaviour is actually observed at run time in a sandbox. We did not build it, we cannot influence it, and it is CC0-licensed so anyone can check our numbers against it.

The protocol. Every sample is copied to a clean directory and scanned once in each of the two install layouts a skill can arrive in. The benchmark's own provenance file is removed first: it is present on 97% of benign samples and on none of the malicious ones, and leaving it in would measure the benchmark's packaging rather than the skill.

The definitions — and there are three. They are not interchangeable, and a figure moved from one to another is how this project has produced wrong conclusions internally. Every number below names the one it belongs to.

One important caveat travels with every precision figure on this page. The corpus is half malicious by construction. A real population of repositories is not, and precision falls steeply as the true rate of malicious skills falls. Recall does not. Read our precision as a comparison against the other tools measured on the same corpus, never as a prediction of how often a flag you see in your own CI will be correct.

Out of the box the engine is a precise, low-recall critical gate: it fails on a Critical finding and nothing else. The stricter settings trade that precision for recall, and the false-positive rate rises with it.

Two readings we would rather state than have a reader derive: 978 of the 3944 known-malicious samples are graded A, so a green badge is wrong about roughly a quarter of the malicious population. And 43.65% of the benign samples draw a badge worse than A.

Both install layouts — a skill directory as it sits in a repository, and one installed under .claude/skills/ — produce identical results on every one of the 7944 samples: same grades, same files scanned, same findings, same exit codes. That was not true before. A 23.7-point recall gap between them existed, was found by measuring both rather than the convenient one, and is now closed.

The corpus labels 1300 of its malicious samples with the behaviour they perform. This is the most useful table on the page, because the two halves of it are very different products.

Code-level attacks — what the skill's code does — we catch at 97.9%. Agent-control attacks — what the skill says to the model — we catch at 17.0%.

The second number is the honest limit of this instrument. The limits section below says why it is where it is, and why the machinery we already have cannot move it.

Note also that the default column is much lower than the any-finding column across almost every code-level family. Out of the box we reliably catch reverse shells and persistence, and little else. The high code-level recall requires the stricter setting, which carries the higher false-positive rate in the table above. That trade is yours to make, and it is the reason both columns are printed.

Read it as a class result, not a leaderboard. Static rule engines cluster at high precision and low recall; LLM-assisted engines invert that. The highest-recall tool in the table reaches 98.4% by flagging 23.4% of clean skills, and a gate built on it would block a quarter of clean builds.

Four caveats before anyone quotes a row of this. The twelve figures are the paper's, measured by its authors; ours are ours, measured by us on the same corpus and the same protocol, which is better than self-assertion and weaker than an independent evaluation. The tools were measured at whatever setting their authors chose, so the comparison is between configurations as much as between engines. Every precision figure here rests on a 50/50 corpus and none of them predicts precision in the wild. And our three rows are one product at three settings, not three products.

We are not the most precise engine in this table and we do not claim to be. Snyk reaches 93.2% precision and VirusTotal 99.5%, both far above us. The trade is visible in the same rows: they recall 28.7% and 21.6% of the malicious set, where our two stricter settings recall 69.2% and 75.2%.

The median recall among the static engines here is 28.7%, and 2 of our 3 settings sit above it. The default sits below, at 24.8%. A deterministic gate with recall well above the static median is an unusual place to be, and it is the position we are actually defending — not a claim to precision we do not hold.

Agent-control attacks, at 17.0% recall. Role hijack, safety bypass, instruction override, system-prompt leakage, goal hijacking and content manipulation. This is the category people buy an agent-skill scanner for, and it is where we are weakest.

It will not be fixed by the AI triage layer, and it is worth saying why rather than leaving the impression that it might be. That layer is suppression-only by design: it can set a finding aside, never create one. Low recall is a missing-finding problem, and a mechanism that can only remove has nothing to remove when no rule fired. Closing this family needs deterministic rules that read what a skill says to the model, and that work belongs in the engine.

OWASP ASI coverage is partial, not complete. Inter-agent communication, cascading failures and human-agent trust exploitation are runtime properties that a static read of one repository cannot observe. Memory and context poisoning is partial: we read poisoned instruction files and do not read persisted agent memory. Our mapping claims ASI01 goal hijack, and at the recall above the honest label for it is partial.

A grade is not a security audit. 25 static checks over agent configuration is a narrow instrument aimed at a specific surface. It does not read your application code, it does not run anything, and an A means no rule fired rather than that a skill is safe. The badge says the same three things in the same words — 25 static checks, agent configuration only, not a security audit — because a reader who only ever sees the badge deserves the qualification in the same glance as the letter.

The corpus cannot measure everything. A benchmark measures what a change costs on shapes the benchmark contains. It cannot tell us what leaving a hole open costs, because the attack that walks through the hole is not in it. Four live bypasses were once found by adversarial review of code that a full corpus run had reported as unchanged. Both methods are needed, and neither substitutes for the other.