Methodology

These are static findings about submitted files. A declaration is not proof of effective enforcement: installed harness version, managed and user settings, provider and session conditions may change what runs. A line that describes an attack and a line that directs one can be the same bytes, and that ambiguity is the source of most accepted false positives below.

The paths above are supported formats and recognized roots, not a claim to inspect every repository subtree. Common dependency and build directories are excluded. If no supported files are found, nothing is checked and no grade is issued.

Supported schemas include Claude settings and hooks, Codex TOML configuration, and supported Copilot repository hooks and path instructions. Invalid analyzed configuration fails the check without a grade; unsupported or invalid configuration cannot establish a clean runtime. External references, including unavailable referenced files, are not fetched from the scanner host.

Repository contents cannot reveal managed and user settings, account-synced skills, cloud or home state, installed plugins, or a live agent session unless explicitly submitted. Unknown settings source, trust, provider and session conditions remain unknown; a grade describes checked evidence, not all effective agent behavior. Unreadable supported files can leave a scan incomplete; a grade does not prove they were read.

Evidence class is derived from what a rule is gated to, not declared: 9 rules read a config value with defined semantics (parsed), and the rest match a pattern over prose or script text (heuristic), where the same bytes can be an instruction or a description of one.

The ASI and CWE columns are our own mapping onto OWASP's Top 10 for Agentic Applications and CWE. They are a navigation aid, not a certification by either body.

An A on an axis means nothing was counted against it. It is the absence of a detection, not a positive assessment.

"Counted" is the precise word. Where AI triage is enabled we ask a model to check each finding, and a finding it judges to be a benign example — documentation showing what an attack looks like, a test fixture — is set aside and does not affect the grade. Nothing is dropped: every set-aside finding is still shown, grouped under its own heading, so you can see exactly what was not counted.

Triage only reviews existing findings and can only demote them. It cannot find a threat no static rule detected, create a finding, raise a severity, or turn a passing grade into a failing one. That is a deliberate limit, and the limits section below says what it costs us.

This is a property of the current ruleset, not of the grading algorithm. The cap table would issue B on security the day a rule emits a Low-severity finding there. We publish the scale as it is rather than renaming grades, because renaming would change grades already recorded on past scans and shown on badges already issued.

The corpus. MalSkillBench (arXiv 2606.07131), 3944 malicious and 4000 benign skills. Malice is verified by execution, not asserted: a candidate enters the benchmark only if its declared behaviour is actually observed at run time in a sandbox. We did not build it, we cannot influence it, and it is CC0-licensed so anyone can check our numbers against it.

The protocol. Every sample is copied to a clean directory and scanned once in each of the two install layouts a skill can arrive in. The benchmark's own provenance file is removed first: it is present on 97% of benign samples and on none of the malicious ones, and leaving it in would measure the benchmark's packaging rather than the skill.

The definitions — and there are three. They are not interchangeable, and a figure moved from one to another is how this project has produced wrong conclusions internally. Every number below names the one it belongs to.

One important caveat travels with every precision figure on this page. The corpus is half malicious by construction. A real population of repositories is not, and precision falls steeply as the true rate of malicious skills falls. Recall does not. Read our precision as a comparison against the other tools measured on the same corpus, never as a prediction of how often a flag you see in your own CI will be correct.

Out of the box the engine is a precise, low-recall critical gate: it fails on a Critical finding and nothing else. The stricter settings trade that precision for recall, and the false-positive rate rises with it.

Two readings we would rather state than have a reader derive: 978 of the 3944 known-malicious samples are graded A, so a green badge is wrong about roughly a quarter of the malicious population. And 43.65% of the benign samples draw a badge worse than A.

Both install layouts — a skill directory as it sits in a repository, and one installed under .claude/skills/ — produce identical results on every one of the 7944 samples: same grades, same files scanned, same findings, same exit codes. That was not true before. A 23.7-point recall gap between them existed, was found by measuring both rather than the convenient one, and is now closed.

The corpus labels 1300 of its malicious samples with the behaviour they perform. This is the most useful table on the page, because the two halves of it are very different products.

Code-level attacks — what the skill's code does — we catch at 97.9%. Agent-control attacks — what the skill says to the model — we catch at 17.0%.

The second number is the honest limit of this instrument. The limits section below says why it is where it is, and why the machinery we already have cannot move it.

Note also that the default column is much lower than the any-finding column across almost every code-level family. Out of the box we reliably catch reverse shells and persistence, and little else. The high code-level recall requires the stricter setting, which carries the higher false-positive rate in the table above. That trade is yours to make, and it is the reason both columns are printed.

Read it as a class result, not a leaderboard. Static rule engines cluster at high precision and low recall; LLM-assisted engines invert that. The highest-recall tool in the table reaches 98.4% by flagging 23.4% of clean skills, and a gate built on it would block a quarter of clean builds.

Four caveats before anyone quotes a row of this. The twelve figures are the paper's, measured by its authors; ours are ours, measured by us on the same corpus and the same protocol, which is better than self-assertion and weaker than an independent evaluation. The tools were measured at whatever setting their authors chose, so the comparison is between configurations as much as between engines. Every precision figure here rests on a 50/50 corpus and none of them predicts precision in the wild. And our three rows are one product at three settings, not three products.

We are not the most precise engine in this table and we do not claim to be. Snyk reaches 93.2% precision and VirusTotal 99.5%, both far above us. The trade is visible in the same rows: they recall 28.7% and 21.6% of the malicious set, where our two stricter settings recall 69.2% and 75.2%.

The median recall among the static engines here is 28.7%, and 2 of our 3 settings sit above it. The default sits below, at 24.8%. A deterministic gate with recall well above the static median is an unusual place to be, and it is the position we are actually defending — not a claim to precision we do not hold.

Agent-control attacks, at 17.0% recall. Role hijack, safety bypass, instruction override, system-prompt leakage, goal hijacking and content manipulation. These instruction-based attacks remain a weak point of the scanner.

It will not be fixed by the AI triage layer, and it is worth saying why rather than leaving the impression that it might be. Triage only reviews existing findings and is suppression-only by design: it can set a finding aside, but it cannot create one or discover a threat no deterministic rule found. Low recall is a missing-finding problem, and a mechanism that can only remove has nothing to remove when no rule fired. Closing this family needs deterministic rules that read what a skill says to the model, and that work belongs in the engine.

OWASP ASI coverage is partial, not complete. Inter-agent communication, cascading failures and human-agent trust exploitation are runtime properties that a static read of one repository cannot observe. Memory and context poisoning is partial: we read poisoned instruction files and do not read persisted agent memory. Our mapping claims ASI01 goal hijack, and at the recall above the honest label for it is partial.

A grade is not a security audit. 27 static checks over supported skill and coding-agent configuration files are a narrow instrument. Ordinary application code is outside the general audit scope, submitted files are not executed, and an A means no rule fired rather than that a skill or configuration is safe. The badge says the same three things in the same words — 27 static checks, agent configuration only, not a security audit — because a reader who only ever sees the badge deserves the qualification in the same glance as the letter.

The benchmark measurements on this page are measurements on skills. They do not establish measured coverage across coding-agent frameworks, settings, hooks or MCP configurations.

The corpus cannot measure everything. A benchmark measures what a change costs on shapes the benchmark contains. It cannot tell us what leaving a hole open costs, because the attack that walks through the hole is not in it. Four live bypasses were once found by adversarial review of code that a full corpus run had reported as unchanged. Both methods are needed, and neither substitutes for the other.