How to Evaluate an Agent Skill Before You Install It

The usual version of this story happens near midnight: a client demo in the morning, the design nearly there, and a skill zip from a stranger’s repo that promises to close the last gap. Before you unzip it and hand it your files and your shell, here is the whole review. To evaluate an Agent Skill before you install it, read its SKILL.md and every bundled script, confirm it declares the capabilities and any paid routes it needs, check that it does not fetch instructions from untrusted URLs, and prefer skills whose claims are backed by graded test runs rather than self-report. A skill can direct your agent to run code and use tools, so treat installing one like installing software from an author you are choosing to trust.

The good news is that a skill is plain files, so you can read all of it before it ever runs. That review is the difference between the demo that breaks and the deliverable you can defend. Here is the routine in order, and a checklist you can steal.

Why a skill deserves a real review

A skill is not passive text. It gives your agent new instructions and often new code to run. Anthropic’s own Agent Skills documentation puts it plainly: use skills only from trusted sources, because a malicious skill can direct the agent to invoke tools or execute code in ways that do not match the skill’s stated purpose. The same guidance recommends auditing every bundled file, watching for unexpected network calls or file access, and being especially careful with skills that pull content from external URLs, since fetched content can carry its own instructions.

Hold that picture next to the midnight scene. The thing you are about to install can be told, by its own author or by a page it fetches later, to do more than it advertises, while it has your files and your shell open. That is the demo that breaks in a way you cannot explain to a client. That reading is cheap insurance against it.

Read the SKILL.md first

Open SKILL.md and read it end to end. You are checking three things.

• Scope. Does the workflow match what the listing promises? A concept skill that suddenly wants to touch credentials or push to a remote is a red flag.
• Clarity. Good instructions are specific and ordered. Vague, hand-wavy steps usually mean the skill was never run against real tasks.
• Honesty about limits. A skill you can trust states what it needs, what it does not include, and what it will not do without your approval.

If the SKILL.md reads like a sales page instead of a procedure, be skeptical. The body of the file is where the actual behavior lives, and marketing language there is often cover for instructions that were never tested.

Audit the scripts and resources

Next, read the scripts/ directory and any reference files. You do not need to be a security researcher to catch the common problems.

• Look for network calls. A script that quietly downloads and runs remote code deserves a hard stop.
• Look for writes outside the working directory, environment-variable harvesting, or anything reaching for keys and tokens.
• Confirm the scripts do what the instructions say they do, and nothing extra.

Skills that fetch data from external URLs carry the most risk, because a page can change after you install the skill and start feeding new instructions to your agent. If a skill depends on live external content, make sure you understand what it pulls and from where before you let it run.

Check what it claims, and how it proves it

Most listings claim to make your agent better at something. The question worth asking is how that claim was tested. Self-reported quality is weak evidence. A skill author saying “this works well” is not a result, it is a hope with good formatting.

Stronger evidence looks like this: the author ran a set of representative tasks with the skill and without it, then graded the outputs from the actual artifacts produced, not from the author’s own opinion. That is the difference between a demonstration of completeness and a marketing line. Be wary of performance percentages, revenue figures, or customer counts offered with no method behind them. If a number has no visible test attached, treat it as decoration and move on.

The checklist you can steal

At Aaria, every listing has to pass a ten-point admission gate before it appears in the catalog, and the full gate is published so you can hold any skill to it, ours included. You do not have to shop here to use it. Borrow these checks for anything you are about to install:

• The upload zip is under 50 MB and places SKILL.md at its root with valid frontmatter carrying a name and a description.
• Every relative link resolves inside the bundle, with no private paths, secrets, or harness-specific commands presented as universal.
• The skill states its required capabilities, optional paid routes, fallback behavior, and support boundary.
• At least three representative prompts have been run with and without the skill.
• Assertions are graded from generated artifacts, not self-reported by the author.
• Included scripts pass syntax checks and at least one observable smoke run where applicable.
• The listing copy makes no unsupported performance, revenue, compatibility, or customer claims.
• The version, license, installation, and support documents match the shipped bundle.

A skill that clears those points is one you can reason about at midnight. A skill that dodges them is asking for trust it has not earned.

The pre-install pass, in order

When the clock is against you, run this and nothing fancier:

1. Read SKILL.md top to bottom. Does the scope match the promise?
2. Skim every script for network calls and out-of-scope file access.
3. Confirm the listing declares capabilities, paid routes, and what it does not include.
4. Find the proof. Is there a test method, or just a claim?
5. Install into a project directory first, not your global directory, and watch the first run.

That last step is the one people skip and regret. A project-scoped install does not fence off your files or your shell; the agent keeps whatever access it already had. What it does is keep the skill out of every other session and let you watch each action on that first run, so a bad one is easy to spot and easy to pull. Scrolldio, for example, is built to probe the tools already in your session and ask for approval before any step that could create charges on your own accounts, which is exactly the behavior you want to confirm on that first run.

None of this is about paranoia. It is about being able to defend what you shipped. When a skill is plain files you have read, its claims are graded, and its first run happened where you could watch it, the demo in the morning is one you can stand behind. That is the whole distance between installing on faith and installing on evidence.

FAQ

Are agent skills safe to install? A skill is only as safe as its source and its contents. Because a skill can tell your agent to run code, install skills from authors you trust and read the files first. Anthropic recommends treating a skill like software you are installing.

What should I look for in a SKILL.md before installing? Check that the scope matches the promise, the instructions are specific and ordered, and the file honestly states what the skill needs, what it excludes, and what it will not do without your approval.

How do I know if a skill’s quality claims are real? Look for a test method. Strong evidence is a set of representative tasks run with and without the skill, graded from the artifacts produced. Treat percentages or customer counts with no visible method as marketing.

Should I install skills globally or per project? Start per project when you can. A project-scoped install lets you watch the first run before you give the skill access to every session.