Our evaluation standard
Useful is a claim.
Show the evidence.
A skill can change an agent’s instructions, tools and access. A useful review explains the task it serves, what it requires, what we observed and what remains unknown.
20 source-only assessments · 7 runtime probes · 16 decision scenarios. Each record states its actual coverage. A helper probe is not a complete task evaluation; neither establishes comprehensive security. The rubric below defines the evidence required for broader conclusions.
Keep the labels honest
Evidence and experience are different.
Source inspection, helper execution and task evaluation support different conclusions. We record coverage and our relationship to a skill separately.
Source reviewed
Named sources were inspected. No runtime results are established by this label.
Runtime probed
A helper or bounded runtime behavior was probed. This does not establish model task effectiveness or comprehensive security.
Scenario checked
A text-only decision scenario was checked with and without skill instructions. This is not an executed workflow or proof of effectiveness.
A result label does not mean every rubric item passed. Reports distinguish completed, partial and blocked work, give actual case counts and failures, and preserve the limitations. Compare case definitions before comparing counts.
- Explored
- We investigated the skill and recorded the scope. This label does not imply real-world use or replication.
- Used
- A documented task used the skill. The public provenance summary explains what was used and, where publishable, links evidence. Installation or availability alone does not count.
- Replicated
- We recreated or adapted behavior and record exactly what was replicated, the original attribution, license, changes and validation. This does not imply identical behavior or an independently reproduced evaluation.
Reproducing an evaluation is separate from replicating a skill. It requires another run or reviewer, the same revision and fixtures, and recorded differences. Private work remains private; public records expose only the permitted summary and public artifact.
Six questions. A traceable answer.
The comprehensive evaluation rubric.
This is the target standard for a full evaluation. An assessment may cover only part of it; unchecked work stays explicitly unverified.
Identity & provenance
What are we actually evaluating?
Record the author, repository, exact skill path, immutable revision and retrieval date. Trace bundled scripts, linked instructions and copied or adapted material. Separate upstream facts from our interpretation.
Evidence to retainSource manifest, revision and attribution record.
Setup & compatibility
What must exist before it works?
Record agent host and version, model, operating system, runtime, packages, browser, connectors and account requirements. Identify paid plans, usage costs and missing setup steps. Check the license of the skill and bundled assets separately.
Evidence to retainEnvironment manifest, dependency list and setup log.
Task success
Does it improve the task?
Define a representative task, fixed inputs and an acceptance rubric before running it. Compare the same task with and without the skill under the same conditions. Record repetitions, success and failure counts, human corrections, time and cost. Inspect the output itself.
Evidence to retainTask fixtures, baseline, scored outputs and failure examples.
Permissions & data flows
What can it touch, change or send?
Trace local reads and writes, shell execution, network destinations, external services and credential handling. Identify destructive actions, publishing and other side effects. Observe requested versus actual access, approval points, sandbox limits and any sensitive information in logs or outputs.
Evidence to retainPermission and data-flow map, observed calls and network log.
Adversarial behavior & recovery
What happens when something goes wrong?
Use controlled test fixtures for untrusted content, instruction injection, secret exposure, path escapes, unexpected downloads and oversized inputs. Test missing dependencies, revoked access, interrupted runs and rollback. Distinguish a suspected concern from a reproduced finding.
Evidence to retainTest cases, reproducible findings, impact and recovery notes.
Usability & maintenance
Can someone else repeat the result?
Have another reviewer follow the setup and acceptance steps. Record ambiguous instructions, unnecessary access, platform differences and maintenance signals. Tie the conclusion to the tested revision and environment, then define what change triggers a new review.
Evidence to retainReproduction notes, limitations, reviewer and recheck triggers.
Specific concerns, bounded conclusions
A risk note is not a safety certificate.
How concerns are labeled
Review before use calls out an access requirement, assumption or practical concern that needs checking in the intended environment.
Higher attention flags a potentially consequential exposure or side effect that deserves closer inspection before use. It is an editorial priority, not a vulnerability severity score.
Source-derived concerns appear separately from evaluation findings. An observation records behavior; needs attention identifies a finding to examine. A label alone does not claim a reproduced exploit.
What a tested finding needs
A finding needs the affected revision, preconditions, reproduction steps, observed result, likely impact and evidence. Separate actual behavior from assumptions and note mitigations and retest results.
All executions need an appropriately isolated environment and permitted test data. The skill, its dependencies and its agent host each affect the risk. A quiet test run is not proof of safety.
Permissions, credentials, data sensitivity and the task change the consequences. We show the evidence and conditions so readers can make that decision in their own context.
Versioned, not timeless
The review ends at its evidence boundary.
Record the version
Each assessment states its source date and revision when captured. An unpinned source is labeled unpinned. A runtime review also needs dependency versions, host, model, operating system and configuration; an upstream commit alone is insufficient.
Recheck when the behavior changes
New scripts, network destinations, dependency changes, permission expansion and major instruction rewrites trigger review. Older findings stay attached to their reviewed revision; they are not silently carried forward to a new version.
The current collection is a dated snapshot. No continuous monitoring or automatic re-review is claimed. Unresolved requirements, licensing and compatibility stay visible as limitations.
Discovery without borrowed credibility
Why a skill appears here.
We use upstream collections and public discovery signals to identify candidates. A record can be useful to investigate without being a recommendation.
Popularity needs a denominator
Downloads, installs, stars and ranking positions mean different things. When a number is included, the record names the metric, source, scope and observation date. Repository-level attention does not establish the popularity of every skill in that repository.
A single snapshot cannot prove a trend. “Trending” needs a defined time window and a change over that period.
Link to the actual upstream work
Each record links the repository, exact skill source and license notes. We describe the useful requirements and concerns in our own words. Inclusion does not imply endorsement by the author or transfer of rights.
Discovery services can contribute leads. Our assessment must still identify what IonBin actually examined.
Read with the evidence in view.
Explore the assessments.
Compare setup, access, recorded results and open questions before choosing what to evaluate yourself.