From IonBin Explore tools and practical workflows. See what’s new
Re-run your evaluation set when a model changes
AI evaluation & quality control · Dev, web & IT · checked 2026-10-18
Compare a baseline and candidate on the same versioned cases before changing a model or prompt. promptfoo can run checks in CI and export results; a reviewer inspects regressions, grader behavior and cost before approving the change.
Independent review. IonBin has no partnership with any tool on this page and was not paid for this page. Our evaluation is our own, based on public documentation and our research. Product names are trademarks of their owners and are used for identification only. How disclosure works.
Goal
Make prompt and model changes reviewable with task-specific evidence.
Connects through: API
Steps
01
You
Freeze the dataset, rubric and baseline configuration, then record the candidate change.
Approves, rejects or requests revisions using the evidence and the predefined pass bar.
Human stepConfirmed
Confirmed means the vendor documents it (linked). It doesn't mean IonBin has tested it. AI-platform and human steps need no source.
Human approval
A named release owner approves the change; automated scores do not publish it.
Watch out for
Changing the grader or dataset between runs changes the comparison. Record both, and inspect high-severity failures even when the average improves.
Starter prompt
Compare these baseline and candidate results using the same case IDs. List regressions, unchanged failures and improvements separately. Preserve the counts and flag missing runs. Do not change the acceptance criteria to make the candidate pass.
Written for this workflow. Paste it into your AI platform and adjust it to your situation.
promptfoo: verify the API caller, network access and credentials; platform code execution alone does not establish this connection; Langfuse: verify the API caller, network access and credentials; platform code execution alone does not establish this connection
promptfoo: verify the API caller, network access and credentials; platform code execution alone does not establish this connection; Langfuse: verify the API caller, network access and credentials; platform code execution alone does not establish this connection
promptfoo: verify the API caller, network access and credentials; platform code execution alone does not establish this connection; Langfuse: verify the API caller, network access and credentials; platform code execution alone does not establish this connection
ChatGPT
To verify
promptfoo: verify the API caller, network access and credentials; platform code execution alone does not establish this connection; Langfuse: verify the API caller, network access and credentials; platform code execution alone does not establish this connection
Gemini
To verify
promptfoo: verify the API caller, network access and credentials; platform code execution alone does not establish this connection; Langfuse: verify the API caller, network access and credentials; platform code execution alone does not establish this connection
Perplexity
To verify
promptfoo: verify the API caller, network access and credentials; platform code execution alone does not establish this connection; Langfuse: verify the API caller, network access and credentials; platform code execution alone does not establish this connection
Readiness considers the connection evidence and unresolved steps. Confirmed: documented access and the proposed steps meet the recorded requirements. MCP and API pairings require an explicit reviewed platform note; a named plugin or CLI alone does not confirm MCP. Inferred: an explicit page note describes a plausible pairing that still needs testing. To verify: a connection or step is unresolved. An API and code execution alone do not establish compatible network access, credentials or permissions. None of this is IonBin testing.
What we checked
promptfoo documents CI execution, assertions and exported evaluation results. Langfuse documents dataset items with inputs and expected outputs. These capabilities support the proposed sequence; IonBin has not tested the combined pipeline or a release gate.
Set it up
Keep the case set, rubric, prompt and provider configuration versioned. Record model and grader identifiers rather than writing only a vendor name.
Define the acceptance rule before comparing results. An average can hide a serious failure, so identify which case types must pass.
Run the baseline and candidate against the same cases and inspect failures alongside a sample of passes. Consider repeat runs when output variation could change the decision.
Store access-controlled result artifacts and the release decision. Add corrected failure cases to the development set; preserve held-out cases for an independent check.
Only release after the designated reviewer accepts the evidence.
A local runner or private repository does not make every call local: configured model providers receive inputs, and shared reports or CI artifacts can expose outputs. Review the data path before running customer material. API usage, including grader calls, may incur charges.
Before relying on this workflow
The steps above are documentation-reviewed. Complete these checks in your own environment before treating the implementation as tested:
Run a pilot that proves an intentionally failing case prevents the intended release.
Check CI secrets, artifact visibility, provider limits and grader costs in the chosen environment.
Related agent skills
Sentry’s Prompt Optimizer helps when a model change breaks a prompt, because it reruns candidates against the same cases. For open models, Hugging Face’s Community Evals skill runs standard benchmarks locally. Your own cases still matter more for business tasks. IonBin reviewed both from source only.
No affiliate links on this page. No paid placements in workflows. How IonBin makes money.
Choose whether to share analytics
With your permission, Google Analytics uses cookies to measure page visits and help us improve IonBin. It stays off until you choose. We do not enable advertising tracking.
You can change this choice anytime below. Turning analytics off reloads open IonBin tabs where analytics is running. Save unfinished form text first. Privacy details.