Your AI agent passed the demo. How often does it pass on a Tuesday?

Most teams with an LLM feature know it works because someone tried it and it looked right. That is a demo, not a measurement.

An agent can answer the same request correctly four times and get it wrong on the fifth. It can reach the right answer by a path that costs ten times what it should. And a prompt change on Friday can quietly break something that worked on Thursday, with nothing in your pipeline to notice.

I am Ana Sousa, a senior QA automation engineer with 14+ years in testing, ISTQB Test Manager and Security, and an MSc in Computer Science. I work from Aveiro, Portugal, under my own company, Crialia. I treat your agent like any other system under test: a defined task set, a scorer you can read, repeated runs, numbers with error bars, and a gate in CI.

• Heads of Engineering, Heads of QA and AI leads at product companies and scale-ups that have shipped an LLM feature or a support agent, and nobody is measuring it.
• Consultancies and training providers who want to offer AI testing to their own clients and need the method, or a team trained in it.

It is probably not for you if you have no agent in a testable state yet, or if what you need is someone to build the agent. I measure agents. I do not build them as part of this service.

On request: red-team and guardrail review

I can review how your agent behaves under hostile or out-of-scope input and whether its guardrails hold. This is not yet offered as a package and has no published price. Ask on the scoping call.

• A golden task set, built with you. Real tasks your agent is meant to handle, with agreed expected outcomes. Your domain knowledge goes into it; it stays yours after the audit.
• A three-layer scorer. Outcome: did the agent get the right result. Trajectory: did it get there by a reasonable path, with the right tool calls. Components: do the individual parts, such as retrieval or a single tool call, do their job.
• Pass rate with confidence intervals. A single percentage on a small task set hides how uncertain it is. You get the interval, so you know how much the number can be trusted.
• pass^k repeats. The same task run several times. An agent that passes once is not the same as an agent that passes every time, and your customers only meet the second kind.
• Cost and p95 latency. What each task costs to run, and how slow the slow cases are, not just the average.
• A failure taxonomy. Every failure sorted by cause, so you fix the category, not the anecdote.
• A CI regression gate. A check in your pipeline that fails the build when the agent drops below the agreed bar on the task set.
• Final report and handover. Written findings, the scorer and task set in your repository, and a session with your team so they can run and extend it without me.

I have no published client case studies yet. What I can show you is my own public lab, including the part where my first result was wrong.

The setup. Three models, 40 tasks, one run per task. One research agent working over a fixture web, a controlled set of pages built for the test.

The first result was wrong. The first ranking put Opus last, at 12%. The cause was my scorer, not the model: it failed any answer that mentioned a distrusted figure, even when the answer mentioned it to reject it.

How I checked. I hand-labelled the 94 flagged passages. I then built an LLM judge to replace the rule, and set its pass bar before running it: 90% recall on answers that asserted the distrusted figure. It reached 25%, and its results changed from run to run. It missed the bar, so I did not use it. The original rule stayed, and the corrected numbers use the hand label for each flagged passage instead of failing it automatically.

The corrected outcome rates. Haiku 55%, Sonnet 42%, Opus 25 to 28% (25% if the 16 answers I never hand-labelled count as failures, 28% if they are ignored). The 95% confidence intervals overlap, which means these numbers do not support a confident ranking of the three.

The limits. 40 tasks. One run per task. One agent. One domain. This is not a benchmark and it does not tell you which model to use. What it shows is how I work. I set the bar before I look at the result, and when the first number turned out to be my scorer's fault, I published the correction.

The full reports are public: github.com/anapsousa/agent-reliability-lab, starting with the stance-rescore report (2026-10-06).

The same discipline goes into your audit. A scorer that fails correct answers would make your numbers worthless, and you would never know.

1. Free scoping call, 20 to 30 minutes. You describe the agent, what it is for, and what worries you. I ask about task volume, tools, risk and integrations. No preparation needed beyond knowing your system.
2. Written proposal. Size, scope, what I need from your side, timeline and a fixed price. You decide with the number in front of you.
3. The work. We build the task set together, I build the scorer and run the evaluation, and the gate goes into your pipeline.
4. Report and handover. You get the findings and the tooling, and your team learns to run it.

These prices are for companies outside Portugal. Portuguese companies have their own page with their own prices: Página em português.

All prices are indicative starting points, excl. VAT; VAT treatment confirmed in the proposal. Durations are working days. The final figure is confirmed after the free scoping call, because it depends on your case.

What moves the price. More tasks in the golden set, more tools the agent can call, higher risk when it fails, and more integrations to set up for testing all move it up. A narrow agent with a few tools and a clean test environment sits at the bottom of its band.

What the price buys. The task set, the scorer, the numbers with their intervals, the failure taxonomy, the CI gate and the handover. You keep all of it.

This service is new. The first two clients get 25% off.

In exchange, I publish a case study of the audit and you give a testimonial. If your company cannot be named, an anonymised case study works. What appears in the case study is agreed with you before we start, and the text only goes out with your approval.

• Not a guarantee your agent will pass. You get an honest measurement. Some audits will show the agent is not ready.
• Not a model benchmark. I measure your agent on your tasks, not models in general.
• Not a build service. The report says where the failures come from. Fixing them is your team's work, or a separate proposal.
• Not a security penetration test. Red-team and guardrail review is on request and not yet packaged.
• Not publicly funded or certified training. The workshop is a private, paid engagement.