Agent Evaluations

Agent Evaluations let you define quality criteria for your agent's outputs, configure automatic retries when criteria aren't met, and sample production runs for ongoing quality monitoring. Evaluation criteria are configured per-agent from the Evaluations tab on the agent detail page.

Agent evaluations are one of three safety layers in Seclai. They validate output quality, while the prompt scanner blocks injection attacks at ingress, and governance policies screen content against your compliance rules. See the Safety & Quality Overview for a comparison of all three.

Overview

Example scenarios where agent evaluations help:

• A customer-facing Q&A agent sometimes returns vague one-sentence answers when the user's question deserves a detailed explanation. An eval-and-retry criterion requiring "at least 3 actionable points" catches thin answers and automatically retries for a better response.
• Your data transformation agent occasionally produces malformed JSON that breaks downstream systems. An eval-and-retry criterion validating output structure catches the error before delivery and retries the step.
• A weekly report agent has been running reliably for months but you want to know if quality degrades over time. A sample-and-flag criterion scoring "factual accuracy and completeness" monitors 20% of runs without adding any latency.
• A multi-step research agent produces a final summary that occasionally contradicts facts stated in earlier steps. An eval-and-retry criterion checking for internal consistency catches contradictions and re-generates the summary.
• Your content generation agent sometimes drifts off-topic when given ambiguous prompts. A sample-and-flag criterion for "relevance to the original request" surfaces these drift cases for human review so you can improve the prompt.

Governance policies answer "is this content safe and compliant?" — agent evaluations answer "is this content good enough?" They are complementary: an output can pass all governance policies (no PII, no harmful content) but still be unhelpful, incomplete, or off-topic.

The evaluation system has two components:

1. Evaluation Criteria — Rules you define that specify what "good output" looks like for a step or the entire agent.
2. Evaluation Results — Records created each time a criterion is evaluated against an agent run, including pass/fail status, scores, and whether retries were triggered.

You can define multiple criteria per agent. Each criterion targets a specific terminal step by step ID.

Performance & Cost Impact

Each evaluation mode has different implications for agent latency and credit usage:

Eval & Retry is the only mode that directly affects agent execution time. When a step fails evaluation and triggers a retry, the step is re-executed — so worst-case latency is (1 + max_retries) × (step_time + evaluation_time). For most agents, this tradeoff is worthwhile: catching a bad output and auto-correcting it is far better than delivering it to users.

Sample & Flag is designed for zero-latency monitoring. Evaluations run asynchronously after the agent completes, so your agents stay fast while you still get quality signals.

Tip: Use Eval & Retry for steps where output quality is critical (customer-facing content, data transformations, API responses). Use Sample & Flag for ongoing monitoring of steps that are generally reliable but need periodic spot-checks.

Criteria Types

Seclai supports three types of evaluation criteria, each serving a different purpose:

Output Expectations

Output Expectations validate that step outputs match expected formats, schemas, or content patterns. Use them to enforce structural requirements on your agent's outputs.

The Expectation Config is a JSON object that defines what to check:

{
  "expected_format": "json",
  "contains": ["summary", "recommendation"],
  "custom_prompt": "The output must include a numbered list of at least 3 items."
}

Output expectations run on every matching step execution and produce a pass/fail result.

Eval & Retry

Eval & Retry criteria evaluate step outputs and automatically retry the step if evaluation fails. This is useful for steps where occasional failures are acceptable if a retry succeeds.

Key settings, set once for the whole agent in its evaluation settings:

• Max Retries — Maximum number of retry attempts (1–10, default 3).
• Retry on Failure — Whether a failed evaluation triggers a retry. When off, the first failed evaluation fails the step.

When an evaluation fails and retries are enabled, Seclai will:

1. Record the failed evaluation result
2. Re-execute the step with the same inputs
3. Re-evaluate the new output
4. Repeat until the evaluation passes or max retries is reached

If the output still fails evaluation after the last retry, the step fails, and the run fails with it.

Only model steps that are safe to run twice are retried: extract-data steps, and prompt steps in simple mode without tools, media generation or a cloud drive. Every other step fails on its first failed evaluation instead:

• Deterministic steps, such as display results and text or extraction steps, would produce the same output again.
• Steps that send, write or call out, such as sending an email, calling a webhook or another agent, or publishing content, have already done so by the time their output is evaluated.
• Prompt steps in advanced (JSON) mode send a request the platform does not interpret, so it cannot tell whether a re-run would repeat something.

Each retry is tracked in the evaluation results with an incrementing retry_count, and retry_triggered marks the failed results that caused one. A run is judged by its last evaluation: a failure that was retried and then passed does not count as an evaluation failure in alerts, run filters or pass rates. The model calls of every attempt and every evaluation are charged.

A prompt step that has an Eval & Retry criterion itself does not stream its response token by token, even on a priority run, so a response that fails that criterion is never shown.

Sample & Flag

Sample & Flag criteria don't run on every execution. Instead, they evaluate a subset of runs based on a sample frequency and flag runs that fail evaluation for human review.

This is ideal for production monitoring — you get quality signals without adding latency to every run.

Flagged runs appear in the evaluation results with flagged: true and can be filtered in both the UI and API.

Configuring Evaluations

To configure evaluation criteria:

1. Navigate to your agent's detail page
2. Click the Evaluations tab
3. Click Add Criteria
4. Fill in the criteria details:
   • Type — Output Expectation, Eval & Retry, or Sample & Flag, chosen once for the agent in its evaluation settings

• Step ID — Target the terminal step to evaluate
• Type-specific settings — The expectation config for this criterion; retry settings and sample frequency are the agent's, shared by all its criteria
• Evaluation Prompt — (Optional) Custom prompt for LLM-based evaluation

5. Click Create

You can enable/disable criteria at any time using the toggle on each criteria card without deleting them.

Evaluation Results

Each time a criterion is evaluated, an evaluation result is created. Results include:

Results are shown in a unified table below the step cards and can be filtered by status, step, and timeframe.

Aggregated Results View

The results table is aggregated across criteria, so pagination and sorting are applied once across all matching rows. This makes it easier to triage failures by recency without jumping between criteria.

Filtering Results

The Evaluations tab provides multiple ways to filter and explore evaluation results.

Time Frame Selection

Filter results by time using the time frame selector:

• Relative — Last N minutes, hours, days, weeks, or months
• Absolute — Specific start and end dates

Sample Frequency

For Sample & Flag criteria, choose how often runs are evaluated:

Choose a frequency that balances visibility with performance for your use case.

Evaluation Prompt

Use a custom Evaluation Prompt to instruct the LLM evaluator. This prompt is sent along with the step output to determine pass/fail:

Evaluate whether the output meets the following criteria:
1. Contains a clear summary paragraph
2. Includes at least 3 actionable recommendations
3. Uses professional tone throughout
4. Does not contain placeholders or TODO items

Return PASS if all criteria are met, FAIL otherwise.

API Access

Evaluation criteria and results can be managed via the API using your API key.

List Evaluation Criteria

AGENT_ID=...

curl "https://api.seclai.com/agents/$AGENT_ID/evaluation-criteria" \
  -H "X-API-Key: $SECLAI_API_KEY"

Create Evaluation Criteria

curl -X POST "https://api.seclai.com/agents/$AGENT_ID/evaluation-criteria" \
  -H "X-API-Key: $SECLAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "description": "Output format check",
    "step_id": "step_1",
    "evaluation_mode": "output_expectation",
    "expectation_config": {
      "expected_format": "json",
      "contains": ["result", "status"]
    }
  }'

Get Single Evaluation Criteria

CRITERIA_ID=...

curl "https://api.seclai.com/agents/evaluation-criteria/$CRITERIA_ID" \
  -H "X-API-Key: $SECLAI_API_KEY"

Update Evaluation Criteria

curl -X PATCH "https://api.seclai.com/agents/evaluation-criteria/$CRITERIA_ID" \
  -H "X-API-Key: $SECLAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "enabled": false
  }'

Delete Evaluation Criteria

curl -X DELETE "https://api.seclai.com/agents/evaluation-criteria/$CRITERIA_ID" \
  -H "X-API-Key: $SECLAI_API_KEY"

List Evaluation Results

curl "https://api.seclai.com/agents/evaluation-criteria/$CRITERIA_ID/results?status=failed&flagged_only=true&page=1&limit=20" \
  -H "X-API-Key: $SECLAI_API_KEY"

List Aggregated Agent Evaluation Results

curl "https://api.seclai.com/agents/$AGENT_ID/evaluation-results?status=failed&step=step_1&page=1&limit=20" \
  -H "X-API-Key: $SECLAI_API_KEY"

Create Evaluation Result

Record an evaluation result against a criteria (e.g. from an external test harness or CI pipeline):

curl -X POST "https://api.seclai.com/agents/evaluation-criteria/$CRITERIA_ID/results" \
  -H "X-API-Key: $SECLAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "agent_run_id": "RUN_UUID",
    "status": "passed",
    "score": 0.95,
    "details": { "reason": "Output matched expected JSON schema" },
    "retry_triggered": false,
    "retry_count": 0,
    "flagged": false
  }'

Get Run Evaluation Results

RUN_ID=...

curl "https://api.seclai.com/agents/$AGENT_ID/runs/$RUN_ID/evaluation-results" \
  -H "X-API-Key: $SECLAI_API_KEY"

Get Evaluation Summary

curl "https://api.seclai.com/agents/evaluation-criteria/$CRITERIA_ID/summary" \
  -H "X-API-Key: $SECLAI_API_KEY"

Summary Metrics

For account-level monitoring, use the summary endpoint:

curl "https://api.seclai.com/agents/evaluation-results/non-manual-summary?days=30" \
  -H "X-API-Key: $SECLAI_API_KEY"

This includes only eval_and_retry and sample_and_flag criteria. In these summaries:

• error counts results whose evaluator returned no verdict. They are not in failed, and pass_rate and failure_rate are over passed and failed results only. Before API version 2026-10-21, failed and the rates include them.
• flagged counts results explicitly marked for review.

MCP Tools

Evaluation criteria can also be managed through the MCP (Model Context Protocol) server. Connect your MCP client to your agent's MCP endpoint and use these tools:

Example MCP tool call:

{
  "tool": "create_evaluation_criteria",
  "arguments": {
    "agent_id": "AGENT_UUID",
    "step_id": "step_1",
    "description": "JSON output validation",
    "evaluation_mode": "output_expectation",
    "expectation_config": {
      "expected_format": "json"
    },
    "evaluation_prompt": "Verify the output is valid JSON with a 'result' key"
  }
}

Exporting

Export an agent's evaluation results as a downloadable file. Supported formats are JSON, JSONL, and CSV.

• UI: Open the agent's Evaluations tab and click the Export button. You'll see an estimate of the record count before confirming.
• API: POST /authenticated/resource-exports with resource_type: "agent_evals" and the agent ID.
• MCP: Use the create_resource_export tool with resource_type: "agent_evals".

Exports include evaluation status, scores, details, retry information, and timestamps for each result.

See Export Formats → Agent Evaluations for the full file schema and available filter options.

Next Steps

• Agents — Learn about creating and configuring agents
• Agent Steps — Understand agent step definitions and execution
• Alerts — Set up notifications for agent failures
• API Introduction — Get started with the Seclai API