config-evals Skill

1---2name: config-evals3description: >-4  Builds and maintains configuration-based evaluations on a workflow with the5  eval-config tool. Use when the user asks to set up, add, view, change, or6  remove an evaluation, score, grade, or judge a workflow's output, or measure7  answer quality against a test dataset. This is the only eval form Instance AI8  handles — it does not touch on-canvas evaluation nodes.9recommended_tools:10  - eval-config11  - data-tables12platforms:13  - daytona14---15 16# Config-based Evaluations17 18Use this skill to attach a configuration-based evaluation to a workflow with the19`eval-config` tool. A config eval pairs a workflow with a name, a start node, an20end node, one or more judged metrics, and a Data Table dataset. Nothing is added21to the canvas — the config lives off-canvas via the evaluation-config API.22 23Config evals are the only evaluation form you work with. Do not add, read,24rewire, or reason about on-canvas evaluation nodes (EvaluationTrigger,25Evaluation/checkIfEvaluating/setOutputs/setMetrics). If the user asks for those,26build a config eval instead and briefly say that is how you set up evaluations.27 28## What a Config Eval Needs29 30- `name` — a human-readable evaluation name.31- `startNodeName` — the node where a run begins; it is fed one test-input row.32  Must be a node with an incoming connection — **never a trigger** (see step 2).33- `endNodeName` — the node whose output is judged.34- `dataTableId` — a Data Table holding the test dataset. Create and populate it35  with the `data-tables` tool first, then link it here by id.36- `metrics` — one or more judged metrics (see below).37 38## Default Procedure39 401. Identify the target workflow and read it. Trace the main path from trigger to41   the node that produces the answer.422. Pick the nodes:43   - `startNodeName` is the first node **after** the trigger — the node that44     receives the input the dataset varies. Never use the trigger itself: an45     eval run swaps the trigger for a dataset-driven one, so the start node must46     have an incoming connection or the run fails to compile. For a chat/agent47     workflow this is usually the agent node (often the same as `endNodeName`).48   - `endNodeName` is the node whose output you want scored (usually the AI agent49     or the final response node).503. Resolve the dataset. Call `data-tables(action="list")` to find an existing51   dataset, or create and seed one with `data-tables` before creating the config.52   Never invent a `dataTableId`; use one returned by `data-tables`.534. Choose metrics and build the `actualAnswer` / `expectedAnswer` / `userQuery`54   expressions (see Metrics).555. Call `eval-config` (`action="create"`), or `update` when changing an existing56   config. The tool shows an approval card automatically — call it and respect57   the result; do not ask for chat approval first.586. Close with facts: evaluation name, workflow, start/end nodes, dataset name and59   id, and the metrics configured.60 61## Metrics62 63Each metric is LLM-judged and needs a judge model: a `credentialId`, a `model`,64and an `outputType` (`numeric`, the default, or `boolean`). Reuse an LLM65credential the workflow already uses when one fits.66 67Do **not** set `provider` unless you know the exact chat-model node type — it is68derived automatically from the credential you pass (each credential type maps to69one provider). Just pick the credential and the model.70 71Two presets are available:72 73- **`correctness`** — compares the produced answer to a ground-truth answer.74  Requires `expectedAnswer` (an n8n expression resolving to the ground-truth75  value, typically a dataset column, e.g. `={{ $json.expected_output }}`).76- **`helpfulness`** — judges the produced answer against the user's query.77  Requires `userQuery` (an n8n expression for the input the user asked, e.g.78  `={{ $json.input }}`).79 80Every metric also needs `actualAnswer`: an n8n expression resolving to the81workflow's produced answer at the end node, e.g. `={{ $json.output }}`.82 83`userQuery` and `expectedAnswer` name **dataset columns** (the input the user84asked; the ground-truth answer). `actualAnswer` names a field of the workflow's85**produced output**. Write all of them as `={{ $json.<name> }}` — the evaluation86reads dataset columns from the dataset row and `actualAnswer` from the end node87automatically. Do not reference the trigger or any node by name.88 89### Expression fields must begin with `=`90 91`actualAnswer`, `userQuery`, and `expectedAnswer` are n8n **expressions** — they92read a value out of each test row at runtime. The leading `=` is what tells n8n93to evaluate the `{{ … }}` template. **Without it the string is stored as literal94text**: the field shows `{{ $json.output }}` verbatim and the judge scores that95raw string instead of the resolved value.96 97- Correct: `={{ $json.output }}`, `={{ $json.expected_output }}`98- Wrong: `{{ $json.output }}` (no `=` → treated as fixed text)99 100Only add `=` when the value references workflow data via `{{ … }}`. A genuinely101fixed constant (rare for these fields) is written as plain text without `=`.102 103Pick `correctness` when the dataset has a known right answer to compare against;104pick `helpfulness` when there is no single ground truth and quality is judged105relative to the request. Use `prompt` only to override the default judge prompt.106 107## Dataset Boundary108 109- Build the dataset with the `data-tables` tool: one column for each input the110  evaluation varies, plus a ground-truth column when using `correctness`.111- The config only references the dataset by `dataTableId`; the `eval-config` tool112  does not create or populate rows. If no suitable dataset exists, create one113  first, then create the config.114- Do not weaken the evaluation to fit a thin dataset — seed the dataset to match115  the metrics, or ask the user for the expected answers.116 117## More Detail118 119Use [references/config-eval-playbook.md](references/config-eval-playbook.md) for120tool-call recipes, worked examples, and output shapes.121