SKILL.md
SKILL.mdBrowse 4 files
13,992 tokens
57,747 bytes
Token encoding: o200k_base
Snapshot 8bff5da
1---2name: n8n:create-instance-ai-eval3description: >-4 Authors a new Instance AI workflow or Agent eval case — written locally as5 JSON, calibrated against a real build, then pushed to the LangTracer suite CI6 runs — build cases, behaviour/process cases, credential cases, and seeded7 (mid-conversation) cases — with intent-driven expectations. Use when adding8 or changing an Instance AI eval, or debugging why one is flaky.9---10 11# Create an Instance AI eval12 13Each eval is **one JSON case**. Author workflow cases in14`packages/@n8n/instance-ai/evaluations/data/workflows/`. Author standalone Agent15cases in `packages/@n8n/instance-ai/evaluations/data/agents/` and follow the16[`create-agent-builder-eval` skill](../create-agent-builder-eval/SKILL.md).17The disk loader auto-discovers `*.json` in both directories. A LangTracer suite18is the durable home. Cases validate against19[`harness/schema.ts`](../../../packages/@n8n/instance-ai/evaluations/harness/schema.ts)20(`.strict()` — unknown keys fail at load). The eval21[README](../../../packages/@n8n/instance-ai/evaluations/README.md) is the22exhaustive field reference; this skill is the opinionated *how*.23 24> **Committing new case JSONs into the repo is no longer the recommended25> approach.** Author the file locally (uncommitted), calibrate it against a real26> build, then **push it to a lang-tracer suite** with `eval:langtracer-push`27> (see [Push to a lang-tracer suite](#push-to-a-lang-tracer-suite)) —28> `--suite baseline` for the consolidated corpus n8n CI runs, or a dedicated29> capability suite like `agents`.30> The suite is the home for the case; the eval CLI reads it back via31> `--source langtracer`. You still write the JSON file — it's just the input to32> the push, not a committed artifact.33>34> **Seeded cases.** An `inline` seed pushes with the case — the case-write API35> stores it verbatim, so the suite is its home like any other case. Only a36> `seed.mode: "replay"` case is refused (listed under `skipped:`): it's37> reconstructed from a LangSmith trace at run time, so it dies when that trace is38> pruned and has no durable home. Don't commit a replay case either — derive a39> synthetic case from it. See [`case-shapes.md`](case-shapes.md).40 41## Set the autonomy level first42 43**Before you source, draft, or run anything, decide how hands-on the driver44wants to be — and say it back.** This skill runs at one of two autonomy levels.45If the request makes the level clear ("just author and calibrate it yourself" vs.46"stop me at each step", or an explicit mode), adopt it, state it in one line, and47note how to override (e.g. "say 'stop me at calibration' to add a checkpoint").48**If it's not clear, ask the driver one question** offering the two levels49*before* doing any work.50 51The skill has four natural decision **gates** — **selection** (which real52failure to encode), **shape + expectations** (archetype, must-haves, scope53trim), **calibration** (classify each red and resolve keep/loosen/drop), and54**push** (kind + tier). The level decides what happens at each gate:55 56| Level | Who decides when to stop | Behaviour |57|---|---|---|58| **autonomous** | agent | Runs all four gates start-to-finish; reports a **decision log** at the end for the driver to review — with the pushed case, its suite, and the source thread as **links** ([Share links, never bare ids](#share-links-never-bare-ids)); includes the observation id when the driver opted to record one; and includes a Linear ticket **proposal** for any kept capability-gap red ([Capability gap → propose a Linear ticket](#capability-gap--propose-a-linear-ticket)). |59| **checkpoint** | driver, per gate | Stops at each gate with a compact **proposal + recommendation**; driver says "go" or redirects. At the **calibration** gate, hands the driver a link to the just-built thread on the live instance plus login credentials so they can review the real conversation and workflow themselves before confirming (below). |60 61**Calibration is special-cased at both levels.** A calibration verdict that62flips a case's *meaning* — a real capability-gap red vs. a harness-caused red, or63any loosening that would let a known-bad build pass — is **surfaced explicitly**64(interactively in checkpoint; in the decision log in autonomous), never silently65committed. It's the one call where a quiet mistake corrupts the suite, so it66never fully auto-commits.67 68**Recording a source observation is optional, not a fifth gate.** After a real69thread passes selection, offer to save why it was selected and what the developer70observed with LangTracer's `create_observation` tool (see71[`sourcing-cases.md`](sourcing-cases.md#optional-record-the-selection-as-an-observation)).72If the driver declines, has not stated a preference in autonomous mode, or the73write fails, continue with the eval. Never block drafting, calibration, or push74on an observation. `create_observation` and `update_observation` write only to75LangTracer; do not add a LangSmith feedback or sync step.76 77**Checkpoint calibration — review the real thread on the instance.** Because the78calibration verdict is trust-critical, in checkpoint mode you don't ask the79driver to trust your reading of the run. You built the case against a live80instance with `--keep-workflows` (step 4), so the thread and the workflow are81still there — hand the driver a direct link and let them look:82 83- **Thread:** `<base-url>/assistant/<threadId>` — the exact conversation the case84 ran (the run prints the `threadId`; the built workflow prints as `BUILT (<id>)`85 and opens at `<base-url>/workflow/<id>`).86- **Login:** the email + password the eval signs in with (the owner you seeded on87 the instance — see [`running-evals.md`](running-evals.md); the default local88 seed is `nathan@n8n.io` / `PlaywrightTest123`).89 90Present, per red: the assertion, whether it went green/red, your proposed91classification (real capability gap / harness limitation / noise) and92keep/loosen/drop, and the review link. The driver logs in, reads the thread and93the workflow, and confirms or redirects before you write the verdict back into94the case `description`.95 96## Share links, never bare ids97 98Every lang-tracer entity has a shareable web page, but the CLI and the MCP hand99you **numeric ids** — `eval:langtracer-push` prints `+ created <slug> (#621)`,100`get_eval_run` returns a run number, `list_conversations` returns thread ids.101An id is unclickable: the driver has to go find it. **Whenever you name a case,102suite, thread, cluster, or run in anything a human reads** — a checkpoint103proposal, the end-of-run decision log, a PR description, a Linear ticket, a Slack104message — render it as a link, keeping the id in the label:105 106```107pushed as [#621](https://lang-tracer.n8n-maintenance.workers.dev/test-cases/621)108```109 110Build links off the **web base** (`LANGTRACER_URL`, in production111`https://lang-tracer.n8n-maintenance.workers.dev`). Never off the API bases —112`${LANGTRACER_URL}/api/v1` and `/api/mcp` are machine endpoints, and a link into113either 404s for the driver or dumps JSON.114 115| Entity | URL | Where the id comes from |116|---|---|---|117| Test case | `<base>/test-cases/<id>` | push output `(#<id>)`; `create_test_case` / `search_test_cases` |118| Suite | `<base>/suites/<suiteId>` | push header `Suite "<slug>" (#<id>)`; `list_suites` |119| Source conversation | `<base>/conversations/<threadId>` | `list_conversations` / `get_conversation` |120| Cluster report | `<base>/clusters/<id>` | `list_cluster_runs` / `get_latest_cluster_run` |121| Eval run (sweep) | `<base>/results?sweep=<runId>` | `list_eval_runs` / `get_eval_run` (`runId` *is* `sweeps.id`, the "run #N") |122 123Two links that are **not** lang-tracer and don't take this base: the built thread124(`<base-url>/assistant/<threadId>`) and workflow (`<base-url>/workflow/<id>`) live125on the **n8n instance** the eval ran against. When both are relevant — reviewing a126calibration red, writing a capability-gap ticket — give both, labelled, so nobody127has to guess which host a link points at.128 129## Where the best cases come from130 131The strongest cases encode a **real** failure, not an invented premise. Two132connections help you find and verify one: **LangTracer** clusters real133conversations into capability-gap themes (discover what actually fails, at134scale), and **LangSmith** holds the raw traces (verify exactly what happened in a135run). LangTracer is the discovery layer; the durable artifact is almost always a136synthetic case you author from what you learn (use `seed.mode: "replay"` only per137[`case-shapes.md`](case-shapes.md)). See138[`sourcing-cases.md`](sourcing-cases.md) for connecting the MCPs and the139discover → verify → encode workflow.140 141## Pick the case shape first142 143The corpus is four archetypes. Decide which you're writing before you draft — it144determines the fields, the grading, and how you validate. They compose (a seeded145case can still assert outcome), but the primary shape drives the work.146 147| Archetype | Question it answers | Primary fields |148|---|---|---|149| **Build** (default) | Does the workflow the agent builds actually *work*? | `outcomeExpectations` + `executionScenarios` |150| **Behaviour / process** | Does the agent *converse* correctly (ask the right clarifying question, not re-ask, honour a correction, respect plan approval)? | `processExpectations` + multi-turn director script; often **build-only** |151| **Credential** | Does the build behave correctly given a specific credential view? | `credentials[]` |152| **Seeded** | Start mid-thread, with prior work already in place, and drive the turn under test | `seed` (authored `mode: "inline"`; `"replay"` for a local check) |153 154**Build** is documented in full below. The other three, the director-script155vocabulary, and the seeding modes are in [`case-shapes.md`](case-shapes.md).156 157## Core principle (all shapes)158 159**Write expectations from intent, then calibrate against a real build.** Decide160up front what makes *any* correct solution correct — the must-haves implied by161what the user actually said — then build the workflow once for real to calibrate162granularity: loosen what's over-specified, confirm the must-haves are163achievable, and catch requirements the agent legitimately satisfies a different164way. Don't transcribe one observed build into assertions — that overfits the165eval into "did the agent reproduce that run" instead of "did it solve the166problem."167 168**Keep the conversation in the user's voice.** State the goal and real169constraints the way a user would — don't name node types, wire up the structure,170or restate your `outcomeExpectations` in the prompt. If the conversation spells171out the build, the case only tests whether the agent can follow instructions and172the expectations become tautological; the gap between what the user asks for and173how a correct workflow realizes it is the capability under test. Even when the174anchor *is* honoring a user's stated technical preference, phrase it as their175need + constraint ("I need field X and the built-in node doesn't expose it, so176pull it straight from the API") — not as an implementation spec ("use an HTTP177Request node").178 179**Write the conversation in English** unless the user asked otherwise (or the180case exists specifically to test non-English handling). Sourced real threads are181frequently non-English — translate the intent into English when you rewrite the182prompt in the user's voice; the failure mode is the anchor, not the original183language.184 185**Trim to the smallest multi-turn conversation that reproduces the issue.**186Real sourced threads are long (dozens of turns of setup, debugging, and187tangents) — do **not** transcribe them. Distill to the fewest turns that still188drive the build or behaviour under test. Every retained turn must earn its place:189a turn stays only if it is *load-bearing* — a value the agent must ask for190(withheld until asked, via a director note), a correction/push-back the case191exists to test, or a plan approval that gates the build. If removing a turn192doesn't change what's tested, remove it. **Collapse to a single turn** whenever193the whole request can be stated at once without a load-bearing exchange; keep it194multi-turn *only* for those exchanges, and keep each director script in one turn195(don't fabricate assistant "done" turns to sequence steps — see196[`case-shapes.md`](case-shapes.md)). A minimal conversation isolates the197capability; a transcribed one buries it in noise and tests instruction-following.198 199**Size the build, not just the assertions.** Real sourced prompts are often200kitchen-sink ("production-ready, runs forever, 3 feed posts *and* 8 stories a201day", "generate 50 articles daily") and reliably blow the ~900s build budget (see202"Known harness limitations"). A **faithful trim is a legitimate authoring move**:203reduce batch sizes, drop one of several parallel pipelines, or merge adjacent AI204steps so the case builds within budget — then note the reduction in the case205`description` ("the original request also asked for an 8-stories/day pipeline;206scoped to feed posts so it builds in budget"). Keep the capability under test; cut207the combinatorial bulk. A case that never builds tests nothing.208 209## Workflow210 211These steps map to the four gates from [Set the autonomy level first](#set-the-autonomy-level-first):212sourcing (before step 1) is the **selection** gate, including the optional offer213to record a source observation; steps 1–2 are the **shape +214expectations** gate; steps 5–6 are the **calibration** gate; steps 7–8 are the215**push** gate. In *autonomous* mode you flow through all of them and summarize in216a decision log; in *checkpoint* mode you pause at each with a proposal, and at217calibration you hand the driver the thread link + login to review the real build218(see [Set the autonomy level first](#set-the-autonomy-level-first)). Calibration219(step 6) always surfaces meaning-flipping verdicts explicitly regardless of level.220 2211. **State the must-haves first.** From the conversation alone, list what every222 correct workflow must do (trigger type, essential operations, gating223 condition) — those become draft `outcomeExpectations`. Required fields:224 `conversation` (≥1 turn, first `user`), `complexity`, `tags`, and **at least225 one** of `executionScenarios` / `processExpectations` / `outcomeExpectations`.2262. **Draft the case** from the template below; validate it loads (see227 "Validate").2283. **Smoke-test the *environment* with one case before any batch.** Run a single229 case end-to-end first. This validates auth / model / `--base-url` / the built230 dist for ~1/Nth the cost — distinct from validating a *case*. If that one case231 crashes at execution (especially with an identical error you'd expect to hit232 every case), fix the environment before running the batch (see "A red is233 signal" → environment check). Running 15 cases only to discover a stale-dist234 crash on all of them wastes a full run.2354. **Build it once** against a running instance (see236 [`running-evals.md`](running-evals.md)) with `--keep-workflows` so the built237 workflow stays for inspection.2385. **Inspect** — read the built workflow (the run prints `BUILT (<id>)`; fetch239 via `GET /rest/workflows/<id>`) and the HTML report's transcript to see what240 the agent actually did.2416. **Calibrate — sharpen assertions; never dull them to force a green.** Fix242 assertions that are genuinely mis-sized: relax one that pins a choice the243 conversation left open (so a valid *alternative* build wrongly fails), tighten244 one a wrong build would slip past, and phrase `executionScenarios` to match how245 the workflow runs on mocked data. But when a scenario goes red because the246 build has a real gap, or because the harness can't exercise it, **that red is247 the result — keep it and surface why** (see "A red is signal", below). Never248 delete a scenario, weaken an assertion, or drop to build-only just to make the249 run green. And when the case comes back **green**, that is a result to earn,250 not to accept: confirm the precondition actually fired, then re-derive it from251 the raw thread before calling the case a regression guard (see252 [First reproduce, then reclassify](#first-reproduce-then-reclassify)).2537. **Push to the suite — do NOT commit the JSON.** Once calibrated, push the case254 into its curated lang-tracer suite with `eval:langtracer-push` (see255 [Push to a lang-tracer suite](#push-to-a-lang-tracer-suite)); the suite is the256 case's home, not the repo. Leave the `data/{workflows,agents}/*.json` file uncommitted257 (or delete it once it's in the suite). Committing new case JSONs into the repo258 is no longer the approach. (An `inline` seed pushes with the case; only a259 `replay` case is refused — it's a local throwaway; see260 [`case-shapes.md`](case-shapes.md).) For a sourced case,261 finish by **linking it to its source thread/finding** over the MCP — see262 [Link the pushed case to its source](#link-the-pushed-case-to-its-source-provenance-step--always-do-this).2638. **Hand back links, and a ticket proposal if the case found a gap.** Report the264 pushed case as `<base>/test-cases/<id>`, not `#<id>` ([Share links, never bare265 ids](#share-links-never-bare-ids)), and if calibration kept a real266 capability-gap red, propose a Linear ticket for it ([Capability gap → propose a267 Linear ticket](#capability-gap--propose-a-linear-ticket)) rather than leaving the268 gap as a red case nobody owns.269 270`--iterations N` is available to measure flakiness (pass@k / pass^k) — reach for271it when you suspect a case is non-deterministic or before promoting it to a272gated tier, not as a routine step (each iteration is a full build + execution).273 274Gut-check: if you can't picture a plausible *wrong* build that this case275reliably turns **red**, the assertions are too loose to guard anything.276 277**Confirm the precondition fired, not just the green.** For any *conditional*278assertion — "when X happened, the agent did Y" (most `processExpectations`, and279any behaviour case) — a pass has two readings: the agent did Y, or **X never280happened** and the assertion passed vacuously. A behaviour case that hinges on281the mock producing a specific failure (e.g. an AI node simulated to empty so a282downstream parse node fails) is the classic trap: if the mock instead returns283parseable data, the failure never occurs and the case guards nothing while284showing green. Calibration must read the execution trace and the agent's285`finalText` (`buildTrace.finalText` in the verifier snapshot, or the HTML report)286and verify X actually materialised — the direct-loop `eval-results.json` does not287persist per-expectation judge reasoning, so pass/fail alone can't tell you which288reading you got.289 290**The negative form is the easiest to fool yourself with.** An assertion phrased291as "the agent did NOT call `X` with a bad argument" passes when the agent called292`X` correctly *and* when **it never called `X` at all**. Those are opposite293results and the judge reports the same green. So for any assertion about tool294misuse, confirm the tool was actually invoked before believing the pass: parse295`testCases[].transcriptPerRun[][].steps[]` for the call. Note the transcript296groups multi-action tools under a bare `toolName` (`nodes`, `workflows`,297`credentials`), so read `args.action` to get the real one — filtering on298`nodes[explore-resources]` finds nothing and looks like a clean pass. Measured on299this corpus: a batch of five tool-misuse cases scored 100% on its first300calibration run, and three of them were passing vacuously because the tool under301test was never called.302 303**A sourced failure that does not reproduce is not yet a regression guard — first304re-derive the precondition.** Behaviour does drift across versions, and a case305that flips from *capability-gap* (red) to *regression guard* (green, catches a306re-introduction) is a legitimate and valuable outcome. But reach it by307elimination, not by default: a green far more often means *your case never set up308the situation* than *the builder improved*. See below.309 310## First reproduce, then reclassify311 312A case built from a real failure that comes back green is the most common313outcome of a first calibration run, and "the build must have improved" is the314most common wrong conclusion. The usual cause is that you authored from a315*summary* of the thread — the theme label, the observation description, your own316one-line note — and the trigger you assumed is not the trigger that fired. Go317back to the raw thread before you downgrade anything.318 319**Find the turn, not the topic.** Locate the exact tool call that failed, then320read the assistant text immediately before it. The agent usually states its321intent in the open, and that sentence is the precondition. Then ask what *state*322made that call necessary — not what the conversation was about.323 324Worked example from this corpus. Sourced finding: "the agent invents325`nodes[explore-resources]` method names." Assumed trigger: the user swaps model326provider. A case built on a clean provider swap came back green — the agent set327the model id directly and never called the tool at all. The raw thread said it328plainly:329 330> "The **Groq Chat Model** has an invalid model (`llama3-8b-8192` isn't offered331> by your Groq credential). **Let me list valid models and fix it.**"332 333The precondition was never the swap. It was *an existing model id that the334provider rejects at runtime, with a credential already connected* — that is the335state that makes enumerating models necessary. Rebuilt on it, the same case336reproduced the failure on the first run, with the agent inventing two method337names in a row.338 339Three moves turn a non-reproducing case into a reproducing one. Try them in340order before settling for a guard:341 3421. **Fix the precondition.** Rebuild the seed and the live turn to recreate the343 state the source thread was in, not the subject it was discussing.3442. **Move the assertion to the first call.** A mechanism where the agent345 *self-corrects* grades green on the end state and is still a real defect — the346 wasted round-trip and the guessed schema are the finding. Grading first-call347 correctness turned a mechanism previously dismissed as "self-corrects, weak348 eval" into a gap that reproduced in 2 of 2 runs.3493. **Keep what the attempt actually caught.** A reproduction run often reds on a350 *different* real defect than the one you targeted. That is still a351 capability-gap finding — keep the red, retarget the description, and say352 plainly in it that the originally targeted mechanism did not reproduce.353 354**Know when to stop.** Some mechanisms are structurally unreachable in this355harness, and no amount of re-deriving fixes that. The clearest example: a seed356restores a **fresh** workspace file at the start of the graded turn, so the357agent's first `old_str` always comes from a file it just read. Failures that need358*accumulated drift* across many turns — the `str_replace` byte-fidelity family,359the largest agent-caused tool-call failure in production at 15% of threads — do360not reproduce even in a multi-turn chain of edits, each followed by a build. Cap361the effort at about three attempts, then write the negative result into the case362`description` as the finding it is, and ask whether the real defect belongs in a363ticket rather than an eval.364 365## A red is signal — surface it, don't work around it366 367Calibration exists to right-size assertions, **not** to make a case pass. When a368run turns a scenario or expectation red, classify the red first — then keep it.369 370**First rule out the environment.** Before reading any red as a signal about a371case, check the shape of the failures across the run. If **every case fails the372same way** — every scenario with the *same* execution error while builds succeed,373*or* every **build** erroring identically before it starts (an `Agent error:374Something went wrong…` with **zero tool calls**) — that is almost never the cases;375it's a broken environment, most often a **stale dist** after a branch or worktree376switch. Two shapes to know:377 378- **Stale `packages/core` / `packages/cli` dist** — a refactor moved a runtime379 export and the built dist still calls the old one, so *builds succeed but every380 execution fails the same way* (e.g. `(0 , n8n_workflow_1.createDeferredPromise)381 is not a function` after `createDeferredPromise` moved to `@n8n/utils`).382- **Stale/half-built `@n8n/instance-ai` dist** — every run errors *before building*383 (`Agent error…`, zero tool calls) and the instance log shows `Cannot find module384 '@/utils/...'` from `dist/skills/*.js`: the build's `tsc-alias` step (which385 rewrites `@/` path aliases to relative requires) didn't complete, so the dist is386 internally inconsistent.387 388- **Out-of-sync `node_modules`** — `pnpm build` itself dies early with `Cannot find389 module '@n8n/<pkg>'` even though that package is a declared `workspace:*`390 dependency *and* has a `dist/`. The workspace symlink is missing from the391 consumer's `node_modules` (typical after a branch or worktree switch). Confirm392 with `ls -d packages/<consumer>/node_modules/@n8n/<pkg>`; fix with a plain393 `pnpm install` — no need for the heavier `pnpm reset --full`.394 395Fix it, don't calibrate around it: run a full ordered `pnpm build` (a targeted396`--filter` build can fail on unrelated stale-dep type errors; for the instance-ai397shape, `cd packages/@n8n/instance-ai && pnpm build` runs `tsc && tsc-alias`), then398**restart the instance** — the running node process holds the old dist in memory,399so rebuilding on disk changes nothing until restart (and `kill` by env-var pattern400misses it — kill the actual `lsof -t -iTCP:<port>` PID). Re-probe one case, confirm401it builds and executes, then re-run the batch. Only once uniform environment402failures are excluded do the three categories below apply:403 404- **Real build / capability gap** — the agent's workflow is wrong or missing405 something the user asked for (a miswired branch, a missing retry, wrong field406 keys). This is exactly what the eval is for. **Keep it red.** Don't loosen the407 assertion or drop the scenario; a currently-red gap is the capability signal408 today, and a re-introduction guard once the builder improves. Then **propose a409 Linear ticket** for the gap — see [Capability gap → propose a Linear410 ticket](#capability-gap--propose-a-linear-ticket).411- **Harness limitation** — the build is correct but the mock/execution layer412 can't exercise the path (see "Known harness limitations", below). **Keep the413 scenario and say so in its `description`** — that this red is harness-caused,414 not a build defect — so nobody misreads it as a product bug. Keep it out of415 gated tiers if it hard-fails every run; when the harness gains the capability it416 starts earning its keep with no re-authoring.417- **Genuine non-determinism** — the *same* build flips green/red across runs.418 This is the only real "noise". Confirm it with `--iterations N` before calling419 it flaky, then de-tier and note it; deletion is the last resort.420 421**Annotate every kept red in the case `description` with a scannable prefix** so a422future reader tells the two apart at a glance. Use `Harness note: …` for a423harness-caused red (name the limitation and why the build is still correct), and424`Capability-gap finding: current build reds because <X> — a real builder bug425(flips to a regression guard once fixed)` for a real gap. Consistent prefixes keep426the corpus greppable and stop harness reds from being misread as product bugs.427 428The one move to never make is **working around a red by weakening what the case429checks** — deleting a failing scenario, loosening an assertion until a wrong430build would pass, or quietly converting to build-only. That makes the suite look431greener than the product is, which is the opposite of the eval's job: bugs and432harness gaps are the deliverable, so **highlight them, don't engineer around433them**. If you catch yourself editing a case so that a known-bad build would now434pass, stop.435 436**Who confirms the classification depends on the autonomy level.** In437*checkpoint* mode the keep/loosen/drop decision is the driver's to confirm: you438hand them the thread link + login (see [Set the autonomy level439first](#set-the-autonomy-level-first)) so they can review the real conversation440and workflow, then you write the agreed `Harness note:` / `Capability-gap441finding:` prefix back into the case `description`. In *autonomous* mode the agent442proposes it explicitly in the end-of-run decision log. Either way the443classification is stated in the open, never silently committed — misreading a444harness red as a real gap (or the reverse) is the one calibration mistake that445quietly corrupts the suite.446 447### Capability gap → propose a Linear ticket448 449A kept capability-gap red is a **product bug you just characterised better than450any bug report would**. But a red case in a suite doesn't assign itself to anyone:451without a ticket the gap sits in CI as permanent noise, and the next person to452read the run assumes someone already owns it. So once a red is classified as a453real gap (and the driver has confirmed it, per the autonomy level), **propose a454Linear ticket for it.**455 456**Propose, don't create.** Per [AGENTS.md](../../../AGENTS.md), never open a457Linear ticket unasked. Put the draft in front of the driver — interactively in458checkpoint mode, in the decision log in autonomous mode — with a title, a team,459and the body, and let them say go. Skip the proposal in two cases:460 461- **The gap already has a ticket.** Check the case's linked issues on its page,462 and run `get_linear_ticket_context <TEAM-N>` on any candidate identifier the463 driver or the source thread mentions, before you draft a duplicate.464- **The red isn't a capability gap.** A `Harness note:` red is a465 lang-tracer/harness issue, and genuine non-determinism is a case-hygiene chore.466 Neither belongs in the builder's queue.467 468The draft body should carry what makes the gap actionable, all of it already in469hand from calibration:470 471- **The eval case**, as a link — `<base>/test-cases/<id>` (see [Share links, never472 bare ids](#share-links-never-bare-ids)). This is the reproducer; it's the most473 valuable line in the ticket.474- **What failed, verbatim** — the failing `outcomeExpectation` /475 `processExpectation` or scenario name, plus the judge's stated reason. Not a476 paraphrase: the exact text is what the fixer will grep for.477- **What the build did instead** — the specific defect (miswired branch, wrong478 field key, missing gate), and links to the real evidence: the source conversation479 (`<base>/conversations/<threadId>`) and the built thread + workflow on the eval480 instance (`<base-url>/assistant/<threadId>`, `<base-url>/workflow/<id>`).481- **Blast radius, if you know it** — the cluster theme or the number of real482 conversations behind the gap (`<base>/clusters/<id>`) is what turns "one red483 case" into a prioritisable bug.484 485**File it from the case page so the link is made.** The `<base>/test-cases/<id>`486page has a *Create Linear issue* dialog that creates the ticket **and** links it to487the case; that link is what makes `get_linear_ticket_context <TEAM-N>` later return488the case, its scenarios, the source conversation, and its analysis in one call. The489case↔ticket link is only writable from that UI — lang-tracer's MCP and `/api/v1`490are read-only for it — so if the ticket gets created some other way (a Linear MCP,491if your harness has one, or Linear directly), say plainly that it isn't linked, ask492the driver to link it on the case page, and meanwhile put the identifier + URL in493the case `description` via `update_test_case` so the provenance isn't lost.494 495Then extend the description prefix with the ticket, so the corpus stays greppable496in both directions: `Capability-gap finding: current build reds because <X> — a497real builder bug (flips to a regression guard once fixed). Tracked in498[<TEAM-123>](<ticket url>)`. And when the *build itself* is wrong — not just a499scenario red under a correct build — push it with `--set-kind capability_gap` into500a suite of that kind (see501[Push to a lang-tracer suite](#push-to-a-lang-tracer-suite)).502 503## Example504 505Minimal build case:506 507```json508{509 "description": "What this case tests.",510 "conversation": [{ "role": "user", "text": "<the build prompt>" }],511 "complexity": "medium",512 "tags": ["build", "<nodes>", "<concepts>"],513 "triggerType": "schedule",514 "outcomeExpectations": ["<a must-have any correct workflow satisfies>"],515 "executionScenarios": [516 {517 "name": "happy-path",518 "description": "<what this run exercises>",519 "dataSetup": "<what the external services return>",520 "successCriteria": "<observable proof the run succeeded>"521 }522 ]523}524```525 526A fuller case with a multi-turn director script (withhold a value until asked,527push back on a wrong plan):528 529```json530{531 "description": "Scheduled GitHub-bugs digest to Slack. Repo and channel are withheld until the agent asks; the plan must filter to the 'bug' label before it's approved.",532 "conversation": [533 { "role": "user", "text": "Every weekday at 9am, fetch this week's open bugs from our GitHub repo and post a short summary to Slack." },534 { "role": "assistant", "text": "Which repo and which Slack channel should I use?" },535 { "role": "user", "text": [536 "[Withhold the repo and channel until the agent asks; then say the repo is 'acme/widgets' and the channel is '#eng-bugs'.",537 "When the agent shows a plan or setup card, reject it unless it filters issues to the 'bug' label — a digest of ALL issues is wrong. Once it filters to bugs, approve.]"538 ] }539 ],540 "messageBudget": 8,541 "complexity": "medium",542 "tags": ["behaviour", "build", "schedule", "http-request", "slack"],543 "triggerType": "schedule",544 "processExpectations": [545 "The agent asked for the repo and Slack channel before building, since the prompt named neither.",546 "The agent's final plan filtered issues to the 'bug' label — if its first attempt didn't, it corrected after the user pushed back rather than summarizing all issues."547 ],548 "outcomeExpectations": [549 "A Schedule Trigger runs the workflow on a recurring weekday-morning cadence.",550 "Open issues are fetched from GitHub (HTTP Request or GitHub node) and filtered to the 'bug' label before the summary is built.",551 "One Slack message summarizing the fetched bugs is posted to the #eng-bugs channel the user gave."552 ],553 "executionScenarios": [554 {555 "name": "posts-bug-digest",556 "description": "Three open bugs are returned; a summary is posted to Slack",557 "dataSetup": "The GitHub issues request returns three open issues labelled 'bug' ('Login 500', 'Timezone off by one', 'CSV export truncates'). The Slack postMessage call returns { \"ok\": true, \"ts\": \"1700000000.0003\" }.",558 "successCriteria": "The run completes without errors and posts one Slack message to #eng-bugs that references the three fetched bug titles."559 }560 ]561}562```563 564What each piece is doing:565 566- **`conversation[0]` is sent to the builder raw.** The opening turn is the real567 prompt — never put a `[director note]` in it (it would leak verbatim).568- **The `[bracketed]` turn is a director script** for the user-proxy — behaviour,569 never spoken. Here it withholds values until asked and rejects a plan that570 misses the label filter. Keep the whole script in one turn and encode ordering571 inside it (don't fabricate assistant "done" turns to sequence steps — see572 [`case-shapes.md`](case-shapes.md)). `applies-each-change-when-asked` (in the573 `baseline` LangTracer suite) is a good real example.574- **`dataSetup` describes only what external services return.** That's the layer575 the harness controls (below).576 577## `dataSetup` and the mock layer578 579The harness mocks by **intercepting outbound HTTP requests to external services**580and having an LLM answer them from the node's config and API docs. It does **not**581let you set a node's output directly, and it does **not** mock n8n internals582(Code/Set/Merge/IF/Switch run for real on the mocked data; triggers and DB583nodes get LLM-generated pin data). So:584 585- Write `dataSetup` as **what each external service returns** ("the GitHub586 request returns three issues labelled bug"), not as node outputs or internal587 state.588- The strongest scenarios exercise **external-service responses** — that's what589 the harness reproduces most faithfully.590- **Data Table *reads* are pinned to the scenario.** A read op (`get` /591 `rowExists` / `rowNotExists`) is treated as the scenario's "stored state" and592 pinned with data derived from your `dataSetup`, so change-detection / dedup /593 "last seen" scenarios *can* be exercised — describe the stored rows in594 `dataSetup`. Two caveats: the pinned rows are LLM-generated (steered, not595 byte-exact — don't assert exact values off them), and *writes/inserts* aren't596 pinned (they hit the real per-thread table, recreated schema-only with **no597 rows**), so read-after-write within one run isn't faithful — the read reflects598 `dataSetup`, not what the run just wrote. A third caveat: only Data Table599 *reads* are seedable this way — **dedup / change-detection built on workflow600 static data** (`removeItemsSeenInPreviousExecutions`, `$getWorkflowStaticData`)601 is **not** seedable, because static data starts empty every run, so such a602 scenario reds vacuously (it sees everything as "new"). To get a seedable603 change-detection scenario, steer the build toward a Data Table; otherwise604 accept the static-data red as a harness limit and carry the logic in605 `outcomeExpectations`. Note the agent may *choose* static-data dedup on its own.606- Don't assert exact counts that depend on mock generation ("exactly 7 posts").607 Say "fewer than the original 10".608 609### Known harness limitations that turn scenarios red regardless of the build610 611These produce a **reliable** red on a *correct* build. Don't engineer around them612— write the scenario for the behaviour you want and note in `description` that the613red is harness-caused (per "A red is signal", above):614 615- **Resource-locator fields left empty for setup** (Google Sheets / Drive /616 Calendar and similar node pickers). The agent legitimately leaves the617 document/folder/calendar ID blank for the user to pick at setup; the mock618 substitutes `__evalMockResource`, and the node then crashes looking it up619 ("Sheet with ID __evalMockResource not found", or "Cannot read properties of620 undefined"). Any scenario whose success path runs *through* such a node621 hard-fails before anything downstream executes.622- **Trigger and Data-Table-read pin data is *LLM-generated*, so not byte-exact.**623 Both are steered by your `dataSetup` (see the mock-layer section above — you624 *can* influence what a trigger emits or what a stored-row read returns), but625 because the values are generated, a scenario that asserts exact values or counts626 off them is flaky. Assert shape/branch/relative facts, not exact figures. (The627 residual hard red here: polling / form triggers still occasionally fail to load628 entirely — "workflow not found".)629- **Mock response shape** — the LLM-generated mock response can omit the real630 envelope, crashing a downstream parse/format node. Recurring, reproducible631 shapes to expect (all produce a red on a *correct* build):632 - **OpenAI structured output** — historically the mock returned a plain633 `{content: "..."}` instead of the Responses envelope634 (`output[0].content[0].text`), so a **Structured Output Parser** /635 **Information Extractor** / **Text Classifier** got nothing and errored with636 **`Model output doesn't fit required format`**. The Responses-envelope637 normalizer (PR #33578, merged) fixes the flat-envelope case, so many of these638 now execute cleanly. A **narrower residual red remains** for structured-output639 schemas declared with strict **`additionalProperties: false`**: the normalized640 `output` wrapper (and any extra fields the mock invents, e.g. `subject`/`date`)641 violate the strict schema, so the node still rejects the mock output. Both the642 old and residual forms are the *mock*, not the build — carry correctness in643 `outcomeExpectations` and note the red as harness-caused.644 - **Gmail** mock returns headers as top-level capitalized fields (`From`,645 `Subject`) instead of under `payload.headers`, so a Code/Filter node reading646 the sender/subject gets empty strings (e.g. a "drop no-reply senders" safety647 gate lets everything through). Assert the *wiring/ordering* of such a gate in648 `outcomeExpectations`, not its runtime effect in a scenario.649 - A less-common API (e.g. Gemini's top-level `candidates`) can omit its envelope650 the same way.651 - **Google Drive resumable upload** — the initiate-upload mock omits the652 `Location` header carrying the session URL, so a Drive file-upload node fails653 with a 400. Any build that uploads a generated image/file to Drive can red on654 this.655- **Agent-tool nodes can't be executed standalone.** An AI-Agent *tool* node656 (`toolHttpRequest` and other `supplyData`-only LangChain nodes with no `execute`657 method) only runs when the agent invokes it; the harness executing it directly658 fails with `has a "supplyData" method but no "execute" method`. A near-universal659 red for chat-trigger / AI-agent build cases whose tool is an HTTP-request tool —660 the build is correct, so carry correctness in `outcomeExpectations` (agent wired661 to trigger + model + tool) and note the execution red as harness-caused.662- **Poll/wait loops can't be fast-forwarded.** A workflow that submits an async663 job then polls for completion (generate → poll status until ready → download)664 can't advance the mocked status deterministically, and a `Wait` node runs in665 real time, so the scenario reds with an execution timeout (`framework_issue`).666 The build is correct — carry it in `outcomeExpectations` and note the red as667 harness-caused.668- **A build can time out and produce no scored result at all** — the run reports669 `BUILD FAILED: Run timed out` and zero graded expectations. Don't assume "spec670 too big": the more common cause is a **single-prompt case where the agent asks671 a clarifying `ask-user` question** and the build hangs on the unanswered672 question until the per-iteration timeout (see [`case-shapes.md`](case-shapes.md)673 — only confirmations auto-approve). Before treating a timeout as spec size,674 **classify it**: read the agent's final response in the report / trace (did it675 ask a question? flag an infeasibility? or genuinely churn through a huge676 build?), and **re-run the case solo (`--concurrency 1`)** — concurrency both677 masks a stalled build (it hits the cap) *and* can time out a perfectly healthy678 build purely by queueing it behind the per-instance build cap (default 4), so a679 solo run either surfaces the real reason in seconds or simply passes outright.680 Fix per cause: a solo run that now passes → it was **concurrency contention**,681 not the case (split big batches across lanes — see682 [`running-evals.md`](running-evals.md)); a clarifying-question stall → author683 multi-turn with a director note that pre-answers it; a genuine infeasibility → it's an684 infeasibility/honesty behaviour case (`processExpectations`), not a build case;685 a true oversized spec → the timeout is itself a finding, but note it so the686 zero isn't mistaken for a scored failure.687 688## outcomeExpectations vs processExpectations689 690Both are natural-language assertions graded by the same Sonnet judge, and each691**counts as a pass-rate unit**. They judge different surfaces:692 693- **`outcomeExpectations`** — the **resulting workflow**, judged from the694 workflow JSON. Assert node choices and configuration, connection topology and695 branch wiring, data/expression references, trigger cadence, gating conditions.696 They run everywhere, including prebuilt/MCP runs (no transcript needed).697- **`processExpectations`** — **how the agent behaved during the build**, judged698 from the transcript. Assert clarifying questions asked (or not re-asked),699 tool-call behaviour, plan/approval handling, batching, honouring a correction,700 ordering. They need a transcript, so they're **skipped in prebuilt/MCP runs**.701 702Rule of thumb: an assertion about *the artifact* is an outcome expectation; an703assertion about *the conversation or the agent's choices along the way* is a704process expectation. A case with **no** `executionScenarios` is a valid705**build-only** case, graded by these expectations plus the workflow checks.706 707## Sizing each assertion708 709Right-size against **what the agent was actually told**. An assertion is710well-sized when every correct build passes it and a wrong or lazy build fails711it — and it holds the agent only to what the conversation specified, not to one712run's arbitrary choices. Two failure modes:713 714- **Too tight** — pins a choice the conversation *left open*. If the prompt never715 named a vendor, "calls flightaware.com" fails a valid build that used a716 different source. **But if the conversation specified it, pin it** — when the717 user said "email me via Gmail," "sends via a Gmail node" is correct and718 *required*, not too tight.719- **Too loose** — a non-solution would also pass. "Fetches data from somewhere"720 passes a workflow that fetches but never compares — it doesn't prove the721 change-detection the prompt asked for.722 723Quick check — the *substitution test*: would a reasonable alternative724implementation *of what the user asked for* still pass? Examples (flight-status725case, where the source and channel were **left unspecified**):726 727| Verdict | Assertion | Why |728|---|---|---|729| ❌ too tight | "Has an HTTP Request node calling `flightaware.com`" | Vendor was unspecified; a valid AeroDataBox build fails. (If the user *had* said "scrape FlightAware", this would be correct.) |730| ❌ too tight | "Publishes via HTTP Request nodes" | Pins the *transport* when a first-party node is the idiomatic path — e.g. the Facebook Graph API node is the correct way to reach the Instagram Graph API, so a valid build using it fails. Assert the capability ("publishes to Instagram via the Graph API, through the Facebook Graph API node or HTTP Request"), not the mechanism. |731| ❌ too loose | "Fetches flight data from somewhere" | A workflow that fetches but never compares passes — doesn't prove change-detection. |732| ✅ right | "Persists the previously-seen status and compares it to the freshly-fetched one" | The defining behaviour; substitution-proof across vendors and storage choices. |733| ✅ right | "Alert is sent only on the change-detected branch, gated by a conditional" | Proves the gate without pinning node or channel. |734 735Put intent the conversation only *implied* (a preferred but unstated channel) in736`processExpectations`, not `outcomeExpectations`.737 738## Robust design vs harness flakiness739 740Two different things — keep them apart:741 742- **Robust assertion design (always do this).** The agent's unspecified choices743 vary run to run. Source-agnostic `outcomeExpectations` for an unspecified744 source aren't a concession to flakiness — they're the *correct* assertion.745- **Harness limitations (surface them, don't hide them).** Some paths hard-fail746 on a correct build regardless of `dataSetup` — empty resource-locator fields747 that crash Sheets/Drive/Calendar nodes, polling triggers failing to load (see748 "Known harness limitations" above). (State-bearing Data Table *reads* are no749 longer in this bucket — they're pinned from `dataSetup`; only the write path and750 exact-value assertions stay unreliable.) The fix is to *document*, not to *work751 around*: note the limitation in `description` and keep a hard-failing scenario752 out of gated tiers.753 Only when a scenario flips **non-deterministically** run to run is it genuine754 noise worth removing — a scenario that reliably fails for a documented harness755 reason is a standing record of what the harness can't yet test, and stays.756 757## Negative execution scenarios758 759Don't stop at the happy path — but only assert graceful handling the prompt760actually implied. Most agent-built workflows don't add error handling by761default, so "the workflow crashes on bad input" is a legitimate builder finding,762not a test-case bug. Where graceful handling *is* expected, phrase763`successCriteria` as the *absence* of the wrong action ("no alert is sent", "run764completes without error") as much as the presence of the right one: empty /765not-found source, source error / timeout, malformed response.766 767## Relationship to the always-on workflow checks768 769Every successful build is also graded by ~28 always-on binary checks across 7770dimensions (structure, topology, parameter correctness, intent, AI wiring,771craftsmanship, security) —772[`binaryChecks/checks/`](../../../packages/@n8n/instance-ai/evaluations/binaryChecks/checks).773Those are broad and low-visibility. **Writing a targeted expectation for your774specific case is still worth it even when a binary check nominally covers it** —775a named case-level assertion gives far better visibility into *this* behaviour776than one row buried in a 28-check rubric. Don't skip an assertion just because a777generic check exists.778 779When a scenario fails, the verifier tags a **failure category** (`builder_issue`,780`mock_issue`, `framework_issue`, `verification_failure`, `build_failure`). Treat781it as a **hint, not ground truth** — we've seen a genuine node misconfiguration782tagged `mock_issue`, and a real mock problem tagged as a build error. Open the783HTML report and check the actual execution and the generated workflow before784concluding whether the failure is your case, the build, or the harness.785 786## Outputs of a run787 788- **`workflow-eval-report.html`** (in the run's `.data/` dir) — the highest-value789 view: full conversation transcript with tool calls, per-node execution traces,790 the exact intercepted requests and the mock responses, Phase-1 hints, verifier791 reasoning, and the workflow-check rubric. Human-oriented; start here when792 debugging.793- **`eval-results.json`** — structured results (the machine-readable artifact;794 the direct loop produces this even with no LangSmith). Good for an LLM or795 script to parse. **For per-case attribution under concurrency, parse this, not796 the streamed verbose log** — with more than one lane the log lines interleave797 across cases, so a `[scenario] FAIL` line in the stream can't be reliably tied to798 its case. Authoritative fields: `testCases[].buildSuccessCount`,799 `buildExpectationResultsPerRun[][].{pass,reason}`, and800 `scenarios[].runs[].{passed,failureCategory,rootCause,execErrors}`.801- **`eval-pr-comment.md`** — the rendered PR comment (aggregate + regression802 comparison), always written.803 804## Validate (before running)805 806```bash807cd packages/@n8n/instance-ai808pnpm exec tsx -e "import {loadWorkflowTestCasesWithFiles} from './evaluations/data/workflows/index.ts'; console.log(loadWorkflowTestCasesWithFiles('<slug>')[0].fileSlug)"809 810# For a standalone Agent case:811pnpm exec tsx -e "import {loadAgentEvalTestCasesWithFiles} from './evaluations/data/agents/index.ts'; console.log(loadAgentEvalTestCasesWithFiles('<slug>')[0].fileSlug)"812```813 814## Push to a lang-tracer suite815 816Once a case is calibrated, push it (and any others) up into a curated lang-tracer817suite instead of committing the JSON. `eval:langtracer-push` **upserts** over the818REST API: it creates cases missing from the suite, updates ones whose content819drifted, leaves the rest unchanged, and never prunes. It's the inverse of820`--source langtracer` (which pulls a suite down).821 822```bash823cd packages/@n8n/instance-ai824# preview first — no writes:825pnpm exec dotenvx run -f .env.eval -- pnpm eval:langtracer-push --suite baseline --dry-run --changed826# then push (drop --dry-run):827pnpm exec dotenvx run -f .env.eval -- pnpm eval:langtracer-push --suite baseline --changed828```829 830- **Selectors** (at least one required — no accidental push-all): positional831 `<slugs...>` (exact file slugs), `--changed` (new/untracked + staged + modified832 `data/{workflows,agents}/*.json`, ideal right after authoring an uncommitted case),833 `--filter`/`--tier` (with `--exclude` as a modifier).834- **Multiple positional slugs? Skip pnpm — call the script directly.** `pnpm835 eval:langtracer-push … slugA slugB` forwards the slugs as one joined argument836 (`"slugA slugB"`), so no case file matches and nothing is pushed. Either use a837 no-positional selector through pnpm (`--changed`), or run the script directly so838 each slug is its own argv: `pnpm exec dotenvx run -f .env.eval -- pnpm exec tsx839 evaluations/cli/langtracer-push.ts --suite <slug> <slug1> <slug2> …`.840- **Env:** `LANGTRACER_URL` + `LANGTRACER_API_KEY` (an `lt_` bearer; one key works841 for MCP + REST) — put them in `.env.eval` and run under `pnpm exec dotenvx`.842- **Options:** `--set-kind regression|capability_gap` (default `regression`, must843 match the suite's kind), `--contains-user-data` (default is `synthetic`). A case844 whose **build is correct** (outcome expectations green) but that carries a845 **currently-red execution scenario** from a builder bug is still a `regression`846 case — it guards the fix; reserve `capability_gap` for cases where the *build847 itself* is wrong.848- **Scenarios sync on update too:** `PATCH /cases/:id` reconciles849 `executionScenarios` by name (update in place, insert new, delete missing —850 lang-tracer #48), so scenario edits re-push like any other field. A lang-tracer851 deployment predating that change silently ignores the key; if a pushed scenario852 edit doesn't land, update the scenario in the lang-tracer UI.853- **Report what was pushed as links, not `#ids`.** The CLI prints `+ created854 <slug> (#621)` / `~ updated <slug> (#621, rev 3)` and a suite header — that's the855 id, and nothing more. Turn each one into `<base>/test-cases/<id>` (and the suite856 into `<base>/suites/<suiteId>`) in whatever you hand the driver, so they can open857 the case they just authored instead of hunting for it. See [Share links, never858 bare ids](#share-links-never-bare-ids).859- **An `inline` seed pushes with the case:** the case-write API stores it860 verbatim, so a seeded case lives in a suite like any other. Only a `replay`861 case is refused — the push lists it under `skipped:`, because it's862 reconstructed from a LangSmith trace at run time and dies when that trace is863 pruned or deleted. Don't commit a replay case either; derive a durable864 synthetic case as the artifact instead.865 866### Link the pushed case to its source (provenance step — always do this)867 868A sourced case that isn't linked back to the conversation/finding it encodes is869an orphan: six months later nobody can tell what real failure it guards. The870push CLI doesn't carry provenance, so after pushing, link the case over the871lang-tracer MCP with one **`update_test_case`** call on the new case id (the872push prints it):873 8741. **`sourceThreadId`** — the source conversation's thread id (plus875 **`sourceRunId`** when the case anchors to one specific run/step within it).876 This is the DB-level link every by-version rollup, conversation float, and877 `?sourceThreadId=` query joins on — the tags/description below are the878 human-readable layer on top, not a substitute. The thread must already be879 imported into lang-tracer (running `get_conversation_analysis` on it, as the880 sourcing flow does, is enough); `source_kind` is derived server-side, and881 the link is only editable on authored cases — promotion-recorded provenance882 is immutable.8832. **`expectedBehavior`** — the rule the case enforces, one paragraph — and884 **`failurePattern`** — what actually happened in the source thread, with885 turn references. Copy/adapt these from the analysis's `extractedCases`886 entry when the case came from `get_conversation_analysis`.887 888Then **`add_case_tags`** (additive; targets the LT-side `tags` array, not889`evalTags`, so nothing round-trips into eval runs): add a capability tag (e.g.890`instruction-persistence`). Tag normalization is aggressive (lowercase, kebab);891colon-form tags get silently dropped. And keep the thread id + turn refs in892the case `description` too (the drafter's habit of "Sourced from thread <id>"893is the convention) — the description is the only field shown everywhere.894 895## Running896 897You need a running n8n instance with Instance AI enabled and a working sandbox;898point the eval at it. The harness runs in three modes — **direct loop** (no899LangSmith; `eval-results.json` only), **LangSmith** (also records an experiment900+ regression comparison), and **prebuilt** (`--prebuilt-workflows`, score901existing workflows). Narrow a run with `--filter <slug>` / `--tier <name>` /902`--exclude`. See [`running-evals.md`](running-evals.md) for the run recipes,903parallel lanes, tiers, and baselines, and the904[README](../../../packages/@n8n/instance-ai/evaluations/README.md) for the full905flag list. Run with `--keep-workflows` when you want to review a build by hand —906in *checkpoint* mode calibration this is how the driver opens the built thread907(`<base-url>/assistant/<threadId>`) and workflow on the instance.908 909## Other eval harnesses (not this skill)910 911Use the [`create-agent-builder-eval` skill](../create-agent-builder-eval/SKILL.md)912for standalone Agent cases. Three other harnesses have their own data dirs and913CLIs: **`eval:subagent`** (workflow-build compatibility corpus,914binary-check scored), **`eval:discovery`** (asserts first-hop tool/dispatch915routing, no n8n server), **`eval:pairwise`** (head-to-head build comparison vs916`ai-workflow-builder.ee`). Authoring them is out of scope here — see the README917sections of the same names.918 Discovery context
Discovered by repository scan. No exact path reference found in the snapshot’s root AGENTS.md.
