Verification and cutover

Prove parity through the application's supported boundary. Do not require high-risk exercises for behavior the traced source path does not use.

Build a contract ledger

For each traced source behavior, maintain one row:

Use these evidence states:

Match testing to the slice

Ordinary agent or tool

• Characterize the public request and result before rewriting TypeScript as Python.
• Use TestModel or FunctionModel to prove prompts, dependency-backed tool calls, validated output, step limits, errors, messages, and usage without network access.
• Exercise tool failures that the source loop exposes to the model separately from failures that terminate the request.
• Add a provider recording or live probe only when provider transport, native tools, structured-output mode, retry behavior, or deltas are part of the contract.

AI SDK UI and cross-language boundaries

• Keep the installed AI SDK version explicit. Capture the actual request body and a source golden stream containing only the headers, framing, event types, stable fields, and ordering the client consumes.
• Run the same language-neutral fixture against the Python endpoint. Test text reconstruction, tool input/output states, approvals, custom data, errors, abort, finish, and trailing events used by the application.
• Validate client-provided messages and files. Prove authenticated identity, server instructions, provider metadata, tool executors, and previous trusted history come from server-owned state.
• If reconnect is supported, disconnect mid-stream and continue in a fresh process or server instance. Assert cursor behavior, the no-active-stream response, terminal detection, and no duplicate message, tool call, or effect.
• If no existing transport boundary exists, migrate the nearest caller too. Treat a newly introduced Python service as an architectural change, not automatic parity.

Structured output and streaming

• Test valid complete output, invalid complete output, and any retry. Do not validate a deep-partial stream as though it were the final object.
• Record only the event fields consumed by the caller. Assert reconstruction, ordering, terminal detection, cancellation propagation, provider errors, tool errors, retries, and whether a final structured-output step occurs.

Approvals and protected effects

• Assert the pending call ID, validated arguments, policy category, and any signed or server-owned correlation data.
• Exercise deny and approve, including invalid, stale, and replayed decisions and any decline reason exposed to the model or caller.
• Assert no protected effect before resolution, zero after denial, and exactly one after approval, including across retry and process restart.
• Define the durable ordering explicitly: persist the authorized decision and an idempotency key before starting the effect, then persist the effect result before resuming the agent. Test crashes between each write and effect boundary.
• Test the actual lifecycle in scope: AI SDK's next request, Eve's parked durable session, inline Pydantic AI handling, or a later Pydantic AI run with persisted messages and DeferredToolResults.
• Map source approval, tool-call, session, and application request IDs explicitly. If Eve accepts unrelated turns while an approval is pending, store those turns on a separate branch because unresolved Pydantic AI tool calls cannot have a new prompt appended after them.
• Exercise authenticated identity, authorization lookup, audit, and tenant isolation. Approval is not authorization.

Eve state, sessions, and recovery

• Test UI messages, normalized agent messages, defineState data, memory, plans, session records, and event cursors independently.
• Kill and restart at every promised model, tool, wait, background-result, and external-effect boundary. Assert step identity, replay unit, event IDs/cursors, terminal status, and exactly-once effects.
• If batching model calls changes Eve's replay unit, characterize that configuration explicitly.
• Exercise concurrent resumes and stale inputs. Validate resume payloads and prove only the selected session, user, and tenant can continue the run.
• Label Harness StepPersistence as equivalent only when these boundary tests pass; persisted snapshots alone are not proof of Eve protocol compatibility.

Skills, subagents, connections, and sandboxes

• For a used Eve skill, test description-based selection, loaded SKILL.md, and every required reference, asset, or script. A successful instruction load does not prove supporting files are available.
• For subagents, test input projection, separate history and state, shared or isolated dependencies/tools/sandbox, result handback, background delivery, event forwarding, usage limits, cancellation, errors, recursion policy, and restart.
• For MCP/OpenAPI connections, test runtime discovery, qualified names, schema conversion, auth failure and resume, token ownership, revocation, remote errors, and lifecycle cleanup.
• For untrusted execution, use a disposable workspace in the actual container, VM, or cloud sandbox. Exercise .., absolute paths, escaping symlinks, secrets, network policy, resource limits, timeouts, termination, and cleanup. A mocked filesystem or command allow-list is not sandbox evidence.

Channels, schedules, observability, and evals

• Exercise the real channel or schedule entrypoint with identity, authorization, deduplication, ordering, retry, timezone, cancellation, and delivery failure.
• Compare required spans, run/session IDs, usage, metadata, content/privacy policy, and dashboards before switching telemetry.
• Re-run observed evaluation cases and essential metrics through the supported public boundary. Keep sampling, scheduling, trace correlation, and result storage application-owned unless separately replaced.

Dependency and cutover checks

1. Resolve the Python project from a clean environment against supported Pydantic AI and Harness versions. Prefer pydantic-ai-slim with only required extras when the dependency surface is bounded.
2. Run the original source characterization tests and focused target tests. Capture retained HTTP, stream, storage, provider, and durable-session boundaries with language-neutral fixtures and run them against both implementations.
3. Search imports, agents, tools, UI routes, transport adapters, stores, sessions, schedules, channels, connections, sandbox configuration, build scripts, and deployment for retained ai, @ai-sdk/*, or eve dependencies.
4. Remove TypeScript packages, Node runtime/build steps, Vercel services, and storage or event assumptions only when no retained path needs them.
5. Report which checks used deterministic fakes, recordings, live providers, fresh-process restart tests, or real sandboxes, and state every remaining limitation.

Completion criterion

Cut over only when every ledger row is verified, an accepted intentional-change, external-owner, or not-applicable; no required row is unverified or blocked; the supported boundary passes; dependencies resolve cleanly; and rollback remains possible.