evals/evals.json
evals/evals.jsonBrowse 2 files
984 tokens
4,583 bytes
Token encoding: o200k_base
Snapshot 63ca342
← Back to SKILL.md
1{2 "skill_name": "changeset",3 "evals": [4 {5 "id": 1,6 "prompt": "Draft the exact contents of one changeset from this isolated context. Do not inspect the repository, run commands, or create files. Return only raw changeset Markdown without a code fence or explanation.\n\nContext:\n- The exact package name is `@astrojs/node`.\n- This is a backward-compatible bug fix.\n- The Node adapter currently produces invalid server URLs for IPv6 hosts such as `::1`; the fix brackets IPv6 addresses so generated URLs and startup output are valid.\n- Internally, the patch adds a `formatAddress()` helper and updates `server-address.test.js`.\n- The focused test command passed.",7 "expected_output": "A patch changeset for `@astrojs/node` with a short user-facing message beginning with a present-tense verb and omitting implementation and test-process details.",8 "files": [],9 "assertions": [10 "The output begins and ends its YAML front matter with `---`.",11 "The front matter contains exactly one package entry: `'@astrojs/node': patch`.",12 "The body contains one concise changelog paragraph whose first word is `Fixes`.",13 "The message identifies IPv6 hosts and valid generated server URLs or startup URLs as the user-visible impact.",14 "The message does not mention `formatAddress()`, `server-address.test.js`, a test command, or passing tests.",15 "The output contains no prose before the opening front matter and no code fence."16 ]17 },18 {19 "id": 2,20 "prompt": "Draft the exact changeset contents for this feature. Work only from the inline context; do not inspect files, run commands, or create anything. Return raw changeset Markdown with no surrounding explanation.\n\nContext:\n- The exact package name is `@astrojs/cloudflare`.\n- This is a backward-compatible public feature.\n- A new optional adapter property, `imageService: 'cloudflare'`, lets sites generate Cloudflare Image Resizing URLs for Astro image transformations.\n- Existing projects behave as before when the property is omitted.\n- Include a minimal JavaScript configuration example containing `cloudflare({ imageService: 'cloudflare' })` because usage is not obvious from the option name.\n- The internal implementation class and test commands are irrelevant to users.",21 "expected_output": "A minor changeset beginning with an Adds statement, naming the option and user capability, followed by a minimal JavaScript example.",22 "files": [],23 "assertions": [24 "The front matter contains exactly `'@astrojs/cloudflare': minor` and no other package entry.",25 "The first prose sentence begins with `Adds`.",26 "The prose formats `imageService` as inline code and explains that it generates Cloudflare Image Resizing URLs.",27 "The body states or demonstrates that the option is configured with the value `cloudflare`.",28 "The changeset contains a fenced `js` example with `cloudflare({ imageService: 'cloudflare' })`.",29 "The message does not discuss internal classes, tests, commands, or passing checks.",30 "The message contains no level-two or level-three Markdown headings; any heading used is level four or deeper."31 ]32 },33 {34 "id": 3,35 "prompt": "Before creating anything, assess this changeset request and respond with the decision and a brief reason. Do not inspect the repository, run commands, or edit files.\n\nThe only changed paths are `examples/view-transitions/src/pages/index.astro` and `examples/view-transitions/README.md`. No package source or package metadata changed. I want an `'astro': minor` changeset so the new example appears in core release notes, but no maintainer has reviewed the bump.",36 "expected_output": "A refusal to create the changeset because examples-only changes are exempt and cannot justify a core package bump, with an additional warning that core minor bumps require maintainer review and are blocked by CI.",37 "files": [],38 "assertions": [39 "The response states that an examples-only change does not require a changeset.",40 "The response states that the example does not justify bumping the `astro` package.",41 "The response notes that minor bumps to core `astro` require maintainer review and are blocked by CI.",42 "The response does not emit YAML front matter or a candidate `'astro': minor` changeset.",43 "The response does not claim that a changeset was created.",44 "No `.changeset` file or other file is created."45 ]46 }47 ]48}49 