Expo Skill Eval

Evaluates skills in plugins/expo/skills/ for trigger accuracy, generated code quality, and/or runtime rendering in Expo Go.

Requirements: macOS with Xcode (iOS simulators), Android SDK with at least one AVD, and bun. No other device tooling is assumed.

Workspace root: /private/tmp/expo-skill-eval-<skill-name>/iteration-N/ (e.g. /private/tmp/expo-skill-eval-expo-ui/iteration-4/).

Before starting — clarify scope

Confirm all of the following up front, before any pipeline work — don't skip any (only skip a given item if the request already states that choice). Batch them into AskUserQuestion calls of ≤4 questions each, in this order:

1. Which skill to eval (if not clear from the request).
2. Prompts — which prompts drive the eval. Built-in prompts (from the skill's eval cases) are pre-selected all; drop any, add a custom text prompt, or build from an uploaded screenshot (a target UI the skill must reproduce). See Prompts below.
3. What to verify — one multi-select of three options: Runtime + screenshots / Trigger accuracy / Code checks (no device). See What to verify below.
4. Expo SDK — latest (default, auto-detected) or a pinned version.
5. Runner — Expo Go (default) or development build.
6. Platforms — iOS / Android / web (always offer all three).
7. Permission flag for claude -p — skip-permissions (default) or accept-edits.
8. Viewer delivery — local only (default) or publish a shareable Artifact.
9. If trigger accuracy is selected — confirm the published expo plugin is disabled (or not installed).

Each is detailed below. Items 4–6 (SDK, runner, platforms) fit naturally in one AskUserQuestion call.

If the skill to eval is not clear from the request, list available skills from plugins/expo/skills/ and ask which one to evaluate.

How the skill under test is loaded — two mechanisms, one per phase (don't pick one globally): executor runs reference it by file path (SKILL_PATH = plugins/expo/skills/<skill>/SKILL.md, read explicitly), while the trigger eval loads it as a plugin (--plugin-dir plugins/expo, so the model can auto-select it from its description). Both point at the local, in-repo version — that's what you're evaluating. You do not need any special flag to launch the harness session itself (the harness finds the skill by repo path); the mechanisms apply to the claude -p subprocesses it spawns. See steps 1 and 3 for why each phase differs. One pre-run check (required when the trigger eval is in scope): if the published expo plugin is installed/enabled, disable it (via /plugin) before launching the harness and re-enable after. A single disable is a global-config change that both this session and the spawned claude -p subprocesses inherit. Why it's required for the trigger eval: that phase loads the local skill via --plugin-dir, and a second installed expo collides with it — the model may trigger the published expo:expo-ui, and since detection only sees the tool-call name you'd silently score the published description instead of your local edits (the collision could also just error). The executor / runtime / static phases are not affected — they read the skill under test by its local SKILL_PATH with no --plugin-dir — so a run with no trigger eval can skip the disable. Disabling expo does not disable expo-skill-eval (a standalone project skill, not part of the expo plugin), so the harness stays available.

Surface this to the user as an explicit up-front confirmation — the same way you confirm which skill to eval. When the trigger eval is in scope, ask the user to confirm the published expo plugin is disabled (or not installed) before you start step 1; if it's still enabled, pause and have them disable it via /plugin. Don't run the trigger eval until they confirm — the harness can't reliably detect installed plugins on its own (reading the global plugin config or claude plugin list would prompt), so this is a manual confirmation, not an auto-check.

Pick the prompts — built-in, custom, or a target screenshot. The prompts are the inputs that drive the executor (with-skill and without-skill); they are separate from what you verify. Confirm them with AskUserQuestion (skip if the request already names a prompt):

• Built-in prompts — representative prompts you generate by reading the skill under test (its SKILL.md + references/) and references/runtime-matrix.md, covering the skill's standard use cases. (If the skill already ships eval cases under evals/evals.json, fold their prompt fields in too — but most skills don't, so you usually derive them.) Pre-select all so the default run exercises the skill's standard cases; let the user deselect any.
• Custom text prompt — a one-off prompt the user types. Don't spend a dedicated option slot on this: AskUserQuestion auto-adds a "Type something" / Other entry, and anything typed there becomes a custom text case.
• Build from an uploaded screenshot — the user gives the path to a target screenshot (a UI to reproduce). The executor is told to open it — claude -p reads PNGs with its Read tool — and build an app matching it; the case records the path as reference_image, and grading compares the generated app to that target (step 6). This is the strongest visual test for a UI skill: "build this."

Respect AskUserQuestion's 4-option-per-question cap with this priority (the bug to avoid: the upload option silently dropped once the four slots fill up):

1. Always reserve a slot for "Build from an uploaded screenshot." It's the whole point of the visual eval and must never be the option that gets dropped.
2. Don't add an explicit "Custom text prompt" option — the auto "Type something" / Other entry already covers it.
3. Fill the remaining ≤3 slots with the built-in/representative prompts, pre-selected. If there are more than 3, collapse them into one pre-selected "All built-in prompts (default)" option and offer subset-picking in a short follow-up, so the upload option still fits.

Present it as a multi-select. When "Build from an uploaded screenshot" is picked, ask for the target image path in a follow-up. Each selected prompt (built-in, typed, or image) becomes one eval case (run with-skill and without-skill).

Always confirm what to verify unless the request makes it unambiguous. Present these options and let the user pick one or more (defaults in bold based on the skill's references/runtime-matrix.md entry):

Option	What it does	When to suggest as default
Runtime + screenshots	Full pipeline: fixture → executor → static gate → run the app on iOS/Android and screenshot it. The runner (Expo Go or dev build) is a separate question — don't name it here.	Default for any skill that renders an app screen (the expo-go/dev-build rows in references/runtime-matrix.md). Requires a booted simulator/emulator.
Trigger accuracy	Run realistic prompts via claude -p, check whether the skill is read. Measures recall (should-trigger queries only).	Always useful as a standalone check.
Code checks (no device)	tsc --noEmit + diff-aware lint + expo export, plus the grader checks the generated code against any custom expectations you provide. No device.	Default for static-only and n/a skills, and whenever you want to verify code patterns (correct import path, a Host wrapper, …) without running the app.
Present these as ONE multi-select question — "What do you want to verify?" These are grading dimensions (how to judge what gets built) — distinct from the Prompts phase (what to build). The user may pick any combination. When a prompt is an uploaded screenshot (see Prompts), include "Runtime + screenshots" so the harness captures the