# FORM AI benchmark

The first task is an editable exterior espresso machine. The runners measure provider APIs or a logged-in Codex subscription against FORM's real local MCP server. They never treat response bytes as tokens. The illustration and automated test fixture are hand-authored and are not results. Codex trials and direct provider API trials have separate protocols and must not be pooled.

## Run with the models available in Codex

```sh
npm run benchmark:codex -- --effort medium --repetitions 1 --core-label baseline --output .artifacts/ai-benchmark/codex-pilot
npm run benchmark:ai -- report --input .artifacts/ai-benchmark/codex-pilot/runs
npm run build:landing
```

This uses the existing ChatGPT login from `codex login`; it does not need provider API keys and consumes subscription quota. There is no available per-run USD price, so cost is null, never advertised as free or $0. The external API spending budget does not apply to subscription usage. Use `--codex /absolute/path/to/codex` if the binary is not on PATH. `--models id,id` selects account-available models; add a new model by its ID without adding a price entry. Model availability and supported reasoning efforts are queried from `model/list` before inference. The default pilot selects Luna, Terra, Sol, GPT-5.5, and Astra when present. Product-tier labels are descriptive, not parameter counts.

Every suite freezes kernel, MCP, interchange, and evaluator sources and records their fingerprints. Runtime dependencies are linked to the installed versions; do not change dependencies during a suite. A fresh local FORM server and empty Codex workspace are used for every trial. Runs are ephemeral, with personal config ignored and unrelated tools, plugins, agents, browsing, and local shell execution disabled per invocation. The same task instructions are passed as a user brief under Codex's own model-specific system context; this is not identical to the direct API system prompt. Codex can also wrap or route tool calls differently by model. The measured result is model + Codex harness performance.

The pilot uses medium reasoning effort (the setting need not represent equal compute across models), a default service tier, a 10-minute wall limit, and at most 60 successful/dispatched FORM tool calls before the runner stops further work. It does not impose the native API runner's per-response token or model-turn limits. Failed calls are recorded. Wall time includes CLI/MCP startup through Codex process exit, excluding evaluation and rendering. Local tool durations are measured separately; Codex does not expose individual provider-request latency/count here. `requests` records a started Codex invocation and `turns` records completed user turns, not internal inference requests. `providerMs` is an unused zero placeholder for this mode. Caching is allowed naturally and the pilot runs in a fixed order, so latency and cache effects can depend on order.

Token totals are taken from completed `codex exec --json` turn events. Input includes repeatedly supplied context and tool results; cached input is a subset, not additional tokens. Output includes reasoning when reported that way; a separate reasoning count is exposed only if present in the event. Missing or interrupted accounting remains unknown. The requested model ID is recorded; this CLI does not expose a resolved model snapshot ID, so automatic claims of core-version savings stay disabled. Pricing fields in private model records are unused placeholders; `estimatedCostUsd` and `providerCostUsd` remain null. Private events and tool traces are retained alongside the CAD artifact. No raw transcripts are published.

A one-trial pilot is a feasibility comparison, not a stable ranking. Run repeated trials and independently review the CAD results before making marketing claims about overall quality. The 60-point geometry result remains separate from the pending 40-point visual review. Claude, Grok, and Gemini require their own API credentials and are not represented by this Codex pilot.

Only the isolated local FORM server is pre-authorized for mutations; the trial workspace uses `workspace-write`. Existing documents and remote connectors are not available. The initial setup suite was invalidated after Codex blocked a batch tool before it reached FORM, and is retained privately with `INVALIDATED.json`. Its results must not be included in the pilot comparison.

Reviews may be marked `method: "ai"` for an explicitly provisional AI assessment. The page labels these separately and includes the reviewer's identity and notes. An assessment performed in the coordinating Codex session is unblinded, is not an independent human judgment, and should be replaced or corroborated before making broad quality claims. Review all three fixed views and retain limitations of the preview renderer in the notes.

The September 21, 2026 pilot contains one medium-effort trial for each of the five discovered Codex models. All five saved CAD artifacts were scored with `espresso-v1-evaluator-2`. This revision corrects a label-resolution bug: a name such as “drip tray below group head” previously caused the tray to be mistaken for the group head. The checker now prefers primary labels and assigns distinct solids to roles. The rubric and point weights did not change. Every measurement retains its original geometry grade, the correction reason, and the corrected evaluator fingerprint under `evaluation`; all five records use the same revised evaluator. Only GPT-5.5's geometry score changed, from 45 to 60. No model was rerun for this scoring correction.

Official references: [Codex non-interactive runs and JSON usage](https://learn.chatgpt.com/docs/non-interactive-mode), [model discovery](https://learn.chatgpt.com/docs/app-server), and [per-invocation configuration](https://learn.chatgpt.com/docs/config-file/config-reference).

## Run a $10 pilot

Copy `benchmarks/.env.example` to a local, git-ignored `.env.benchmark` file with `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `XAI_API_KEY`, and `GEMINI_API_KEY`. Keys are loaded only in the local runner; the marketing site has no provider credentials or paid-run endpoint.

```sh
npm run benchmark:ai -- plan --env-file .env.benchmark
npm run benchmark:ai -- run --env-file .env.benchmark --repetitions 1 --budget-usd 10 --core-label baseline --output .artifacts/ai-benchmark/pilot
```

`plan` makes no model API requests. It checks the configured models, presence of keys, source fingerprint, and the actual MCP tool registry. `run` validates every selected key and pricing date before starting. Use `--models gpt-5.6-luna,claude-haiku-4-5,grok-4.3,gemini-3.5-flash-lite` for a smaller first pass. Availability is account-dependent: the configured IDs are examples from official documentation, not proof of API access. Prefer pinned snapshots when available; aliases and reported response model IDs are saved.

The pilot runs smaller product tiers first. Subsequent repetitions rotate the order. The $10 budget applies to a single invocation, not to all historical runs. It uses a conservative pre-request reserve (serialized UTF-8 bytes plus 4,096 tokens of overhead, at the highest input/cache-write rate, plus maximum output cost). Actual measured cost then reduces the remainder. This reserve is a planning bound, not a provider-enforced billing cap. Provider tokenization, hidden overhead, or billing changes can differ; configure provider billing limits if an absolute account cap is required. HTTP errors and timeouts stop further paid requests because their billing may be unknown. There are no hidden retries. Budget stops and unstarted trials are recorded, not silently dropped.

Increase to `--repetitions 3` after a successful smoke pass. Three is a minimum descriptive sample; use more trials before making a general performance claim. A new invocation writes a new suite; it does not silently resume or overwrite earlier trials. An existing output directory is refused to preserve its manifest. For a one-command pilot with a new output directory each time, use `npm run benchmark:pilot -- --env-file .env.benchmark`.

## Fixed protocol v1

- Exact system instruction and task: `benchmarks/espresso-machine.v1.json`.
- Empty, independent local MCP process for every trial. Existing projects and cloud accounts are untouched.
- Native OpenAI and xAI Responses APIs, Anthropic Messages, and Gemini generateContent. Raw continuation blocks retain reasoning state and thought signatures where provided.
- All discovered CAD tools, excluding camera and viewport capture. The model receives no visual feedback during construction. Local MCP serialization, validation, and tool descriptions are included in the experiment.
- Default limits: 20 model turns, 60 tool calls, 8,192 output tokens per request, 10 minutes elapsed per trial, 3 minutes per model request, and 176,000 serialized request bytes. All are saved and part of the comparison cohort.
- Provider-native defaults for reasoning and temperature, unless the model's `settings` explicitly override them. Equal labels such as “medium effort” are not assumed to mean equal compute across providers. Output ceilings include thinking where the API defines them that way. Adjusting limits or prompts creates a different cohort.
- Provider caching can occur naturally. Cached reads and cache writes are measured separately. This is not a cold-cache latency experiment. No explicit persistent caches, web search, or paid provider tools are enabled.
- Wall time starts immediately before the first model request and ends after the final answer, limit, or failure. MCP startup, scoring, rendering, and artifact export are excluded. Provider and tool durations are recorded separately. Requests include network latency; these are local-machine observations, not universal speed claims.

The system is deliberately headless. It measures CAD tool use through APIs, not ChatGPT, Claude Desktop, Grok's web app, or Gemini's consumer app. Add a separately versioned visual-feedback protocol later if desired.

## Tokens and cost

Use actual provider usage for every turn, including repeated history and schemas. Save the raw response usage as evidence. Never estimate measured tokens from characters. Input and output totals are disjoint. Cached-read and cache-write input are subsets of total input. OpenAI/xAI reasoning is already included in output; Gemini thoughts are added to candidate output exactly once. Anthropic's separately reported cache counts are added to uncached input. Anthropic does not expose a separate reasoning token count here; zero in that field means no separately reported count, not proof of no reasoning.

Estimated USD cost = uncached input × input rate + cached input × cache rate + cache writes × write rate + output × output rate, divided by one million. Pricing is captured with each run. xAI's exact `cost_in_usd_ticks`, when returned, is stored separately and preferred in the website. Other prices are list-price estimates, not invoices: taxes, negotiated discounts, free tiers, and separate account charges are excluded. Missing usage and unsupported pricing ranges yield unavailable cost rather than zero. Partial accounting remains visible in private records but is not reported as a complete total.

The initial price snapshot was checked September 20, 2026 and deliberately expires October 20, 2026 to require rechecking. The runner only prices requests up to 180,000 input tokens, below long-context breakpoints for the configured models. The byte limit and conservative reserve reject larger requests. To add longer contexts, implement and test per-request pricing tiers first. Estimated cost must be summed per response, not computed once using a run's cumulative input-token total.

Official references:

- [OpenAI pricing](https://developers.openai.com/api/docs/pricing), [function calling](https://developers.openai.com/api/docs/guides/function-calling), and [cache accounting](https://developers.openai.com/api/docs/guides/prompt-caching).
- [Anthropic pricing](https://platform.claude.com/docs/en/about-claude/pricing) and [tool use](https://platform.claude.com/docs/claude/docs/tool-use).
- [Gemini pricing](https://ai.google.dev/gemini-api/docs/pricing), [function calling](https://ai.google.dev/gemini-api/docs/function-calling), and [usage metadata / JSON function schemas](https://ai.google.dev/api/generate-content).
- [xAI pricing](https://docs.x.ai/developers/pricing) and [reported cost](https://docs.x.ai/developers/cost-tracking).

## Quality: 60 automatic + 40 reviewed

Automatic checks are computed from the saved geometry, not the agent's self-report:

| Check                                                       | Points |
| ----------------------------------------------------------- | -----: |
| At least ten visible editable objects                       |      5 |
| Every visible object is closed with volume above 1 mm³      |     15 |
| Ten required labels identify distinct solids                |     10 |
| Housing dimensions within 5%; bottom within 1 mm of Z=20    |     10 |
| Whole assembly within envelope; ground within 0.5 mm        |      5 |
| Group-head bounds above and overlapping drip-tray footprint |      5 |
| Five grate bars and two controls by separate part labels    |      5 |
| FORM JSON export reopens unchanged                          |      5 |

Naming and bounding boxes are objective proxies, not semantic or manufacturing validation. Disconnected, awkward, occluded, or misleadingly named geometry still needs review. No exact solid interference test is claimed.

Visual review contributes four equally weighted scores from 0–10: recognizability, coherent assembly/open brewing area, visible required details, and proportions/materials/finish. Anchors: 0 absent or unusable, 5 recognizable with major flaws, 8 meets the brief with minor flaws, 10 fully convincing with no visible issue. Prefer independent reviewers; the initial implementation stores one named review per artifact. It does not claim inter-rater agreement or automated aesthetic judging.

```sh
npm run benchmark:ai -- review --input .artifacts/ai-benchmark/pilot --output .artifacts/espresso-review
```

Open the generated `index.html`. Reviewers see randomly ordered sample numbers and front/right/isometric views, without provider labels. Keep `mapping.private.json` with the operator; do not give it to reviewers. The geometry renderer uses fixed orthographic cameras, fits each artifact, and uses approximate painter ordering. Ambiguous intersections should be checked in FORM. Fill each sample's `review.json` with the rubric scores, reviewer, ISO date, and notes. Keep its artifact hash unchanged.

```sh
npm run benchmark:ai -- import-reviews --input .artifacts/ai-benchmark/pilot --reviews .artifacts/espresso-review
npm run benchmark:ai -- report --input .artifacts/ai-benchmark/pilot
npm run build:landing
```

The importer validates all reviews before writing any. Changed artifacts invalidate reviews. Overall quality stays pending until reviewed; it never substitutes a geometry score for a 100-point quality score.

## Website and retained evidence

`/benchmarks` reads `apps/landing/public/benchmarks/results.json` at build time. Filters group by provider, exact source fingerprint, protocol, and model settings. Medians include failures; unknown totals make that aggregate unavailable. Build-check rate means the agent completed and every automatic check passed. It is not aesthetic success. A missing visual review leaves the group's quality pending. Different settings are kept separate. Mixed reported model versions are labeled and block core-savings claims. Provider failures without a returned model ID remain in the model’s attempted-trial denominator. Setup/budget stops before any API request are excluded from trial counts and retained in downloadable records. Trial counts and individual artifacts remain visible.

`report` exports every discovered run, including failures and limit stops, with JSON and CSV. It copies CAD files, fixed SVG views, and per-run measurement metadata, but excludes private model reasoning, raw traces, credentials, and machine paths. It writes local files; it does not deploy the marketing site. Pass a dedicated suite directory, or a parent containing the exact suites you want to publish. The private suite manifest records planned and unstarted trials. Explain incomplete sampling when interpreting a budget-limited pilot; run counts describe attempted trials, not unstarted slots.

Local evidence: `suite.json`, then per-run `run.json`, `trace.jsonl`, `protocol.json`, `model.form.json`, three SVG views, and optional `review.json`. The protocol file retains the exact task, schemas, config, runtime and machine metadata. Trace files retain raw model responses and may contain reasoning text; keep `.artifacts` private. Every exported measurement has model/pricing settings, reported model ID, git commit, dirty-state flag, core content hash, task hash, protocol hash, and evaluator/harness hash.

## Compare core releases

Use the same benchmark checkout and evaluator to launch the MCP server from different source checkouts. Each core checkout must have compatible Node dependencies installed. `--core-path` selects the target; `--core-label` is display text, not identity. Content hashes include uncommitted kernel/MCP/interchange sources and the dependency lockfile.

```sh
npm run benchmark:ai -- run --env-file .env.benchmark --models gpt-5.6-luna --core-path /absolute/path/to/old-form --core-label before --budget-usd 10 --output .artifacts/ai-benchmark/before
npm run benchmark:ai -- run --env-file .env.benchmark --models gpt-5.6-luna --core-path /absolute/path/to/new-form --core-label after --budget-usd 10 --output .artifacts/ai-benchmark/after
# Review both sets before claiming quality-preserving savings.
npm run benchmark:ai -- compare --before .artifacts/ai-benchmark/before --after .artifacts/ai-benchmark/after --output .artifacts/core-comparison.json
```

Keep the benchmark runner and its evaluator unchanged across both invocations. Running each core's own changed evaluator would confound the comparison; such harness mismatches are deliberately refused. Repeat old and new cores near each other in time and use stable model snapshots to reduce model/service drift. The comparison requires matching task, protocol, model/settings/reported version, and harness; different single core fingerprints; at least three trials per side; all automatic checks passing; complete usage; and reviewed median quality not decreasing. It reports median percentage differences, not statistical significance. This strict initial gate avoids advertising token savings caused by incomplete models or missing runs.

## Add a model or task

Add one entry to `benchmarks/models.json`: provider, API model ID, label, relative tier, environment-variable name, settings, and verified USD-per-million rates with source and expiry. No adapter code changes are needed for additional models on the four supported APIs. Refresh the price snapshot before future runs. Product tiers describe the product lineup, not known parameter counts.

To change the task, version the task JSON, update the geometry rubric and fixtures, and record a new protocol cohort. Do not silently rewrite the v1 prompt while comparing it to v1 results. The first implementation intentionally ships one task.

Validation:

```sh
npm run test:benchmark
npm run typecheck:benchmark
npm run benchmark:assets
npm run build:landing
```

Automated tests use mocked provider responses with the real MCP process and CAD engine. They check normalization, cost, continuation state, limits, actual geometry, artifact/review integrity, aggregation, and comparison eligibility. Passing these tests does not certify live model access or native API schema acceptance; the first keyed pilot is the integration check.