THE FORM BENCHMARK / 001

Same brief.
Different minds.

What does it take for AI to give an idea form? One espresso machine. Different models. Measured tokens, build time, and geometry, with every result open to inspection.

5 recorded trials · 2026-09-21

STUDY 001ESPRESSO MACHINE
An illustrative espresso machine with a metal housing, dark portafilter handle, steam wand, and drip tray
240 × 180 × 260 mm housingIllustration of the task · not an AI result
01 / EFFICIENCYTokens & cost

Provider usage, with caching and reasoning accounted for.

02 / SPEEDTime to build

Elapsed time across the full modeling conversation.

03 / CRAFTQuality of the result

Geometry checks plus a separate visual review.

04 / PROGRESSCore vs. core

Repeat the same task after a FORM upgrade.

THE MEASUREMENTS

A little less guesswork.

First pilot · models available in Codex.
Recorded settings and results for every attempt.

Median measurements across all attempts, grouped by model settings, protocol, and core version.
Model / coreTrialsBuild checksTokensCost*TimeGeometryQuality
GPT-5.6-LunaCodex subscription · codex-pilot-2026-09-21 · 01a3457Protocol 8f2be13 · settings 841c24b1100%184,470Subscription1.9m60/ 6082/ 100Provisional AI review
GPT-5.6-SolCodex subscription · codex-pilot-2026-09-21 · 01a3457Protocol 8f2be13 · settings 841c24b1100%173,847Subscription2.1m60/ 6081/ 100Provisional AI review
GPT-5.6-TerraCodex subscription · codex-pilot-2026-09-21 · 01a3457Protocol 8f2be13 · settings 841c24b1100%133,137Subscription2.3m60/ 6086/ 100Provisional AI review
GPT-5.5Codex subscription · codex-pilot-2026-09-21 · 01a3457Protocol 8f2be13 · settings 841c24b1100%144,982Subscription1.4m60/ 6085/ 100Provisional AI review
GPT-6-AstraCodex subscription · codex-pilot-2026-09-21 · 01a3457Protocol 8f2be13 · settings 841c24b1100%219,651Subscription1.8m60/ 6094/ 100Provisional AI review

*Direct API runs use list-price estimates unless provider cost is supplied. Subscription runs consume Codex quota; a dollar cost is unavailable, not zero. “—” means unavailable accounting. Build checks require completion and every automatic check passing; visual quality is scored separately. Failed attempts remain in the denominator.

JSON ↓CSV ↓
Inspect individual trials and CAD results
Isometric CAD result from GPT-5.6-Luna

GPT-5.6-Luna

completed · 1.9m · Codex subscription

181,753 input + 2,717 output tokens · 124,416 cached input

Geometry 60/60 · Quality 82/100

Provisional AI review · Codex coordinating assistant (unblinded AI review)

Recognizable box-style machine with two accent controls and a dark handle. The group head and portafilter appear as front-facing discs; the straight wand and shallow tray are schematic. The rear tank is largely hidden and the brewing area is not recessed. Assessed from front, right, and isometric geometry previews; shading and occlusion limit visual certainty.

Editable CAD ↓Measurements ↗
Isometric CAD result from GPT-5.6-Sol

GPT-5.6-Sol

completed · 2.1m · Codex subscription

168,312 input + 5,535 output tokens · 136,576 cached input

Geometry 60/60 · Quality 81/100

Provisional AI review · Codex coordinating assistant (unblinded AI review)

The prominent rear tank, two controls, grate, and accented handle make the intended appliance recognizable. Wand sections are visibly separated in side and isometric views, weakening assembly coherence. The front-facing group-head arrangement and block body remain schematic. Assessed from three fixed geometry previews; shading and occlusion limit visual certainty.

Editable CAD ↓Measurements ↗
Isometric CAD result from GPT-5.6-Terra

GPT-5.6-Terra

completed · 2.3m · Codex subscription

129,706 input + 3,431 output tokens · 93,952 cached input

Geometry 60/60 · Quality 86/100

Provisional AI review · Codex coordinating assistant (unblinded AI review)

Clear front fascia, accent controls, a contrasting rear tank, and an angled wand give the machine more definition. The group head and portafilter still read as front-facing discs; the tray and grate are shallow and the body remains a simple block. Assessed from front, right, and isometric geometry previews; shading and occlusion limit visual certainty.

Editable CAD ↓Measurements ↗
Isometric CAD result from GPT-5.5

GPT-5.5

completed · 1.4m · Codex subscription

141,247 input + 3,735 output tokens · 109,056 cached input

Geometry 60/60 · Quality 85/100

Provisional AI review · Codex coordinating assistant (unblinded AI review)

A U-shaped housing creates an actual open brewing area, with a visible rear tank, six grate bars, and accent controls. The group head and portafilter remain front-facing, and the wand tip appears detached in side and isometric views. Assessed from three fixed geometry previews; shading and occlusion limit visual certainty.

Editable CAD ↓Measurements ↗
Isometric CAD result from GPT-6-Astra

GPT-6-Astra

completed · 1.8m · Codex subscription

216,797 input + 2,854 output tokens · 183,680 cached input

Geometry 60/60 · Quality 94/100

Provisional AI review · Codex coordinating assistant (unblinded AI review)

The recessed brewing bay, vertically stacked group head and portafilter, joined angled wand, dial pointers, cup-warmer rails, and tank level marks form a coherent espresso-machine concept. The body is still blocky; some grate and tray overlaps are ambiguous in the preview. Assessed from three fixed geometry views; clearances and material realism are not certified.

Editable CAD ↓Measurements ↗

MEASURE FORM’S PROGRESS

Did a core upgrade use fewer tokens?

Hold the model, settings, and brief constant. Compare token usage only when geometry passes and reviewed quality is maintained.

Version comparisons unlock after trials on two FORM core versions.

A SINGLE, SHARED CHALLENGE

Make coffee.
Start with the machine.

An exterior CAD concept with named, editable parts. Enough detail to test spatial reasoning, assembly, and finish.

Download the exact brief
ESPRESSO MACHINE / V1

Build a recognizable countertop espresso machine.

A 240 × 180 × 260 mm housing, base, group head, portafilter and handle, steam wand, drip tray and grate, two controls, and rear water tank.

Millimeters · Z upEmpty documentEditable solids
Read the full prompt

Build a recognizable countertop espresso machine as an editable FORM assembly. The main housing must be 240 mm wide (X), 180 mm deep (Y), and 260 mm tall (Z), within 5% on each axis. Its bottom is at Z=20 mm. Add a base resting on Z=0; a front group head; a portafilter with a handle projecting toward the front; a separate steam wand to the right of the group head; a drip tray below the group head with at least five visible grate bars; two front controls; and a water tank at the rear. Use separately editable, closed solid parts with descriptive names containing these labels: housing, base, group head, portafilter, handle, steam wand, drip tray, grate, control, water tank. Use metallic housing, dark handles, and a contrasting control accent. The assembly must fit within 320 mm width, 380 mm depth, and 340 mm height, with no geometry below the ground plane. Keep the brewing area visibly open, put the group head above the drip tray, and make the machine visually coherent from the front, right, and an isometric view. This is an exterior concept model; internal plumbing and manufacturing readiness are not required.

Shared system instruction

You are a CAD modeling agent using FORM. Build the requested design in the empty document using the supplied tools. Read describe_capabilities first. Use millimeters, Z up, and front toward -Y. You have a headless workspace: camera and screenshot tools are unavailable. Inspect geometry with the available tools. Work independently without clarification. Finish with a short description when the model is complete. Do not claim checks you have not performed.

HOW WE MEASURE

The method is part of the result.

Full protocol
01

A fresh start, every time.

Each trial starts with an empty document and the same brief. Models use FORM’s native CAD tools through a headless runner. Codex and direct API trials are kept in separate protocol groups. No screenshots or human help during a run.

02

Measure the whole attempt.

Provider-reported input and output tokens include the conversation history and tool schemas on every turn. Reasoning and cached tokens are counted without duplication. Direct API cost estimates use saved pricing. Codex subscription trials have no available per-run dollar cost.

03

Inspect what was made.

Automatic checks contribute 60 points for geometry, dimensions, part coverage, placement, and editable export. A separate review adds up to 40 points for recognizability, assembly, detail, and finish.

04

Keep the evidence.

We retain failures and show trial counts. Medians include every attempt with complete measurements. Each result links to its CAD file and fixed views. Quality remains pending until the visual review is complete.

AN OPEN QUESTION, A REPEATABLE TEST.

Better models. Better tools.
Keep measuring both.

Make something in FORM