EEBench puts AI circuit design to the test with real SPICE simulations

OpenAI’s GPT-6 Astra demo of a KiCad session put electronics design in front of a mainstream model launch, but the team behind EEBench argues the harder question is whether AI-generated circuits actually work. Their benchmark skips the GUI entirely, using the declarative atopile format so an agent manipulates components, connections, and electrical constraints as code — then builds, simulates, and inspects failures without burning context on menu-clicking. That lets the eval spend its budget on electronics rather than computer-use overhead.

What makes EEBench notable is its insistence on real-world messiness. Tasks are graded deterministically against SPICE simulations and a bill of materials drawn from actual manufacturer parts, with datasheet specs carried into the models. A capacitor must hold a processor’s rail above its 3.0V brownout threshold for 20ms after power loss, accounting for the fact that ceramic parts lose much of their rated capacitance under bias; a harder task demands synthesizing a multiple-feedback op-amp filter that survives worst-case tolerance corners. Crucially, designs are scored on cost efficiency too, but cost only counts once the circuit functions — mirroring the trade-offs between performance, price, and part availability that define real hardware engineering.

The September 1 leaderboard shows Claude Opus 5 leading at 61.6% across 13 tasks, with Grok 4.6 at 57.1% and Claude Fable 5.1 at 56.4%; OpenAI’s GPT-5.5 and GPT-5.6 Sol trail at 42.3% and 39.4%, and no GPT-6 Astra score exists yet. The methodology is public, and the same graded harness doubles as a reinforcement-learning environment, with the team now working directly with frontier labs on simulation-backed training. xAI’s decision to cite EEBench in the Grok 4.6 model card — and Musk’s promised Grok 4.7 trained on SpaceX engineering data — suggests AI labs are beginning to treat electronics as a serious capability category.