OpenAI's Jalapeño Chip Shows Industry-Leading Inference Speed and Efficiency in First Benchmarks

Posted on 29th Aug 2026 06:06:42 in Artificial Intelligence, Machine Learning

Tagged as: OpenAI, Jalapeno, AI chip, inference, Broadcom, Nvidia

OpenAI has published the first measured performance results for Jalapeño, its first custom inference chip, and the numbers put the startup squarely in competition with the fastest commercial AI accelerators on the market. Presented at the Hot Chips conference in late August 2026, the results show a processor that delivers more useful AI work per watt of power while simultaneously cutting the time users wait for a response — a combination that existing hardware has historically forced engineers to trade off.

Developed in close collaboration with Broadcom and unveiled in June 2026, Jalapeño is the opening chapter of a multigenerational silicon strategy that positions OpenAI as more than a model lab. The company now designs models, products, serving software, chips, memory, networking, and rack-scale systems together — and the first benchmark data suggests that full-stack control translates into real efficiency gains. For businesses and developers who pay for AI by the token, faster and more efficient inference means lower costs, snappier agents, and more reliable access when demand spikes.

The Benchmark Numbers: A Chip That Sits on the Pareto Frontier

OpenAI measured Jalapeño on InferenceX, a public benchmark from SemiAnalysis that captures the full process of serving an AI request, and compared it against leading commercially available systems — including an Nvidia Blackwell-class GB300 reference system. The comparison was normalised using each accelerator's published chip power rating: Jalapeño is rated at 700 watts, while the GB300 comparison package is rated at 1,400 watts. In practice, Jalapeño's measured sustained power stayed at or below 550 watts on the workloads tested.

Across three public models — OpenAI's GPT-OSS 120B, DeepSeek R1 670B, and Kimi K2.5 1T — Jalapeño delivered the following results:

• 1.5 to 1.9 times more AI work per watt at peak throughput compared with the comparison systems.
• 1.7 to 3.6 times lower end-to-end latency across the tested operating range.
• 2.1 to 4.1 times higher performance on highly interactive, low-latency workloads.

On Kimi K2.5 1T, the largest public model in the test, Jalapeño achieved roughly 1.5 times higher peak performance per watt and 3.4 times lower end-to-end latency. The chip placed on the Pareto frontier for throughput-per-kilowatt on both DeepSeek R1 and Kimi K2.5 — meaning no comparison system beat it on both axes at once. OpenAI added that in internal testing, the advantage widens further on its own frontier models, suggesting the architecture grows more valuable as workloads get larger and more demanding.

Inside the Architecture: Designing Around Prefill, Decode, and the KV Cache

Language-model inference is not a single workload. Prefill — the phase where the system digests a prompt — is compute-intensive, while decode — generating the response token by token — is constrained more by memory bandwidth. Communication between cores and chips adds further latency when data has to move between resources. A system that excels at one phase can lose its advantage while waiting for data or shuffling model state around.

OpenAI architected Jalapeño around exactly these phases. The chip minimises data movement and communication delays, allowing model state — including the KV cache used while generating a response — to be explicitly placed and kept local, while the system activates the right combination of compute, memory, and networking for each phase of inference. The network is integral to the design: a large domain keeps an entire workload inside one connected system, which keeps end-to-end requests fast and efficient. The result, OpenAI argues, is a balanced accelerator that performs well at both prefill and decode and can adapt as the balance between them shifts — a defining requirement of agentic workloads, where many sequential steps make latency compound across a whole task.

AI Helped Build the Chip — and the Chip Was Built So AI Can Program It

One of the more striking details in the disclosure is how much of Jalapeño's development loop was itself automated. The team moved from initial design to manufacturing tape-out in nine months — a pace OpenAI calls the fastest ASIC development cycle achieved in high-performance semiconductors. OpenAI's models explored implementation options, shortened design, measurement, and verification loops, and optimised the chip's arithmetic circuits to fit more compute into the design on schedule.

The relationship runs both ways. Jalapeño was designed as a clear, predictable programming target for humans and AI alike, with local tensors, explicit communication, and predictable synchronisation. Using Codex together with GPT-Astra, OpenAI's team brought three open-weight models that were never part of the original production plan to high performance within two months. On selected GPT-OSS attention and mixture-of-experts blocks, AI-generated implementations ran 1.5 to 1.8 times faster than the existing human-written kernels. Those figures apply to selected blocks rather than the full model, but they hint at a powerful new development loop: AI that designs hardware and then writes the software to program it.

What This Means for the Inference Market — and for Nvidia

The benchmark comparison is a direct challenge to Nvidia's dominance in inference silicon, but it comes with important caveats. The results are first-party disclosures on a benchmark OpenAI helped popularise, and the comparison system is a currently available Blackwell-class reference — by the time Jalapeño reaches full deployment, Nvidia's own roadmap will have advanced. OpenAI itself was careful to frame the chip as additive rather than replacement: the company said it will continue to widely deploy accelerators from Nvidia and other partners for both training and inference workloads.

The broader significance is strategic. By controlling the full stack, OpenAI can optimise every layer — model, kernel, serving system, chip, memory, network — against the same goal. For the industry, that raises the stakes on custom silicon: Anthropic has reportedly explored its own chip partnerships, and hyperscalers including Google, Amazon, and Microsoft have been investing in proprietary accelerators for years. Jalapeño's results give the strongest public evidence yet that a model lab with full-stack control can beat general-purpose commercial accelerators on the metric that matters most in production: useful work per watt at interactive latencies.

The Road Ahead: Small Volumes in 2026, a Multigenerational Platform

OpenAI plans to begin deploying Jalapeño inside its own compute infrastructure by the end of 2026, with Richard Ho, the company's head of hardware, describing the initial volumes as "very small" and more significant deployment coming in 2027. Broadcom CEO Hock Tan framed the partnership as targeting gigawatt-scale data centres with Microsoft and other partners beginning in 2026, with Celestica handling board, rack, and system integration and Broadcom contributing silicon implementation and Tomahawk networking technology.

Jalapeño is explicitly the first generation of a long-term platform: OpenAI says Gen 2 is deep in development and Gen 3 is already taking shape, with each generation building on what the previous one learned. As the company prepares for deployment, it is continuing production qualification, maturing the software stack, and validating performance across more models. The strategic logic is simple: every efficiency gain in inference lowers the cost of serving intelligence, which makes advanced AI cheaper for students, developers, small businesses, and enterprises alike — and funds the next, more capable generation of models.

Sources

• OpenAI — Jalapeño's First Results Show Industry-Leading Speed and Efficiency in AI Inference
• TechCrunch — OpenAI's Jalapeño Chip Is Built for Fast Inference at Scale, Benchmarks Show
• HPCwire — OpenAI Details Jalapeño Inference Chip Performance and Multigenerational Roadmap
• OpenAI — OpenAI and Broadcom Unveil LLM-Optimized Inference Chip