On June 12, 2025, the world was reminded how fragile our AI infrastructure is. When GCP and Cloudflare went down for several hours, they took half the internet with them, including most of the big AI providers like Google and Anthropic.
Credits: Down Detector
The cascade made AI apps like Cursor, Gemini, and Claude essentially unusable. For millions of users, their AI assistants simply vanished. To be fair, outages at this scale are rare: the last major one was when Fastly brought down Amazon, Reddit, Spotify, and others for 49 minutes in 2021. Still, it's a reminder that nearly every AI application today depends on the cloud and a stable internet connection.
We have the hardware
Sam Altman recently said a single ChatGPT query uses about as much energy as running an oven for one second (≈0.34Wh per prompt), which scales to gigawatt-hours at billions of queries. That might not sound like much, but imagine firing up your oven every time you asked your phone a question. That's where Neural Processing Units (NPUs) come in. These chips are built to run AI inference (using a trained model, as opposed to training it) at 5–10x less power than traditional processors.
Values shown in the original illustration: CPU 15 to 125 watts, GPU 30 to 300 watts, NPU 0.05 to 10 watts. These are the source illustration's ranges, not universal device specifications.
"NPU" has become the catch-all label for on-chip AI horsepower, but every vendor slaps its own badge on the same idea. Apple ships a Neural Engine (ANE), Qualcomm has Hexagon, MediaTek has an APU, Arm has Ethos, Intel has AI Boost, AMD has XDNA, Google has TPU in the cloud and Edge TPU on devices, Graphcore has an IPU, Horizon Robotics has a BPU, and the list keeps growing. Under the stickers? All of them are matrix engines built to chew through neural-net math, and some even stretch to full-blown training.
This echoes the early days of computing, when architectural incompatibility was the norm. IBM System/360 code (1964) wouldn't boot on a DEC PDP-11 (1970), and today an ANE binary won't magically light up on a rival NPU. Through the 1970s minicomputer revolution, incompatible architectures forced developers to pick a platform and fragmented the software ecosystem. Different decade, same fragmentation, just with tensors instead of punched cards.
The industry is trying to solve this with abstraction layers, much like graphics APIs eventually unified GPU programming. Google's Android Neural Networks API (NNAPI) tried to abstract NPU access across Android devices, but it was deprecated as an NDK API in Android 15, and developers are moving to TensorFlow Lite (now LiteRT) or hardware-specific SDKs. Apple's Core ML provides a unified interface for the Neural Engine. ONNX promises model portability across platforms. But unlike graphics APIs, which had decades to mature, or CUDA, which dominated through sheer market force, NPU standards are fragmenting faster than they're converging. Cross-platform options like ONNX or Foundry Local exist, but integrating them into native apps often reveals they're outdated, supporting models months behind state-of-the-art.
Representative documented integrations available by December 31, 2025. ExecuTorch 1.0 delegates to Core ML, Qualcomm, MediaTek, Intel OpenVINO, Samsung Exynos, Arm Ethos-U, NXP, and Cadence. ONNX Runtime uses execution providers. Windows ML manages a Windows runtime and vendor providers. LiteRT gained QNN and NeuroPilot accelerators during 2025. Apple Foundation Models exposes Apple's own on-device language model. MLX is shown on its CPU/GPU path, not connected to the Neural Engine. NNAPI is marked deprecated.
Map updated through December 2025. ExecuTorch 1.0, Windows ML, and LiteRT’s Qualcomm and MediaTek accelerators added more shared deployment paths. These integrations reduce some duplication, but each target still has its own coverage and constraints.
But we can't use it
The fragmentation is real, and it's holding everyone back. Apple's new Foundation Models framework shows what's possible with the right infrastructure. Getting speech-to-text and local LLMs running on a Mac used to take us two weeks; now it takes less than a day, especially with the new SpeechAnalyzer and SpeechTranscriber APIs, whose ASR (automatic speech recognition) models are much better than Apple's previous speech recognition. But that only works in Apple's tightly controlled ecosystem, and users need to be on OS 26 or later. Apple also mentioned that model updates may ship with OS updates. Once a year is worryingly slow for foundation models, but much more often becomes a pain to maintain.
Microsoft can't seem to decide how to brand its toolchain either. It went from Windows AI to Windows App SDK to Windows Copilot Runtime to Windows AI Foundry, its latest rebrand. The constant renaming shows how much confusion there is in the space. Even Phi Silica, the model that's supposed to ship with Windows 11, is still in developer preview after eight months and only runs on Windows devices with Qualcomm chips.
Microsoft has to support multiple chipmakers, which is part of why it's still behind Apple. Foundry Local supports Phi-4-mini-reasoning (Apr 2025) and DeepSeek-R1 (Jan 2025) on Qualcomm NPUs, but other chips still run inference on CPU or GPU. Phi-4-mini-reasoning is a different model from the standard Phi-4-mini, and developers report implementation problems even with officially supported versions.
Foundry Local models supported on a Snapdragon Microsoft Surface Laptop as of June 2025
On top of that, state-of-the-art models change every couple of months, and you have to work with chipmakers to make sure small changes in training carry over correctly when running inference on each platform.
The conversion pipeline itself is a nightmare. Most models start as PyTorch checkpoints trained on NVIDIA GPUs. Getting them onto edge devices means navigating a maze of format conversions between PyTorch, Safetensors, and ONNX. Each step can introduce precision loss, unsupported operations, and performance degradation. A model that worked perfectly in training might fail outright or run 10x slower on the target device.
Then there's the language gap. Machine learning engineers live in Python: models are trained and tuned there. Application developers work in their platform's native language, and apps for endpoint devices have to ship in native formats. That means supporting Swift, .NET, TypeScript, Rust, C++, Go, Kotlin, and more. The SDK fragmentation alone is enough to keep most developers on cloud APIs.
And users want everything
You can offer a private, cost-efficient, offline experience, but it has to be nearly as valuable as cloud-first solutions or most users won't stick around. Only those who care deeply about privacy or have strict compliance requirements will make the trade-off.
And it's not just LLMs or ASR. Once users get a taste of AI-rich features, they want more. Build an AI-native app and you'll get requests for speaker diarization, vision, OCR, text-to-speech, and the like, all of which have traditionally run in the cloud on GPUs. Each needs its own model with a different architecture, so the fragmentation compounds. It's hard enough getting one LLM to run efficiently across chips. Now multiply that by every AI capability users expect, in every language developers ship in.
The path to intelligence everywhere
Despite these challenges, the momentum is undeniable. Small language models are getting good enough for many everyday tasks. The power efficiency gains are real. And demand for offline, private, always-available AI keeps growing.
So how do we actually get intelligence everywhere? We're at an inflection point. The hardware is here, and the models are already capable. What we need now is the connective tissue: the standards, frameworks, and tools that make edge AI as seamless as cloud AI. As with application code on CPUs (x86, x64, ARM), the world is unlikely to converge on one architecture. But it needs to narrow to two or three, not one per chipmaker. Even then, the hardware should ideally be hidden from developers behind a platform-agnostic layer, the way Docker or the JVM made deploying application code simple.
The building blocks for running models on NPUs, and the techniques for converting them, shouldn't be the moat. They need to be open-sourced and shared so everyone can move faster. We all win if more inference happens on-device.
The path forward isn't mysterious. It's about removing artificial barriers and building the boring but essential infrastructure that lets intelligence flow to every device. By 2030, we'll look back at today's cloud-dependent AI the way we look at dial-up: quaint, necessary for its time, but a stepping stone to something far more powerful and pervasive. Software took off the moment code became pure logic: portable bytes that ignored the underlying silicon. When AI models reach that same portability and cost-efficiency, intelligence will truly be everywhere we are.