At Fluid Inference, our goal is simple: make deploying and running local AI models as easy as running application code. For too long, developers have had to rely on the cloud even for basic AI tasks. Intel's new AI PCs with integrated NPUs are changing that, making it possible to run sophisticated AI workloads directly on consumer devices.
Developer interest in local AI is exploding, from indie developers to engineers at major tech companies. Developers are far more educated about AI models than they used to be, and hardware like Intel's NPUs is finally powerful enough to make local deployment practical.
That's why our collaboration with Intel is so exciting. Together, we optimized Whisper v3 Turbo, Qwen3, and Phi-4-mini for Intel® Core™ Ultra processors with integrated NPUs. These models, usually associated with GPU-heavy cloud infrastructure, now run in real time on Intel AI PCs.
For more details, read the full article on Intel's community blog.
Agents optimized three models in weeks
Our agent-based optimization system takes three inputs: the models to optimize (Whisper v3 Turbo, Qwen3, Phi-4-mini), Intel NPU hardware specifications, and performance requirements.
A lead agent coordinates the pipeline and works with specialized agents. A researcher agent analyzes model architectures and identifies NPU-specific optimization opportunities. The optimization & quantization agent implements them using tools like OpenVINO™, handling precision reduction and graph optimizations. Finally, a benchmark agent validates performance on real hardware, measuring latency, throughput, and accuracy.
The agents share results and refine their strategies based on real hardware performance data, which allows rapid iteration. We optimized all three models for Intel NPUs in a matter of weeks.
Models and requirements connect to a lead agent. It coordinates research, quantization, and benchmarking agents, all of which exchange results with tools: OpenVINO, GitHub source code, online benchmarks, and resource monitoring.
Results: faster transcription, offline summaries, lower power
Whisper v3 Turbo runs 40% faster on Intel NPUs than on CPU (0.31s down to 0.19s per segment) with no loss in accuracy. It processes audio in real time, which is crucial for live applications.
For language tasks, Qwen3 and Phi-4-mini deliver about 70-75% of GPT-4's quality on summarization and Q&A, which is impressive for models running entirely offline. Power consumption dropped significantly compared to GPU inference, though exact numbers vary by workload.
These aren't just benchmark numbers. A Fortune 100 company already uses our NPU-optimized models in its next-generation AI application.
Resources for developers
All our NPU-optimized models are on Hugging Face. We're also building a native .NET library to make it easier to deploy GenAI workloads in Windows applications.
If you want to deploy local AI, with your own models or open-source ones, reach out through fluidinference.com. We'd love to help you bring AI directly to your users' devices.