The bottleneck for on-device AI isn’t the models or the hardware. It’s the porting and optimization work in between, and that work is learnable. We’re building Möbius, an AI agent that brings models to edge devices, as a loop: every conversion teaches it patterns it reuses on the next, so each new model gets cheaper to port. To be clear, no model is improving itself here. The agent’s knowledge compounds across ports.
In this post, we test that loop on Silero VAD, one of the most popular voice activity detection models, converting it from PyTorch to Apple’s native Core ML runtime. Like Codex or Claude Code, Möbius works alongside a human. The approach is designed to generalize to other runtimes like OpenVINO and TensorRT.
Why voice activity detection matters for edge AI
Voice activity detection (VAD) answers a simple question many times per second: does this audio slice contain speech? Because VADs rely on acoustics, they’re language-agnostic and typically under 5MB, which makes them ideal for local, always-on use. They sit in front of automatic speech recognition (ASR), text-to-speech (TTS), and voice agents, improving quality and keeping latency predictable.
Our manual conversion failed after two weeks
Our team has ported several models, including FluidInference/parakeet-tdt-0.6b-v3-coreml and Pyannote models for speaker diarization. We previously converted Silero VAD v5 by hand. A team member new to model porting, mentored by a researcher with over 20 years in ML, spent about two weeks on it. The result was broken: ±0.15 to 0.2 probability drift, no speedup, and inconsistent behavior.
Our previous model’s probability distribution when running against the same time used to test later. You see the blue and green are decently far apart.
This time, we pointed our Möbius prototype at v6 from scratch, with only select context from data we’ve collected. The goal wasn’t another porting example. We wanted to show that with the right context and knowledge, an AI agent can search, reason, verify, and optimize, turning weeks of work into hours while finding optimizations most humans miss. Just as Vercel abstracted infrastructure for web apps, we want bringing models to edge applications to be just as easy.
Why we’re building Möbius: edge software lags the hardware
Möbius is named after the Möbius strip
Our team’s background is in traditional ML and distributed systems: recommendation systems and content moderation serving over a billion users, and years of scaling infrastructure across AWS, Azure, and GCP. Edge is newer territory for us, but many distributed systems concepts translate surprisingly well to edge optimization.
After building edge AI applications and shipping them to users, we quickly realized the hardware is catching up, but the software layer is far behind and not improving fast enough. Near-realtime workloads on consumer CPUs and GPUs were too slow and drained the battery. Meanwhile, enterprises and healthcare providers started asking us how to run modern models locally for privacy and compliance.
Some solutions for running local models exist, but most are only partially open or integrate poorly with native applications. Instead of waiting for someone else to fix that, we decided to tackle it ourselves and share our models and SDKs with everyone.
The ecosystem is fragmented and moving fast, with new models shipping on different architectures. Rather than a traditional static compiler, we’re using AI itself. Möbius is an agent that captures these optimization patterns, making edge AI accessible to everyone, not just hardware or ML engineers, and letting teams apply their existing distributed systems expertise to the constraints of on-device inference.
Core ML is more than Apple’s Neural Engine
Before getting into how Möbius solved this, it helps to understand what Core ML is. It’s the only developer interface to Apple’s Neural Engine (ANE), but it’s also Apple’s complete on-device ML runtime, targeting the CPU, GPU, and ANE.
Core ML uses heuristics, operator support tables, and runtime constraints to decide where each part of your model runs. The model configuration’s computeUnits setting lets you limit or prefer certain compute units, but placement isn’t guaranteed.
Core ML’s strength is its ecosystem integration, not just hardware access. Ahead-of-time compilation compiles models before runtime, which helps startup latency and enables graph-level optimizations. The unified memory architecture lets the CPU, GPU, and ANE share one memory pool, so data isn’t copied between compute units.
On CPU, Core ML uses Accelerate and Basic Neural Network Subroutines (BNNS) for many operations and may fuse them. It supports weight palettization for compression, as well as traditional quantization, depending on the OS version and target. Automatic compute unit selection dispatches each operation to the best unit based on support and cost.
The original illustration's twelve floating-point weights become twelve two-bit indices and a four-entry lookup table. All displayed input weights, stored indices, and lookup values are transcribed from the original figure.
The original example uses a scale of 2.35 and an optional bias of zero. The displayed INT8 weights reconstruct approximate floating-point weights through scale times INT8 plus bias. All values are transcribed from the original illustration.
As with any abstraction, there are trade-offs. Apple doesn’t expose its internal decision logic, and there are strict requirements for input/output dimensions, internal state, and static data structures. Not all operations convert cleanly, so you often have to remap them to Core ML’s supported set.
Model conversion is translation, not export
Porting a model to another runtime is like rewriting code in a different programming language. You lower the same operator graph into the target runtime’s supported operations without changing the math or state semantics.
The challenges are familiar. Not all operators have direct equivalents, especially control flow. Most accelerators require static graphs with fixed input dimensions, which clashes with dynamic graphs and internal state. And tracing has limits, so not every operation exports cleanly.
Despite these hurdles, porting to a native runtime is worth it. You get vendor-tuned optimizations, and performance can be significantly better, sometimes even when the final model runs only on CPU, thanks to ahead-of-time graph compilation and optimized memory layouts.
In theory, someone could write custom kernels that outperform Core ML, but that’s rare. On Apple devices, skipping Core ML means losing Apple’s internal optimizations, which are often opaque and restricted.
Möbius: a coding agent for model porting
Rather than building new models or writing kernels from scratch, we’re developing Möbius as a specialized agent for model conversion. Like other coding agents, it works alongside engineers, handling the repetitive and complex parts while humans guide and verify. It’s an orchestration layer that combines frontier LLMs like Claude with domain-specific tooling, memory for continuous learning, and models we’re training specifically for runtime optimization.
This is the loop from the intro: each conversion teaches the system new patterns. We deliberately collect true positives: cases where the agent got blocked, along with their solutions. We’ve found this steers the agent toward good outcomes better than true negatives, which more often lead it down the wrong path.
The Silero VAD repository, whitepapers, documentation, examples, and knowledge feed a research agent. It works with a planning agent, which hands the plan to an execution agent.
Our roadmap leads toward more autonomy, but today the prototype is an AI coding partner. A developer still drives, but instead of spending weeks debugging numerical differences and rewriting operations, they guide the agent through the conversion in hours. The goal is to give any developer the kind of model optimization that used to require specialized ML engineering expertise.
512 input samples pass through a sampling-rate check and an 8kHz versus 16kHz conditional branch. Each branch has six convolution layers, five ReLU activations, normalization, and state updates. The result is speech probability and updated state, matching the original schematic.
Mapping the model and breaking open the black box
The prototype starts by understanding what it’s working with: gathering requirements and clarifying the model’s use case. Typically, users provide access to the existing model plus the target platform and programming language for inference.
For this test, our team member’s prompt was a link to the Silero VAD GitHub repository and the requirement that the model run on macOS 13+ and iOS 15+. A higher minimum OS version might unlock more optimizations, but users are often slow to upgrade, so we prefer to support more of them first.
The agent then studies the model and researches how to optimize it for those requirements. After about 10 minutes, the prototype had mapped the model’s execution flow and chosen 16kHz, matching our existing Parakeet TDT v3 Core ML transcription model.
The tricky part is that Silero VAD ships as a TorchScript (JIT) model, pre-traced and heavily optimized for mobile and libtorch execution. JIT models are packaged to run faster and more reliably outside a developer’s laptop, but they lose the granular details of the original model, which makes them harder to convert.
The ONNX path failed after 30 minutes when the agent hit the If operator, which Core ML doesn’t support. Since Core ML Tools has dropped ONNX support, the agent abandoned this approach after trying several version permutations.
This is where the prototype showed promise. After finding the convert-gguf.py script from whisper.cpp, it realized it could rebuild the nn.Module classes from scratch in PyTorch using the provided weights. It loaded the weights, created separate Short-Time Fourier Transform (STFT), encoder, and decoder models, traced each one, and verified that each matched the baseline. Our manual attempt never considered this strategy.
Finding and fixing three critical bugs
The initial conversion looked successful, but with real audio, everything fell apart. Outputs differed from the baseline by up to 30%, which makes a VAD unusable.
Accuracy compared to baseline model over time
When we prompted the agent to review its work, we discovered it had been comparing against its simplified baseline, not the actual PyTorch JIT model, a common failure mode for coding agents. Once corrected, the agent systematically identified three root causes that compounded into complete failure.
This kind of systematic debugging is the real value of AI-assisted porting. Our manual attempt, aided by ChatGPT and Claude, tried band-aids like Squeeze-and-Excitation modifications without finding these underlying issues. An agent with the right context and tooling helped us find the real problems instead of patching symptoms.
256ms windows made it 3.5× faster
With correctness solved, the prototype turned to speed. The initial Core ML model wasn’t much faster than PyTorch JIT, which defeated one purpose of converting.
Here the agent found something our team initially missed: processing 256ms windows instead of 32ms chunks gave dramatic speedups. This isn’t traditional batching of independent samples in parallel. It’s just passing a longer audio window to the model in a single call.
The base model processes eight successive 32ms slices in eight calls, each with 64 samples of context. The 256ms model handles the same eight slices in one call: 4096 new samples at 16kHz plus an initial 64-sample context, with context carried between slices.
The speedup comes from Core ML’s ahead-of-time graph optimizations, optimized BNNS and Accelerate kernels, and a single-call unified graph that avoids per-window dispatch overhead, which improves cache utilization and memory access patterns.
To combine the eight probabilities from each 256ms window, the agent chose noisy-OR over mean or max, treating each 32ms slice as an independent expert vote. Noisy-OR better models whether there was voice anywhere in the last 256ms: unlike mean, it stays sensitive to brief speech, and unlike max, it combines evidence across slices, so several moderate probabilities add up to a confident detection. Together with unified graph execution in Core ML, this led to a 3.5× inference speedup.
Why this model runs faster on CPU than ANE
The agent also tried quantization and palettization, reducing the model from float32 to float16 or int8. Quantization gives big speedups for most large language models, but that’s generally not true for Core ML models. At around 2MB, Silero VAD isn’t memory-bound, so dequantization overhead outweighs any savings.
Performance comparison
Like any optimization, these techniques trade away some accuracy, and how much depends on the model’s architecture. The trade-offs show up clearly when comparing size against latency and accuracy against compression rate.
The agent didn’t try other techniques like pruning, which we still need to explore.
Accuracy comparison
Profiling the mlpackage in Xcode gave an unexpected result: the Core ML model ran 100% on CPU. Comparing CPU-only and CPU+NE settings confirmed it, with latency differing by < 0.05ms.
Accuracy and compression trade-off graph generated by Möbius
Xcode performance benchmark
We had expected some operations to run on the ANE. Profiling showed that although the operations could technically run on the GPU and ANE, Core ML’s execution engine decided CPU-only was optimal.
Grey checkmarks mean they could technically support GPU and Neural Engine, but Core ML chose CPU
The likely reason is model size. At just 2MB, the overhead of moving data to the ANE outweighs any compute benefit, and operations like STFT and custom convolutions are simply more efficient on CPU at this scale. Apple documents little about Core ML’s internal decisions, so this is our best understanding of its behavior.
From two weeks to twelve hours
Even running entirely on CPU, converting to the native runtime unlocked significant performance gains.
With the Möbius prototype, development time dropped from two weeks to 12 hours, a 93% reduction. The developer still made key decisions and verified results, while the agent handled the heavy lifting: debugging, testing variations, and systematically exploring optimizations. Accuracy stayed near-perfect at 0.99998 correlation, versus ±0.15 to 0.2 drift for the broken manual attempt, measured on a 60-minute video with multiple speakers and varying volume. Most importantly, we got a 3.5× speedup versus none from the manual conversion.
Quality comparison generated by Möbius
Performance comparison generated by Möbius
The agent didn’t just speed up human work. Together, the human-AI team found optimizations our manual attempt missed: 256ms window processing with noisy-OR aggregation wasn’t in our playbook. The agent systematically explored the solution space, verified each step, and found a better path.
The 256ms approach doesn’t work for use cases that need close to 32ms processing, but even the 32ms Core ML model runs at 33 RTFx, about 8% faster than the baseline PyTorch model, with much less performance jitter.
Where Möbius still struggles: complex models and LLMs
Our prototype handles simpler models like VAD well, but more complex architectures are still a challenge. Models with dynamic shapes like Kokoro still need significant human intervention. We’re working to extend Möbius to these cases and exploring how it can rewrite certain operations to make better use of the ANE.
For scale, Kokoro has over 2,000 operations compared to Silero VAD’s 48. Xcode’s model structure visualizations make the difference obvious.
Model Structure of VAD by Xcode
Kokoro Model is so complicated that you have to zoom out 50x to see the whole model in Xcode
Native restrictions also limit some workloads on accelerators like the ANE and other NPUs. For LLMs, context windows are often restricted to under 5,000 tokens, and memory bandwidth becomes the main bottleneck. For now, Apple’s MLX framework or llama.cpp remain better options for local LLM inference on Apple devices.
Each successful conversion adds to our knowledge base, and we’re already seeing community results like CAM++ achieving 3× speedups after converting to Core ML.
From this project, we learned how to handle PyTorch JIT models when the original PyTorch model isn’t available by working directly with the weights. The VAD model is too small to benefit from the ANE, but the native runtime still gives us more room to optimize. Eventually, Möbius may be able to redesign models to run entirely on the Neural Engine.
The real bottleneck is optimization expertise
Getting models to run well on edge devices is crucial for AI’s future. Devices have limited resources, so every bit of performance matters, especially for ambient, always-on workloads. Unified runtimes offer portability but often lack the performance and efficiency of native runtimes, which hardware makers optimize and which give more room to tune models, even on CPU. Yet few people have the expertise for this work, which means understanding multiple frameworks and debugging subtle numerical differences. Properly structured agents with domain knowledge can handle that complexity.
This case study shows the potential of AI-assisted model conversion. We’ve tested the approach on several models beyond Silero VAD, but this is early work with much more to explore. The same approach that made VAD 3.5× faster extends naturally to emerging edge scenarios: fleet management running vision models on heterogeneous vehicle hardware, ambient monitoring with multimodal models on battery-powered sensors, and real-time data collection on industrial IoT devices.
As powerful multimodal models like Moondream3 and VoxCPM-0.5B emerge, the gap between cloud capabilities and edge constraints keeps widening, making systematic optimization more important. What works for a 2MB VAD on Core ML informs how we’ll optimize tomorrow’s foundation models for tomorrow’s edge accelerators.
The bottleneck in edge AI deployment isn’t the models or the hardware. It’s the optimization expertise gap. Just as GitHub Copilot changed how developers write code, we believe tools like Möbius can change how we optimize models for the edge, gradually blurring the line between ML engineers and software engineers.
At FluidInference, we’re building Möbius to be the Claude Code or Codex of model porting. We’ll share more as development continues and plan to open source the platform. Join our Discord to hear about it first, and if you’re working on edge AI or model optimization, we’d love to connect at [email protected].
Use the new VAD model: https://github.com/FluidInference/FluidAudio
Access Möbius’s models and conversion code: github.com/FluidInference/mobius
Join our community: https://discord.gg/WNsvaCtmDe