A model is also a movement problem
A local inference engine is balancing arithmetic, memory bandwidth, device memory, and the cost of moving weights. Once the model is larger than the comfortable working set, the schedule matters as much as the kernel.
A prototype schedule
Flatline groups GGUF tensors into layers. During its inference loop, it can ask the operating system to prefetch the next layer, run Metal work for the current layer, wait for the GPU command buffer, and mark the completed layer as no longer needed.
The design is intentionally simple: keep the current work available, prepare what comes next, and release what has finished. A production runtime would need more careful synchronization, profiling, and model-specific scheduling.
Why this matters for products
On-device AI is not only about shrinking a model. It is about building a runtime that respects the device. Scheduling, quantization, buffer ownership, and observability determine whether a technically capable model feels dependable in a real application.