Local is a product decision
Running a model on the device can remove round trips, protect sensitive input, reduce inference bills, and keep a workflow available without a network. Users should be able to feel those benefits. Excessive latency, memory use, or weaker output can make a local model the wrong choice for the product.
Define the user promise first. “Your meeting audio never leaves this Mac” is a meaningful architectural constraint. “We use local AI” without a clear user benefit is just implementation language.
Workloads that are strong local candidates
Compact, frequent, latency-sensitive tasks tend to fit on-device execution well, especially when the operating system already provides an optimized framework.
- Speech recognition, transcription, and text-to-speech with device-appropriate models.
- Classification, extraction, embeddings, and focused language tasks.
- Private search over a bounded local collection.
- Image and audio processing that benefits from Apple hardware acceleration.
- Features that must continue during travel, poor connectivity, or a cloud outage.
Measure the runtime, not the model card
A model download size does not tell you its peak memory use. The runtime may allocate weights, caches, intermediate tensors, audio buffers, and decoded output at the same time. Unified memory also belongs to the operating system and every other process on the Mac.
Benchmark the slow path on the lowest supported device. Measure first-run setup, warm and cold latency, peak memory, sustained thermal behavior, cancellation, and recovery after failure. Product routing rules should come from those measurements.
Design a hybrid boundary users can understand
A hybrid product can keep capture, playback, search, and common transformations local while reserving larger reasoning tasks for a controlled cloud service. The interface should make the boundary clear before data moves, not bury it in a privacy policy.
The engineering boundary matters too: local features should remain healthy when cloud authentication or a provider fails. Treat local runtime health and cloud AI health as separate systems with separate diagnostics.