Laya runs structured LLM inference offline on Apple's M4 via CoreML

A developer demo shows Laya, a structured question-answering inference tool, running entirely offline on an Apple M4 Mac through CoreML (repo: github.com/mizorewww/laya-coreml). Rather than returning free text, Laya takes an input string and a set of typed questions and answers each one — in the example, a customer message about payouts failing for three days is scored for urgency, returning a confidence value of roughly 0.79. Profiling of the local Python 3.12 process (ARM64, macOS 27, launched via uv) puts the physical memory footprint around 560MB, peaking near 778MB, which is modest enough to run on consumer Apple Silicon.

The author also wrapped the same functionality behind a Cloudflare-hosted HTTP endpoint, exposing a simple JSON API where callers POST a state string plus a dictionary of questions and get typed answers back. That pairing illustrates the appeal: the model can run locally for privacy and cost reasons, or be fronted by an edge API for shared access, using an identical request format either way.

Details remain thin — this is an informal gist and screen recording rather than a documented release — but the significance is the pattern it represents. Small, task-specific extraction and classification models are increasingly viable on-device via CoreML, letting teams pull structured signals from unstructured text without shipping data to a hosted LLM provider.