2025 · Local AI / Systems
Self-Hosted LLM Inference Server
Open-weight models running on Apple Silicon, behind a custom FastAPI streaming router.
Local-first private LLM infrastructure
The problem
Sending internal codebases, documents, and customer logs to third-party commercial LLM endpoints creates two problems: sensitive data leaves the local network, and recurring per-token pricing scales aggressively with usage.
The constraint
A script that loads weights and answers one prompt is a demo, not a service. It must survive crashes unattended, authenticate callers, manage multiple model weights, and run on hardware you can buy once and put on a shelf rather than renting expensive cloud GPUs.
What I built
An M4 Mac Mini (32 GB unified memory, ~100 GB/s bandwidth) running Apple's MLX-LM array framework with 4-bit and 8-bit quantized Qwen 2.5 models. Built a Python FastAPI router with streaming Server-Sent Events (SSE), API key auth, hard context caps to prevent SSD swap, and a macOS launchd supervisor with KeepAlive for automatic crash recovery.
The outcome
Zero per-token bills and zero data leaving the network. The server runs quietly on a desk with 45+ tokens/second throughput on 7B/14B models.