Deploying Quantized Open-Weight LLMs on Local Edge Infrastructure

Why I stopped calling vendor APIs

Sending internal codebases, private documents, and customer logs to third-party commercial LLM endpoints creates two major problems:

1. Data leaves your local perimeter.
2. Costs scale with every token generated.

Open-weight models (Qwen 2.5, DeepSeek) running on local hardware eliminate data leaks and produce zero ongoing API fees.

---

Quantization Benchmarks: 4-Bit vs 8-Bit

Format	VRAM Footprint (14B Model)	Memory Bandwidth Saturation	Quality / Accuracy
FP16 (Unquantized)	~28 GB (Exceeds safe limits)	Saturated / Disk Paging	100% baseline
8-Bit (INT8)	~15 GB	Optimal on 32GB hardware	99.2% accuracy on coding benchmarks
4-Bit (INT4)	~8.5 GB	Maximum throughput (45+ t/s)	97.8% accuracy, fastest streaming

For daily code generation and document analysis, 4-bit quantized Qwen 2.5-Coder delivers the best balance of responsiveness and memory headroom.

---

The FastAPI Streaming Router

from fastapi import FastAPI
from fastapi.responses import StreamingResponse
import mlx_lm

app = FastAPI(title="Local LLM Server")
model, tokenizer = mlx_lm.load("Qwen/Qwen2.5-Coder-7B-Instruct-4bit")

@app.post("/v1/stream")
async def stream_inference(prompt: str):
    def event_generator():
        for chunk in mlx_lm.stream_generate(model, tokenizer, prompt, max_tokens=512):
            yield f"data: {chunk.text}\n\n"
    return StreamingResponse(event_generator(), media_type="text/event-stream")