How Should We Buy LLM Compute?

For a product like AnswerLayer, customers in healthcare, finance, legal, and public sector can't send their data to third-party APIs. Data sovereignty is becoming the dominant paradigm—governments worldwide are mandating local storage and restricting cross-border transfers. Europe has issued over €5.65 billion in GDPR fines since 2018, with the EU AI Act adding new obligations in 2026.

Local inference keeps sensitive data entirely on-premises. No API logs, no third-party training pipelines, no policy changes from providers. For regulated industries, this isn't optimization—it's compliance.

Local hardware isn't always the answer. Cloud and API providers offer real advantages: no hardware procurement, no maintenance, no capacity planning. For teams without dedicated infrastructure expertise, managed services eliminate entire categories of operational burden.

SLAs matter too. Cloud providers guarantee uptime, offer redundancy across regions, and handle failover automatically. Local hardware is a single point of failure unless you invest in redundancy—which multiplies the cost.

The API providers above offer access to both proprietary and open-source models. What's changed is that open models now compete on quality. According to the Artificial Analysis leaderboard, open models like DeepSeek R1, Qwen 2.5, and Llama 4 now rival closed models on key benchmarks. Qwen has overtaken Llama as the most-downloaded base model for fine-tuning. For many applications—summarization, extraction, code generation—the gap with proprietary models has effectively closed.

There's another factor to consider: today's API prices are heavily subsidized by VC funding. OpenAI has raised over $78B and Anthropic over $33B—much of it deployed to capture market share at below-cost pricing. When these subsidies end (and they will), API costs could rise significantly. Google's pricing is an exception—not VC-dependent—but still subject to competitive pressure that may not last.

For applications where data stays local anyway, the choice isn't about quality anymore. It's about cost predictability, latency, and control.

One insight worth highlighting: bigger isn't always better. Research shows that fine-tuned smaller models significantly outperform zero-shot generative AI models like ChatGPT and Claude on text classification tasks. Together AI documented a case where a fine-tuned Gemma 3 27B outperformed Claude Sonnet by 60% on healthcare scribing—at 10-100× lower inference cost.

With techniques like LoRA, teams can fine-tune models for specific domains with modest compute. The pattern is consistent: task-specific training data often beats raw parameter count—and these specialized models can run on hardware that fits under a desk.

This calculator makes several simplifying assumptions. It assumes consistent daily usage—real workloads are often bursty. Hardware depreciation and electricity costs are excluded, which favors local hardware. Cloud providers may impose minimum commitments not reflected here. We focus on the Mac Studio and DGX Spark because they're readily available—traditional GPU workstations (RTX 4090/5090) face ongoing supply constraints that make them difficult to source at reasonable prices. The calculator shows throughput (tokens/second) but not time-to-first-token latency, which matters for interactive applications.