How we cut median latency to 38ms with speculative decoding
A look at the runtime changes that let us serve large models faster without sacrificing output quality.
Blog
Engineering deep-dives, product updates and notes from the team.
Zero data retention, explained
What "we never train on your data" actually means at the infrastructure level, and how we prove it.
Building reliable RAG: chunking strategies that actually work
Lessons from indexing hundreds of millions of documents across customer workloads.
Introducing multi-region failover
Your requests now automatically route to the nearest healthy region with transparent failover.
The economics of running open models at scale
How we think about GPU utilization, batching and cost per token.