Run Llama on Groq for High-Speed Inference

Groq provides cloud inference for open-source models including Llama 3.1 on their custom LPU (Language Processing Unit) hardware, delivering 400-900 tokens per second — 10-30x faster than typical GPU inference. Groq's API is OpenAI-compatible, making it a drop-in replacement for applications that already use the OpenAI SDK. This example shows a Groq API request for Llama 3.1 70B that would run in under a second. Groq's key differentiator is latency, not just throughput. Time-to-first-token (TTFT) on Groq is typically under 200ms, compared to 2-5 seconds on standard GPU inference for large models. This makes Groq particularly valuable for real-time applications — live voice assistants, interactive coding tools, and streaming chat interfaces where users can see the response begin immediately. The free tier provides generous token allowances for testing. The models available on Groq include Llama 3.1 8B, 70B, and 405B, as well as Mixtral, Gemma, and other open-source models. Model availability changes as Groq partners with model providers, so check their models page for the current list. The API uses model IDs like "llama-3.1-70b-versatile" rather than Ollama-style format strings.

{
  "model": "llama-3.1-70b-versatile",
  "messages": [
    {
      "role": "system",
      "content": "You are a helpful assistant. Be concise and accurate."
    },
    {
      "role": "user",
      "content": "Explain the CAP theorem in distributed systems and give a real-world example of a system that prioritizes availability over consistency."
    }
  ],
  "temperature": 0.5,
  "max_tokens": 600,
  "stream": true
}