LLM Caching

When developing and testing LLM applications, you often make the same requests repeatedly during debugging and iteration. Caching stores responses on the edge using Cloudflare Workers, eliminating redundant API calls and reducing both latency and costs.

​

Why use Caching

• Save money during development: Avoid repeated charges for identical requests while testing and debugging
• Reduce response latency: Serve cached responses instantly instead of waiting for LLM providers
• Handle traffic spikes: Protect against rate limits and maintain performance during high usage

Dashboard view of cache hits, cost and time saved

​

Quick Start

{
  "Helicone-Cache-Enabled": "true"
}

const response = await openai.chat.completions.create(
  {
    model: "gpt-4o-mini",
    messages: [{ role: "user", content: "Hello world" }]
  },
  {
    headers: {
      "Helicone-Cache-Enabled": "true"
    }
  }
);

// This exact same request will return a cached response
const cachedResponse = await openai.chat.completions.create(
  {
    model: "gpt-4o-mini", 
    messages: [{ role: "user", content: "Hello world" }]
  },
  {
    headers: {
      "Helicone-Cache-Enabled": "true"
    }
  }
);

​

Configuration Options

​

Basic Settings

Control caching behavior with these headers:

​

Advanced Settings

​

Use Cases

import OpenAI from "openai";

const openai = new OpenAI({
  apiKey: process.env.OPENAI_API_KEY,
  baseURL: "https://oai.helicone.ai/v1",
  defaultHeaders: {
    "Helicone-Auth": `Bearer ${process.env.HELICONE_API_KEY}`,
    "Helicone-Cache-Enabled": "true",
    "Cache-Control": "max-age=86400" // Cache for 1 day during development
  },
});

// This request will be cached
const response = await openai.chat.completions.create({
  model: "gpt-4o-mini",
  messages: [{ role: "user", content: "Explain quantum computing" }]
});

// Subsequent identical requests return cached response instantly

import OpenAI from "openai";

const openai = new OpenAI({
  apiKey: process.env.OPENAI_API_KEY,
  baseURL: "https://oai.helicone.ai/v1",
  defaultHeaders: {
    "Helicone-Auth": `Bearer ${process.env.HELICONE_API_KEY}`,
    "Helicone-Cache-Enabled": "true",
    "Cache-Control": "max-age=86400" // Cache for 1 day during development
  },
});

// This request will be cached
const response = await openai.chat.completions.create({
  model: "gpt-4o-mini",
  messages: [{ role: "user", content: "Explain quantum computing" }]
});

// Subsequent identical requests return cached response instantly

// Use a consistent identifier for the code snippet
const codeIdentifier = `code-${codeSnippet.length}-${codeSnippet.slice(0, 20)}`;

const response = await openai.chat.completions.create(
  {
    model: "gpt-4o-mini",
    messages: [{ 
      role: "user", 
      content: `Explain this code:\n\n${codeSnippet}` 
    }]
  },
  {
    headers: {
      "Helicone-Cache-Enabled": "true",
      "Helicone-Cache-Seed": codeIdentifier,    // Same code = same cache
      "Cache-Control": "max-age=604800"         // Cache for 1 week
    }
  }
);

// Multiple developers asking about the same function get instant responses

const response = await openai.chat.completions.create(
  {
    model: "gpt-4o-mini",
    messages: [{ 
      role: "user", 
      content: "How do I authenticate with the API?" 
    }]
  },
  {
    headers: {
      "Helicone-Cache-Enabled": "true",
      "Cache-Control": "max-age=86400",          // Cache for 24 hours
      "Helicone-Cache-Bucket-Max-Size": "1"     // Consistent answers
    }
  }
);

// Common documentation questions get instant, consistent responses
// Perfect for chatbots, help widgets, and FAQ systems

​

Understanding Caching

​

How Cache Keys Work

Helicone generates cache keys by hashing:

• Cache seed: If specified (for namespacing)
• Request URL: The full endpoint URL
• Request body: Complete request payload including all parameters
• Relevant headers: Authorization and Helicone-specific cache headers
• Bucket index: For multi-response caching

What triggers cache hits:

// ✅ Cache hit - identical requests
const request1 = { model: "gpt-4o-mini", messages: [{ role: "user", content: "Hello" }] };
const request2 = { model: "gpt-4o-mini", messages: [{ role: "user", content: "Hello" }] };

// ❌ Cache miss - different content  
const request3 = { model: "gpt-4o-mini", messages: [{ role: "user", content: "Hi" }] };

// ❌ Cache miss - different parameters
const request4 = { model: "gpt-4o-mini", messages: [{ role: "user", content: "Hello" }], temperature: 0.5 };

​

Cache Response Headers

Check cache status in response headers:

const response = await openai.chat.completions.create(
  { /* your request */ },
  { 
    headers: { "Helicone-Cache-Enabled": "true" }
  }
);

// Access raw response to check headers
const chatCompletion = await client.chat.completions.with_raw_response.create(
  { /* your request */ },
  { 
    headers: { "Helicone-Cache-Enabled": "true" }
  }
);

const cacheStatus = chatCompletion.http_response.headers.get('Helicone-Cache');
console.log(cacheStatus); // "HIT" or "MISS"

const bucketIndex = chatCompletion.http_response.headers.get('Helicone-Cache-Bucket-Idx');
console.log(bucketIndex); // Index of cached response used

​

Cache Bucket Behavior

Cache buckets store multiple responses for the same request:

Bucket Size 1 (default):

• Same request always returns same cached response
• Deterministic behavior

Bucket Size > 1:

• Same request can return different cached responses
• Useful for creative prompts where variety is desired
• Response chosen randomly from bucket

​

Cache Limitations

• Maximum duration: 365 days
• Maximum bucket size: 20 (enterprise plans support more)
• Cache key sensitivity: Any parameter change creates new cache entry
• Storage location: Cached in Cloudflare Workers KV (edge-distributed), not your infrastructure

​

Related Features