Configuration Reference

Define one or more routers. Each router name becomes part of the URL path when making requests.

routers:
  production:
    load-balance:
      chat:
        strategy: latency
        providers:
          - openai
          - anthropic
  
  development:
    load-balance:
      chat:
        strategy: weighted
        providers:
          - provider: anthropic
            weight: '0.9'
          - provider: openai
            weight: '0.1'

Usage: Set your OpenAI SDK baseURL to http://localhost:8080/production or http://localhost:8080/experimental

Use latency for automatic load balancing that routes to the provider with the lowest latency.

routers:
  production:
    load-balance:
      chat:
        strategy: latency
        providers:
          - openai
          - anthropic
          - gemini
          - ollama

Use weighted to distribute requests based on specific percentages.

routers:
  production:
    load-balance:
      chat:
        strategy: weighted
        providers:
          - provider: anthropic
            weight: '0.95'
          - provider: openai
            weight: '0.05'

Important: Weights must sum to exactly 1.0

List of target providers for load balancing.

For Latency Strategy:

providers:
  - openai
  - anthropic
  - gemini

For Weighted Strategy:

providers:
  - provider: anthropic
    weight: '0.7'
  - provider: openai
    weight: '0.3'

Define the cache storage backend. Must be configured at the top level.

cache-store:
  type: "in-memory"

cache-store:
  type: "in-memory"
  max-size: 10000  # Number of cached entries

cache-store:
  type: "in-memory"

cache-store:
  type: "in-memory"
  max-size: 10000  # Number of cached entries

cache-store:
  type: "redis"

cache-store:
  type: "redis"
  host-url: "redis://localhost:6379"

Configure response caching for a router.

cache-store:
  type: "in-memory"

routers:
  production:
    cache:
      directive: "max-age=3600, max-stale=1800"
      buckets: 10
      seed: "unique-cache-seed"

HTTP cache-control directive string.

cache:
  directive: "max-age=3600, max-stale=1800"

How it works: Defines cache freshness (max-age) and staleness tolerance (max-stale) in seconds for all requests. Optionally override with cache-control request headers.

Number of responses stored per cache key before random selection begins.

cache:
  buckets: 10

How it works: Stores n number of different responses for identical requests, then randomly selects from the stored responses to add variability.

Unique seed for cache key generation.

cache:
  seed: "unique-cache-seed"

How it works: Creates isolated cache namespaces - different seeds maintain separate cache spaces for the same requests.

Configure application-wide rate limits that apply to all requests.

global:
  rate-limit:
    store: in-memory
    per-api-key:
      capacity: 500
      refill-frequency: 1s

How it works: These limits are checked first for every request across all routers.

Configure additional rate limiting specific to this router (applied after global limits).

routers:
  production:
    rate-limit:
      per-api-key:
        capacity: 100
        refill-frequency: 1m

How it works: If global limits are configured, they’re checked first. Then these router-specific limits are applied as an additional layer.

Storage backend for rate limit counters.

store: in-memory

Options:

• in-memory - Local memory storage (default)
• redis - Redis storage (coming soon…)

Rate limits applied per API key.

per-api-key:
  capacity: 500
  refill-frequency: 1s

Maximum number of requests in the bucket (burst capacity).

per-api-key:
  capacity: 1000

How it works: This is the maximum number of requests that can be made instantly before rate limiting kicks in.

Time to completely refill the capacity bucket.

per-api-key:
  refill-frequency: 1s

How it works: With capacity=500 and refill-frequency=1s, you get 500 requests per second sustained rate.

How often to clean up expired rate limit entries.

cleanup-interval: 5m

Note: Only available for global rate limiting configuration.

Configure an application-wide retry policy that applies to all requests.

global:
  retries:
    strategy: "constant"
    delay: "50ms"
    max-retries: 3

How it works: If a transient error is encountered on any endpoint, the request will be retried with the given policy.

Configure retry policy specific to this router (overrides global retry settings).

routers:
  production:
    retries:
      strategy: "exponential"
      min-delay: "100ms"
      max-delay: "30s"
      max-retries: 5
      factor: 2.0

How it works: Router-specific retry settings take precedence over global retry configuration.

Retry backoff strategy.

retries:
  strategy: "constant"

Options:

• constant - Fixed delay between retry attempts with jitter
• exponential - Exponentially increasing delays with jitter and configurable bounds

Fixed delay between retry attempts (for constant strategy).

retries:
  strategy: "constant"
  delay: "50ms"

How it works: Each retry waits exactly this duration (plus jitter) before attempting again.

Maximum number of retry attempts.

retries:
  strategy: "constant"
  max-retries: 3

How it works: After this many failed attempts, the request fails permanently.

Minimum delay for exponential backoff strategy.

retries:
  strategy: "exponential"
  min-delay: "100ms"

How it works: First retry waits this duration, then each subsequent retry increases exponentially.

Maximum delay cap for exponential backoff strategy.

retries:
  strategy: "exponential"
  max-delay: "60s"

How it works: Delays will never exceed this value, even with exponential growth.

Maximum number of retry attempts.

retries:
  strategy: "exponential"
  max-retries: 5

How it works: After this many failed attempts, the request fails permanently.

Exponential backoff multiplication factor.

retries:
  strategy: "exponential"
  factor: 1.5

How it works: Each retry delay is multiplied by this factor. With factor=2.0: 100ms → 200ms → 400ms → 800ms.

Enable Helicone features such as LLM observability and authentication for secure API access.

helicone:
  # Set to `features: observability` to enable observability and auth
  features: auth

Enable authentication (required) and request logging to your Helicone dashboard.

helicone:
  features: observability

When enabled: Must set the HELICONE_CONTROL_PLANE_API_KEY environment variable with your Helicone API key when deploying the AI Gateway when using Helicone add ons.

Helicone API endpoint URL.

helicone:
  base-url: "https://api.helicone.ai"

Note: Only change this if you’re self-hosting Helicone. Use the default for Helicone Cloud.

WebSocket URL for control plane connection.

helicone:
  websocket-url: "wss://api.helicone.ai/ws/v1/router/control-plane"

Note: Only change this if you’re self-hosting Helicone. Use the default for Helicone Cloud.

Configure provider settings to override defaults.

providers:
  anthropic:
    base-url: "https://api.anthropic.com"
    version: "2023-06-01"
    models:
      - claude-3-5-haiku
  
  ollama:
    base-url: "http://192.168.1.100:11434"
    models:
      - llama3.2
      - deepseek-r1
      - custom-fine-tuned-model
  
  bedrock:
    base-url: "https://bedrock-runtime.us-west-2.amazonaws.com"
    models:
      - anthropic.claude-3-5-sonnet-20241022-v2:0
      - anthropic.claude-3-haiku-20240307-v1:0

API endpoint URL for the provider.

providers:
  openai:
    base-url: "https://api.openai.com"

List of supported models for this provider.

providers:
  openai:
    models:
      - gpt-4
      - gpt-4o
      - gpt-4o-mini

API version (required for some providers like Anthropic).

providers:
  anthropic:
    version: "2023-06-01"

Router-specific model mappings for fallback when requested model isn’t available.

routers:
  production:
    model-mappings:
      gpt-4o: claude-3-opus
      claude-3-5-sonnet: gemini-1.5-pro
      gpt-4o-mini: claude-3-5-sonnet

Global fallback mappings used when router-specific mappings aren’t defined.

default-model-mapping:
  gpt-4o: claude-3-opus
  gpt-4o-mini: claude-3-5-sonnet
  claude-3-5-sonnet: gemini-1.5-pro

Logging level in env logger format.

telemetry:
  level: "info"

Common patterns:

• "info" - General information for all modules, recommended for production
• "info,ai_gateway=debug" - Debug for dependencies, info for gateway, recommended for development

Telemetry data export destination.

telemetry:
  exporter: "otlp"

Options:

• stdout - Export telemetry data to standard output (default)
• otlp - Export telemetry data to OTLP collector endpoint
• both - Export to both stdout and OTLP collector

OTLP collector endpoint URL.

telemetry:
  otlp-endpoint: "http://localhost:4317"

Add helicone-provider header showing which provider handled the request.

response-headers:
  provider: true

When enabled: Responses include header like helicone-provider: openai or helicone-provider: anthropic.

Add helicone-provider-req-id header showing the provider’s request ID.

response-headers:
  provider-request-id: true

When enabled: Responses include header like helicone-provider-req-id: req-12345 for request tracing.

Health monitoring strategy.

discover:
  monitor:
    health:
      ratio: 0.1
      window: 60s
      grace-period:
        min-requests: 20

Options:

• error-ratio - Monitor based on error rate thresholds (only option currently)

Error ratio threshold (0.0-1.0) that triggers provider removal.

discover:
  monitor:
    health:
      ratio: 0.15

How it works: If errors/requests exceeds this ratio, provider is marked unhealthy and removed from load balancing.

Time window for measuring error ratios.

discover:
  monitor:
    health:
      window: 60s

How it works: Rolling window size for calculating error rates.

Minimum requests required before health monitoring takes effect.

discover:
  monitor:
    health:
      grace-period:
        min-requests: 20

How it works: Providers won’t be marked unhealthy until they’ve handled at least this many requests.