How to Integrate a Model Provider to the AI Gateway

Overview

Adding a new provider to Helicone involves several key components:

Authors: Companies that create the models (e.g., OpenAI, Anthropic)
Models: Individual model definitions with pricing and metadata
Providers: Inference providers that host models (e.g., OpenAI, Vertex AI, DeepInfra, Bedrock)
Endpoints: Model-provider combinations with deployment configurations

Prerequisites

OpenAI-compatible API (recommended for simplest integration)
Access to provider’s pricing and inference documentation
Model specifications (context length, supported features)
API authentication details

Step 1: Understanding the File Structure

All model support configurations are located in the packages/cost/models directory:

packages/cost/models/
├── authors/           # Model creators (companies)
├── providers/         # Inference providers
├── build-indexes.ts   # Builds maps for easy data access
├── calculate-cost.ts  # Cost calculation utilities
├── provider-helpers.ts # Helper methods
└── registry-types.ts  # Type definitions (requires updates)

Step 2: Create Provider Definition

We will use DeepInfra as our example.

For OpenAI-Compatible Providers

Create a new file in packages/cost/models/providers/[provider-name].ts:

import { BaseProvider } from "./base";

export class DeepInfraProvider extends BaseProvider {
  readonly displayName = "DeepInfra";
  readonly baseUrl = "https://api.deepinfra.com/";
  readonly auth = "api-key" as const;
  readonly pricingPages = ["https://deepinfra.com/pricing/"];
  readonly modelPages = ["https://deepinfra.com/models/"];

  buildUrl(): string {
    return `${this.baseUrl}v1/openai/chat/completions`;
  }
}

Make sure to look up the correct endpoints and override anything that is not OpenAI API default. This handles auth because the BaseProvider class handles the standard Bearer ${apiKey} authentication pattern automatically when you set auth = "api-key", which is the common pattern for OpenAI-compatible APIs.

For Non-OpenAI Compatible Providers

For non-OpenAI compatible providers, you’ll need to override additional methods. You can find options by reviewing the BaseProvider definition.

export class CustomProvider extends BaseProvider {
  // ... basic configuration

  buildBody(request: any): any {
    // Custom body transformation logic
    return transformedRequest;
  }

  buildHeaders(authContext: AuthContext): Record<string, string> {
    // Custom header logic
    return customHeaders;
  }
}

Step 3: Add Provider to Index

Update packages/cost/models/providers/index.ts:

import { DeepInfraProvider } from "./deepinfra";

export const providers = [
	/// ...
	deepinfra: new DeepInfraProvider(),
]

Step 4: Add Provider to the Web’s Data

Update web/data/providers.ts to include the new provider:

...,
  {
    id: "deepinfra",
    name: "DeepInfra",
    logoUrl: "/assets/home/providers/deepinfra.webp",
    description: "Configure your DeepInfra API keys for fast and affordable inference",
    docsUrl: "https://docs.helicone.ai/getting-started/integration-methods",
    apiKeyLabel: "DeepInfra API Key",
    apiKeyPlaceholder: "...",
    relevanceScore: 40,
  },
  ...

Step 5: Update provider helpers

Include provider in packages/cost/models/provider-helpers.ts within the heliconeProviderToModelProviderName function, so the mapping is done by the AI Gateway correctly.

case "DEEPINFRA":
    return "deepinfra";
case "NOVITA":
    return "novita";

Also, go to the getUsageProcessor function within packages/cost/usage.ts and add the provider. If your provider require a custom usage processor (non-OpenAI compatible), you will need to add it here.

export function getUsageProcessor(
  provider: ModelProviderName
): IUsageProcessor | null {
  switch (provider) {
    case "openai":
    case "azure":
    case "chutes":
    case "deepinfra":
		//.... <add new provider>
    default:
      return null;
  }
}

Step 6: Add provider to priorities list

We need to add the provider to the list of priorities so the gateway knows how much to prioritize each provider. Go to packages/cost/models/providers/priorities.ts and include your provider within the PROVIDER_PRIORITIES constant variable.

export const PROVIDER_PRIORITIES: Record<ModelProviderName, number> = {
  // Priority 1: BYOK (Bring Your Own Key) - Reserved for user's own API keys
  // Priority 2: Helicone-hosted inference
  helicone: 2,
  // Priority 3: Premium direct providers
  anthropic: 3,
  openai: 3,
  //... <add new provider>
  deepinfra: 4,
} as const;

Step 7: Update provider setup for tests

Head to worker/test/setup.ts and include your new provider within the supabase-js mock.

vi.mock("@supabase/supabase-js", () => ({
  createClient: vi.fn(() => ({
	  // ....
    deepinfra: {
          org_id: "0afe3a6e-d095-4ec0-bc1e-2af6f57bd2a5",
          provider_name: "deepinfra",
          decrypted_provider_key: "helicone-deepinfra-api-key",
          decrypted_provider_secret_key: null,
          auth_type: "api_key",
          config: null,
          byok_enabled: true,
        },
    // ...
  })
})

Step 8: Define Authors (Model Creators)

Create author definitions in packages/cost/models/authors/[author-name]/:

Folder Structure

authors/mistralai/    # Author name
└── mistral-nemo      # Model family
	└── endpoints.ts    # Model-provider combinations
	└── models.ts       # Model definitions
└── index.ts          # Exports
└── metadata.ts       # Metadata about the author

models.ts

Include the model within the models object. This can contain all model versions within that model family, in this case, the mistral-nemo model family. Make sure to research each value and include the tokenizer in the Tokenizer interface type if it is not there already.

import type { ModelConfig } from "../../../types";

export const models = {
  "mistral-nemo": {
    name: "Mistral: Mistral-Nemo",
    author: "mistralai",
    description:
      "The Mistral-Nemo-Instruct-2407 Large Language Model (LLM) is an instruct fine-tuned version of the Mistral-Nemo-Base-2407. Trained jointly by Mistral AI and NVIDIA, it significantly outperforms existing models smaller or similar in size.",
    contextLength: 128_000,
    maxOutputTokens: 16_400,
    created: "2024-07-18T00:00:00.000Z",
    modality: { inputs: ["text", "image"], outputs: ["text"] },
    tokenizer: "Tekken",
  },
} satisfies Record<string, ModelConfig>;

export type MistralNemoModelName = keyof typeof models;

endpoints.ts

Now, update the packages/models/[author]/[model-family]/endpoints.ts file with model-provider endpoint combinations. Make sure to review the provider’s page itself since the inference cost changes per provider. Make sure the initial key "mistral-nemo:deepinfra" is human-readable and friendly. It’s what users will call!

import { ModelProviderName } from "../../../providers";
import type { ModelProviderConfig } from "../../../types";
import { MistralNemoModelName } from "./models";

export const endpoints = {
  "mistral-nemo:deepinfra": {
    providerModelId: "mistralai/Mistral-Nemo-Instruct-2407",
    provider: "deepinfra",
    author: "mistralai",
    pricing: [
      {
        threshold: 0,
        input: 0.0000002,
        output: 0.0000004,
      },
    ],
    rateLimits: {
      rpm: 12000,
      tpm: 60000000,
      tpd: 6000000000,
    },
    contextLength: 128_000,
    maxCompletionTokens: 16_400,
    supportedParameters: [
      "max_tokens",
      "temperature",
      "top_p",
      "stop",
      "frequency_penalty",
      "presence_penalty",
      "repetition_penalty",
      "top_k",
      "seed",
      "min_p",
      "response_format",
    ],
    ptbEnabled: false,
    endpointConfigs: {
      "*": {},
    },
  }
} satisfies Partial<
  Record<`${MistralNemoModelName}:${ModelProviderName}` | MistralNemoModelName, ModelProviderConfig>
>;

Two important things to note here:

Some providers have multiple deployment regions:

endpointConfigs: {
  "global": {
    pricing: [/* global pricing */],
    passThroughBillingEnabled: true,
  },
  "us-east": {
    pricing: [/* regional pricing */],
    passThroughBillingEnabled: true,
  },
}

Pricing Configuration

pricing: [
  {
    threshold: 0,              // Context length threshold
    inputCostPerToken: 0.0000005, // Always per million tokens
    outputCostPerToken: 0.0000015,
    cacheReadMultiplier: 0.1,   // Cache read cost (10% of input)
    cacheWriteMultiplier: 1.25, // Cache write cost (125% of input)
  },
  {
    threshold: 200000,         // Different pricing for >200k context
    inputCostPerToken: 0.000001,
    outputCostPerToken: 0.000003,
  },
],

Step 9: Add model to Author registries (if needed)

If the model family hasn’t been created, you will need to add it within the AI Gateway’s registry.

index.ts

Update packages/cost/models/authors/[author]/index.ts to include the new model family. You don’t need to update anything if the model family has already been created.

/**
 * Mistral model registry aggregation
 * Combines all models and endpoints from subdirectories
 */

import type { ModelConfig, ModelProviderConfig } from "../../types";

// Import models
import { models as mistralNemoModels } from "./mistral-nemo/models";

// Import endpoints
import { endpoints as mistralNemoEndpoints } from "./mistral-nemo/endpoints";

// Aggregate models
export const mistralModels = {
  ...mistralNemoModels,
} satisfies Record<string, ModelConfig>;

// Aggregate endpoints
export const mistralEndpointConfig = {
  ...mistralNemoEndpoints,
} satisfies Record<string, ModelProviderConfig>;

metadata.ts

Update packages/cost/models/authors/[author]/metadata.ts to fetch models. You don’t need to update anything if the author has already been created.

/**
 * Mistral metadata
 */

import type { AuthorMetadata } from "../../types";
import { mistralModels } from "./index";

export const mistralMetadata = {
  modelCount: Object.keys(mistralModels).length,
  supported: true,
} satisfies AuthorMetadata;

registry-types.ts

Update types for the new model family in packages/cost/models/registry-types.ts.

import { mistralEndpointConfig } from "./authors/mistralai";
import { mistralModels } from "./authors/mistralai";

const allModels = {
	...,
  ...mistralModels
};

const modelProviderConfigs = {
	...,
  ...mistralEndpointConfig
};

Add your new model to the packages/cost/models/registry.ts:

import { mistralModels, mistralEndpointConfig } from "./authors/mistral";

const allModels = {
	//...
	  ...mistralModels
} satisfies Record<string, ModelConfig>;

const modelProviderConfigs = {
	// ...
	  ...mistralEndpointConfig
} satisfies Record<string, ModelProviderConfig>;

Step 10: Create Tests

Create test files in worker/tests/ai-gateway/ for the author. Feel free to use the existing tests there as reference.

Step 11: Snapshots

Make sure to rerun snapshots before deploying.

cd <your-path-to-the-repo>/helicone/helicone/packages && npx jest -u

Common Issues & Solutions

Issue: Complex Authentication

Solution: Override the auth() method with custom logic:

auth(authContext: AuthContext): ComplexAuth {
  return {
    "Authorization": `Bearer ${authContext.providerKeys?.custom}`,
    "X-Custom-Header": this.buildCustomHeader(authContext),
  };
}

Issue: Non-Standard Request Format

Solution: Override the buildBody() method:

buildBody(request: OpenAIRequest): CustomRequest {
  return {
    // Transform OpenAI format to provider format
    prompt: request.messages.map(m => m.content).join('\\n'),
    max_tokens: request.max_tokens,
  };
}

Issue: Multiple Pricing Tiers

Solution: Use threshold-based pricing:

pricing: [
  { threshold: 0, inputCostPerToken: 0.0000005 },
  { threshold: 100000, inputCostPerToken: 0.000001 },
  { threshold: 500000, inputCostPerToken: 0.000002 },
]

Deployment Checklist

Provider class created with correct authentication
Models defined with accurate specifications
Endpoints configured with correct pricing
Registry types updated
Tests written and passing
Snapshots updated
Documentation updated
Pass-through billing tested (if applicable)
Fallback behavior verified

Getting Started

AI Gateway

Observability & Analytics

Prompt Management

Legacy Integrations

References

How to Integrate a Model Provider to the AI Gateway

Overview

Prerequisites

Step 1: Understanding the File Structure

Step 2: Create Provider Definition

For OpenAI-Compatible Providers

For Non-OpenAI Compatible Providers

Step 3: Add Provider to Index

Step 4: Add Provider to the Web’s Data

Step 5: Update provider helpers

Step 6: Add provider to priorities list

Step 7: Update provider setup for tests

Step 8: Define Authors (Model Creators)

Folder Structure

models.ts

endpoints.ts

Step 9: Add model to Author registries (if needed)

index.ts

metadata.ts

registry-types.ts

Step 10: Create Tests

Step 11: Snapshots

Common Issues & Solutions

Issue: Complex Authentication

Issue: Non-Standard Request Format

Issue: Multiple Pricing Tiers

Deployment Checklist

​Overview

​Prerequisites

​Step 1: Understanding the File Structure

​Step 2: Create Provider Definition

​For OpenAI-Compatible Providers

​For Non-OpenAI Compatible Providers

​Step 3: Add Provider to Index

​Step 4: Add Provider to the Web’s Data

​Step 5: Update provider helpers

​Step 6: Add provider to priorities list

​Step 7: Update provider setup for tests

​Step 8: Define Authors (Model Creators)

​Folder Structure

​models.ts

​endpoints.ts

​Step 9: Add model to Author registries (if needed)

​index.ts

​metadata.ts

​registry-types.ts

​Step 10: Create Tests

​Step 11: Snapshots

​Common Issues & Solutions

​Issue: Complex Authentication

​Issue: Non-Standard Request Format

​Issue: Multiple Pricing Tiers

​Deployment Checklist

Overview

Prerequisites

Step 1: Understanding the File Structure

Step 2: Create Provider Definition

For OpenAI-Compatible Providers

For Non-OpenAI Compatible Providers

Step 3: Add Provider to Index

Step 4: Add Provider to the Web’s Data

Step 5: Update provider helpers

Step 6: Add provider to priorities list

Step 7: Update provider setup for tests

Step 8: Define Authors (Model Creators)

Folder Structure

models.ts

endpoints.ts

Step 9: Add model to Author registries (if needed)

index.ts

metadata.ts

registry-types.ts

Step 10: Create Tests

Step 11: Snapshots

Common Issues & Solutions

Issue: Complex Authentication

Issue: Non-Standard Request Format

Issue: Multiple Pricing Tiers

Deployment Checklist