> ## Documentation Index
> Fetch the complete documentation index at: https://docs.helicone.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# How to Integrate a Model Provider to the AI Gateway

> Tutorial to integrate a new model provider into the AI Gateway

## Overview

Adding a new provider to Helicone involves several key components:

* **Authors**: Companies that create the models (e.g., OpenAI, Anthropic)
* **Models**: Individual model definitions with pricing and metadata
* **Providers**: Inference providers that host models (e.g., OpenAI, Vertex AI, DeepInfra, Bedrock)
* **Endpoints**: Model-provider combinations with deployment configurations

## Prerequisites

* OpenAI-compatible API (recommended for simplest integration)
* Access to provider's pricing and inference documentation
* Model specifications (context length, supported features)
* API authentication details

## Step 1: Understanding the File Structure

All model support configurations are located in the `packages/cost/models` directory:

```
packages/cost/models/
├── authors/           # Model creators (companies)
├── providers/         # Inference providers
├── build-indexes.ts   # Builds maps for easy data access
├── calculate-cost.ts  # Cost calculation utilities
├── provider-helpers.ts # Helper methods
└── registry-types.ts  # Type definitions (requires updates)

```

## Step 2: Create Provider Definition

We will use `DeepInfra` as our example.

### For OpenAI-Compatible Providers

Create a new file in `packages/cost/models/providers/[provider-name].ts`:

```tsx theme={null}
import { BaseProvider } from "./base";

export class DeepInfraProvider extends BaseProvider {
  readonly displayName = "DeepInfra";
  readonly baseUrl = "https://api.deepinfra.com/";
  readonly auth = "api-key" as const;
  readonly pricingPages = ["https://deepinfra.com/pricing/"];
  readonly modelPages = ["https://deepinfra.com/models/"];

  buildUrl(): string {
    return `${this.baseUrl}v1/openai/chat/completions`;
  }
}
```

Make sure to look up the correct endpoints and override anything that is not OpenAI API default.

This handles auth because the `BaseProvider` class handles the standard `Bearer ${apiKey}` authentication pattern automatically when you set `auth = "api-key"`, which is the common pattern for OpenAI-compatible APIs.

### For Non-OpenAI Compatible Providers

For non-OpenAI compatible providers, you'll need to override additional methods. You can find options by reviewing the `BaseProvider` definition.

```tsx theme={null}
export class CustomProvider extends BaseProvider {
  // ... basic configuration

  buildBody(request: any): any {
    // Custom body transformation logic
    return transformedRequest;
  }

  buildHeaders(authContext: AuthContext): Record<string, string> {
    // Custom header logic
    return customHeaders;
  }
}

```

## Step 3: Add Provider to Index

Update `packages/cost/models/providers/index.ts`:

```tsx theme={null}
import { DeepInfraProvider } from "./deepinfra";

export const providers = [
	/// ...
	deepinfra: new DeepInfraProvider(),
]
```

## Step 4: Add Provider to the Web's Data

Update `web/data/providers.ts` to include the new provider:

```tsx theme={null}
...,
  {
    id: "deepinfra",
    name: "DeepInfra",
    logoUrl: "/assets/home/providers/deepinfra.webp",
    description: "Configure your DeepInfra API keys for fast and affordable inference",
    docsUrl: "https://docs.helicone.ai/getting-started/integration-methods",
    apiKeyLabel: "DeepInfra API Key",
    apiKeyPlaceholder: "...",
    relevanceScore: 40,
  },
  ...
```

## Step 5: Update provider helpers

Include provider in `packages/cost/models/provider-helpers.ts` within the `heliconeProviderToModelProviderName` function, so the mapping is done by the AI Gateway correctly.

```tsx theme={null}
case "DEEPINFRA":
    return "deepinfra";
case "NOVITA":
    return "novita";
```

Also, go to the `getUsageProcessor` function within `packages/cost/usage.ts`  and add the provider. If your provider require a custom usage processor (non-OpenAI compatible), you will need to add it here.

```tsx theme={null}
export function getUsageProcessor(
  provider: ModelProviderName
): IUsageProcessor | null {
  switch (provider) {
    case "openai":
    case "azure":
    case "chutes":
    case "deepinfra":
		//.... <add new provider>
    default:
      return null;
  }
}
```

## Step 6: Add provider to priorities list

We need to add the provider to the list of priorities so the gateway knows how much to prioritize each provider.

Go to `packages/cost/models/providers/priorities.ts` and include your provider within the `PROVIDER_PRIORITIES` constant variable.

```tsx theme={null}
export const PROVIDER_PRIORITIES: Record<ModelProviderName, number> = {
  // Priority 1: BYOK (Bring Your Own Key) - Reserved for user's own API keys
  // Priority 2: Helicone-hosted inference
  helicone: 2,
  // Priority 3: Premium direct providers
  anthropic: 3,
  openai: 3,
  //... <add new provider>
  deepinfra: 4,
} as const;
```

## Step 7: Update provider setup for tests

Head to `worker/test/setup.ts` and include your new provider within the `supabase-js` mock.

```tsx theme={null}
vi.mock("@supabase/supabase-js", () => ({
  createClient: vi.fn(() => ({
	  // ....
    deepinfra: {
          org_id: "0afe3a6e-d095-4ec0-bc1e-2af6f57bd2a5",
          provider_name: "deepinfra",
          decrypted_provider_key: "helicone-deepinfra-api-key",
          decrypted_provider_secret_key: null,
          auth_type: "api_key",
          config: null,
          byok_enabled: true,
        },
    // ...
  })
})
```

## Step 8: Define Authors (Model Creators)

Create author definitions in `packages/cost/models/authors/[author-name]/`:

### Folder Structure

```
authors/mistralai/    # Author name
└── mistral-nemo      # Model family
	└── endpoints.ts    # Model-provider combinations
	└── models.ts       # Model definitions
└── index.ts          # Exports
└── metadata.ts       # Metadata about the author

```

### models.ts

Include the model within the `models` object. This can contain all model versions within that model family, in this case, the `mistral-nemo` model family.

Make sure to research each value and include the tokenizer in the `Tokenizer` interface type if it is not there already.

```tsx theme={null}
import type { ModelConfig } from "../../../types";

export const models = {
  "mistral-nemo": {
    name: "Mistral: Mistral-Nemo",
    author: "mistralai",
    description:
      "The Mistral-Nemo-Instruct-2407 Large Language Model (LLM) is an instruct fine-tuned version of the Mistral-Nemo-Base-2407. Trained jointly by Mistral AI and NVIDIA, it significantly outperforms existing models smaller or similar in size.",
    contextLength: 128_000,
    maxOutputTokens: 16_400,
    created: "2024-07-18T00:00:00.000Z",
    modality: { inputs: ["text", "image"], outputs: ["text"] },
    tokenizer: "Tekken",
  },
} satisfies Record<string, ModelConfig>;

export type MistralNemoModelName = keyof typeof models;

```

### endpoints.ts

Now, update the `packages/models/[author]/[model-family]/endpoints.ts` file with model-provider endpoint combinations.

Make sure to review the provider's page itself since the inference cost changes per provider.

Make sure the initial key `"mistral-nemo:deepinfra"` is human-readable and friendly. It's what users will call!

```tsx theme={null}
import { ModelProviderName } from "../../../providers";
import type { ModelProviderConfig } from "../../../types";
import { MistralNemoModelName } from "./models";

export const endpoints = {
  "mistral-nemo:deepinfra": {
    providerModelId: "mistralai/Mistral-Nemo-Instruct-2407",
    provider: "deepinfra",
    author: "mistralai",
    pricing: [
      {
        threshold: 0,
        input: 0.0000002,
        output: 0.0000004,
      },
    ],
    rateLimits: {
      rpm: 12000,
      tpm: 60000000,
      tpd: 6000000000,
    },
    contextLength: 128_000,
    maxCompletionTokens: 16_400,
    supportedParameters: [
      "max_tokens",
      "temperature",
      "top_p",
      "stop",
      "frequency_penalty",
      "presence_penalty",
      "repetition_penalty",
      "top_k",
      "seed",
      "min_p",
      "response_format",
    ],
    ptbEnabled: false,
    endpointConfigs: {
      "*": {},
    },
  }
} satisfies Partial<
  Record<`${MistralNemoModelName}:${ModelProviderName}` | MistralNemoModelName, ModelProviderConfig>
>;
```

Two important things to note here:

* Some providers have multiple deployment regions:

```tsx theme={null}
endpointConfigs: {
  "global": {
    pricing: [/* global pricing */],
    passThroughBillingEnabled: true,
  },
  "us-east": {
    pricing: [/* regional pricing */],
    passThroughBillingEnabled: true,
  },
}

```

* Pricing Configuration

```tsx theme={null}
pricing: [
  {
    threshold: 0,              // Context length threshold
    inputCostPerToken: 0.0000005, // Always per million tokens
    outputCostPerToken: 0.0000015,
    cacheReadMultiplier: 0.1,   // Cache read cost (10% of input)
    cacheWriteMultiplier: 1.25, // Cache write cost (125% of input)
  },
  {
    threshold: 200000,         // Different pricing for >200k context
    inputCostPerToken: 0.000001,
    outputCostPerToken: 0.000003,
  },
],

```

## Step 9: Add model to Author registries (if needed)

If the model family hasn't been created, you will need to add it within the AI Gateway's registry.

### index.ts

Update `packages/cost/models/authors/[author]/index.ts` to include the new model family.

You don't need to update anything if the model family has already been created.

```jsx theme={null}
/**
 * Mistral model registry aggregation
 * Combines all models and endpoints from subdirectories
 */

import type { ModelConfig, ModelProviderConfig } from "../../types";

// Import models
import { models as mistralNemoModels } from "./mistral-nemo/models";

// Import endpoints
import { endpoints as mistralNemoEndpoints } from "./mistral-nemo/endpoints";

// Aggregate models
export const mistralModels = {
  ...mistralNemoModels,
} satisfies Record<string, ModelConfig>;

// Aggregate endpoints
export const mistralEndpointConfig = {
  ...mistralNemoEndpoints,
} satisfies Record<string, ModelProviderConfig>;

```

### metadata.ts

Update `packages/cost/models/authors/[author]/metadata.ts` to fetch models.

You don't need to update anything if the author has already been created.

```jsx theme={null}
/**
 * Mistral metadata
 */

import type { AuthorMetadata } from "../../types";
import { mistralModels } from "./index";

export const mistralMetadata = {
  modelCount: Object.keys(mistralModels).length,
  supported: true,
} satisfies AuthorMetadata;

```

### registry-types.ts

Update types for the new model family in `packages/cost/models/registry-types.ts`.

```tsx theme={null}
import { mistralEndpointConfig } from "./authors/mistralai";
import { mistralModels } from "./authors/mistralai";

const allModels = {
	...,
  ...mistralModels
};

const modelProviderConfigs = {
	...,
  ...mistralEndpointConfig
};
```

Add your new model to the `packages/cost/models/registry.ts`:

```tsx theme={null}
import { mistralModels, mistralEndpointConfig } from "./authors/mistral";

const allModels = {
	//...
	  ...mistralModels
} satisfies Record<string, ModelConfig>;

const modelProviderConfigs = {
	// ...
	  ...mistralEndpointConfig
} satisfies Record<string, ModelProviderConfig>;
```

## Step 10: Create Tests

Create test files in `worker/tests/ai-gateway/` for the author.

Feel free to use the existing tests there as reference.

## Step 11: Snapshots

Make sure to rerun snapshots before deploying.

```bash theme={null}
cd <your-path-to-the-repo>/helicone/helicone/packages && npx jest -u
```

## Common Issues & Solutions

### Issue: Complex Authentication

**Solution**: Override the `auth()` method with custom logic:

```tsx theme={null}
auth(authContext: AuthContext): ComplexAuth {
  return {
    "Authorization": `Bearer ${authContext.providerKeys?.custom}`,
    "X-Custom-Header": this.buildCustomHeader(authContext),
  };
}

```

### Issue: Non-Standard Request Format

**Solution**: Override the `buildBody()` method:

```tsx theme={null}
buildBody(request: OpenAIRequest): CustomRequest {
  return {
    // Transform OpenAI format to provider format
    prompt: request.messages.map(m => m.content).join('\\n'),
    max_tokens: request.max_tokens,
  };
}

```

### Issue: Multiple Pricing Tiers

**Solution**: Use threshold-based pricing:

```tsx theme={null}
pricing: [
  { threshold: 0, inputCostPerToken: 0.0000005 },
  { threshold: 100000, inputCostPerToken: 0.000001 },
  { threshold: 500000, inputCostPerToken: 0.000002 },
]

```

## Deployment Checklist

* Provider class created with correct authentication
* Models defined with accurate specifications
* Endpoints configured with correct pricing
* Registry types updated
* Tests written and passing
* Snapshots updated
* Documentation updated
* Pass-through billing tested (if applicable)
* Fallback behavior verified