Add evaluators to an experiment using v2 API

Add Evaluators (v2)

Add Experiment Evaluators (v2)

api key

Helicone OSS LLM Observability

Helicone is the open-source LLM observability platform for developers to monitor, debug, and improve **production-ready** applications.

Introduction

Introduction to Helicone - Open Source LLM Observability

Use Vercel AI SDK to integrate with Helicone to log your OpenAI usage.

Vercel AI

Vercel AI SDK Integration

Vercel AI SDK Integration - Helicone OSS LLM Observability

Connect Helicone with any LLM deployed on Anyscale, including Llama, Mistral, Gemma, and GPT.

Anyscale

Anyscale Integration

Anyscale Integration - Helicone OSS LLM Observability

Connect Helicone with Together AI, a platform for running open-source language models. Monitor and optimize your AI applications using Together AI's powerful models through a simple base_url configuration.

Together AI

Together AI Integration

Together AI Integration - Helicone OSS LLM Observability

Integrate Helicone with Hyperbolic, a platform for running open-source LLMs. Monitor and analyze interactions with any Hyperbolic-deployed model using a simple base_url configuration.

Hyperbolic

Hyperbolic Integration

Hyperbolic Integration - Helicone OSS LLM Observability

Connect Helicone with OpenAI-compatible models on Deepinfra. Simple setup process using a custom base_url for seamless integration with your Deepinfra-based AI applications.

Deepinfra

Deepinfra Integration

Deepinfra Integration - Helicone OSS LLM Observability

Integrate Helicone with OpenRouter, a unified API for accessing multiple LLM providers. Monitor and analyze AI interactions across various models through a single, streamlined interface.

OpenRouter

OpenRouter Integration

OpenRouter Integration - Helicone OSS LLM Observability

Integrate Helicone with Crew AI, a multi-agent framework supporting multiple LLM providers. Monitor AI-driven tasks and agent interactions across providers.

Crew AI

Crew AI Integration

Crew AI Integration - Helicone OSS LLM Observability

Connect Helicone with Fireworks AI, a platform that provides a fast, affordable, and customizable solution for generative artificial intelligence that helps product developers run, fine-tune, and share large language models.

Fireworks AI

Fireworks AI Integration

Fireworks AI Integration - Helicone OSS LLM Observability

Connect Helicone with DeepSeek AI, a platform that provides powerful language models including MoE and Code models for various AI applications.

DeepSeek AI

DeepSeek AI Integration

DeepSeek AI Integration - Helicone OSS LLM Observability

Connect Helicone with Mistral AI, a platform that provides state-of-the-art language models including Mistral-Large and Mistral-Medium for various AI applications.

Mistral AI

Mistral AI Integration

Mistral AI Integration - Helicone OSS LLM Observability

Connect Helicone with Perplexity AI, a platform that provides powerful language models including Sonar and Sonar Pro for various AI applications.

Perplexity AI

Perplexity AI Integration

Perplexity AI Integration - Helicone OSS LLM Observability

Connect Helicone with Nebius AI, a platform that provides powerful image generation models including Flux and Stability AI's SDXL.

Nebius AI

Nebius AI Integration

Nebius AI Integration - Helicone OSS LLM Observability

Connect Helicone with Novita AI, a platform that provides powerful LLM models including DeepSeek, Llama, Mistral, and more.

Novita AI

Novita AI Integration

Novita AI Integration - Helicone OSS LLM Observability

Connect Helicone with X AI, a platform that provides powerful language models including MoE and Code models for various AI applications.

X AI

X AI Integration

X AI Integration - Helicone OSS LLM Observability

Combine Helicone's LLM analytics with PostHog, a comprehensive product analytics platform. View LLM insights alongside other application data for holistic performance analysis.

PostHog

PostHog Integration

PostHog Integration - Helicone OSS LLM Observability

Integrate Helicone with Ragas, an open-source framework for evaluating Retrieval-Augmented Generation (RAG) systems. Monitor and analyze the performance of your RAG pipelines.

Ragas

Ragas Integration

Ragas Integration - Helicone LLM Observability

Integrate Helicone with Open WebUI, an extensible, offline-capable interface for various LLM runners. Monitor interactions across Ollama, OpenAI-compatible APIs, and custom LLM setups.

Open WebUI

Open WebUI Integration

Open WebUI Integration - Helicone OSS LLM Observability

Integrate Helicone with MetaGPT, a multi-agent framework that simulates a software company workflow. Monitor AI-driven software development processes and agent interactions.

MetaGPT

MetaGPT Integration

MetaGPT Integration - Helicone OSS LLM Observability

Integrate Helicone with Open Devin, an AI-powered platform for autonomous software engineering. Monitor interactions between AI agents and human developers in your projects.

Open Devin

Open Devin Integration

Open Devin Integration - Helicone OSS LLM Observability

Integrate Helicone with Embedchain, an Open Source Framework for personalizing LLM responses. Monitor interactions across different LLMs in AI applications.

Mem0 Embedchain

Mem0 Embedchain Integration

Mem0 Embedchain Integration - Helicone OSS LLM Observability

Dify is an open-source LLM app development platform. Its intuitive interface combines AI workflow, RAG pipeline, agent capabilities, model management, observability features and more, letting you quickly go from prototype to production. Here is how to get Observability and logs for your dify instance.

Dify LLM Observability and Logs

Dify

Integrate Helicone with Firecrawl to enhance your web scraping and AI applications with comprehensive observability and analytics.

Firecrawl

Firecrawl Integration

Learn how to integrate Helicone with your Bubble application for LLM observability and logging.

Bubble

Bubble.io LLM Observability and Logs

Group and visualize multi-step LLM interactions. Track request flows across multiple traces and gain insights into complex AI workflows by adding only 2 simple headers.

Sessions

Sessions - Helicone OSS LLM Observability

Experiments is a spreadsheet-like experience designed to tune your LLM prompts for production. 

Experiments (NEW)

Experiments

Experiments | Helicone OSS LLM Observability

Add custom metadata to LLM requests for advanced segmentation and analysis. Tag requests with session IDs, conversation context, or application data to gain deeper insights into your AI application performance.

Custom Properties

Custom Properties - Helicone OSS LLM Observability

Reduce latency and save costs on LLM calls by caching responses on the edge. Configure cache duration, bucket sizes, and use cache seeds for consistent results across requests.

Caching

LLM Caching

LLM Cache - Helicone OSS LLM Observability

Configure Helicone to selectively exclude sensitive request and response data from your LLM application logs for privacy and compliance needs.

Omit Logs

Omit Logs - Helicone OSS LLM Observability

Monitor individual user interactions with your LLM applications. Track per-user request volumes, costs, and usage patterns across your AI services.

User Metrics

User Metrics - Helicone OSS LLM Observability

Curate datasets and fine-tune your LLMs with Helicone to improve performance on specific tasks.

Datasets & Fine-Tuning

Datasets & Fine-Tuning - Helicone OSS LLM Observability

Access multiple LLM providers through Helicone's unified gateway. Ensure high availability with automatic fallback options across OpenAI, Anthropic, and other AI services.

Gateway Fallbacks

Gateway Fallbacks for LLM Providers

Gateway Fallbacks for LLM Providers - Helicone OSS LLM Observability

Configure Helicone to automatically retry failed LLM requests, overcoming rate limits and server issues using intelligent exponential backoff.

Retries

Retries - Helicone OSS LLM Observability

Set custom rate limits for model provider API calls. Control usage by request count, cost, or custom properties to manage expenses and prevent unintended overuse.

Custom Rate Limits

Custom LLM Rate Limits

Custom LLM Rate Limits - Helicone OSS LLM Observability

Centralize and secure provider API key management. Distribute Helicone proxy keys for simplified access control and enhanced security of your LLM applications.

Key Vault

Key Vault - Helicone OSS LLM Observability

Enable OpenAI's moderation feature in your LLM applications to automatically detect and filter harmful content in user messages.

Moderations

Moderations - Helicone OSS LLM Observability

Enable robust security measures in your LLM applications to protect against prompt injections, detect anomalies, and prevent data exfiltration.

LLM Security

LLM Security - Helicone OSS LLM Observability

Provide customers with a dedicated portal to view and manage their LLM usage. Oversee customer access, set rate limits, and monitor usage while allowing customers to track their own metrics.

Customer Portal

Customer Portal - Helicone OSS LLM Observability

Compare Helicone's Proxy and Async integration methods. Understand the features, benefits, and use cases for each approach to choose the best fit for your LLM application.

Proxy vs Async

Proxy vs Async Integration

Proxy vs Async Integration - Helicone OSS LLM Observability

Helicone ensures high availability for your LLM applications using Cloudflare's global network. Learn about our deployment practices and how we maintain reliability.

Reliability

Availability and Reliability

Availability - Helicone OSS LLM Observability

Helicone minimizes latency for your LLM applications using Cloudflare's global network. Detailed benchmarking results and performance metrics included.

Latency

Latency Impact

Latency Impact - Helicone OSS LLM Observability

Helicone ensures top-tier data security and privacy through our SOC2 compliant cloud solution, with options for enhanced control and data ownership.

Data Security & Privacy

Data Security & Privacy - Helicone OSS LLM Observability

Understanding Helicone's open-source status and how to contribute

Open Source

Learn how Helicone calculates the cost per request for nearly all models, including both streamed and non-streamed requests. Detailed explanations and examples provided.

Cost Calculation

How We Calculate Cost

How We Calculate Cost - Helicone OSS LLM Observability

Comprehensive guide to all Helicone headers. Learn how to access and implement various Helicone features through custom request headers.

Header Directory

Helicone Header Directory

Helicone Header Directory - Open Source LLM Observability

Retrieve all requests visible in the request table at Helicone.

Get Requests

Get Requests - Helicone OSS LLM Observability

Get Requests (Point Queries)

Get Requests (Point Queries) - Helicone OSS LLM Observability

Retrieve a single request visible in the request table at Helicone.

Get Single Request

Get Single Request - Helicone OSS LLM Observability

Get Requests by IDs

Get Requests by IDs - Helicone OSS LLM Observability

Submit Feedback

Submit Feedback - Helicone OSS LLM Observability

Create or update a property of a specific request.

Upsert Request Property

Upsert Request Property - Helicone OSS LLM Observability

Submit assets for a specific request. - If you don't know what this is, you probably don't need this.

Request

Prompt

User

Evals

Webhooks

Evaluator

Experiment

Trace

Log

Property

Dataset

Customer

Dashboard

Settings

Session

Add Experiment Evaluators (v2)

Authorizations

Path Parameters

Body

Response