Upstash QStash LLM Integration

​

Introduction

Upstash QStash is a serverless message queue and task scheduler that seamlessly integrates with Helicone, allowing you to call any LLM asynchronously. This integration is particularly powerful for serverless environments, enabling you to overcome timeout limitations and efficiently manage LLM operations.

​

Key Features

• Asynchronous LLM Calls: Make non-blocking LLM requests, ideal for serverless architectures.
• Support for Any LLM: Use with any LLM provider while maintaining Helicone observability.
• Serverless-Friendly: Perfect for environments with execution time limits.
• Extended Timeouts: Up to 2 hours for long-running LLM tasks.
• Automatic Retries: Handles rate limit resets efficiently.
• Batching Support: Optimize large-scale LLM operations.

​

Integration Steps

import { Client, custom } from "@upstash/qstash";

const client = new Client({ token: "<QSTASH_TOKEN>" });

await client.publishJSON({
  api: {
    name: "llm",
    provider: custom({
      token: "YOUR_LLM_PROVIDER_TOKEN",
      baseUrl: "https://api.your-llm-provider.com",
    }),
    analytics: { name: "helicone", token: "YOUR_HELICONE_API_KEY" },
  },
  body: {
    model: "your-chosen-model",
    messages: [{ role: "user", content: "Your message here" }],
  },
  callback: "https://your-callback-url.com",
});

By combining Upstash QStash LLM with Helicone, you can build robust, serverless-compatible AI applications while maintaining comprehensive observability into your LLM operations.

For more details on QStash LLM features and usage, visit the Upstash QStash LLM documentation.