LiteLLM

Coralogix's AI Observability integrations are designed to provide deep insight into applications leveraging large language models. Through a dedicated integration with the LiteLLM SDK, Coralogix delivers a unified view of calls across various LLM providers, enabling teams to track performance, costs, and errors in a single place. This helps teams standardize monitoring, compare model performance, and optimize the entire system for efficiency and accuracy.

Overview

This library offers customized instrumentation for the LiteLLM SDK, optimized to support the development of production-ready applications. It provides streamlined integration and offers detailed visibility into LLM calls through LiteLLM's unified interface. This enables effective debugging, performance analysis, and a clear understanding of all LLM interactions, regardless of the underlying provider (OpenAI, Azure, Anthropic, Cohere, etc.).

Requirements

• Python version 3.9 and above.
• Coralogix API keys.

Installation

Run the following command:

pip install llm-tracekit[litellm]

Authentication

Authentication data is passed during the instrumentor object's creation. You can do this in one of two ways:

Passing arguments to the constructor

You can pass the token, endpoint, and other parameters directly when initializing the LiteLLMInstrumentor.

from llm_tracekit import LiteLLMInstrumentor

instrumentor = LiteLLMInstrumentor(
    coralogix_token="<your_coralogix_token>",
    coralogix_endpoint="<your_coralogix_endpoint>",
    application_name="<ai-application>",
    subsystem_name="<ai-subsystem>"
)

Using environment variables

If arguments are not passed to the constructor, the instrumentor will automatically use the following environment variables:

• CX_TOKEN: Your Coralogix API key
• CX_ENDPOINT: The endpoint associated with your Coralogix domain
• CX_APPLICATION_NAME: Your application's name
• CX_SUBSYSTEM_NAME: Your subsystem's name

Usage

This section describes how to set up instrumentation for the LiteLLM SDK.

Set up tracing

Instrument

To instrument all clients, create an instance of LiteLLMInstrumentor and call the instrument method.

from llm_tracekit import LiteLLMInstrumentor

# Arguments can be passed here or set as environment variables
instrumentor = LiteLLMInstrumentor()
instrumentor.instrument()

Uninstrument

To uninstrument clients, call the uninstrument method.

instrumentor.uninstrument()

Full example

import litellm
from llm_tracekit import LiteLLMInstrumentor

# Activate instrumentation
# Coralogix connection details are read from environment variables:
# - CX_TOKEN
# - CX_ENDPOINT
# - CX_APPLICATION_NAME
# - CX_SUBSYSTEM_NAME
instrumentor = LiteLLMInstrumentor()
instrumentor.instrument()

# LiteLLM SDK Usage Example
response = litellm.completion(
    model="gpt-4o-mini",
    messages=[{"content": "What is the capital of Italy?", "role": "user"}]
)

print(response)

Enable message content capture

By default, message content, such as the contents of the prompt, completion, function arguments and return values, are not captured.

To capture message content as span attributes, set the environment variable OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT to true.

Most Coralogix AI evaluations require message contents to function properly, so enabling message capture is strongly recommended.

Key differences from OpenTelemetry

User prompts and model responses are captured as span attributes instead of log events, as detailed below.

Semantic conventions