Python OpenTelemetry Instrumentation

This tutorial shows simple examples of how to instrument Python applications for metrics and traces using Open Telememtry and send them to Coralogix. The examples will utilise the Flask web framework, however, it is possible to instrument using other frameworks such as Tornado, Django and more.

Open Telemetry supports Auto and Manual Instrumentation for Python applications. While Auto Instrumentation is the easiest way to instrument an application for traces, it is important to note that the level of control is significantly reduced, for example, you will not be able to explicitly define relationships between functions or effectively propagate context to other applications.

This document will provided examples for both Auto and Manual instrumentation.

Prerequisites

• A Coralogix account, set up on the Coralogix domain
• Coralogix Send-Your-Data API Key
• Python 3.9+

Auto Instrumentation

Auto instrumentation works by running your application within another wrapper application provided by OpenTelemetry. This wrapper handles collecting traces from all compatible parts of your application.

Traces

STEP 1. Create a Python environment.

# create python virtual environment
python -m venv env

# activate python environment
source env/bin/active

# create app.py
touch app.py

STEP 2. Add a simple Flask app to app.py.

from random import randint
from flask import Flask

app = Flask(__name__)

@app.route("/roll")
def roll_dice():
    return str(do_roll())

def do_roll():
    res = randint(1, 6)
    return f"you've rolled {res}"

if __name__ == "__main__":
    app.run(host='0.0.0.0', port='5001')

STEP 3. Add a requirements.txt file and install dependencies.

flask
opentelemetry-distro
opentelemetry-exporter-otlp
opentelemetry-instrumentation-flask

pip install -r requirements.txt

STEP 4. Now we can run our application using the opentelemetry-instrument wrapper, but before that we need to configure the auto instrumentation tool. The wrapper is configured either by CLI arguments or Environment variables. Here we will use environment variables.

# set exporter headers. These are the headers used when calling the Coralogix Endpoint
export OTEL_EXPORTER_OTLP_HEADERS='Authorization=Bearer%20[your-private-key], CX-Application-Name=dev-traces, CX-Subsystem-Name=dev-traces'

# set the coralogix endpoint for traces
export OTEL_EXPORTER_OTLP_ENDPOINT='ingress.coralogix.com:443'

# set the exporter protocol, in this case we're using grpc
export OTEL_TRACES_EXPORTER=otlp_proto_grpc

Note that the values of the Environment variable OTEL_EXPORTER_OTLP_HEADERS must be URL Encoded, hence we use the value %20 to represent a space between Bearer and the token.

For a list of all the Environment variable options for configuring the opentelemetry-instrument tool, see here.

Run Auto Instrumentation

Once we’ve set our environment variables, we run our application using the following.

opentelemetry-instrument \
    --traces_exporter console,otlp \ # export traces to console and otel protocol
    --metrics_exporter none \ # disable metrics
    --service_name py-auto-instr-test \ # service name is required
    python app.py

The application will be served on port 5001. A trace can be triggered by calling the endpoint localhost:5001/roll

Metrics

In order to add metrics we simply need to add otlp as argument to the --metrics_exporter flag.

opentelemetry-instrument \
    --traces_exporter console,otlp \
    --metrics_exporter console,otlp \ # send metrics to console and otel exporters
    --service_name py-auto-instr-test \
    python app.py

The metrics collected will depend the support for underlying framework being instrumented. For a list of all supported frameworks, see here.

Manual Instrumentation

Now we will implement the same example above, this time using Manual Instrumentation.

Traces

STEP 1. Create a file, app.py and import the required packages.

from flask import Flask, request
import os

# open telemetry imports
from os import environ
from opentelemetry import metrics
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import SimpleSpanProcessor, ConsoleSpanExporter
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
import random

STEP 2. Define the headers necessary for calling the Coralogix endpoint and use them to define an Exporter and Tracer. An Exporter is the mechanism responsible for exporting/sending traces during runtime.

Note that the headers variable is a string and not a dictionary.

# define headers. Single string with comma separated key-value pairs
# Authorization, CX-Application-Name and CX-Subsystem-Name are mandatory when calling the coralogix endpoint directly.
headers = ', '.join([
    f'Authorization=Bearer%20{environ.get("CX_TOKEN")}',
    "CX-Application-Name=manual-instro-traces",
    "CX-Subsystem-Name=manual-instro-traces",
])


# create a tracer provider
tracer_provider = TracerProvider()

# set up an OTLP exporter to send spans to coralogix directly.
exporter = OTLPSpanExporter(
    endpoint=environ.get('CX_ENDPOINT'),
    headers=headers,
)

# set up a span processor to send spans to the exporter
span_processor = SimpleSpanProcessor(exporter)
# span_processor = ConsoleSpanExporter(exporter)

# add the span processor to the tracer provider
tracer_provider.add_span_processor(span_processor)

# create a tracer
tracer = trace.get_tracer_provider().get_tracer(__name__)

# set trace provider
trace.set_tracer_provider(tracer_provider)

STEP 3. Define the Flask application and instrument the functions.

# initialise flask app
app = Flask(__name__)

# example of adding a span to a function using a context manager
# this is useful when you want to dynamic attribuf"http://{os.getenv('OTEL_COLLECTOR_SERVICE_HOST')}:4318"tes to the span
@app.route("/roll")
def roll():
    # create a span
    with tracer.start_as_current_span("roll_v2") as span:
        span.set_attribute("http.method", "GET")
        span.set_attribute("http.host", "localhost")
        span.set_attribute("http.url", "/v2/rolldice")
        res = dice()
        span.set_attribute("roll.value", res)
        return f"you rolled {res}\n"

# example of adding a span to a function using a decorator
@tracer.start_as_current_span("diceFunc")
def dice():
    return random.randint(1, 6)

if __name__ == "__main__":
    app.run(host='0.0.0.0', port='5001')

The code above has 2 functions, roll which is the HTTP Handle for the /roll path and dice which is a simple function that returns a number between 1 and 6.

The roll function is instrumented by defining a trace within the function body, however, the dice function is instrumented using a decorter. There is no rule on which to use, either approach can be used depending on your use case and codebase.

STEP 5. To run the app, set the required Environment variables:

• CX_TOKEN
• CX_ENDPOINT

Important:

When using a Python Virtual Environment, these Environment variables must be set inside that environment.

For details on the various Coralogix Endpoints, see here

python app.py

Validation

You can confirm that your traces are being sent to Coralogix.

In your Coralogix dashboard, click Explore > Tracing. You should now see the traces generated by your application.

Metrics

Like traces, metrics can also be instrumented manually, however, you should only consider manually instrumenting metrics when there are specific metrics that you wish to extract from your application runtime context.

Note that the dependencies are the same as those utilised in the requirements.txt above for traces.

STEP 1. Define the metrics exporter and metrics provider.

from opentelemetry import metrics
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import (
    PeriodicExportingMetricReader,
)

from opentelemetry.exporter.otlp.proto.grpc.metric_exporter import OTLPMetricExporter
from os import environ


# define headers. Single string with comma separated key-value pairs
# Authorization, CX-Application-Name and CX-Subsystem-Name are mandatory 
# when calling the coralogix endpoint directly.
headers = ', '.join([
    f'Authorization=Bearer%20{environ.get("CX_TOKEN")}',
    "CX-Application-Name=manual-instro-traces",
    "CX-Subsystem-Name=manual-instro-traces",
])

# set up an OTLP exporter to send spans to coralogix directly.
exporter = OTLPMetricExporter(
    endpoint=environ.get('CX_ENDPOINT'),
    headers=headers,
)

metric_reader = PeriodicExportingMetricReader(exporter)
provider = MeterProvider(metric_readers=[metric_reader])

# Sets the global default meter provider
metrics.set_meter_provider(provider)

# Creates a meter from the global meter provider
meter = metrics.get_meter("my.meter.name")

Above we defined the headers and created an OTLP exporter. We then created a PeriodicExportingMetricReader which is responsible for sending metrics to the exporter. Finally we created a MeterProvider and set it as the global default provider. We also created a Meter from the provider. Meters are used to create metrics.

STEP 2. Create a counter metric that will be used to count the number of requests made to the /roll endpoint.

# Creates a counter instrument
http_req_counter = meter.create_counter(
    "http.request.counter", unit="1", description="counts the number of requests done"
)

Above, using our meter we created a counter metric called http.request.counter. The counter will count the number of requests made to the /roll endpoint.

This is one example of a metric that can be created using the meter, for more information on the meter and a list of other metric types that can be defined, see here.

STEP 3. Finally, we add our application logic and instrument the /roll endpoint.

from random import randint
from flask import Flask

app = Flask(__name__)

@app.route("/roll")
def roll_dice():
    # increment the counter
    http_req_counter.add(1, {"request.path": '/roll'})
    return str(do_roll())

def do_roll():
    res = randint(1, 6)
    return f"you've rolled {res}"

if __name__ == "__main__":
    app.run(host='0.0.0.0', port='5001')

Note that counter is incremented each time the /roll endpoint is called.

STEP 4. Execute

To run the app, as before, set the required Environment variables:

• CX_TOKEN
• CX_ENDPOINT

python app.py

Validation

You can confirm that your metrics are being sent to Coralogix.

In your Coralogix dashboard, go to Grafana > Explore > Metric Browser, then search for the name of your metric.

Support

For more information on Python Opentelemetry instrumentation, please view the official Open Telemetry documentation.

Need help?

Our world-class customer success team is available 24/7 to walk you through your setup and answer any questions that may come up.

Feel free to reach out to us via our in-app chat or by sending us an email at [email protected].