Skip to content

OpenTelemetry Lambda Auto Instrumentation

Coralogix offers coralogix-*-wrapper-and-exporter-* Lambda layers enable you to generate logs, metrics, and traces, providing insights into triggers, invocation times, and interconnections. These features work out-of-the-box, without requiring any modification of the monitored Lambda functions' code. Once the configuration is complete, view your data using our cutting-edge Serverless Monitoring feature.

Note

  • This integration is one of two options - complete and basic - for monitoring Lambda, a requirement for enjoying Serverless Monitoring. This tutorial demonstrates how to set up Lambda monitoring to get complete telemetry, including traces.

  • To set up Lambda monitoring to get only basic telemetry, including logs, you are only required to set up the Coralogix AWS Lambda Telemetry Exporter. View the relevant documentation here.

Overview

The Lambda instrumentation layers (coralogix-*-wrapper-and-exporter-*) contain modified versions of OpenTelemetry Lambda auto instrumentation wrappers, bundled together with the Coralogix AWS Lambda Telemetry Exporter.

OpenTelemetry Lambda auto instrumentation wrappers instrument the handler function, AWS SDK, and other client libraries to produce traces out-of-the-box. They also provide a configured Otel SDK through which the function code can produce custom spans and metrics.

The telemetry produced by the auto instrumentation wrapper is forwarded to the Coralogix AWS Lambda Telemetry Exporter, which supplements it with information obtained from the Lambda Telemetry API and sends it all to Coralogix.

Once fully set up, the auto instrumentation will produce traces, as shown below. The traces will help you discover:

  • What invoked/triggered the lambda function

  • What Lambda functions and other AWS services were called by the function

  • How long did different parts of the invocation take

Differences compared to upstream OpenTelemetry Lambda instrumentation

  • Coralogix instrumentation captures request and response payloads and puts them under rpc.request.payload / rpc.response.payload span attributes. By default the length of the captured payload is limited to ~50kB. That number can be adjusted with OTEL_PAYLOAD_SIZE_LIMIT. Setting OTEL_PAYLOAD_SIZE_LIMIT=0 effectively disables this feature.
  • Coralogix instrumentation creates an extra “trigger” span as a parent of the invocation/handler span which indicates what triggered the lambda function.
  • Coralogix instrumentation extracts trace context from incoming invocations/messages and injects trace context into downstream calls. This results in coherent traces spanning multiple functions, or a set of linked traces (in case of batched messaging).
  • Coralogix instrumentation preconfigures traces and metrics exporter, which makes it easy for users to define custom metrics and spans. (The support for this varies between languages and versions of the upstream OTLP auto instrumentation)
  • Coralogix instrumentation doesn’t export traces/metrics periodically. It only does it at the end of an invocation.
  • Coralogix instrumentation sends extra copies of spans at the beginning of an invocation (we call this the Early Spans feature) which improves traces in case of timeouts and crashes.
  • Coralogix node.js instrumentation has a custom way of instrumenting the handler function which works with both CommonJS and ES modules.
  • Coralogix node.js instrumentation supports extra env variables with which instrumentation libraries can be enabled/disabled.
  • Coralogix node.js instrumentation warms-up OTEL exporter during initialization to improve cold invocation performance.

Supported features & Limitations

Supported triggers

The instrumentation will generate a trigger spans corresponding to those triggers.

trigger python node.js java
Api Gateway Rest (V1)
Api Gateway HTTP (V2)
S3
SQS
Dynamo DB -
Cognito -
EventBridge -
SNS - -
Kinesis - -

Supported trace propagation scenarios

The instrumentation will propagate trace context via the request/message, so that both ends of the communication will either become visible in Coralogix as one trace or two linked traces.

scenario python node.js java
Lambda invoking Lambda via AWS SDK
SQS
SNS - -
EventBridge with JSON payload -
Kinesis with JSON payload - -

Included OTEL instrumentation libraries

Python: instrumentation-aiohttp-client, util-http, instrumentation-asgi, instrumentation-boto, instrumentation-asyncpg, instrumentation-celery, instrumentation-dbapi, instrumentation-django, instrumentation-elasticsearch, instrumentation-fastapi, instrumentation-falcon, instrumentation-flask, instrumentation-grpc, instrumentation-jinja2, instrumentation-mysql, instrumentation-psycopg2, instrumentation-pymemcache, instrumentation-pymongo, instrumentation-pymysql, instrumentation-pyramid, instrumentation-redis, instrumentation-requests, instrumentation-sqlalchemy, instrumentation-sqlite3, instrumentation-starlette, instrumentation-tornado, instrumentation-wsgi

Node.js: instrumentation-dns, instrumentation-express, instrumentation-graphql, instrumentation-grpc, instrumentation-hapi, instrumentation-http, instrumentation-ioredis, instrumentation-koa, instrumentation-mongodb, instrumentation-mysql, instrumentation-net, instrumentation-pg, instrumentation-redis, instrumentation-aws-sdk

Java: aws-lambda-events, aws-sdk

Recommendations

Python

When using AWS Lambda with the Serverless framework, your Python dependency layer should be last, ensuring that AWS Lambda provides your application dependencies instead of the wrapper’s. Not doing so might cause application crashes with [ERROR] AttributeError: partially initialized module 'charset_normalizer' due to collisions between the Python wrapper and application dependencies.

Node.js

Node.js auto instrumentation can't instrument code that has been moved or renamed by bundlers like esbuild or webpack. It will still instrument the handler function.

Java

Java auto-instrumentation adds significant overhead during cold invocations. In order to maintain reasonable latencies we strongly recommend configuring Java functions with:

  • at least 1024MB of memory
  • JAVA_TOOL_OPTIONS=-XX:+TieredCompilation -XX:TieredStopAtLevel=1 (as suggested by AWS documentation)

Set up Lambda Monitoring

Follow these steps in order to monitor AWS Lambda functions with Coralogix.

STEP 1. Set up the Coralogix AWS resource metadata collection.

STEP 2. Set up the OpenTelemetry Lambda auto instrumentation, as described in this tutorial. This is available for Node.js, Python, and Java. For other runtimes, opt for the Coralogix AWS Lambda Telemetry Exporter setup to view basic telemetry.

Amazon Resource Name

To deploy or update the Coralogix OpenTelemetry wrapper, select the Amazon Resource Name (ARN) corresponding to your AWS region, Lambda runtime and Architecture from the following lists.

Python x86_64
Python arm64
Node.js x86_64
Node.js arm64
Java x86_64
Java arm64

Installation: published layer ARN

In your AWS Management console, navigate to AWS Lambda.

STEP 1. Access the Lambda function that you would like to monitor. Check the list of Layers. If you already have a coralogix-aws-lambda-telemetry-exporter-* layer added, remove it. In the following steps, you will install a coralogix-*-wrapper-and-exporter-* layer that already contains the telemetry exporter.

STEP 2. Access the Lambda function that you would like to monitor. Select Layers > Add a layer.

STEP 3. Specify an ARN by pasting one from the list above > Add

STEP 4. Access the Lambda function that you would like to monitor and configure the environment variables: Configuration > Environment variables

  • Set AWS_LAMBDA_EXEC_WRAPPER = /opt/otel-handler to enable the auto-instrumentation wrapper.

  • Set CX_DOMAIN to the Coralogix domain within which you’ve set up your account. For example coralogix.us (do NOT add protocol, port, team name etc. to the domain).

  • Set CX_API_KEY to the Coralogix Send-Your-Data API key.

  • Set CX_REPORTING_STRATEGY = REPORT_AFTER_INVOCATION

    • If the function is invoked frequently consider setting it to LOW_OVERHEAD instead.

    • See AWS Lambda Telemetry Exporter documentation to learn more about reporting strategies.

STEP 5. [Optional] Fine-tune the telemetry-exporter to match your preferences. See all available configuration options here.

Installation: container Image Lambda

This instruction applies to container images based on the official AWS Lambda images.

STEP 1. Add the coralogix auto instrumentation to your docker image. See the examples below.

# Python auto-instrumentation (wrapper-and-exporter) based on official AWS Lambda image
FROM coralogixrepo/coralogix-python-wrapper-and-exporter:20 as coralogix
FROM public.ecr.aws/lambda/python:3.12

WORKDIR /opt
COPY --from=coralogix /opt/ .

WORKDIR $LAMBDA_TASK_ROOT
COPY app.py ./

# This only works with official AWS Lambda base images
ENV AWS_LAMBDA_EXEC_WRAPPER=/opt/otel-handler
CMD ["app.lambda_handler"]
# Node.js auto-instrumentation (wrapper-and-exporter) based on official AWS Lambda image
FROM coralogixrepo/coralogix-nodejs-wrapper-and-exporter:25 as coralogix
FROM public.ecr.aws/lambda/nodejs:20

WORKDIR /opt
COPY --from=coralogix /opt/ .

WORKDIR $LAMBDA_TASK_ROOT
COPY app.js ./

# This only works with official AWS Lambda base images
ENV AWS_LAMBDA_EXEC_WRAPPER=/opt/otel-handler
CMD ["app.lambdaHandler"]
# Java auto-instrumentation (wrapper-and-exporter) based on official AWS Lambda image
FROM coralogixrepo/coralogix-java-wrapper-and-exporter:13 as coralogix
FROM public.ecr.aws/lambda/java:21

WORKDIR /opt
COPY --from=coralogix /opt/ .

WORKDIR $LAMBDA_TASK_ROOT
COPY java-simple-all.jar $LAMBDA_TASK_ROOT/lib/

# This only works with official AWS Lambda base images
ENV AWS_LAMBDA_EXEC_WRAPPER=/opt/otel-handler
CMD ["app.App::handleRequest"]

STEP 2. Access the Lambda function that you would like to monitor and configure the environment variables: Configuration > Environment variables

  • Set CX_DOMAIN to the Coralogix domain within which you’ve set up your account. For example coralogix.us (do NOT add protocol, port, team name etc. to the domain)

  • Set CX_API_KEY to the Coralogix Send-Your-Data API key.

  • Set CX_REPORTING_STRATEGY = REPORT_AFTER_INVOCATION (If the function is invoked frequently consider setting it to LOW_OVERHEAD instead. See AWS Lambda Telemetry Exporter to learn more about reporting strategies)

STEP 3. [Optional] Fine-tune the telemetry-exporter to match your preferences. See all available configuration options here.

Validating the results

Assuming you’ve already configured the Coralogix AWS resource metadata collection, you’re ready to take advantage of the Serverless Monitoring feature. Validate that your setup works.

STEP 1. Invoke the function.

STEP 2. In your Coralogix navigation pane, click APM. Select the Serverless tab. Click on the function of interest.

STEP 3. View Lambda invocations in the INVOCATIONS table.

Note: It may take up to 1 minute for your telemetry to fully process.

STEP 4. Click on an invocation to drill down using our Spans View.

Additional resources

Documentation AWS Lambda Telemetry Exporter
AWS Resource Metadata Collection
Serverless Monitoring
GitHub Node.js wrapper
Python wrapper
Java wrapper

Support

Need help?

Our world-class customer success team is available 24/7 to walk you through your setup and answer any questions that may come up.

Feel free to reach out to us via our in-app chat or by sending us an email at [email protected].