Using Lambda Extensions to Streamline Observability

Lambda Observability

Lambda is a top-rated compute service available on the AWS cloud service network. Its popularity largely derives from its ease of use, allowing users to run Lambda functions reliably without provisioning or managing servers. Lambda can be triggered manually or by any linked events in the AWS network, including DynamoDB streams, SQS, Kinesis, and more. 

To use Lambda in a production setting, you must consider more than just the development time you save using Lambda. Developers and DevOps professionals need to watch functions for code errors and potential infrastructure optimizations. These settings include memory, duration, concurrency. As well, tracking error rates and causes, throttles from the AWS platform, and AWS delivery failures will contribute to the overall health of your system.

Having clear and easily-analyzed metrics, logs, and traces are keys to a well-observed architecture. AWS provides some observability capabilities for Lambda with AWS CloudWatch. Lambda Extensions provides various partner programs that enhance your Lambda observability.

Lambda Observability using CloudWatch

CloudWatch has several internal tools helpful in observing the health of your Lambda functions. These include viewing and searching logs, configuring and using metrics, setting alarms, and configuring rules to trigger other events.

CloudWatch features have grown significantly over the last several years enabling developers to have more insight into their Lambda functions. Some enterprises also want to augment the capabilities of CloudWatch using third-party tools that have different features than those provided by AWS. Coralogix’s log analytics tool can automatically detect unusual behavior in functions and alert you of the need for intervention. Developers can use AWS and third-party Lambda observability tools together, or they may choose to use one or the other.

Turning Off CloudWatch in Lambda

CloudWatch is the defacto observability tool for Lambda. It is automatically linked to Lambda no matter what method you use to deploy your function. You generally need to turn it off actively if you do not want to use it. If you use the Serverless Framework, the deployment process will automatically add log policies and groups for each Lambda function in the service. Users suggest declining permissions for CloudWatch in the IAM roles section to remove logging events, as shown below. 

 {
	"Effect": "Deny",
	"Action": [
		"logs:CreateLogStream",
                "logs:CreateLogGroup",
                "logs:PutLogEvents"
       ],
      "Resource": [ "arn:aws:logs:*" ],
 }

Lambda Observability using Extensions

Lambda Extensions is used to plug your Lambda function into supporting tools. AWS partners like Coralogix may provide tools, or you may write your own. You would choose to utilize an Extension instead of in-line functionality when users want to send telemetry data to another vendor or when users need to collect and use data over a long period or across concurrent Lambdas.

Lambda Extension Use Cases

Lambdas are short-lived, stateless functions, making it challenging to observe valuable data. Lambda Extensions provide more ways to integrate with and monitor your Lambda functions and your execution environment. 

Since Extensions initialize before your Lambda function is invoked and shut down after your Lambda function is complete, you will be able to capture metrics during all phases of your Lambda’s lifecycle. Available metrics include previously-hidden data before and after function invocations.

Lambda’s being stateless can sometimes mean maintaining databases simply to hold state. You can use Lambda Extension to store configurations and settings and reduce the need for extra infrastructure. 

Extensions can also be used to log and alert you on security issues that may arise during a Lambda’s execution. You can add up to ten Extensions per Lambda function using up to five Lambda layers or a container image.

Available Extensions for Lambda Observability

Extensions built by third-party teams can be found in the AWS Serverless Application Repository. These include simple examples to help you test out using Extensions and Extensions that enhance Lambda Observability. This repository does include applications that are not used for Extensions, so be aware of your chosen application’s use.

Third-party teams like Coralogix have provided Lambda Extensions that connect your function’s logs with their machine learning tools. The Lambda Extension allows you to send log and metric data directly to analytics platforms. If all the analytics are done in a third-party tool, you may turn off CloudWatch to reduce the cost incurred running the Lambda. Previous to Lambda Extensions, log and metric data had to be sent using CloudWatch, but now that service can be bypassed.

Once Lambda log and metric data are in the Coralogix platform, our service can provide insights into the health and maintenance of your AWS Infrastructure. Our AWS Obervability solution can collect data from across your AWS account, using it to detect and provide alerts on anomalous activity. Its Log Analytics platform can provide insights specific to your Lambda function’s behavior to ensure it is working as expected.

How to Add Extensions

Lambda Extensions can be added no matter how you build your Lambdas. When using Lambda container images, you build the Extension image using Docker’s CLI or AWS SAM with a Dockerfile. When using deployment methods such as Serverless Framework or Terraform, you use the same methods of deploying Lambda Layers to deploy Lambda Extensions. Environment variables need to be added here to invoke Extension capabilities. Instructions on how to add Extensions built by third-party teams like Coralogix are readily available.

Internal and External Extensions

Lambda Extensions come in two flavors: internal and external. External Extensions run independently of the Lambda execution environment and are not limited by Lambda’s runtime or coding language. Internal Extensions use the same runtime process as the Lambda and use wrapper scripts or other in-process tools that depend on your chosen language.

Internal Extensions allow you to modify the runtime process. You can use them to add parameters to the runtime and collect metrics otherwise not available. They can also be used to pre-load functionality without modifying your actual Lambda function. This feature is useful when you have many Lambdas or Lambda services that should use the same startup routines. Internal Extensions may be run using environment variables dependent on the used language or run using wrapper scripts.

External Extensions run as a separate process from your Lambda. They can start before Lamba’s runtime and continue after the Lambda runtime completes. They can also be triggered to run before the container running the Lambda stops to provide metrics. 

Performance and Execution of Lambda Extensions

Lambdas use a secure and isolated execution environment. Lambda Extensions run in the same environment as your Lambda function. They are extracted from a Lambda layer during the init phase of execution, and the Lambda will start all attached Extensions in parallel. 

The Extensions must be initialized completely before your Lambda function can be invoked. Ensuring a fast initialization time for your Extension is critical; otherwise, you may add latency to your Lambda functions and slow down regular processing times. 

The size of the Extension will count towards the total Lambda deployment package size. The Extension will also share memory, CPU, and storage with your Lambda. Offloading intensive computations to an Extension can increase your function’s duration and memory usage, and allocating more memory to a function can increase the function’s throughput.

Cost 

AWS charges users for any Lambda execution time consumed by the extension. The cost is broken down in one millisecond increments just as regular Lambda runtimes. There is no charge for the installation of your Extensions to Lambda.

Note that the cost of running Lambdas does increase per unit time as you increase the memory allocation. While increasing memory may be required to improve the speed of calculations, cost should be considered in this infrastructure design. You need to optimize memory allocation throughout time and cost based on these settings and the needs of the Lambda function.

Security

Lambda extensions share resources with the Lambda function. Resources such as memory allocation, CPU, and environment variables are shared. This includes sharing IAM roles, meaning that the Lambda extension has access to everything your Lambda function does. Keeping with AWS’s recommended security principles of allowing access to only required infrastructure is critical to securing your Lambda function and its associated extensions. Further, if the extension needs access to some AWS service, it should be included in the IAM roles of the Lambda, even if that access is only needed for the extension.

Summary

Lambda Extensions is a new, fully available service used to plug in extra functionality to Lambda runtimes. The typical use-case it provides is to streamline data exports to third-party observability tools like Coralogix’s machine learning platform. Using Lambda Extensions can save on your AWS bill because it allows you to bypass CloudWatch, which was previously required to export data automatically.