How to Setup AWS CloudWatch Agent Using AWS Systems Manager

What is AWS Systems Manager

Before we jump into this, it’s important to note that older names, and still in use in some areas of AWS, are often referred to SSM which stands for Simple Systems Manager. AWS Systems Manager is designed to be a control panel for your AWS resources so you can manage them externally without having to SSH into the resources individually. 

What is important to remember with AWS Systems Manager is that features contained within the tool may occur additional pricing. You’ll need to consider this on your own use case and calculate the cost/benefit analysis accordingly. 

AWS Systems Manager allows you to group, view, and act on resources within your AWS environment. It allows you to group resources into Applications, Business Units, or any other category you care to consider. What this means is that you can view and manage Operating System Patch Levels and Installed S/W so that you can see how complies with the desired state. 

You can also set up configuration rules against specific resources and allow AWS Systems Manager to inform when they don’t match and/or act automatically to resolve issues that have been identified. For those of you reading this who are familiar with Ansible, Chef, or Puppet, you may notice some overlap with these technologies. For example, this could include noticing version differences of software installed on the machines. 

Why you need AWS CloudWatch Agent with AWS Systems Manager

Setting up the AWS CloudWatch Logs Agent via the command line is an extremely manual process. For a small handful of servers, you can probably manage to do this manually, but what you’ll soon realize is that for any form of consistency and simplifying things as they scale, you need to automate this process. This is where AWS Systems Manager comes into play. 

AWS Systems Manager is that central control panel that allows you to see how things are configured and what is out of alignment against what is configured VS what should be configured. To get a simplified understanding of how this fits together, below is a high-level architecture of how things work;

What you’ll notice with the simplified diagram above is that the core difference with installing the CloudWatch Agent via AWS Systems Manager is that this reduces the effort required significantly and ensures that all your machines are controlled from a top-down approach rather than an error-prone bottom-up approach. 

This allows you to securely connect to a managed EC2 instance without having to worry about managing SSH keys and/or opening ports on your firewalls/security groups. This enables you to easily view CloudWatch metrics along with any alarms you have configured via AWS CloudWatch automatically without having to configure each instance with the specific metrics that you want to collect and the frequency on which they are collected. 

When you need to consider utilizing AWS Systems Manager

You should start looking at AWS Systems Manager to automate systems when you are ready to start looking at infrastructure as low code technologies that allow you to view observability metrics on your systems. You’ll know when this time comes as you’ll come to the conclusion that fiddling around at the command line becomes too much of a burden for tracking changes to ensure you are keeping your instances patched in line with the latest releases. 

AWS Systems Manager allows you to centrally define configuration details and policies for your infrastructure resources under its control. This means that if a system becomes non-compliant, such as in the scenario whereby a user has manually SSH’d into a system and changed a version of a piece of software running, then AWS Systems Manager should detect this so you can act accordingly. 

While tools like AWS Systems Manager are great, you need to walk before you can run. These tools add complexity to your architecture which can often mean more effort to set up and more effort to maintain. So while they do bring added benefits, it’s not all roses. 

It’s important to get the basics right before jumping in with these tools. You need to ensure you fully understand your technology stack so that you can create a runbook in a basic tool such as Word if you have to, so you have full visibility of what is set up and why. This should include information on the setup process and commands you have to run and configure to ensure your system is fit for purpose. Once you have this, you’re ready to get started with automation.  

How to install AWS CloudWatch Agent Using AWS Systems Manager

One of the major differences with AWS Systems Manager VS Ansible is that with AWS Systems Manager you need to install an Agent on the instance that you want to control, whereas with Ansible this accesses the system over SSH which is a little more convenient. 

Select Host Management Configuration Type

There is a number of configuration types within AWS Systems Manager that you can select from depending on your needs. To get started it’s recommended that you start with the Quick Setup and select the Host Management option so you can start to get a feel for the capabilities. 

Select Option to Install AWS CloudWatch Agent

By default, the next screen only has options for Systems Manager automatically selected. So simply check the boxes next to Amazon CloudWatch to also install the CloudWatch Agent when you install the Systems Manager Agent. 

What you can easily skip over when first setting this up is that out of the box this process will configure the bare minimum set of metrics to be collected which you have no control over. This can be handy when you are wanting to quickly get set up, but you’ll soon want to customize the metrics to a level that suits your needs. So if there is a specific piece of data you are wanting to view, make sure you read the manual from Amazon for how to configure the different metrics within CloudWatch Agent so you can report on what you need with ease. 

Select a Target Instance to Install CloudWatch Agent On

You can now select the target for where you want to install AWS Systems Manager Agent and CloudWatch Agent. If you are just getting started it is recommended to start small so you can become familiar with how things work, so start with a single instance. Once you are more comfortable you can scale this out to meet your needs;

Remember, it’s going to be exponentially more difficult to clean up a situation where you install this incorrectly on ‘all instances’ rather than just a single instance to start with. Once you’re comfortable with setting up the CloudWatch Agent via AWS Systems Manager on a single instance, perhaps then take a step forwards and do small batches either using the Instance Tag or the Resource Group so you can easily roll this out to all instances safely. 

Successfully Completed

Once you’ve successfully run through the setup process you’ll notice that things have been configured for you;

All being well, this should have worked for you and you should start to see additional metrics being collected via the CloudWatch Agent that is now installed on your instance(s). 

It’s worth mentioning that, like everything with AWS services, it all depends on the technology stack. When this was being tested, certain operating systems worked, others didn’t. In some cases, the AWS Systems Manager Agent failed to install on the instance which meant that the CloudWatch Agent couldn’t be installed either – and this was on a supported system, according to the documentation.  

Benefits from installing AWS Systems Manager

One of the biggest benefits of using AWS Systems Manager is that you can start to standardize your infrastructure configurations, including configuring AWS CloudWatch Agent to consistently report on the data that you are interested in. All of this helps you bring to life your observability data to allow you to be proactive in managing your systems. 

Always remember though, you need to manage this carefully. AWS Systems Manager is great on paper, but you need to test this thoroughly with your own systems to ensure it supports the features and functionality that you expect. You will likely have a diverse technology stack across your systems to get the data you need which is fairly natural but it is something you need to keep a close tab on. 

Once you’ve got AWS Systems Manager setup, alongside installing great tools such as the CloudWatch Agent and other tools that collect observability metrics, you can also start to use features such as the Patch Manager functionality within AWS Systems Manager which enables you to control your instances by setting patch schedules and much more.