Kubernetes Service Monitoring with Prometheus: A Practical Guide

What Is Prometheus?

Prometheus is an open-source systems monitoring and alerting toolkit originally developed by SoundCloud. It collects metrics from various services and applications, providing insights into their performance and reliability. Prometheus uses a query language called PromQL to filter and aggregate time-series data, allowing users to create dashboards and alerts.

Key features of Prometheus include:

• Multidimensional data model: Stores data as time-series, identified by metric names and key-value pairs called labels.
• Flexible query language (PromQL): Allows for real-time analysis of the metrics.
• Autonomous single server nodes: Does not rely on distributed storage, making it simpler to deploy and manage.
• Service discovery: Can discover targets dynamically or through static configurations.
• Pull-based metrics collection: Scrapes metrics from instrumented jobs, enabling it to handle large-scale monitoring efficiently.

Prometheus is widely used in cloud-native environments and integrates with Kubernetes, making it a popular choice for monitoring dynamic and complex infrastructure.

This is part of a series of articles about Kubernetes monitoring

What Is a Prometheus CRD (Custom Resource Definition)?

Prometheus custom resource definitions (CRDs) extend Kubernetes capabilities, allowing users to define and manage custom resources specific to Prometheus. These CRDs include AlertmanagerConfigs, PodMonitors, Probes, ServiceMonitors, and ThanosRulers.

The use of CRDs simplifies the configuration and management of Prometheus within a Kubernetes cluster, enabling dynamic service discovery and monitoring configuration, which traditional static configurations lack.

CRDs provide a standard approach to embedding Prometheus configurations within Kubernetes manifests. This integration allows for declarative infrastructure management, where configurations are applied using Kubernetes-native tools.

Chris Cooney

Head of Developer Experience, Coralogix

Chris is passionate about all things observability, organizational leadership, and cutting-edge engineering. He’s worked on everything, from internal developer platforms, enormous Kubernetes clusters, embedded systems running industrial batteries on the UK balancing mechanism and more.

Tips from the expert:

In my experience, here are tips that can help you better monitor Kubernetes services with Prometheus:

• Optimize label cardinality: While labels are powerful for filtering and aggregating data, high cardinality can cause performance issues in Prometheus. Regularly review and limit the number of unique label values, especially for labels that could produce a large number of unique combinations.
• Leverage recording rules for performance: Use Prometheus recording rules to precompute frequently queried expressions and store them as new time series. This reduces the load on Prometheus when running complex queries, leading to faster query results and lower resource usage.
• Deploy horizontal sharding for scalability: For large clusters, consider horizontally sharding Prometheus instances based on workload types or namespaces. This allows you to scale out monitoring without overloading a single Prometheus instance, improving reliability and performance.
• Monitor the Prometheus stack itself: Ensure that you have metrics and alerts set up to monitor the health and performance of your Prometheus stack, including the Prometheus server, Alertmanager, and any associated exporters or custom integrations. This self-monitoring helps catch issues in your monitoring stack before they impact visibility.
• Automate ServiceMonitor creation with GitOps: Integrate ServiceMonitor creation and updates into a GitOps workflow. By storing these configurations in a version-controlled repository, you can automate deployments, ensure consistency, and maintain an audit trail of all changes made to your monitoring setup.

How a Prometheus CRD Works

Prometheus CRDs function by extending the Kubernetes API to manage Prometheus configurations through Kubernetes-native tools. Here’s an overview of how these CRDs operate:

1. Definition and registration: CRDs are defined using YAML files, specifying the custom resources and their schemas. Once defined, these CRDs are registered with the Kubernetes API server, making Kubernetes aware of the new resource types.
2. Resource creation: Users create custom resources (like ServiceMonitors, PodMonitors, etc.) based on the registered CRDs. These resources are defined in YAML manifests, detailing the configuration and specifications for Prometheus to monitor the desired services, pods, or endpoints.
3. Controller logic: The Prometheus Operator watches for changes in these custom resources. It acts as a controller, ensuring that the desired state defined by the custom resources is maintained. For example, if a new ServiceMonitor is created, the Prometheus Operator updates the Prometheus configuration to start scraping the specified endpoints.
4. Dynamic configuration: The use of CRDs allows for dynamic and declarative configuration. As services are added or removed, corresponding custom resources can be created or deleted, automatically updating Prometheus’ monitoring scope without manual configuration changes.
5. Integration with the Kubernetes ecosystem: By leveraging CRDs, Prometheus configurations benefit from Kubernetes features like namespaces, labels, and selectors. This integration provides a scalable way to manage monitoring configurations, ensuring they are consistent with the rest of the Kubernetes-managed infrastructure.

What Is a Service Monitor in Prometheus?

A ServiceMonitor is a specialized custom resource in Prometheus CRD that defines how services should be monitored. It specifies the endpoints of services and the configurations needed to scrape metrics from those endpoints. By using ServiceMonitors, users can dynamically configure Prometheus to adapt to changes in the service landscape.

ServiceMonitors allow for fine-grained control over which services Prometheus monitors and how it does so. This includes defining selectors to identify target services, specifying endpoints, and configuring scrape intervals and paths. The integration of ServiceMonitors into Kubernetes workflows makes it easier to maintain visibility over complex microservices architectures.

Tutorial: Using Service Monitoring in Kubernetes

In this tutorial, we will walk through the process of setting up a ServiceMonitor in Prometheus to monitor Kubernetes services. By the end of this guide, you will understand how to configure Prometheus to automatically scrape metrics from services within a Kubernetes cluster using ServiceMonitors.

Step 1: Verify Service Labels

First, identify the labels of the service you want to monitor. Use the kubectl describe command to view the details of the prometheus-operated service created by the Prometheus Operator:

kubectl describe service prometheus-operated --namespace prometheus

The output should look similar to this:

Note the label operated-prometheus=true, which we will use in our ServiceMonitor.

Step 2: Create the ServiceMonitor YAML

Next, create a ServiceMonitor resource that targets the service we identified. Create a file named servicemonitor.yaml and add the following configuration:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: prometheus
  namespace: prometheus
  labels:
    serviceMonitorSelector: prometheus
spec:
  endpoints:
  - interval: 20s
    targetPort: 9090
    path: /metrics
  namespaceSelector:
    matchNames:
    - prometheus
  selector:
    matchLabels:
      operated-prometheus: "true"

This YAML defines a ServiceMonitor that scrapes the /metrics endpoint of the prometheus-operated service every 20 seconds.

Step 3: Apply the ServiceMonitor

Apply the ServiceMonitor configuration to your Kubernetes cluster:

kubectl apply -f servicemonitor.yaml

Verify that the ServiceMonitor was created successfully:

kubectl get servicemonitor --namespace prometheus

You should see an output similar to this:

Step 4: Update the Prometheus Resource

To instruct the Prometheus Operator to use the ServiceMonitor, update the Prometheus resource configuration to include the serviceMonitorSelector. Edit the prometheus.yaml file and add the following section:

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: prometheus
  namespace: prometheus
spec:
  serviceMonitorSelector:
    matchLabels:  
      serviceMonitorSelector: prometheus  
# other configurations

Apply the updated Prometheus configuration:

kubectl apply -f prometheus.yaml

Step 5: Verify the Configuration

After a few moments, the Prometheus Operator will update the Prometheus instance with the new scrape configuration. To verify, open the Prometheus UI and navigate to Status, then Targets:

kubectl port-forward svc/prometheus-operated 9090:9090 --namespace prometheus

Open your browser and go to http://localhost:9090/targets. You should see the prometheus-operated service listed as a scrape target.

You can also query metrics in the Prometheus UI. For example, type prometheus_build_info in the query box and click Execute. This should display build information metrics for the Prometheus instance.

Related content: Read our guide to Kubernetes Metrics

Kubernetes Monitoring with Coralogix

Coralogix sets itself apart in observability with its modern architecture, enabling real-time insights into logs, metrics, and traces with built-in cost optimization. Coralogix’s straightforward pricing covers all its platform offerings including APM, RUM, SIEM, infrastructure monitoring and much more. With unparalleled support that features less than 1 minute response times and 1 hour resolution times, Coralogix is a leading choice for thousands of organizations across the globe.

Learn more about Coralogix for Kubernetes