Filtering and reducing costs

The Otel integration has a couple of ways you can reduce costs of telemetry data. One simple way is to enable the reduceResourceAttributes preset, which removes the following list of resource attributes that are typically not used:

• azure.resourcegroup.name
• azure.vm.name
• azure.vm.scaleset.name
• azure.vm.size
• cloud.account.id
• cloud.availability_zone
• cloud.platform
• cloud.provider
• cloud.region
• container.id
• cx.otel_integration.name
• faas.id
• faas.instance
• faas.name
• faas.version
• gcp.cloud_run.job.execution
• gcp.cloud_run.job.task_index
• gcp.gce.instance_group_manager.name
• gcp.gce.instance_group_manager.region
• gcp.gce.instance_group_manager.zone
• host.image.id
• host.type
• k8s.cronjob.uid
• k8s.daemonset.uid
• k8s.deployment.uid
• k8s.hpa.uid
• k8s.job.uid
• k8s.namespace.uid
• k8s.node.uid
• k8s.pod.start_time
• k8s.pod.uid
• k8s.replicaset.uid
• k8s.statefulset.uid
• os.type
• os.version
• process.command
• process.command_line
• process.command_args
• process.executable.name
• process.executable.path
• process.owner
• process.pid
• process.parent_pid
• process.runtime.description
• process.runtime.name
• process.runtime.version
• net.host.name
• net.host.port
• telemetry.distro.name (only removed from metrics and logs)
• telemetry.distro.version (only removed from metrics and logs)
• telemetry.sdk.language (only removed from metrics and logs)
• telemetry.sdk.name (only removed from metrics and logs)
• telemetry.sdk.version (only removed from metrics and logs)

Kubernetes resource attributes are typically coming from Kubernetes Attributes Processor and Kubernetes Cluster receiver.

While net.host.name and net.host.port is coming from Prometheus receiver, instead of using these attributes you can use the service.instance.id attribute, which has a combination of host and port.

Additionally, the reduceLogAttributes preset removes the following log record attributes that are typically not used:

• log.iostream
• log.file.path
• logtag
• time (only a timestamp, raw nanoseconds epoch time is still present)

Coralogix recommends new customers to enable these two presets by default. Existing customers should consider enabling them on new deployments after analyzing the potential impact in alerts and dashboards that they already have.

Important disclaimer: the default denylists in these presets can change between versions of the integration. If stability is important you should consider customizing them. Any change to the lists will be announced in the changelog and you can recheck them at any time.

Custom filtering of attributes

The reduceResourceAttributes preset supports a custom denylist of attributes per pipeline. You can customize the denylists per signal like so:

presets:
  reduceResourceAttributes:
    enabled: true
    pipelines: ["metrics", "traces", "logs"] # the "all" pipeline is also supported
    denylist:
     metrics:
      - custom_attribute_1
      - custom_attribute_2
    traces:
      - custom_attribute_1
      - custom_attribute_2
    logs:
      - custom_attribute_1
      - custom_attribute_2

Note that when customizing the denylists you are completely overriding the default one. This means that if you just want to append a few attributes to it, you will have to repeat the default ones.

Custom filtering of log record attributes

The reduceLogAttributes preset supports a custom denylist of log record attributes. You can customize the denylist like so:

presets:
  reduceLogAttributes:
    enabled: true
    denylist:
      - custom_attribute_1
      - custom_attribute_2

Note that when customizing the denylist you are completely overriding the default one. This means that if you just want to append a few attributes to it, you will have to repeat the default ones.

Custom filtering of metrics

Alternatively, you can also use include / exclude filters to collect only metrics about needed objects. For example, the following configuration allows you to exclude kube-* and default namespace Kubernetes metrics. This filtering is available on many mdatagen based receiver.

For example:

receivers:
  k8s_cluster:
    collection_interval: 10s
    allocatable_types_to_report: [cpu, memory]
    resource_attributes:
      k8s.namespace.name:
        metrics_exclude:
          - regexp: kube-.*
          - strict: default

Dropping data using processors

Alternatively you can use OpenTelemetry Transformation Language with filter processor to filter out unneeded data. The following example drops metrics named my.metric and any metrics with resource where attribute my_label equals abc123:

processors:
  filter/ottl:
    error_mode: ignore
    metrics:
      metric:
        - name == "my.metric"
        - resource.attributes["my_label"] == "abc123"