Fun with OTEL collectors and metrics

As part of an evaluation of Prometheus compatible monitoring solutions, I found the need to push our use of the OTEL Collector to handle some use cases like creating metrics allowlists, renaming metrics, or adding and modifying labels.

Here's some examples, based on what I learned, of the crazy and powerful things you can do with OTEL collector processors to manipulate metrics.

Inserting static labels

As part of a multi-account AWS strategy, we have many Kubernetes clusters, spread across AWS accounts for each of our teams. We wanted to make sure that all metrics coming from Kubernetes clusters contain labels with metadata about which cluster and account they came from (beyond what comes with the k8sattributes processor).

We use the OTEL collector as a daemonset (so it runs on all nodes in our clusters), and all telemetry from our pods goes through them.

NOTE

Since these OTEL collectors are deployed in Kubernetes, we can inject environment variables into the pods with this static information (in our case these environment variables are set via Terraform).

attributes/cluster-metadata:
  actions:
  - action: upsert
    value: "${CLUSTER_ENV}"
    key: env
  - action: upsert
    value: "${CLUSTER_LABEL}"
    key: cluster_label
  - action: upsert
    value: "${CLUSTER_NAME}"
    key: cluster
  - action: upsert
    value: "${CLUSTER_TEAM}"
    key: team
  - action: upsert
    value: "${CLUSTER_REGION}"
    key: region

Inserting dynamic labels

We have a bunch of legacy services, deployed outside Kubernetes, that don't have an instance label (which is idiomatic in Prometheus). These metrics (generated by telegraf) do have a host label, however, so we used that to create an instance label, also using the attributes processor:

attributes/instance-label:
  actions:
  - action: insert
    from_attribute: host
    key: instance

Replacing useless labels with useful ones

When we scrape kube-state-metrics, the pod and namespace labels on the metrics are the pod and namespace of the kube-state-metrics pod itself. This isn't so useful; we don't care about the kube-state-metrics pod names, we only care about the pods that are the subject of the metrics.

Here's a trick where we use the attributes processor to remove the kube-state-metric pod/namespace labels, and then rename the exported pod/namespace labels to replace them.

This way, when users are querying for metrics on their pod, they can just use the pod label, and don't have to worry about the implementation details of how kube-state-metrics is scraped:

# Delete the pod and namespace labels which refer to the kube-state-metrics
# pod itself, not the pods the metrics refer to.
attributes/kube-state-metrics:
  include:
    match_type: regexp
    metric_names: "^kube_.+$"
  actions:
  - action: delete
    key: pod
  - action: delete
    key: namespace

# Rename the exported_pod and exported_namespace labels to pod and namespace
metricstransform/kube-state-metrics:
  transforms:
  - include: "^kube_.*$"
    match_type: regexp
    action: update
    operations:
    - action: update_label
      label: exported_namespace
      new_label: namespace
    - action: update_label
      label: exported_pod
      new_label: pod

You'll need to make sure that the attributes/kube-state-metrics processor runs before the metricstransform/kube-state-metrics processor in your pipeline, so that the old labels are deleted before the new ones are renamed.

Renaming metrics

Sometimes, we'd find older services had metrics that had been named in various problematic ways, so we wanted a way to rename metrics (e.g. to adhere to a naming convention). Here's a use of the metricstransform processor that renames all metrics with a badsuffix to have a goodsuffix instead:

NOTE

The double dollar sign ($$) is intentional; the OTEL collector would interpret ${1} as an environment variable. The second $ escapes the first, so that it's interpreted as a literal $, and used as part of the regular expression capture group.

metricstransform/fix-suffix:
  transforms:
  - include: ^(.*)_badsuffix$
    match_type: regexp
    action: update
    new_name: "$${1}_goodsuffix"

Truncating long label values

Grafana Cloud has a maximum label length of 1024 characters. Any metrics with labels exceeding this length will be dropped before they're ingested. Here's a nifty transform that truncates all label values this length:

warning

Why would anyone have a label value that long? Well, there's no good reason. But sometimes, just sometimes, a distracted programmer may accidentally include an entire stack trace as a label value.

Not naming names.

transform/truncate-labels:
  metric_statements:
  - context: datapoint
    statements:
    - truncate_all(attributes, 1024)

Filtering metrics with arbitrary queries

Here's where we get to the use case that we really needed: a way to drop large swaths of metrics entirely, based on arbitrary queries. The most common pattern we were trying to replicate was an "allowlist", where we drop most metrics, except for those that meet some criteria.

The OTEL collector has a filter processor to do this, and it supports the OpenTelemetry Transformation Language (OTTL), which allows you to write complex expressions to represent your filter criteria:

filter/drop-rules:
  error_mode: ignore
  metrics:
    datapoint:

    # An "allowlist" that drops all metrics from 'some-service' except for
    # the two specified.
    - >
      resource.attributes["service.name"] == "some-service" and
      metric.name != "some_service_important_metric" and
      metric.name != "up"

    # A similar allowlist, but using a regex to match the service name
    # for various sized fluent-bit daemonset pods (we have small, medium, 
    # and large variants of fluent-bit).
    - >
      IsMatch(resource.attributes["service.name"], "fluent-bit-.*") and
      metric.name != "fluentbit_output_dropped_records" and
      metric.name != "up"

    # A drop rule that drops a specific cadvisor metric for all services
    # except for those in a specific namespace.
    - >
      resource.attributes["service.name"] == "cadvisor" and
      metric.name == "container_file_descriptors" and
      (not IsMatch(attributes["namespace"], "^someservice.*$"))

    # A drop rule that shows you can use more complex boolean expressions
    # with parentheses to group conditions.
    - >
      attributes["telegraf"] == "1" and (
        IsMatch(metric.name, "^internal_(agent|gather|memstats|serializer|statsd|write)_.*") or
        IsMatch(metric.name, ".+[-_]request[-_]metrics[-_](median|sum)$") or
        IsMatch(metric.name, ".+_stddev$")
      )

    # Another complex drop rule, where we're dropping metrics matching a
    # regex for all services, but with a list of exceptions.
    - >
      IsMatch(metric.name, ".+[-_]request[-_]metrics[-_]upper$") and not (
        attributes["service"] == "service1" or
        attributes["service"] == "service2" or 
        metric.name == "inconsistently_named_service_request_metrics_upper"
      )

Limitations of the OTEL collector

It can't do aggregations

While the OTEL collector has many powerful processors, it doesn't currently have the ability to do aggregations (i.e. drop a particular label from a metric and create a new metric by aggregating the metrics that had that label). This is a much harder problem to solve than just dropping metrics, since all the OTEL collector instances that could process any metric you'd want to aggregate would need to coordinate with each other, creating some scaling challenges.

Both Grafana Cloud and Chronosphere offer powerful features around metrics aggregation.

It can't convert delta to cumulative counters

When I evaluated Chronosphere, I was delighted to find that it had a feature to change the temporality of metrics (e.g. change a "delta" counter to a "cumulative" counter), and I was hoping to replicate it with the OTEL collector. While the OTEL collector does have a converter processor in the works, it's still early in development.

In our case, we were able to work around this with some hackery in telegraf (setting the delete_counters setting of the statsd plugin to false).