OpenTelemetry Configuration

This guide provides instructions on how to configure OpenTelemetry in Krateo to enable the collection of metrics from the supported components and send them to your preferred observability backend (e.g., Prometheus) via an OpenTelemetry Collector.

Prerequisites

Before configuring OpenTelemetry in Krateo, make sure you have the following prerequisites in place:

1. Run your Krateo components with OpenTelemetry enabled (see Enabling Metrics).
2. An OpenTelemetry Collector configured in your Kubernetes cluster.
3. A compatible observability backend (e.g., Prometheus) set up to receive metrics from the OpenTelemetry Collector.
4. Grafana connected to your observability backend as a data source.

Deploying OpenTelemetry Collector (Helm)

You can deploy a shared Collector in-cluster using the official Helm chart. This example configuration sets up an OTLP HTTP receiver and a Prometheus exporter.

1. Add the Helm repository:

helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
helm repo update

2. Create a otelcol-values.yaml file with the following content:

mode: deployment
presets:
  kubernetesAttributes:
    enabled: true
image:
  repository: "otel/opentelemetry-collector-contrib"
  tag: "0.151.0"
config:
  receivers:
    otlp:
      protocols:
        http:
          endpoint: 0.0.0.0:4318
  exporters:
    prometheus:
      endpoint: 0.0.0.0:9464
  service:
    pipelines:
      metrics:
        receivers: [otlp]
        processors: [batch]
        exporters: [prometheus]
ports:
  otlp-http:
    enabled: true
    containerPort: 4318
    servicePort: 4318
    protocol: TCP
  prom-metrics:
    enabled: true
    containerPort: 9464
    servicePort: 9464
    protocol: TCP

3. Install the Collector:

helm upgrade --install otel-collector open-telemetry/opentelemetry-collector \
	-n monitoring --create-namespace \
	-f otelcol-values.yaml

4. The chart creates the ServiceMonitor automatically, so Prometheus can scrape the Collector metrics endpoint at :9464.

Quick Install: Prometheus & Grafana (Example)

Note

The following steps are provided as an example for setting up a monitoring stack. For production environments, follow your organization's standard practices for deploying and managing observability tools.

If you don't have a monitoring stack yet, one way to get started is by installing the kube-prometheus-stack. It includes Prometheus, Grafana, and the Prometheus Operator.

1. Add the Prometheus community Helm repository:

   helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
   helm repo update

2. Create a kube-prom-values.yaml file with the following content to configure a non-production setup of kube-prometheus-stack:

   grafana:
     enabled: true
     adminPassword: admin
   
   prometheus:
     prometheusSpec:
       scrapeInterval: 15s
       additionalScrapeConfigs:
         - job_name: "otel-collector"
           static_configs:
             - targets: ["otel-collector-opentelemetry-collector.monitoring.svc.cluster.local:9464"]
   
   alertmanager:
     enabled: false
   
   nodeExporter:
     enabled: false
   
   kubeStateMetrics:
     enabled: false

3. Install the stack in the monitoring namespace:

   helm upgrade --install prometheus prometheus-community/kube-prometheus-stack \
     -n monitoring --create-namespace \
     -f kube-prom-values.yaml

4. Once the pods are ready, you can proceed with the Collector deployment.

Enabling Metrics in Components

To enable OpenTelemetry metrics in Krateo components, you need to set the following environment variables:

OTEL_ENABLED: "true"
OTEL_EXPORT_INTERVAL: "30s"
OTEL_EXPORTER_OTLP_ENDPOINT: "http://otel-collector-opentelemetry-collector.monitoring.svc.cluster.local:4318"

Using krateoctl

You can create a custom config file for krateoctl to enable OpenTelemetry metrics. For instance, otel-config.yaml:

components:
  db-maintenance:
    stepConfig:
      install-deviser:
        with:
          values:
            config:
              OTEL_ENABLED: "true"
              OTEL_EXPORT_INTERVAL: "30s"
              OTEL_EXPORTER_OTLP_ENDPOINT: http://otel-collector-opentelemetry-collector.monitoring.svc.cluster.local:4318
  # ... other components

Using the monitoring profile

For development environments, you can use the --profile monitoring flag during installation:

krateoctl install apply --profile monitoring

Accessing Grafana

If you are running Grafana inside your Kubernetes cluster (for example, as part of the kube-prometheus-stack), you can access it using kubectl port-forward.

1. Find the Grafana service name:

   kubectl get svc -n monitoring | grep grafana

2. Port-forward to the service (replace grafana-service-name with the actual name):

   kubectl port-forward -n monitoring svc/grafana-service-name 3000:80

3. Open your browser at http://localhost:3000.

Importing Dashboards

Krateo provides ready-to-use Grafana dashboards for various components.

1. Open Grafana.
2. Go to Dashboards -> New -> Import.
3. Upload the .dashboard.json file provided with the component.
4. Select your Prometheus data source.
5. Save.

Metric Naming and Normalization

When metrics flow from OpenTelemetry to Prometheus, they are normalized:

• Metric names in code use dots (.), which Prometheus converts to underscores (_).
• Counters: Usually appear with a _total suffix (e.g., provider_runtime.startup.success becomes provider_runtime_startup_success_total).
• Histograms: Expose _bucket (cumulative count), _sum, and _count series.
• Average Latency: Use the _sum / _count series for average duration instead of _bucket quantiles for more accurate averages.

Troubleshooting

• Check collector logs: kubectl logs -n monitoring deployment/otel-collector
• Verify Prometheus scrape: Check http://<collector-service>:9464/metrics
• Check controller logs: Look for OpenTelemetry metrics initialized messages.
• Empty Panels: Some metrics (like webhooks) are only emitted during active traffic. Panels will remain empty until real requests are processed.