Open Telemetry

Introduction

This is a quick page about open telemetry which I came across doing the Effect_TS.

Four things

Four things must ye know about Open Telemetry

• Traces - Spans
• Metrics - Performances, counts etc
• Logs - No Explanation required
• Baggage - contextual information we want to pass from span to span

This is an example of a trace with spans.

Trace: "User logs in"
├── Span: "Frontend sends login request"
├── Span: "API receives request"
│   ├── Span: "Validate credentials"
│   └── Span: "Query user DB"

Configuring OTel

For Open Telemetry you need to set up

• Resources - Defines common data
• Metrics - Defines and exposes application metrics
• Traces - Defines the traces to be sent
• Logs - Defines how to capture logs
• Exporters - Where to send the data

Resources

This is where you define common things you want to appear on you traces, logs and metrics. Service Name, Service Version and Service

Metrics

This is some which will hold your metrics you intend to capture. Typically this can be a

• Counter - Counter of requests
• Histogram - Buckets E.g bucket for duration of transactions
• Gauge - Something which fluctuates like CPU, Memory

Traces

This is for shows the spans for a transaction. E.g. Receive Request, Get Data, Send Response. A lot of libraries provide there own implementation called instrumentation where you can choose to turn these on and receive telemetry from there library with little or no configuration. Examples are Postgres, Grpc, Aspnet. These are known as instrumentation

Logs

This was not support or in beta so I guess a bit in flux. For one platform you could attach logs to the spans.

Exporters

This is where you define the platform to send the data to. For my I have used Jaeger, Honecomb and will be looking at prometheus.

Results

Here is the result when it all works

You can see the call stack on the left. From the Grpc Client through to the Get Actor from the database. On the right you can see 6 Span Events. These are actually the logs for the call.
And here it is in Jaeger

Configuring Collector

There are six things you configure in a collector

• Receivers - How is how you collector takes in data
• Processors - Cleansing, Transform
• Extensions - Extra things, e.g. health monitor
• Exporters - Push data out, e.g Loki, Prometheus
• Pipelines - The path that telemetry data—traces, metrics, or logs—follows from ingestion to export
• Connectors - Can export data from one pipeline and feed it into another

connectors:
  spanmetrics:

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [spanmetrics]

    metrics:
      receivers: [spanmetrics]
      exporters: [prometheus]

Example with Node

npm install @opentelemetry/sdk-node \
  @opentelemetry/api \
  @opentelemetry/auto-instrumentations-node \
  @opentelemetry/sdk-metrics \
  @opentelemetry/sdk-trace-node

We can run this with

npx  tsx --import  ./src/opentelemetry/instrumentation.ts  ./src/app.ts