Open Telemetry

Introduction

This is a quick page about open telemetry which I came across doing the Effect_TS.

Four things

Four things must ye know about Open Telemetry

• Traces - Spans
• Metrics - Performances, counts etc
• Logs - No Explanation required
• Baggage - contextual information we want to pass from span to span

This is an example of a trace with spans.

Trace: "User logs in"
├── Span: "Frontend sends login request"
├── Span: "API receives request"
│   ├── Span: "Validate credentials"
│   └── Span: "Query user DB"

Other Terms

• OTLP - Open Telemetry Protocol Sent over gRPC or HTTP.
• Collector - Own Service to accept data and push it out
• Instrumentation - How you make the data. Some frameworks like expressJS, effectTS have automatic but you can use a OpenTelemetry SDK

Configuring Collector

There are six things you configure in a collector

• Receivers - How is how you collector takes in data
• Processors - Cleansing, Transform
• Extensions - Extra things, e.g. health monitor
• Exporters - Push data out, e.g Loki, Prometheus
• Pipelines - The path that telemetry data—traces, metrics, or logs—follows from ingestion to export
• Connectors - Can export data from one pipeline and feed it into another

connectors:
  spanmetrics:

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [spanmetrics]

    metrics:
      receivers: [spanmetrics]
      exporters: [prometheus]

Example with Node

npm install @opentelemetry/sdk-node \
  @opentelemetry/api \
  @opentelemetry/auto-instrumentations-node \
  @opentelemetry/sdk-metrics \
  @opentelemetry/sdk-trace-node

We can run this with

npx  tsx --import  ./src/opentelemetry/instrumentation.ts  ./src/app.ts

Example with CSharp

Configure OpenTelemetry Resource/Tracing/Metrics

So we need to configure these three

public static class IServiceCollectionExtension
{
    public static IServiceCollection AddTelemetry(
        this IServiceCollection serviceCollection,
        ITelemetryConfiguration telemetryConfig,
        IHoneycombConfiguration honeycombConfig)
    {
        // Register telemetry service first
        serviceCollection
            .AddSingleton<ITelemetryServiceFactory, TelemetryServiceFactory>()
            .AddSingleton<ITelemetryService>(sp =>
            {
                try
                {
                    return sp.GetRequiredService<ITelemetryServiceFactory>().CreateTelemetryService();
                }
                catch (Exception ex)
                {
                    Console.WriteLine($"Error creating telemetry service: {ex.Message}");
                    throw; // Re-throw to prevent silent failures
                }
            });

        // Create the builder first
        var builder = serviceCollection
            .AddOpenTelemetry()
            .ConfigureResource(resourceBuilder =>
                resourceBuilder.ConfigurableResource(telemetryConfig)
            )
            .WithTracing(tracerProviderBuilder =>
                tracerProviderBuilder.ConfigureTracing(telemetryConfig, honeycombConfig)
            )
           .WithMetrics(metricsBuilder =>
                metricsBuilder.ConfigureMetrics(telemetryConfig, honeycombConfig)
            );

        return serviceCollection;
    }
}

Configure Resources

In CSharp you have, and I expect in the others too, Resources. These are things you configure once and a attached to the OTel data. They are only sent once per batch. To set these up you do the following. Add an extension to build them.

    public static ResourceBuilder ConfigurableResource(
        this ResourceBuilder resourceBuilder,
        ITelemetryConfiguration telemetryConfig)
    {
        resourceBuilder
            .AddService(
                    telemetryConfig.ServiceName,
                    telemetryConfig.ServiceNamespace,
                    telemetryConfig.ServiceVersion)
            .AddAttributes(BuildAttributes(telemetryConfig));

        return resourceBuilder;
    }

</syntaxhighlight>

Configure Tracing

Tracing is where you configure the Tracing sent to OTel. The Dataset is sent in the header. I had to remove the http Instrumentation as it prevented the Grpc being tied to the other events.

    public static TracerProviderBuilder ConfigureTracing(
        this TracerProviderBuilder tracerProviderBuilder,
        ITelemetryConfiguration telemetryConfig,
        IHoneycombConfiguration honeycombConfig)
    {
        // Configure basic tracing without external exporters first
        tracerProviderBuilder
            .AddSamplerWhen(telemetryConfig.Environment == "Development")
            .AddAspNetCoreInstrumentation()
            .AddGrpcClientInstrumentation(options =>
            {
                options.SuppressDownstreamInstrumentation = true;
            })
            // .AddHttpClientInstrumentation()
            .AddSource(telemetryConfig.ServiceName)
            .AddConsoleExporterWhen(telemetryConfig.EnableConsoleExporter == true && telemetryConfig.Environment == "Development")
            .AddOtlpExporterWhen(honeycombConfig, telemetryConfig.Enabled);

        return tracerProviderBuilder;
    }

Configure Metrics

This is straightforward but here for completeness.

    public static MeterProviderBuilder ConfigureMetrics(
            this MeterProviderBuilder meterProviderBuilder,
            ITelemetryConfiguration telemetryConfig,
            IHoneycombConfiguration honeycombConfig)
    {
        // Configure basic metrics
        meterProviderBuilder
            // Add a resource with service name, version, and namespace
            .AddAspNetCoreInstrumentation()
            // Add HTTP client instrumentation
            .AddHttpClientInstrumentation()
            // Addi runtime instrumentation
            .AddRuntimeInstrumentation()
            // Add SQL client instrumentation if needed
            .AddSqlClientInstrumentation()
            //Adding process instrumentation (cpu, memory etc.)
            .AddProcessInstrumentation()
            // Add custom meters
            .AddMeter(
                "Microsoft.AspNetCore.Hosting",
                "Microsoft.AspNetCore.Server.Kestrel",
                "Microsoft.AspNetCore.Http.Connections",
                telemetryConfig.ServiceName)
            .AddConsoleExporterWhen(telemetryConfig.EnableConsoleExporter == true && telemetryConfig.Environment == "Development")
            .AddOtlpExporter(opts =>
            {
                // Set the endpoint explicitly
                opts.Endpoint = new Uri(honeycombConfig.MetricsEndpoint);
                // Include dataset and environment in headers
                opts.Headers = $"x-honeycomb-team={honeycombConfig.ApiKey},x-honeycomb-dataset={honeycombConfig.Dataset},x-honeycomb-environment={honeycombConfig.Environment}";
                // Set the correct protocol
                opts.Protocol = OtlpExportProtocol.HttpProtobuf;
            });

        return meterProviderBuilder;
    }

Activities

Overview

So I struggled with this as the robot took over and hid how it works. What I ended up doing was wrapping my creation of activity inside a helper class. This had two constructors, one which take a parent context and one that does not. The idea is that if you start and activity and you want to track the stack e.g.

foo1()
  foo2()
    foo3()

You create an Activity with a default parent context for the first call but for the subsequent calls you look in something call Activity.Current, check it is not null and use that for the parent context which should be the previous calls spam id.

    public OperationTracker(
        OperationType type,
        string operationName,
        ActivityContext parentContext,
        TagList? tags,
        TracingProvider tracing,
        MetricsProvider metrics)
    {
        _type = type;
        _operationName = operationName ?? throw new ArgumentNullException(nameof(operationName));
        _parentContext = parentContext;
        _tracing = tracing ?? throw new ArgumentNullException(nameof(tracing));
        _metrics = metrics ?? throw new ArgumentNullException(nameof(metrics));

        // Check for listeners before creating a new activity
        if (!_tracing.ActivitySource.HasListeners())
        {
            Console.WriteLine($"Warning: ActivitySource '{_tracing.ActivitySource.Name}' has no listeners. Activity '{_operationName}' will not be created.");
        }

        _activity = _tracing.ActivitySource.StartActivityWithParentAndTags(
            operationName,
            _parentContext,
            ActivityKind.Server
            );

        if (_activity != null && tags != null)
        {
            _activity.AddTags(tags);
            Console.WriteLine($"{_activity.DisplayName}/{operationName}/{_activity.SpanId}: Making new Activity with Kind {_activity.Kind} and with Parent: Parent SpanId {parentContext.SpanId}");
        }

        // Count the request regardless of sampling
        _metrics.RecordRequest(_type, operationName);
    }

Interceptor

For my Grpc I used interceptors to wrap the calls. This way I could create the initial call for all Grpc endpoints. I my case GetActor, GetActors, GetFilm, Getfilms. This got me used to interceptors again on .NET and reduced the code. The GetActor just does the following

Get Actor
  Get Actor From Database

So there is a Actor Repository

    public async Task<ActorEntity?> GetActorAsync(int actorId, CancellationToken cancellationToken = default)
    {
        _logger.LogDebug("Getting actor with ID {ActorId} from database", actorId);

        using var dbTracker = _telemetryService.TrackOperation(
            OperationType.DatabaseOperation,
            DiagnosticNames.DatabaseGetActor);

        try
        {
            const string selectQuery = "SELECT actor_id, first_name, last_name, last_update FROM actor WHERE actor_id = @actor_id";

            // Validate SQL statement with parameters
            var parameters = new Dictionary<string, object> { { "@actor_id", actorId } };

            // Validate SQL statement
            var validationResult = await _sqlValidator.ValidateAsync(connection, selectQuery, parameters, cancellationToken);
            if (!validationResult.IsValid)
            {
                _logger.LogError("SQL Validation Error: {ErrorMessage}", validationResult.ErrorMessage);
                throw new SqlValidationException($"Invalid SQL statement: {validationResult.ErrorMessage}", 
                    selectQuery);
            }

            // Ensure connection is opened only if it's closed
            if (connection.State != System.Data.ConnectionState.Open)
            {
                _logger.LogInformation("Opening connection for GetActorAsync");
                await connection.OpenAsync(cancellationToken);
            }

            await using var cmd = connection.CreateCommand();
            cmd.CommandText = selectQuery;
            cmd.Parameters.AddWithValue("@actor_id", actorId);

            // Execute reader
            await using var reader = await cmd.ExecuteReaderAsync(cancellationToken);

            ActorEntity? actorEntity = null;

            if (await reader.ReadAsync(cancellationToken))
            {
                actorEntity = BuildActorEntity(reader);
            }

            return actorEntity;
        }
        catch (SqlValidationException ex)
        {
            _logger.LogError(ex, "Error SQL Validation error");
            dbTracker.TrackException(ex);
            throw new RpcException(new Status(StatusCode.Internal, $"Invalid SQL statement: {ex.Message}"));
        }
        catch (Exception ex)
        {
            _logger.LogError(ex, "Error retrieving actor with ID {ActorId}", actorId);
            dbTracker.TrackException(ex);
            throw;
        }
    }

Capturing Data

To capture the data I created a TelephonyService which holds these. You can find these in C# grpc project

• ServiceInfo - Holds basic Service information Name, Version etc
• MetricProvider - Defines and exposes application metrics
• OperationTracker - Combines metrics and tracing for a single operation
• TelemetryService - The central orchestration point for all telemetry operations
• Tracing Provider - Manages distributed tracing across service boundaries

You generally make one set of these per application and once done you add them to your Tracing Provider.

Exporters

You need to configure the exporter appropriate to the destination, I worked with Console for Development, Honeycomb and Jaeger. Not of this is difficult, you just follow the docs. With Honeycomb I ended up making the Service Name the same as the dataset as that how it works for either Tracing or Metrics (cannot remember which)