Back to Blog

Enterprise Logging & Observability with .NET: OpenTelemetry Complete Guide 2025

Nov 15, 2025•
loggingobservabilityopentelemetry.net
•
0

Enterprise applications require comprehensive observability to understand system behavior, diagnose issues, and optimize performance. OpenTelemetry provides a vendor-neutral standard for collecting telemetry data—logs, metrics, and traces—enabling consistent observability across .NET applications.

This comprehensive guide covers enterprise logging and observability for .NET applications using OpenTelemetry, including structured logging, distributed tracing, metrics collection, and production monitoring patterns. You'll learn how to implement observability, correlate telemetry data, and build production-ready monitoring solutions.

Understanding Observability

Three Pillars of Observability

1. Logs

  • Event records
  • Text-based
  • High cardinality
  • Good for debugging

2. Metrics

  • Numerical measurements
  • Aggregated data
  • Low cardinality
  • Good for monitoring

3. Traces

  • Request flows
  • Distributed context
  • Performance analysis
  • Good for troubleshooting

OpenTelemetry Setup

Basic Configuration

Configure OpenTelemetry in .NET:

builder.Services.AddOpenTelemetry()
    .WithTracing(builder => builder
        .AddAspNetCoreInstrumentation()
        .AddHttpClientInstrumentation()
        .AddSource("MyApplication")
        .AddConsoleExporter())
    .WithMetrics(builder => builder
        .AddAspNetCoreInstrumentation()
        .AddHttpClientInstrumentation()
        .AddConsoleExporter())
    .WithLogging(builder => builder
        .AddConsoleExporter());

Exporters

Configure exporters for telemetry:

builder.Services.AddOpenTelemetry()
    .WithTracing(builder => builder
        .AddAspNetCoreInstrumentation()
        .AddOtlpExporter(options =>
        {
            options.Endpoint = new Uri("https://otel-collector:4317");
        }))
    .WithMetrics(builder => builder
        .AddAspNetCoreInstrumentation()
        .AddOtlpExporter(options =>
        {
            options.Endpoint = new Uri("https://otel-collector:4317");
        }));

Structured Logging

Serilog Configuration

Configure Serilog for structured logging:

Log.Logger = new LoggerConfiguration()
    .WriteTo.Console()
    .WriteTo.File("logs/app-.log", rollingInterval: RollingInterval.Day)
    .WriteTo.Seq("http://localhost:5341")
    .Enrich.FromLogContext()
    .Enrich.WithMachineName()
    .Enrich.WithThreadId()
    .CreateLogger();

builder.Host.UseSerilog();

Structured Logging Patterns

Use structured logging:

public class OrderService
{
    private readonly ILogger<OrderService> _logger;

    public async Task<Order> CreateOrderAsync(CreateOrderRequest request)
    {
        using var scope = _logger.BeginScope(new Dictionary<string, object>
        {
            ["OrderId"] = request.OrderId,
            ["CustomerId"] = request.CustomerId
        });

        _logger.LogInformation(
            "Creating order {OrderId} for customer {CustomerId}",
            request.OrderId, request.CustomerId);

        try
        {
            var order = await ProcessOrderAsync(request);
            
            _logger.LogInformation(
                "Order {OrderId} created successfully with {ItemCount} items",
                order.Id, order.Items.Count);
            
            return order;
        }
        catch (Exception ex)
        {
            _logger.LogError(ex,
                "Failed to create order {OrderId} for customer {CustomerId}",
                request.OrderId, request.CustomerId);
            throw;
        }
    }
}

Distributed Tracing

Activity Creation

Create custom activities:

public class OrderProcessingService
{
    private static readonly ActivitySource ActivitySource = new("OrderProcessing");

    public async Task ProcessOrderAsync(Order order)
    {
        using var activity = ActivitySource.StartActivity("ProcessOrder");
        activity?.SetTag("order.id", order.Id);
        activity?.SetTag("order.customer_id", order.CustomerId);

        try
        {
            await ValidateOrderAsync(order);
            await ProcessPaymentAsync(order);
            await FulfillOrderAsync(order);
            
            activity?.SetStatus(ActivityStatusCode.Ok);
        }
        catch (Exception ex)
        {
            activity?.SetStatus(ActivityStatusCode.Error, ex.Message);
            activity?.RecordException(ex);
            throw;
        }
    }
}

Trace Correlation

Correlate traces across services:

public class TracePropagationMiddleware
{
    private readonly RequestDelegate _next;

    public async Task InvokeAsync(HttpContext context)
    {
        var activity = Activity.Current;
        
        if (activity != null)
        {
            // Add trace context to response headers
            context.Response.Headers.Add("trace-id", activity.TraceId.ToString());
            context.Response.Headers.Add("span-id", activity.SpanId.ToString());
        }

        await _next(context);
    }
}

Metrics Collection

Custom Metrics

Create custom metrics:

public class OrderMetrics
{
    private readonly Meter _meter;
    private readonly Counter<long> _ordersCreated;
    private readonly Histogram<double> _orderProcessingTime;

    public OrderMetrics()
    {
        _meter = new Meter("OrderService");
        _ordersCreated = _meter.CreateCounter<long>("orders.created");
        _orderProcessingTime = _meter.CreateHistogram<double>("order.processing.time");
    }

    public void RecordOrderCreated(string customerId)
    {
        _ordersCreated.Add(1, new KeyValuePair<string, object>("customer.id", customerId));
    }

    public void RecordProcessingTime(double duration, string orderType)
    {
        _orderProcessingTime.Record(duration, new KeyValuePair<string, object>("order.type", orderType));
    }
}

Metric Exporters

Export metrics to monitoring systems:

builder.Services.AddOpenTelemetry()
    .WithMetrics(builder => builder
        .AddMeter("OrderService")
        .AddPrometheusExporter(options =>
        {
            options.StartHttpListener = true;
            options.HttpListenerPrefixes = new[] { "http://localhost:9464/" };
        }));

Production Patterns

Log Aggregation

Aggregate logs from multiple services:

builder.Services.AddOpenTelemetry()
    .WithLogging(builder => builder
        .AddOtlpExporter(options =>
        {
            options.Endpoint = new Uri("https://log-aggregator:4318");
        }));

Error Tracking

Track errors with context:

public class ErrorTrackingMiddleware
{
    private readonly RequestDelegate _next;
    private readonly ILogger<ErrorTrackingMiddleware> _logger;

    public async Task InvokeAsync(HttpContext context)
    {
        try
        {
            await _next(context);
        }
        catch (Exception ex)
        {
            var activity = Activity.Current;
            
            _logger.LogError(ex,
                "Unhandled exception in {Path} with trace {TraceId}",
                context.Request.Path,
                activity?.TraceId);

            activity?.RecordException(ex);
            activity?.SetStatus(ActivityStatusCode.Error, ex.Message);

            throw;
        }
    }
}

Performance Monitoring

Monitor application performance:

public class PerformanceMonitoringMiddleware
{
    private readonly RequestDelegate _next;
    private readonly ILogger<PerformanceMonitoringMiddleware> _logger;

    public async Task InvokeAsync(HttpContext context)
    {
        var stopwatch = Stopwatch.StartNew();
        var activity = Activity.Current;

        await _next(context);

        stopwatch.Stop();
        
        activity?.SetTag("http.duration", stopwatch.ElapsedMilliseconds);
        
        if (stopwatch.ElapsedMilliseconds > 1000)
        {
            _logger.LogWarning(
                "Slow request: {Path} took {Duration}ms",
                context.Request.Path,
                stopwatch.ElapsedMilliseconds);
        }
    }
}

Best Practices

1. Use Structured Logging

  • Include context in logs
  • Use consistent log levels
  • Avoid sensitive data
  • Use correlation IDs

2. Instrument Key Operations

  • HTTP requests
  • Database queries
  • External API calls
  • Business operations

3. Set Appropriate Sampling

builder.Services.AddOpenTelemetry()
    .WithTracing(builder => builder
        .SetSampler(new TraceIdRatioBasedSampler(0.1))); // 10% sampling

4. Monitor Resource Usage

  • CPU usage
  • Memory consumption
  • Network I/O
  • Disk I/O

5. Alert on Critical Issues

  • Error rate thresholds
  • Latency spikes
  • Resource exhaustion
  • Service degradation

Advanced Observability Patterns

Custom Metrics

Create custom business metrics:

public class BusinessMetrics
{
    private readonly Meter _meter;
    private readonly Counter<long> _ordersProcessed;
    private readonly Histogram<double> _orderProcessingTime;

    public BusinessMetrics()
    {
        _meter = new Meter("ECommerce.Business");
        _ordersProcessed = _meter.CreateCounter<long>("orders_processed_total");
        _orderProcessingTime = _meter.CreateHistogram<double>("order_processing_seconds");
    }

    public void RecordOrderProcessed(string orderType, double processingTime)
    {
        _ordersProcessed.Add(1, new KeyValuePair<string, object>("order_type", orderType));
        _orderProcessingTime.Record(processingTime, new KeyValuePair<string, object>("order_type", orderType));
    }
}

Custom Traces

Create custom spans for business operations:

public class CustomTracing
{
    private readonly TracerProvider _tracerProvider;

    public async Task ProcessOrderAsync(Order order)
    {
        using var activity = _tracerProvider
            .GetTracer("ECommerce")
            .StartActivity("ProcessOrder");

        activity?.SetTag("order.id", order.Id);
        activity?.SetTag("order.amount", order.Amount);

        try
        {
            await ValidateOrderAsync(order);
            activity?.AddEvent(new ActivityEvent("Order validated"));

            await ProcessPaymentAsync(order);
            activity?.AddEvent(new ActivityEvent("Payment processed"));

            await FulfillOrderAsync(order);
            activity?.SetStatus(ActivityStatusCode.Ok);
        }
        catch (Exception ex)
        {
            activity?.SetStatus(ActivityStatusCode.Error, ex.Message);
            activity?.RecordException(ex);
            throw;
        }
    }
}

Real-World Scenarios

Scenario 1: Multi-Service Tracing

Trace requests across multiple services:

public class DistributedTracingService
{
    public async Task<OrderResponse> ProcessOrderAsync(OrderRequest request)
    {
        using var activity = ActivitySource.StartActivity("ProcessOrder");
        activity?.SetBaggage("order.id", request.OrderId);

        // Call inventory service
        using var inventoryActivity = ActivitySource.StartActivity("CheckInventory");
        var inventory = await _inventoryService.CheckInventoryAsync(request.ProductId);
        inventoryActivity?.SetTag("inventory.available", inventory.Available);

        // Call payment service
        using var paymentActivity = ActivitySource.StartActivity("ProcessPayment");
        var payment = await _paymentService.ProcessPaymentAsync(request);
        paymentActivity?.SetTag("payment.status", payment.Status);

        // Call fulfillment service
        using var fulfillmentActivity = ActivitySource.StartActivity("FulfillOrder");
        var fulfillment = await _fulfillmentService.FulfillOrderAsync(request);
        fulfillmentActivity?.SetTag("fulfillment.tracking", fulfillment.TrackingNumber);

        return new OrderResponse { /* ... */ };
    }
}

Scenario 2: Performance Monitoring

Monitor application performance:

public class PerformanceMonitor
{
    private readonly Meter _meter;
    private readonly Histogram<double> _requestDuration;
    private readonly Counter<long> _requestCount;

    public PerformanceMonitor()
    {
        _meter = new Meter("Application.Performance");
        _requestDuration = _meter.CreateHistogram<double>("http_request_duration_seconds");
        _requestCount = _meter.CreateCounter<long>("http_requests_total");
    }

    public async Task<T> MonitorRequestAsync<T>(
        string endpoint, 
        Func<Task<T>> request)
    {
        var stopwatch = Stopwatch.StartNew();
        _requestCount.Add(1, new KeyValuePair<string, object>("endpoint", endpoint));

        try
        {
            var result = await request();
            _requestDuration.Record(
                stopwatch.Elapsed.TotalSeconds,
                new KeyValuePair<string, object>("endpoint", endpoint),
                new KeyValuePair<string, object>("status", "success"));
            return result;
        }
        catch (Exception ex)
        {
            _requestDuration.Record(
                stopwatch.Elapsed.TotalSeconds,
                new KeyValuePair<string, object>("endpoint", endpoint),
                new KeyValuePair<string, object>("status", "error"));
            throw;
        }
    }
}

Troubleshooting Observability Issues

Common Problems

Problem 1: Missing Traces

Symptoms:

  • Traces not appearing
  • Incomplete trace data

Solution:

public class TraceTroubleshooter
{
    public void ConfigureTracing()
    {
        Sdk.CreateTracerProviderBuilder()
            .AddSource("MyApplication")
            .AddConsoleExporter()
            .AddJaegerExporter(options =>
            {
                options.AgentHost = "localhost";
                options.AgentPort = 6831;
            })
            .SetSampler(new AlwaysOnSampler()) // Ensure all traces are sampled
            .Build();
    }
}

Problem 2: High Cardinality Metrics

Symptoms:

  • Too many unique metric combinations
  • Performance issues

Solution:

public class LowCardinalityMetrics
{
    // Bad: High cardinality
    private readonly Counter<long> _badMetric = _meter.CreateCounter<long>(
        "requests_total",
        "userId", "orderId", "productId"); // Too many unique combinations

    // Good: Low cardinality
    private readonly Counter<long> _goodMetric = _meter.CreateCounter<long>(
        "requests_total",
        "endpoint", "status"); // Limited unique combinations
}

Performance Optimization

Sampling Strategies

Implement intelligent sampling:

public class IntelligentSampler
{
    public SamplingResult ShouldSample(SamplingParameters parameters)
    {
        // Sample all errors
        if (parameters.Tags.ContainsKey("error"))
        {
            return new SamplingResult(SamplingDecision.RecordAndSample);
        }

        // Sample 10% of normal requests
        if (Random.Shared.NextDouble() < 0.1)
        {
            return new SamplingResult(SamplingDecision.RecordAndSample);
        }

        return new SamplingResult(SamplingDecision.Drop);
    }
}

Batch Export

Batch telemetry export:

public class BatchExporter
{
    public void ConfigureBatchExport()
    {
        Sdk.CreateTracerProviderBuilder()
            .AddJaegerExporter(options =>
            {
                options.BatchExportProcessorOptions = new BatchExportProcessorOptions<Activity>
                {
                    MaxQueueSize = 2048,
                    ScheduledDelayMilliseconds = 5000,
                    ExporterTimeoutMilliseconds = 30000,
                    MaxExportBatchSize = 512
                };
            })
            .Build();
    }
}

Extended FAQ

Q: How do I correlate logs, traces, and metrics?

A: Use trace context:

public class CorrelationService
{
    public void LogWithTraceContext(string message)
    {
        var activity = Activity.Current;
        var traceId = activity?.TraceId.ToString();
        var spanId = activity?.SpanId.ToString();

        _logger.LogInformation(
            "Message: {Message}, TraceId: {TraceId}, SpanId: {SpanId}",
            message, traceId, spanId);
    }
}

Q: How do I reduce observability overhead?

A: Use sampling and filtering:

Sdk.CreateTracerProviderBuilder()
    .AddSource("MyApplication")
    .SetSampler(new TraceIdRatioBasedSampler(0.1)) // Sample 10%
    .AddProcessor(new FilteringProcessor(
        new BatchActivityExportProcessor(new ConsoleExporter<Activity>()),
        activity => activity.Kind == ActivityKind.Server)) // Only server activities
    .Build();

Case Studies

Case Study 1: Enterprise Application Monitoring

Challenge: A large enterprise application needed comprehensive observability across 100+ microservices.

Solution: Implemented centralized observability:

public class EnterpriseObservability
{
    public void ConfigureObservability()
    {
        // Configure OpenTelemetry
        Sdk.CreateTracerProviderBuilder()
            .AddSource("Enterprise.*")
            .AddHttpClientInstrumentation()
            .AddSqlClientInstrumentation()
            .AddServiceBusInstrumentation()
            .AddOtlpExporter(options =>
            {
                options.Endpoint = new Uri("https://observability-platform:4317");
            })
            .Build();

        // Configure metrics
        Sdk.CreateMeterProviderBuilder()
            .AddMeter("Enterprise.*")
            .AddRuntimeInstrumentation()
            .AddOtlpExporter(options =>
            {
                options.Endpoint = new Uri("https://observability-platform:4317");
            })
            .Build();

        // Configure logging
        LoggingBuilderExtensions.AddOpenTelemetry(builder =>
        {
            builder.AddOtlpExporter(options =>
            {
                options.Endpoint = new Uri("https://observability-platform:4317");
            });
        });
    }
}

Results:

  • 100% service coverage
  • Sub-second trace query
  • 99.9% observability uptime

Migration Guide

Migrating from Application Insights

Step 1: Replace Application Insights:

// Old: Application Insights
services.AddApplicationInsightsTelemetry();

// New: OpenTelemetry
services.AddOpenTelemetry()
    .WithTracing(builder => builder
        .AddAspNetCoreInstrumentation()
        .AddHttpClientInstrumentation()
        .AddOtlpExporter());

Step 2: Migrate custom telemetry:

// Old: Application Insights
_telemetryClient.TrackEvent("OrderProcessed", properties);

// New: OpenTelemetry
using var activity = _tracerProvider
    .GetTracer("Application")
    .StartActivity("OrderProcessed");
activity?.SetTag("order.id", orderId);

Conclusion

Enterprise observability is essential for understanding and maintaining production systems. By implementing OpenTelemetry, structured logging, distributed tracing, and metrics collection, you can build comprehensive observability into your .NET applications.

Key Takeaways:

  1. Use OpenTelemetry for vendor-neutral observability
  2. Implement structured logging with context
  3. Enable distributed tracing for request flows
  4. Collect metrics for monitoring
  5. Correlate telemetry across services
  6. Monitor production continuously
  7. Custom metrics for business insights
  8. Custom traces for business operations
  9. Performance optimization with sampling
  10. Migration planning from legacy systems

Next Steps:

  1. Set up OpenTelemetry
  2. Configure structured logging
  3. Add distributed tracing
  4. Implement metrics collection
  5. Set up monitoring dashboards
  6. Optimize performance
  7. Plan migration from legacy systems

Observability Best Practices

Implementation Checklist

  • Set up OpenTelemetry
  • Configure structured logging
  • Enable distributed tracing
  • Implement metrics collection
  • Correlate telemetry data
  • Set up monitoring dashboards
  • Configure alerting
  • Test observability setup
  • Document observability procedures
  • Review and optimize

Production Deployment

Before deploying observability:

  1. Test all telemetry collection
  2. Verify trace correlation
  3. Test metric collection
  4. Validate log aggregation
  5. Set up dashboards
  6. Configure alerts
  7. Document procedures
  8. Review performance impact

Additional Resources

OpenTelemetry Documentation

  • OpenTelemetry .NET
  • Instrumentation libraries
  • Exporters
  • Best practices
  • Distributed tracing
  • Structured logging
  • Metrics collection
  • Application performance monitoring

Observability Implementation Guide

Step-by-Step Setup

  1. Set Up OpenTelemetry

    • Install OpenTelemetry packages
    • Configure tracer provider
    • Configure meter provider
    • Set up exporters

  2. Configure Logging

    • Set up structured logging
    • Configure log levels
    • Add log exporters
    • Correlate with traces

  3. Enable Tracing

    • Add instrumentation
    • Create custom spans
    • Configure sampling
    • Set up trace export

  4. Collect Metrics

    • Define custom metrics
    • Add instrumentation
    • Configure metric export
    • Set up dashboards

  5. Set Up Monitoring

    • Create dashboards
    • Configure alerts
    • Set up notifications
    • Review regularly

Observability Patterns

Structured Logging

Implement structured logging:

public class StructuredLogger
{
    private readonly ILogger<StructuredLogger> _logger;

    public void LogOrderProcessed(string orderId, double amount)
    {
        _logger.LogInformation(
            "Order processed: {OrderId}, Amount: {Amount}",
            orderId, amount);
    }
}

Distributed Tracing

Create distributed traces:

public class TracingService
{
    private readonly ActivitySource _activitySource;

    public async Task ProcessOrderAsync(Order order)
    {
        using var activity = _activitySource.StartActivity("ProcessOrder");
        activity?.SetTag("order.id", order.Id);
        
        await ValidateOrderAsync(order);
        await ProcessPaymentAsync(order);
        await FulfillOrderAsync(order);
    }
}

Observability Advanced Patterns

Log Aggregation

Aggregate logs from multiple services:

public class LogAggregator
{
    public async Task<AggregatedLogs> AggregateLogsAsync(
        TimeSpan period, 
        string serviceName)
    {
        var logs = await CollectLogsAsync(period, serviceName);
        
        return new AggregatedLogs
        {
            TotalLogs = logs.Count,
            ErrorLogs = logs.Count(l => l.Level == LogLevel.Error),
            WarningLogs = logs.Count(l => l.Level == LogLevel.Warning),
            InfoLogs = logs.Count(l => l.Level == LogLevel.Information),
            TopErrors = GetTopErrors(logs)
        };
    }
}

Trace Correlation

Correlate traces across services:

public class TraceCorrelator
{
    public async Task<CorrelatedTrace> CorrelateTracesAsync(string traceId)
    {
        var traces = await GetTracesByTraceIdAsync(traceId);
        
        return new CorrelatedTrace
        {
            TraceId = traceId,
            Spans = traces.SelectMany(t => t.Spans).ToList(),
            TotalDuration = CalculateTotalDuration(traces),
            ServiceCount = traces.Select(t => t.ServiceName).Distinct().Count()
        };
    }
}

Observability Dashboard

Metrics Dashboard

Create comprehensive metrics dashboard:

public class MetricsDashboard
{
    public async Task<DashboardData> GetDashboardDataAsync(TimeSpan period)
    {
        return new DashboardData
        {
            RequestMetrics = await GetRequestMetricsAsync(period),
            ErrorMetrics = await GetErrorMetricsAsync(period),
            PerformanceMetrics = await GetPerformanceMetricsAsync(period),
            ResourceMetrics = await GetResourceMetricsAsync(period)
        };
    }
}

Trace Analysis

Analyze distributed traces:

public class TraceAnalyzer
{
    public async Task<TraceAnalysis> AnalyzeTraceAsync(string traceId)
    {
        var trace = await GetTraceAsync(traceId);

        return new TraceAnalysis
        {
            TraceId = traceId,
            TotalDuration = trace.Spans.Max(s => s.EndTime) - trace.Spans.Min(s => s.StartTime),
            ServiceCount = trace.Spans.Select(s => s.ServiceName).Distinct().Count(),
            SlowestSpan = trace.Spans.OrderByDescending(s => s.Duration).First(),
            ErrorSpans = trace.Spans.Where(s => s.Status == SpanStatus.Error).ToList()
        };
    }
}

Observability Best Practices

Logging Best Practices

Follow logging best practices:

public class LoggingBestPractices
{
    private readonly ILogger<LoggingBestPractices> _logger;

    public void LogWithContext(string message, string userId, string action)
    {
        // Use structured logging
        _logger.LogInformation(
            "User {UserId} performed {Action}: {Message}",
            userId, action, message);
    }

    public void LogErrorWithContext(Exception ex, string userId, string action)
    {
        // Include context in error logs
        _logger.LogError(ex,
            "Error occurred when user {UserId} performed {Action}",
            userId, action);
    }
}

Observability Best Practices Summary

Implementation Checklist

  • Set up OpenTelemetry
  • Configure structured logging
  • Enable distributed tracing
  • Implement metrics collection
  • Correlate telemetry data
  • Set up monitoring dashboards
  • Configure alerting
  • Test observability setup
  • Document observability procedures
  • Review and optimize

Production Deployment

Before deploying observability:

  1. Test all telemetry collection
  2. Verify trace correlation
  3. Test metric collection
  4. Validate log aggregation
  5. Set up dashboards
  6. Configure alerts
  7. Document procedures
  8. Review performance impact

Observability Troubleshooting

Common Issues

Troubleshoot observability issues:

public class ObservabilityTroubleshooter
{
    public async Task<DiagnosticsResult> DiagnoseAsync()
    {
        var diagnostics = new DiagnosticsResult();

        // Check OpenTelemetry setup
        diagnostics.OpenTelemetryConfigured = CheckOpenTelemetrySetup();

        // Check exporters
        diagnostics.ExportersWorking = await TestExportersAsync();

        // Check trace collection
        diagnostics.TracesCollected = await CheckTraceCollectionAsync();

        // Check metrics collection
        diagnostics.MetricsCollected = await CheckMetricsCollectionAsync();

        return diagnostics;
    }
}

Observability Performance Impact

Minimizing Overhead

Reduce observability overhead:

public class ObservabilityOptimizer
{
    public void OptimizeSampling()
    {
        Sdk.CreateTracerProviderBuilder()
            .SetSampler(new TraceIdRatioBasedSampler(0.1)) // Sample 10%
            .Build();
    }

    public void OptimizeLogging()
    {
        // Use log level filtering
        builder.AddFilter("Microsoft", LogLevel.Warning)
               .AddFilter("System", LogLevel.Warning)
               .AddFilter("Application", LogLevel.Information);
    }
}

Observability Implementation Examples

OpenTelemetry Setup

Complete OpenTelemetry setup:

public class OpenTelemetrySetup
{
    public void ConfigureObservability(IServiceCollection services)
    {
        // Tracing
        services.AddOpenTelemetry()
            .WithTracing(builder => builder
                .AddAspNetCoreInstrumentation()
                .AddHttpClientInstrumentation()
                .AddSqlClientInstrumentation()
                .AddSource("MyApplication")
                .AddJaegerExporter());

        // Metrics
        services.AddOpenTelemetry()
            .WithMetrics(builder => builder
                .AddAspNetCoreInstrumentation()
                .AddHttpClientInstrumentation()
                .AddRuntimeInstrumentation()
                .AddPrometheusExporter());

        // Logging
        services.AddLogging(builder => builder
            .AddOpenTelemetry(options =>
            {
                options.AddConsoleExporter();
                options.AddOtlpExporter();
            }));
    }
}

Custom Instrumentation

Create custom instrumentation:

public class CustomInstrumentation
{
    private readonly ActivitySource _activitySource;
    private readonly Meter _meter;

    public CustomInstrumentation()
    {
        _activitySource = new ActivitySource("MyApplication");
        _meter = new Meter("MyApplication");
    }

    public async Task ProcessOrderAsync(Order order)
    {
        using var activity = _activitySource.StartActivity("ProcessOrder");
        activity?.SetTag("order.id", order.Id);

        var counter = _meter.CreateCounter<long>("orders_processed_total");
        counter.Add(1, new KeyValuePair<string, object>("status", "processing"));

        try
        {
            await ProcessOrderInternalAsync(order);
            activity?.SetStatus(ActivityStatusCode.Ok);
            counter.Add(1, new KeyValuePair<string, object>("status", "success"));
        }
        catch (Exception ex)
        {
            activity?.SetStatus(ActivityStatusCode.Error, ex.Message);
            activity?.RecordException(ex);
            counter.Add(1, new KeyValuePair<string, object>("status", "error"));
            throw;
        }
    }
}

Observability Data Analysis

Log Analysis

Analyze logs for insights:

public class LogAnalyzer
{
    public async Task<LogAnalysis> AnalyzeLogsAsync(TimeSpan period)
    {
        var logs = await GetLogsAsync(period);

        return new LogAnalysis
        {
            TotalLogs = logs.Count,
            ErrorRate = (double)logs.Count(l => l.Level == LogLevel.Error) / logs.Count,
            TopErrors = logs
                .Where(l => l.Level == LogLevel.Error)
                .GroupBy(l => l.Message)
                .OrderByDescending(g => g.Count())
                .Take(10)
                .Select(g => new ErrorSummary
                {
                    Message = g.Key,
                    Count = g.Count()
                })
                .ToList()
        };
    }
}

Trace Analysis

Analyze traces for performance:

public class TraceAnalyzer
{
    public async Task<TraceAnalysis> AnalyzeTracesAsync(TimeSpan period)
    {
        var traces = await GetTracesAsync(period);

        return new TraceAnalysis
        {
            TotalTraces = traces.Count,
            AverageDuration = traces.Average(t => t.Duration),
            P95Duration = CalculatePercentile(traces.Select(t => t.Duration), 0.95),
            SlowestTraces = traces
                .OrderByDescending(t => t.Duration)
                .Take(10)
                .ToList(),
            ErrorTraces = traces.Where(t => t.HasErrors).ToList()
        };
    }
}

Observability Dashboard Creation

Custom Dashboard Builder

Build custom dashboards:

public class DashboardBuilder
{
    public async Task<Dashboard> CreateDashboardAsync(DashboardConfig config)
    {
        var dashboard = new Dashboard
        {
            Name = config.Name,
            Panels = new List<Panel>()
        };

        // Add metrics panels
        foreach (var metric in config.Metrics)
        {
            dashboard.Panels.Add(await CreateMetricPanelAsync(metric));
        }

        // Add log panels
        foreach (var logQuery in config.LogQueries)
        {
            dashboard.Panels.Add(await CreateLogPanelAsync(logQuery));
        }

        // Add trace panels
        foreach (var traceQuery in config.TraceQueries)
        {
            dashboard.Panels.Add(await CreateTracePanelAsync(traceQuery));
        }

        return dashboard;
    }
}

Alert Configuration

Configure alerts:

public class AlertConfigurator
{
    public async Task<Alert> CreateAlertAsync(AlertConfig config)
    {
        var alert = new Alert
        {
            Name = config.Name,
            Condition = config.Condition,
            Threshold = config.Threshold,
            NotificationChannels = config.NotificationChannels
        };

        // Set up alert evaluation
        await SetupAlertEvaluationAsync(alert);

        // Set up notifications
        await SetupNotificationsAsync(alert);

        return alert;
    }
}

Observability Performance Impact

Performance Monitoring

Monitor observability performance:

public class ObservabilityPerformanceMonitor
{
    public async Task<PerformanceMetrics> MonitorPerformanceAsync()
    {
        var metrics = new PerformanceMetrics();

        // Measure instrumentation overhead
        metrics.InstrumentationOverhead = await MeasureInstrumentationOverheadAsync();

        // Measure export overhead
        metrics.ExportOverhead = await MeasureExportOverheadAsync();

        // Measure storage overhead
        metrics.StorageOverhead = await MeasureStorageOverheadAsync();

        // Measure query performance
        metrics.QueryPerformance = await MeasureQueryPerformanceAsync();

        return metrics;
    }
}

Sampling Strategy

Implement sampling:

public class SamplingStrategy
{
    public bool ShouldSample(Activity activity)
    {
        // Sample all errors
        if (activity.Status == ActivityStatusCode.Error)
        {
            return true;
        }

        // Sample based on trace ID
        var traceId = activity.TraceId.ToString();
        var hash = traceId.GetHashCode();
        return Math.Abs(hash % 100) < 10; // 10% sampling
    }
}

Observability Best Practices Summary

Key Takeaways

  1. Use Structured Logging: Enable better search and analysis
  2. Implement Distributed Tracing: Track requests across services
  3. Collect Metrics: Monitor application and system metrics
  4. Implement Sampling: Reduce overhead while maintaining visibility
  5. Create Dashboards: Visualize observability data for better insights

Performance Optimization

  • Implement sampling to reduce overhead
  • Use batch export for better performance
  • Optimize log levels for production
  • Use async logging to avoid blocking
  • Monitor observability overhead

Common Pitfalls to Avoid

  • Logging too much or too little
  • Not implementing distributed tracing
  • Failing to collect meaningful metrics
  • Not using sampling in high-volume scenarios
  • Not monitoring observability overhead

Observability Implementation Checklist

Pre-Implementation

  • Choose observability tools (OpenTelemetry, Application Insights, etc.)
  • Design logging strategy
  • Plan distributed tracing
  • Design metrics collection
  • Plan dashboard and alerting

Implementation

  • Configure OpenTelemetry
  • Add structured logging
  • Implement distributed tracing
  • Add metrics collection
  • Create dashboards and alerts

Post-Implementation

  • Monitor observability overhead
  • Review and optimize sampling
  • Update dashboards based on needs
  • Update documentation
  • Train team on observability tools

For more observability guidance, explore our OpenTelemetry Observability guide or Enterprise Integration Architecture.

Related posts