Catatan Seekor Jaeger

Catatan Seekor: Jaeger

<� Quick Start: Panduan lengkap distributed tracing menggunakan Jaeger untuk monitoring dan troubleshooting microservices architecture.

=� Daftar Isi

• <� Overview

• [<� Architecture](#-architecture)

• [=� Installation & Setup](#-installation--setup)

• =' Instrumentation

• =� Data Collection

• = Querying & Analysis

• =� Performance Optimization

• =' Integration

• =� Best Practices

• =� Resources

<� Overview

Jaeger adalah open-source distributed tracing system yang dirancang untuk monitoring dan troubleshooting microservices-based architectures. Jaeger membantu mengidentifikasi performance bottlenecks, dependency analysis, dan error tracking dalam distributed systems.

< Mengapa Jaeger?

• = End-to-End Tracing: Trace request flow melalui multiple services

• =� Performance Analysis: Identifikasi latency dan bottleneck issues

• =' Root Cause Analysis: Temukan penyebab masalah dengan cepat

• < Multi-Language Support: Instrumentasi untuk berbagai programming languages

• =� Scalable Architecture: Dapat handle large-scale distributed systems

• = Ecosystem Integration: Integrasi dengan Prometheus, Grafana, OpenTelemetry

<� Architecture

<� Core Components

Jaeger Components:
  - Jaeger Client: Libraries untuk instrumentasi aplikasi
  - Jaeger Agent: Process yang menerima spans dari clients
  - Jaeger Collector: Centralized component untuk6� spans
  - Jaeger Query: Service untuk querying stored traces
  - Jaeger UI: Web interface untuk trace visualization
  - Storage: Backend storage untuk trace data

= Data Flow

Application Spans � Jaeger Client � Jaeger Agent � Jaeger Collector � Storage
                                                          �
                                                Jaeger Query � Jaeger UI

=� Trace & Span Concepts

Trace: Tree of spans yang mewakili request path
Span: Single operation dalam trace dengan timing dan metadata
Trace ID: Unique identifier untuk seluruh trace
Span ID: Unique identifier untuk individual span
Parent Span ID: Identifier untuk parent-child relationship

=� Installation & Setup

=3 Docker Installation

# docker-compose.yml
version: '3'
services:
  jaeger:
    image: jaegertracing/all-in-one:latest
    ports:
      - "16686:16686"  # Jaeger UI
      - "14268:14268"  # HTTP collector
      - "14250:14250"  # gRPC collector
    environment:
      - COLLECTOR_OTLP_ENABLED=true
      - SPAN_STORAGE_TYPE=memory

8 Kubernetes Installation

# jaeger-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: jaeger
spec:
  replicas: 1
  selector:
    matchLabels:
      app: jaeger
  template:
    metadata:
      labels:
        app: jaeger
    spec:
      containers:
      - name: jaeger
        image: jaegertracing/all-in-one:latest
        ports:
        - containerPort: 16686
        - containerPort: 14268
        - containerPort: 14250
---
apiVersion: v1
kind: Service
metadata:
  name: jaeger
spec:
  selector:
    app: jaeger
  ports:
  - name: ui
    port: 16686
    targetPort: 16686
  - name: collector
    port: 14268
    targetPort: 14268

=' Production Setup with Elasticsearch

# jaeger-production.yml
version: '3'
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.14.0
    environment:
      - discovery.type=single-node
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ports:
      - "9200:9200"

  jaeger-collector:
    image: jaegertracing/jaeger-collector:latest
    command: [
      "/go/bin/all-in-one-linux",
      "--es.server-urls=http://elasticsearch:9200",
      "--es.index-prefix=jaeger",
      "--collector.grpc-server.host-port=:14250",
      "--collector.http-server.host-port=:14268"
    ]
    depends_on:
      - elasticsearch
    ports:
      - "14250:14250"
      - "14268:14268"

  jaeger-query:
    image: jaegertracing/jaeger-query:latest
    command: [
      "/go/bin/all-in-one-linux",
      "--es.server-urls=http://elasticsearch:9200",
      "--es.index-prefix=jaeger",
      "--query.http-server.host-port=:16686"
    ]
    depends_on:
      - elasticsearch
    ports:
      - "16686:16686"

=' Instrumentation

=9 Go Instrumentation

package main

import (
    "context"
    "io"
    "net/http"
    "time"

    "github.com/opentracing/opentracing-go"
    "github.com/opentracing/opentracing-go/ext"
    "github.com/uber/jaeger-client-go"
    jaegercfg "github.com/uber/jaeger-client-go/config"
    jaegerlog "github.com/uber/jaeger-client-go/log"
)

func initJaeger(service string) (opentracing.Tracer, io.Closer) {
    cfg := jaegercfg.Configuration{
        ServiceName: service,
        Sampler: &jaegercfg.SamplerConfig{
            Type:  jaeger.SamplerTypeConst,
            Param: 1,
        },
        Reporter: &jaegercfg.ReporterConfig{
            LogSpans: true,
            LocalAgentHostPort: "localhost:6831",
        },
    }

    tracer, closer, err := cfg.NewTracer(
        jaegercfg.Logger(jaegerlog.StdLogger),
    )
    if err != nil {
        panic(fmt.Sprintf("ERROR: cannot init Jaeger: %v\n", err))
    }

    return tracer, closer
}

func httpHandler(w http.ResponseWriter, r *http.Request) {
    // Extract span context from incoming request
    spanCtx, _ := tracer.Extract(opentracing.HTTPHeaders, opentracing.HTTPHeadersCarrier(r.Header))

    // Create new span for this handler
    span := tracer.StartSpan("http-handler", ext.RPCServerOption(spanCtx))
    defer span.Finish()

    // Add tags
    span.SetTag("http.method", r.Method)
    span.SetTag("http.url", r.URL.Path)
    span.SetBaggageItem("user-id", "12345")

    // Business logic
    time.Sleep(100 * time.Millisecond)
    w.Write([]byte("Hello, World!"))
}

func main() {
    tracer, closer := initJaeger("example-service")
    defer closer.Close()
    opentracing.SetGlobalTracer(tracer)

    http.HandleFunc("/", httpHandler)
    http.ListenAndServe(":8080", nil)
}

=

Python Instrumentation

from flask import Flask
from jaeger_client import Config
import opentracing
from opentracing.ext import tags

app = Flask(__name__)

def init_tracer(service_name='flask-app'):
    config = Config(
        config={
            'sampler': {
                'type': 'const',
                'param': 1,
            },
            'logging': True,
            'local_agent': {
                'reporting_host': 'localhost',
                'reporting_port': '6831',
            },
        },
        service_name=service_name,
    )
    return config.initialize_tracer()

tracer = init_tracer()

@app.route('/')
def hello_world():
    # Extract span context from headers
    span_ctx = tracer.extract(opentracing.Format.HTTP_HEADERS, dict(request.headers))

    # Start new span
    span = tracer.start_span('flask-handler', child_of=span_ctx)
    span.set_tag(tags.HTTP_METHOD, request.method)
    span.set_tag(tags.HTTP_URL, request.url)
    span.set_tag('custom.tag', 'custom-value')

    try:
        # Business logic here
        with tracer.start_span('database-query', child_of=span):
            # Simulate database operation
            time.sleep(0.1)

        with tracer.start_span('external-api-call', child_of=span):
            # Simulate external API call
            time.sleep(0.2)

        span.set_tag(tags.HTTP_STATUS_CODE, 200)
        return 'Hello, World!'

    except Exception as e:
        span.set_tag(tags.ERROR, True)
        span.log_kv({'event': 'error', 'error.object': e})
        return 'Error occurred', 500
    finally:
        span.finish()

if __name__ == '__main__':
    app.run(debug=True)

=� Java (Spring Boot) Instrumentation

@SpringBootApplication
@RestController
public class JaegerDemoApplication {

    private final Tracer tracer;

    public JaegerDemoApplication(Tracer tracer) {
        this.tracer = tracer;
    }

    @GetMapping("/hello")
    public String hello(HttpServletRequest request) {
        // Extract span context from incoming request
        SpanContext spanContext = tracer.extract(
            Format.Builtin.HTTP_HEADERS,
            new TextMapExtractAdapter(
                Collections.list(request.getHeaderNames())
                    .stream()
                    .collect(Collectors.toMap(h -> h, request::getHeader))
            )
        );

        // Start new span
        Span span = tracer.buildSpan("hello-endpoint")
                .asChildOf(spanContext)
                .withTag(Tags.SPAN_KIND, Tags.SPAN_KIND_SERVER)
                .withTag(Tags.HTTP_METHOD, request.getMethod())
                .withTag(Tags.HTTP_URL, request.getRequestURL().toString())
                .start();

        try (Scope scope = tracer.scopeManager().activate(span)) {
            // Business logic
            span.setTag("custom.tag", "custom-value");

            // Create child span for database operation
            Span dbSpan = tracer.buildSpan("database-query")
                    .asChildOf(span)
                    .start();
            try {
                // Simulate database operation
                Thread.sleep(100);
                dbSpan.setTag("db.statement", "SELECT * FROM users");
                dbSpan.setTag("db.type", "sql");
            } catch (InterruptedException e) {
                Tags.ERROR.set(dbSpan, true);
                dbSpan.log(Map.of("event", "error", "error.object", e));
            } finally {
                dbSpan.finish();
            }

            Tags.HTTP_STATUS.set(span, 200);
            return "Hello, World!";
        } catch (Exception e) {
            Tags.ERROR.set(span, true);
            span.log(Map.of("event", "error", "error.object", e));
            return "Error occurred";
        } finally {
            span.finish();
        }
    }

    public static void main(String[] args) {
        SpringApplication.run(JaegerDemoApplication.class, args);
    }
}

=� Data Collection

<� Span Attributes

Required Attributes:
  - operation.name: Name of the operation being traced
  - service.name: Name of the service generating the span
  - span.kind: Type of span (server, client, producer, consumer)

Recommended Attributes:
  - http.method: HTTP method for web requests
  - http.url: Full URL for web requests
  - http.status_code: HTTP response status
  - db.type: Database type (sql, mongodb, redis)
  - db.statement: Database query or statement
  - messaging.system: Message broker system
  - messaging.destination: Queue/topic name

=� Baggage Items

// Setting baggage items in Go
span.SetBaggageItem("user.id", "12345")
span.SetBaggageItem("request.id", "req-abc-123")
span.SetBaggageItem("tenant.id", "tenant-xyz")

// Getting baggage items
if userID := span.BaggageItem("user.id"); userID != "" {
    // Use user ID for business logic
}

= Trace Context Propagation

// HTTP Client with Trace Context
function makeTracedRequest(url) {
    const span = tracer.startSpan('http-request', {
        tags: {
            'http.method': 'GET',
            'http.url': url,
            'span.kind': 'client'
        }
    });

    const headers = {};
    tracer.inject(span, opentracing.FORMAT_HTTP_HEADERS, headers);

    fetch(url, { headers })
        .then(response => {
            span.setTag('http.status_code', response.status);
            span.finish();
            return response;
        })
        .catch(error => {
            span.setTag('error', true);
            span.log({ event: 'error', 'error.object: error });
            span.finish();
        });
}

=

Querying & Analysis

< Jaeger UI Features

Search Capabilities:
  - Service-based filtering
  - Operation-based filtering
  - Tag-based filtering
  - Time range selection
  - Duration filtering
  - Trace ID lookup

Visualization Features:
  - Gantt chart visualization
  - Service dependency graph
  - Span details view
  - Log correlation
  - Performance metrics

=

Query API Examples

# Get trace by ID
curl "http://localhost:16686/api/traces/{trace-id}"

# Search traces
curl -X POST "http://localhost:16686/api/traces" \
  -H "Content-Type: application/json" \
  -d '{
    "service": "user-service",
    "operation": "GetUser",
    "start": "2023-01-01T00:00:00Z",
    "end": "2023-01-01T23:59:59Z",
    "limit": 100
  }'

# Get services
curl "http://localhost:16686/api/services"

# Get operations for a service
curl "http://localhost:16686/api/services/user-service/operations"

=� Trace Analysis Patterns

Performance Analysis:
  1. Identify slow traces (high duration)
  2. Locate bottleneck spans
  3. Analyze service dependencies
  4. Compare normal vs error traces

Error Analysis:
  1. Filter traces with error tags
  2. Examine error logs and stack traces
  3. Identify error propagation patterns
  4. Correlate with system metrics

=� Performance Optimization

� Sampling Strategies

Sampling Types:
  - Constant: Sample all or none traces
  - Probabilistic: Sample percentage of traces
  - Rate Limiting: Sample up to N traces per second
  - Adaptive: Dynamic sampling based on load

Best Practices:
  - Use higher sampling for critical services
  - Implement adaptive sampling for production
  - Consider business-critical operations
  - Balance cost vs. observability

=' Performance Tuning

Collector Optimization:
  - Increase collector replicas
  - Use appropriate storage backend
  - Optimize batch processing
  - Configure appropriate timeouts

Storage Optimization:
  - Choose storage based on scale
  - Implement data retention policies
  - Use indexing for efficient queries
  - Monitor storage performance

=� Monitoring Jaeger Itself

Metrics to Monitor:
  - Collector throughput
  - Storage query latency
  - Agent connectivity
  - UI response times
  - Memory usage
  - CPU utilization

Alerting Rules:
  - High error rates
  - Increasing latency
  - Storage capacity issues
  - Component failures

=' Integration

= OpenTelemetry Integration

Migration Benefits:
  - Vendor-neutral observability
  - Unified instrumentation
  - Future-proof technology
  - Enhanced functionality

Migration Path:
  1. Deploy OpenTelemetry Collector
  2. Update instrumentation to OpenTelemetry
  3. Configure Jaeger as backend
  4. Validate trace collection
  5. Retire old instrumentation

=� Prometheus Integration

Metrics Integration:
  - Jaeger metrics in Prometheus
  - Correlate traces with metrics
  - Unified observability dashboard
  - Alerting based on trace data

Configuration:
  jaeger:
    metrics-backend: prometheus
    metrics-http-route: /metrics

=� Grafana Integration

Dashboard Components:
  - Trace search interface
  - Service dependency graphs
  - Performance metrics
  - Error rate visualization
  - Latency heatmaps

Data Sources:
  - Jaeger for trace data
  - Prometheus for metrics
  - Loki for logs

=� Best Practices

<� Instrumentation Best Practices

Span Naming:
  - Use descriptive, consistent names
  - Include operation type and resource
  - Avoid high cardinality values
  - Follow naming conventions

Tag Usage:
  - Use standard semantic conventions
  - Include relevant context
  - Avoid sensitive data
  - Keep tag values concise

=' Production Readiness

Security:
  - Secure Jaeger endpoints
  - Implement authentication
  - Use TLS for communication
  - Audit access logs

Scalability:
  - Horizontal scaling of collectors
  - Appropriate storage sizing
  - Load balancing configuration
  - Monitoring resource usage

=� Operational Excellence

Monitoring:
  - Component health checks
  - Performance metrics
  - Error rate tracking
  - Capacity planning

Maintenance:
  - Regular backups
  - Software updates
  - Storage cleanup
  - Configuration review

=� Resources

=� Official Documentation

• Jaeger Documentation

• OpenTelemetry Documentation

• Jaeger GitHub Repository

=' Client Libraries

• Jaeger Client Libraries

• OpenTelemetry Instrumentation

• Spring Cloud Sleuth

=� Learning Resources

• Distributed Tracing Concepts

• Observability Best Practices

• Microservices Monitoring

---

=� Implementation Checklist

Setup & Configuration

• Deploy Jaeger infrastructure

• Configure appropriate storage backend

• Set up sampling strategy

• Implement monitoring and alerting

• Verify component connectivity

Instrumentation

• Install client libraries

• Implement tracing in critical services

• Add context propagation

• Include relevant span attributes

• Test trace collection

Operations

• Configure data retention policies

• Set up performance monitoring

• Implement security measures

• Create operational procedures

• Document architecture and processes

---

Catatan Seekor: Jaeger - Comprehensive distributed tracing guide for microservices observability