KubeCon + CloudNativeCon Japan 2025

How can real-time event streaming platforms, handling millions of events and complex data processing, maintain peak performance and reliability? Managing the same has previously been complex. The latest agent changes and addition of semantic convention in OpenTelemetry make it ideal to monitor highly distributed event streaming architectures (EDA) like Kafka. In this session we will discuss how these changes help standardize telemetry, explain the usage of span links for capturing several traces for a transaction in EDA.

The talk will also cover how Otel enables automatic anomaly detection particularly useful for identifying issues like Consumer Lag, Increased Latency in Event Processing, and Partition Failures. By leveraging context propagation, Otel tracks end-to-end latency across the entire Kafka ecosystem, including producers, brokers, and consumers.

The talk covers real-world examples from gaming platforms and data systems which have enabled Otel for Kafka monitoring.