Bringing Observability to Your Stream Processing
A presentation at Oracle Openworld 2019 in in San Francisco, CA, USA by Viktor Gamov
1 Bringing Observability to Your Stream Processing September, 17th, 2019 @gamussa | #CodeOne | @ConfluentINc
2 @gamussa | #CodeOne | @ConfluentINc
3 I build highly scalable Hello World apps @gamussa | #CodeOne | @ConfluentINc
3 I build highly scalable Hello World apps @gamussa | #CodeOne | @kennybastani @ConfluentINc
Raffle, yeah 🚀
Raffle, yeah 🚀 Follow @gamussa 📸🖼🏋 Tag @gamussa With #CodeOne @confluentinc
Photo by Khai Sze Ong on Unsplash
Photo by Rainier Ridao on Unsplash \
Photo by Rohit Tandon on Unsplash
Photo by Theodore Moore on Unsplash
Photo by Cristian Grecu on Unsplash
10 @gamussa | #CodeOne | @ConfluentINc
How did we do monitoring? @gamussa | #CodeOne | @ConfluentINc
How did we do monitoring? ● Each system would have their very own monitoring system. @gamussa | #CodeOne | @ConfluentINc
How did we do monitoring? ● Each system would have their very own monitoring system. ● Developers would not worry about monitoring. This was supposed to be “IT stuff”. @gamussa | #CodeOne | @ConfluentINc
How did we do monitoring? ● Each system would have their very own monitoring system. ● Developers would not worry about monitoring. This was supposed to be “IT stuff”. ● Application was deemed OK if all systems were showing green. @gamussa | #CodeOne | @ConfluentINc
How did we do monitoring? ● Each system would have their very own monitoring system. ● Developers would not worry about monitoring. This was supposed to be “IT stuff”. ● Application was deemed OK if all systems were showing green. ● Different monitoring approaches were used throughout the entire IT stack portfolio, requiring many teams to be involved. @gamussa | #CodeOne | @ConfluentINc
WTF 🤬 @gamussa | @ #CodeOne | @ConfluentINc
WTF 🤬 @gamussa | @ #CodeOne | @ConfluentINc
WTF 🤬 troubleshooting. Days, weeks, months! @gamussa | @ #CodeOne | @ConfluentINc
System IS working! WTF 🤬 troubleshooting. Days, weeks, months! @gamussa | @ #CodeOne | @ConfluentINc
13 @gamussa | #CodeOne | @ConfluentINc
13 @gamussa | #CodeOne | @ConfluentINc
14 Intro to observability @gamussa | #CodeOne | @ConfluentINc
15 @gamussa | #CodeOne | @ConfluentINc
16 The Pillars of observability @gamussa | #CodeOne | @ConfluentINc
18 @gamussa | #CodeOne | @ConfluentINc 🤔
18 @gamussa | #CodeOne | @ConfluentINc 🤔
18 A raw sequence of events from a single instance of service @gamussa | #CodeOne | @ConfluentINc 🤔
19 @gamussa | #CodeOne | @ConfluentINc
19 @gamussa | #CodeOne | @ConfluentINc
19 Metrics: numerical measures aggregated in given point of time @gamussa | #CodeOne | @ConfluentINc
Pillars of observability @gamussa | #CodeOne | @ConfluentINc
Pillars of observability ● Distributed Tracing @gamussa | #CodeOne | @ConfluentINc
Pillars of observability ● Distributed Tracing ● detailed execution of the causality-related activities performed by a given transaction. It answers: @gamussa | #CodeOne | @ConfluentINc
Pillars of observability ● Distributed Tracing ● detailed execution of the causality-related activities performed by a given transaction. It answers: ○ Which services were involved? @gamussa | #CodeOne | @ConfluentINc
Pillars of observability ● Distributed Tracing ● detailed execution of the causality-related activities performed by a given transaction. It answers: ○ Which services were involved? ○ If it was slow, who caused that? @gamussa | #CodeOne | @ConfluentINc
Pillars of observability ● Distributed Tracing ● detailed execution of the causality-related activities performed by a given transaction. It answers: ○ Which services were involved? ○ If it was slow, who caused that? ○ If failed, who actually failed? @gamussa | #CodeOne | @ConfluentINc
Evolution of concurrency @gamussa | #CodeOne | @ConfluentINc
Evolution of concurrency ● No Concurrency at all ○ Ex: Apache HTTP Server @gamussa | #CodeOne | @ConfluentINc
Evolution of concurrency ● No Concurrency at all ○ Ex: Apache HTTP Server ● Basic Concurrency ○ Ex: Multi-threaded Applications @gamussa | #CodeOne | @ConfluentINc
Evolution of concurrency ● No Concurrency at all ○ Ex: Apache HTTP Server ● Basic Concurrency ○ Ex: Multi-threaded Applications ● Async Concurrency ○ Ex: Actor-based Programming @gamussa | #CodeOne | @ConfluentINc
Evolution of concurrency ● No Concurrency at all ○ Ex: Apache HTTP Server ● Basic Concurrency ○ Ex: Multi-threaded Applications ● Async Concurrency ○ Ex: Actor-based Programming ● Distributed Concurrency ○ Ex: μServices Architecture Style @gamussa | #CodeOne | @ConfluentINc
Distributed tracing today @gamussa | #CodeOne | @ConfluentINc
Distributed tracing today ● There are many distributed tracing technologies available. @gamussa | #CodeOne | @ConfluentINc
Distributed tracing today ● There are many distributed tracing technologies available. ● Standards are getting created to ensure a single programming model for each μService. @gamussa | #CodeOne | @ConfluentINc
Distributed tracing today ● There are many distributed tracing technologies available. ● Standards are getting created to ensure a single programming model for each μService. ● Deployment mechanisms such as Kubernetes are also taking care of that automatically. @gamussa | #CodeOne | @ConfluentINc
Distributed tracing today ● There are many distributed tracing technologies available. ● Standards are getting created to ensure a single programming model for each μService. ● Deployment mechanisms such as Kubernetes are also taking care of that automatically. ● Network proxies such as Service Meshes are also handling this. @gamussa | #CodeOne | @ConfluentINc
Distributed tracing today ● There are many distributed tracing technologies available. ● Standards are getting created to ensure a single programming model for each μService. ● Deployment mechanisms such as Kubernetes are also taking care of that automatically. ● Network proxies such as Service Meshes are also handling this. ● OSS and proprietary options. @gamussa | #CodeOne | @ConfluentINc
23 @gamussa | #CodeOne | @ConfluentINc
23 @gamussa | #CodeOne | @ConfluentINc
23 @gamussa | #CodeOne | @ConfluentINc
23 @gamussa | #CodeOne | @ConfluentINc
23 @gamussa | #CodeOne | @ConfluentINc
23 @gamussa | #CodeOne | @ConfluentINc
23 @gamussa | #CodeOne | @ConfluentINc
23 Trace that! @gamussa | #CodeOne | @ConfluentINc
24 @gamussa | #CodeOne | @ConfluentINc
Why trace apache kafka? @gamussa | #CodeOne | @ConfluentINc
Why trace apache kafka? ● Apache Kafka is becoming the defacto standard to handle data. @gamussa | #CodeOne | @ConfluentINc
Why trace apache kafka? ● Apache Kafka is becoming the defacto standard to handle data. ● Prediction? It will be the central nervous system of any company. @gamussa | #CodeOne | @ConfluentINc
Why trace apache kafka? ● Apache Kafka is becoming the defacto standard to handle data. ● Prediction? It will be the central nervous system of any company. ● μServices in general already use Kafka to exchange messages and keep their data stores in-sync. @gamussa | #CodeOne | @ConfluentINc
Why trace apache kafka? ● Apache Kafka is becoming the defacto standard to handle data. ● Prediction? It will be the central nervous system of any company. ● μServices in general already use Kafka to exchange messages and keep their data stores in-sync. ● With event streaming becoming even more popular, the Kafka adoption tend to grow even more. @gamussa | #CodeOne | @ConfluentINc
Why trace apache kafka? ● Apache Kafka is becoming the defacto standard to handle data. ● Prediction? It will be the central nervous system of any company. ● μServices in general already use Kafka to exchange messages and keep their data stores in-sync. ● With event streaming becoming even more popular, the Kafka adoption tend to grow even more. ● Because it is so freaking cool! @gamussa | #CodeOne | @ConfluentINc
26 Distributed tracing in kafka @gamussa | #CodeOne | @ConfluentINc
Opentracing java api https://github.com/opentracing-contrib/javakafka-client @gamussa | #CodeOne | @ConfluentINc
Opentracing java api ● Library written in Java to handle distributed tracing via OpenTracing compatible APIs. https://github.com/opentracing-contrib/javakafka-client @gamussa | #CodeOne | @ConfluentINc
Opentracing java api ● Library written in Java to handle distributed tracing via OpenTracing compatible APIs. ● Requires the creation of specific tracer using the distributed tracing technology API. https://github.com/opentracing-contrib/javakafka-client @gamussa | #CodeOne | @ConfluentINc
Opentracing java api ● Library written in Java to handle distributed tracing via OpenTracing compatible APIs. ● Requires the creation of specific tracer using the distributed tracing technology API. ● Uses the GlobalTracer utility class to handle the tracer throughout the JVM application. https://github.com/opentracing-contrib/javakafka-client @gamussa | #CodeOne | @ConfluentINc
Opentracing java api ● Library written in Java to handle distributed tracing via OpenTracing compatible APIs. ● Requires the creation of specific tracer using the distributed tracing technology API. ● Uses the GlobalTracer utility class to handle the tracer throughout the JVM application. https://github.com/opentracing-contrib/javakafka-client ● Supports: Apache Kafka Clients, Kafka Streams, and Spring Kafka. @gamussa | #CodeOne | @ConfluentINc
Support for bundled jvms https://github.com/riferrei/kafka-tracing-support @gamussa | #CodeOne | @ConfluentINc
Support for bundled jvms ● Library written in Java that does the automatic creation of the tracer. https://github.com/riferrei/kafka-tracing-support @gamussa | #CodeOne | @ConfluentINc
Support for bundled jvms ● Library written in Java that does the automatic creation of the tracer. ● Implements the tracing logic using the Kafka Interceptors API. https://github.com/riferrei/kafka-tracing-support @gamussa | #CodeOne | @ConfluentINc
Support for bundled jvms ● Library written in Java that does the automatic creation of the tracer. ● Implements the tracing logic using the Kafka Interceptors API. ● Allows different tracers to be used, by using the TracerResolver class. https://github.com/riferrei/kafka-tracing-support @gamussa | #CodeOne | @ConfluentINc
Support for bundled jvms ● Library written in Java that does the automatic creation of the tracer. ● Implements the tracing logic using the Kafka Interceptors API. ● Allows different tracers to be used, by using the TracerResolver class. ● Provides OOTB support for Jaeger. https://github.com/riferrei/kafka-tracing-support @gamussa | #CodeOne | @ConfluentINc
Support for bundled jvms ● Library written in Java that does the automatic creation of the tracer. ● Implements the tracing logic using the Kafka Interceptors API. ● Allows different tracers to be used, by using the TracerResolver class. ● Provides OOTB support for Jaeger. ● Allows multiple services in the JVM use their own tracer by specifying a configuration properties file. https://github.com/riferrei/kafka-tracing-support @gamussa | #CodeOne | @ConfluentINc
Support for bundled jvms ● Library written in Java that does the automatic creation of the tracer. ● Implements the tracing logic using the Kafka Interceptors API. ● Allows different tracers to be used, by using the TracerResolver class. ● Provides OOTB support for Jaeger. ● Allows multiple services in the JVM use their own tracer by specifying a configuration properties file. https://github.com/riferrei/kafka-tracing-support ○ @gamussa export INTERCEPTORS_CONFIG_FILE= | #CodeOne | @ConfluentINc
29 DEMO @gamussa | #CodeOne | @ConfluentINc
Gamov40 40% OFF*, duh! *Standard Priced Conference pass
Thanks! @gamussa viktor@confluent.io https://slackpass.io/confluentcommunity @gamussa | @ #CodeOne | @ConfluentINc
32
Using microservices—breaking down the domain problem into multiple reusable, highly scalable, and efficient services—is rapidly becoming the de facto standard way to develop applications. However, when it comes to monitoring these services, developers often have to leverage distributed tracing technologies such as OpenTracing to make the transaction flow observable. Although this is relatively easy to accomplish while dealing with request/response microservices, it might not be that easy to bring distributed tracing to event-driven/asynchronous architectures built on Apache Kafka. This session explains how distributed tracing can be implemented in pipelines built with Apache Kafka, via usage of OpenTracing and the Kafka Interceptors API.
Video
Resources
The following resources were mentioned during the presentation or are useful additional information.
Buzz and feedback
Here’s what was said about this presentation on social media.
