Bringing Observability to Your Stream Processing

A presentation at Oracle Openworld 2019 in September 2019 in San Francisco, CA, USA by Viktor Gamov

Slide 1

Slide 1

1 Bringing Observability to Your Stream Processing September, 17th, 2019 @gamussa | #CodeOne | @ConfluentINc

Slide 2

Slide 2

2 @gamussa | #CodeOne | @ConfluentINc

Slide 3

Slide 3

3 I build highly scalable Hello World apps @gamussa | #CodeOne | @ConfluentINc

Slide 4

Slide 4

3 I build highly scalable Hello World apps @gamussa | #CodeOne | @kennybastani @ConfluentINc

Slide 5

Slide 5

Raffle, yeah 🚀

Slide 6

Slide 6

Raffle, yeah 🚀 Follow @gamussa 📸🖼🏋 Tag @gamussa With #CodeOne @confluentinc

Slide 7

Slide 7

Photo by Khai Sze Ong on Unsplash

Slide 8

Slide 8

Photo by Rainier Ridao on Unsplash \

Slide 9

Slide 9

Photo by Rohit Tandon on Unsplash

Slide 10

Slide 10

Photo by Theodore Moore on Unsplash

Slide 11

Slide 11

Photo by Cristian Grecu on Unsplash

Slide 12

Slide 12

10 @gamussa | #CodeOne | @ConfluentINc

Slide 13

Slide 13

How did we do monitoring? @gamussa | #CodeOne | @ConfluentINc

Slide 14

Slide 14

How did we do monitoring? ● Each system would have their very own monitoring system. @gamussa | #CodeOne | @ConfluentINc

Slide 15

Slide 15

How did we do monitoring? ● Each system would have their very own monitoring system. ● Developers would not worry about monitoring. This was supposed to be “IT stuff”. @gamussa | #CodeOne | @ConfluentINc

Slide 16

Slide 16

How did we do monitoring? ● Each system would have their very own monitoring system. ● Developers would not worry about monitoring. This was supposed to be “IT stuff”. ● Application was deemed OK if all systems were showing green. @gamussa | #CodeOne | @ConfluentINc

Slide 17

Slide 17

How did we do monitoring? ● Each system would have their very own monitoring system. ● Developers would not worry about monitoring. This was supposed to be “IT stuff”. ● Application was deemed OK if all systems were showing green. ● Different monitoring approaches were used throughout the entire IT stack portfolio, requiring many teams to be involved. @gamussa | #CodeOne | @ConfluentINc

Slide 18

Slide 18

WTF 🤬 @gamussa | @ #CodeOne | @ConfluentINc

Slide 19

Slide 19

WTF 🤬 @gamussa | @ #CodeOne | @ConfluentINc

Slide 20

Slide 20

WTF 🤬 troubleshooting. Days, weeks, months! @gamussa | @ #CodeOne | @ConfluentINc

Slide 21

Slide 21

System IS working! WTF 🤬 troubleshooting. Days, weeks, months! @gamussa | @ #CodeOne | @ConfluentINc

Slide 22

Slide 22

13 @gamussa | #CodeOne | @ConfluentINc

Slide 23

Slide 23

13 @gamussa | #CodeOne | @ConfluentINc

Slide 24

Slide 24

14 Intro to observability @gamussa | #CodeOne | @ConfluentINc

Slide 25

Slide 25

15 @gamussa | #CodeOne | @ConfluentINc

Slide 26

Slide 26

16 The Pillars of observability @gamussa | #CodeOne | @ConfluentINc

Slide 27

Slide 27

Slide 28

Slide 28

18 @gamussa | #CodeOne | @ConfluentINc 🤔

Slide 29

Slide 29

18 @gamussa | #CodeOne | @ConfluentINc 🤔

Slide 30

Slide 30

18 A raw sequence of events from a single instance of service @gamussa | #CodeOne | @ConfluentINc 🤔

Slide 31

Slide 31

19 @gamussa | #CodeOne | @ConfluentINc

Slide 32

Slide 32

19 @gamussa | #CodeOne | @ConfluentINc

Slide 33

Slide 33

19 Metrics: numerical measures aggregated in given point of time @gamussa | #CodeOne | @ConfluentINc

Slide 34

Slide 34

Pillars of observability @gamussa | #CodeOne | @ConfluentINc

Slide 35

Slide 35

Pillars of observability ● Distributed Tracing @gamussa | #CodeOne | @ConfluentINc

Slide 36

Slide 36

Pillars of observability ● Distributed Tracing ● detailed execution of the causality-related activities performed by a given transaction. It answers: @gamussa | #CodeOne | @ConfluentINc

Slide 37

Slide 37

Pillars of observability ● Distributed Tracing ● detailed execution of the causality-related activities performed by a given transaction. It answers: ○ Which services were involved? @gamussa | #CodeOne | @ConfluentINc

Slide 38

Slide 38

Pillars of observability ● Distributed Tracing ● detailed execution of the causality-related activities performed by a given transaction. It answers: ○ Which services were involved? ○ If it was slow, who caused that? @gamussa | #CodeOne | @ConfluentINc

Slide 39

Slide 39

Pillars of observability ● Distributed Tracing ● detailed execution of the causality-related activities performed by a given transaction. It answers: ○ Which services were involved? ○ If it was slow, who caused that? ○ If failed, who actually failed? @gamussa | #CodeOne | @ConfluentINc

Slide 40

Slide 40

Evolution of concurrency @gamussa | #CodeOne | @ConfluentINc

Slide 41

Slide 41

Evolution of concurrency ● No Concurrency at all ○ Ex: Apache HTTP Server @gamussa | #CodeOne | @ConfluentINc

Slide 42

Slide 42

Evolution of concurrency ● No Concurrency at all ○ Ex: Apache HTTP Server ● Basic Concurrency ○ Ex: Multi-threaded Applications @gamussa | #CodeOne | @ConfluentINc

Slide 43

Slide 43

Evolution of concurrency ● No Concurrency at all ○ Ex: Apache HTTP Server ● Basic Concurrency ○ Ex: Multi-threaded Applications ● Async Concurrency ○ Ex: Actor-based Programming @gamussa | #CodeOne | @ConfluentINc

Slide 44

Slide 44

Evolution of concurrency ● No Concurrency at all ○ Ex: Apache HTTP Server ● Basic Concurrency ○ Ex: Multi-threaded Applications ● Async Concurrency ○ Ex: Actor-based Programming ● Distributed Concurrency ○ Ex: μServices Architecture Style @gamussa | #CodeOne | @ConfluentINc

Slide 45

Slide 45

Distributed tracing today @gamussa | #CodeOne | @ConfluentINc

Slide 46

Slide 46

Distributed tracing today ● There are many distributed tracing technologies available. @gamussa | #CodeOne | @ConfluentINc

Slide 47

Slide 47

Distributed tracing today ● There are many distributed tracing technologies available. ● Standards are getting created to ensure a single programming model for each μService. @gamussa | #CodeOne | @ConfluentINc

Slide 48

Slide 48

Distributed tracing today ● There are many distributed tracing technologies available. ● Standards are getting created to ensure a single programming model for each μService. ● Deployment mechanisms such as Kubernetes are also taking care of that automatically. @gamussa | #CodeOne | @ConfluentINc

Slide 49

Slide 49

Distributed tracing today ● There are many distributed tracing technologies available. ● Standards are getting created to ensure a single programming model for each μService. ● Deployment mechanisms such as Kubernetes are also taking care of that automatically. ● Network proxies such as Service Meshes are also handling this. @gamussa | #CodeOne | @ConfluentINc

Slide 50

Slide 50

Distributed tracing today ● There are many distributed tracing technologies available. ● Standards are getting created to ensure a single programming model for each μService. ● Deployment mechanisms such as Kubernetes are also taking care of that automatically. ● Network proxies such as Service Meshes are also handling this. ● OSS and proprietary options. @gamussa | #CodeOne | @ConfluentINc

Slide 51

Slide 51

23 @gamussa | #CodeOne | @ConfluentINc

Slide 52

Slide 52

23 @gamussa | #CodeOne | @ConfluentINc

Slide 53

Slide 53

23 @gamussa | #CodeOne | @ConfluentINc

Slide 54

Slide 54

23 @gamussa | #CodeOne | @ConfluentINc

Slide 55

Slide 55

23 @gamussa | #CodeOne | @ConfluentINc

Slide 56

Slide 56

23 @gamussa | #CodeOne | @ConfluentINc

Slide 57

Slide 57

23 @gamussa | #CodeOne | @ConfluentINc

Slide 58

Slide 58

23 Trace that! @gamussa | #CodeOne | @ConfluentINc

Slide 59

Slide 59

24 @gamussa | #CodeOne | @ConfluentINc

Slide 60

Slide 60

Why trace apache kafka? @gamussa | #CodeOne | @ConfluentINc

Slide 61

Slide 61

Why trace apache kafka? ● Apache Kafka is becoming the defacto standard to handle data. @gamussa | #CodeOne | @ConfluentINc

Slide 62

Slide 62

Why trace apache kafka? ● Apache Kafka is becoming the defacto standard to handle data. ● Prediction? It will be the central nervous system of any company. @gamussa | #CodeOne | @ConfluentINc

Slide 63

Slide 63

Why trace apache kafka? ● Apache Kafka is becoming the defacto standard to handle data. ● Prediction? It will be the central nervous system of any company. ● μServices in general already use Kafka to exchange messages and keep their data stores in-sync. @gamussa | #CodeOne | @ConfluentINc

Slide 64

Slide 64

Why trace apache kafka? ● Apache Kafka is becoming the defacto standard to handle data. ● Prediction? It will be the central nervous system of any company. ● μServices in general already use Kafka to exchange messages and keep their data stores in-sync. ● With event streaming becoming even more popular, the Kafka adoption tend to grow even more. @gamussa | #CodeOne | @ConfluentINc

Slide 65

Slide 65

Why trace apache kafka? ● Apache Kafka is becoming the defacto standard to handle data. ● Prediction? It will be the central nervous system of any company. ● μServices in general already use Kafka to exchange messages and keep their data stores in-sync. ● With event streaming becoming even more popular, the Kafka adoption tend to grow even more. ● Because it is so freaking cool! @gamussa | #CodeOne | @ConfluentINc

Slide 66

Slide 66

26 Distributed tracing in kafka @gamussa | #CodeOne | @ConfluentINc

Slide 67

Slide 67

Opentracing java api https://github.com/opentracing-contrib/javakafka-client @gamussa | #CodeOne | @ConfluentINc

Slide 68

Slide 68

Opentracing java api ● Library written in Java to handle distributed tracing via OpenTracing compatible APIs. https://github.com/opentracing-contrib/javakafka-client @gamussa | #CodeOne | @ConfluentINc

Slide 69

Slide 69

Opentracing java api ● Library written in Java to handle distributed tracing via OpenTracing compatible APIs. ● Requires the creation of specific tracer using the distributed tracing technology API. https://github.com/opentracing-contrib/javakafka-client @gamussa | #CodeOne | @ConfluentINc

Slide 70

Slide 70

Opentracing java api ● Library written in Java to handle distributed tracing via OpenTracing compatible APIs. ● Requires the creation of specific tracer using the distributed tracing technology API. ● Uses the GlobalTracer utility class to handle the tracer throughout the JVM application. https://github.com/opentracing-contrib/javakafka-client @gamussa | #CodeOne | @ConfluentINc

Slide 71

Slide 71

Opentracing java api ● Library written in Java to handle distributed tracing via OpenTracing compatible APIs. ● Requires the creation of specific tracer using the distributed tracing technology API. ● Uses the GlobalTracer utility class to handle the tracer throughout the JVM application. https://github.com/opentracing-contrib/javakafka-client ● Supports: Apache Kafka Clients, Kafka Streams, and Spring Kafka. @gamussa | #CodeOne | @ConfluentINc

Slide 72

Slide 72

Support for bundled jvms https://github.com/riferrei/kafka-tracing-support @gamussa | #CodeOne | @ConfluentINc

Slide 73

Slide 73

Support for bundled jvms ● Library written in Java that does the automatic creation of the tracer. https://github.com/riferrei/kafka-tracing-support @gamussa | #CodeOne | @ConfluentINc

Slide 74

Slide 74

Support for bundled jvms ● Library written in Java that does the automatic creation of the tracer. ● Implements the tracing logic using the Kafka Interceptors API. https://github.com/riferrei/kafka-tracing-support @gamussa | #CodeOne | @ConfluentINc

Slide 75

Slide 75

Support for bundled jvms ● Library written in Java that does the automatic creation of the tracer. ● Implements the tracing logic using the Kafka Interceptors API. ● Allows different tracers to be used, by using the TracerResolver class. https://github.com/riferrei/kafka-tracing-support @gamussa | #CodeOne | @ConfluentINc

Slide 76

Slide 76

Support for bundled jvms ● Library written in Java that does the automatic creation of the tracer. ● Implements the tracing logic using the Kafka Interceptors API. ● Allows different tracers to be used, by using the TracerResolver class. ● Provides OOTB support for Jaeger. https://github.com/riferrei/kafka-tracing-support @gamussa | #CodeOne | @ConfluentINc

Slide 77

Slide 77

Support for bundled jvms ● Library written in Java that does the automatic creation of the tracer. ● Implements the tracing logic using the Kafka Interceptors API. ● Allows different tracers to be used, by using the TracerResolver class. ● Provides OOTB support for Jaeger. ● Allows multiple services in the JVM use their own tracer by specifying a configuration properties file. https://github.com/riferrei/kafka-tracing-support @gamussa | #CodeOne | @ConfluentINc

Slide 78

Slide 78

Support for bundled jvms ● Library written in Java that does the automatic creation of the tracer. ● Implements the tracing logic using the Kafka Interceptors API. ● Allows different tracers to be used, by using the TracerResolver class. ● Provides OOTB support for Jaeger. ● Allows multiple services in the JVM use their own tracer by specifying a configuration properties file. https://github.com/riferrei/kafka-tracing-support ○ @gamussa export INTERCEPTORS_CONFIG_FILE= | #CodeOne | @ConfluentINc

Slide 79

Slide 79

29 DEMO @gamussa | #CodeOne | @ConfluentINc

Slide 80

Slide 80

Gamov40 40% OFF*, duh! *Standard Priced Conference pass

Slide 81

Slide 81

Thanks! @gamussa viktor@confluent.io https://slackpass.io/confluentcommunity @gamussa | @ #CodeOne | @ConfluentINc

Slide 82

Slide 82

32