Kafka on Kubernetes: Just because you can, doesn’t mean you should!

A presentation at Docker NYC Meetup in July 2019 in New York, NY, USA by Viktor Gamov

Slide 1

Kafka on Kubernetes: Just because you can, doesn’t mean you should! July, 2019 / New York City, 2019 @gamussa @gamussa || #DockerNYC #DockerNYC || @ConfluentINc @ConfluentINc

Slide 2

2 @gamussa | #DockerNYC | @ConfluentINc

Slide 3

3 I build highly scalable Hello World apps @gamussa | #DockerNYC | @ConfluentINc

Slide 4

4 A company is built on DATA FLOWS but All we have is DATA STORES @gamussa | #DockerNYC | @ConfluentINc

Slide 5

5 Pre-streaming @gamussa | #DockerNYC | @ConfluentINc

Slide 6

6 @gamussa | #DockerNYC | @ConfluentINc

Slide 7

7 New World Streaming first • DB/DWH + Many more distributed data systems • Monolith -> Microservices • Batch -> Real-time @gamussa | #DockerNYC | @ConfluentINc

Slide 8

8 Evolution of #devkafkaops Ansible Docker Kubernetes Shell scripts @gamussa | #DockerNYC | @ConfluentINc

Slide 9

9 https://twitter.com/claesmogren/status/1108456688175079424 https://twitter.com/QuinnyPig/status/1150927901782499330 @gamussa | #DockerNYC | @ConfluentINc

Slide 10

10 🙋 @gamussa | #DockerNYC | @ConfluentINc

Slide 11

11 Who run stateless apps in Kubernetes? 🙋 Who thinks it’s a good idea? Who run stateful app in Kubernetes? Who thinks it’s a good idea? @gamussa | #DockerNYC | @ConfluentINc

Slide 12

12 kafkaesque world of Kafka on Kubernetes @gamussa | #DockerNYC | @ConfluentINc

Slide 13

13 #devkafkaops Well, it’s tricky © Translating an existing architecture to Kubernetes External access to brokers and other components Persistent Storage options on prem and clouds Security Configuration and Upgrades @gamussa | #DockerNYC | @ConfluentINc

Slide 14

14 But I just want to deploy kafka @gamussa | #DockerNYC | @ConfluentINc

Slide 15

15 Kubernetes Fundamentals @gamussa | #DockerNYC | @ConfluentINc

Slide 16

16 https://twitter.com/sahrizv/status/1018184792611827712 @gamussa | #DockerNYC | @ConfluentINc

Slide 17

17 @gamussa | #DockerNYC | @ConfluentINc

Slide 18

18 Kubernetes ●Schedules and allocates resources ●Networking between Pods ●Storage ●Service Discovery @gamussa | #DockerNYC | @ConfluentINc

Slide 19

19 https://twitter.com/claesmogren/status/1108456688175079424 @gamussa | #DockerNYC | @ConfluentINc

Slide 20

20 Kubernetes Architecture kubectl https://thenewstack.io/kubernetes-an-overview/ @gamussa | #DockerNYC | @ConfluentINc

Slide 21

21 Pod • Basic Unit of Deployment in Kubernetes • A collection of containers sharing: • Namespace • Network • Volumes @gamussa | #DockerNYC | @ConfluentINc

Slide 22

22 Storage • Persistent Volume (PV) & Persistent Volume Claim (PVC) • Both PV and PVC are ‘resources’ @gamussa | #DockerNYC | @ConfluentINc

Slide 23

23 Storage • Persistent Volume (PV) & Persistent Volume Claim (PVC) • PV is a piece of storage that is provisioned dynamic or static of any individual pod that uses the PV @gamussa | #DockerNYC | @ConfluentINc

Slide 24

24 Storage • Persistent Volume (PV) & Persistent Volume Claim (PVC) • PVC is a request for storage by a User @gamussa | #DockerNYC | @ConfluentINc

Slide 25

25 Storage • Persistent Volume (PV) & Persistent Volume Claim (PVC) • PVCs consume PV @gamussa | #DockerNYC | @ConfluentINc

Slide 26

26 Stateful Workloads @gamussa | #DockerNYC | @ConfluentINc

Slide 27

27 We just need to deploy Kafka on Kubernetes @gamussa | #DockerNYC | @ConfluentINc

Slide 28

28 We will use confluentinc/cp-helmcharts @gamussa | #DockerNYC | @ConfluentINc

Slide 29

29 Workloads Deployment @gamussa | #DockerNYC | @ConfluentINc

Slide 30

30 Helm Charts @gamussa | #DockerNYC | @ConfluentINc

Slide 31

31 Helm Charts @gamussa | #DockerNYC | @ConfluentINc

Slide 32

32 Helm Charts @gamussa | #DockerNYC | @ConfluentINc

Slide 33

33 https://cnfl.io/helm_video @gamussa | #DockerNYC | @ConfluentINc

Slide 34

34 Helm Charts is just a GO Templates. How Charts help with rolling update? @gamussa | #DockerNYC | @ConfluentINc

Slide 35

35 Be like Justin! @gamussa | #DockerNYC | @ConfluentINc

Slide 36

Rolling Upgrade Kafka Broker Upgrades: 1. Stop the broker, upgrade Kafka 2. Wait for Partition Leader reassignment 3. Start the upgraded broker 4. Wait for zero underreplicated partitions 5. Upgrade the next broker @gamussa | #DockerNYC | @ConfluentINc 36

Slide 37

37 We will use StatefulSets with OrderedReady @gamussa | #DockerNYC | @ConfluentINc

Slide 38

38 We need SRE / Operator knowledge to manage the platform. You need Operator! @gamussa | #DockerNYC | @ConfluentINc

Slide 39

39 @gamussa | #DockerNYC | @ConfluentINc

Slide 40

40 Show me your Operator @gamussa | #DockerNYC | @ConfluentINc

Slide 41

41 KLyfft - Kafka app on Kubernetes 2 Web Apps powered by a microservice each Apps are a ride sharing driver and rider location tracking Rider hails a ride and the driver drives towards the rider Real time location events powered by Kafka on Kubernetes Rider @gamussa | #DockerNYC | @ConfluentINc Driver

Slide 42

DO KAFKA ON KUBERNETES DEMO AND EVERYONE LOOSES THEIR MINDS @gamussa | #DockerNYC | @ConfluentINc 42

Slide 43

43 @gamussa | #DockerNYC | @ConfluentINc

Slide 44

44 Automated Security Configuration SASL PLAIN and Mutual TLS Authentication Automate configuration of truststores and keystores with secret objects Automate configuration of Kafka and all Confluent Platform Components @gamussa | #DockerNYC | @ConfluentINc

Slide 45

45 Want to see advanced deployment? vs. @gamussa | #DockerNYC | @ConfluentINc

Slide 46

46 http://gamov.dev/ksummit_nyc19_k8s @gamussa | #DockerNYC | @ConfluentINc

Slide 47

Gamov30 30% OFF*, duh! *Standard Priced Conference pass

Slide 48

Thanks! @gamussa viktor@confluent.io https://slackpass.io/confluentcommunity #kubernetes @gamussa | @ #DockerNYC | @ConfluentINc