Kafka on Kubernetes!

A presentation at DevNexus 2020 in February 2020 in Atlanta, GA, USA by Viktor Gamov

Slide 1

Kafka on Kubernetes: Does it have to be the hard way? february / Atlanta, GA 2019 @gamussa @gamussa || #devnexus #devnexus || @ConfluentINc @ConfluentINc

Slide 2

2 @gamussa | #devnexus | @ConfluentINc

Slide 3

3 I build highly scalable Hello World apps @gamussa | #devnexus | @ConfluentINc

Slide 4

Slide 5

5 Evolution of #devkafkaops Ansible Containers Cloud Scripts @gamussa | #devnexus | @ConfluentINc

Slide 6

Slide 7

7 @gamussa | #devnexus | @ConfluentINc

Slide 8

8 https://twitter.com/claesmogren/status/1108456688175079424 @gamussa | #devnexus | @ConfluentINc

Slide 9

9 https://twitter.com/claesmogren/status/1108456688175079424 https://twitter.com/QuinnyPig/status/1150927901782499330 @gamussa | #devnexus | @ConfluentINc

Slide 10

10 🙋 @gamussa | #devnexus | @ConfluentINc

Slide 11

11 Who run stateless apps in Kubernetes? 🙋 Who thinks it’s a good idea? Who run stateful apps in Kubernetes? Who thinks it’s a good idea? @gamussa | #devnexus | @ConfluentINc

Slide 12

12 #devkafkaops Well, it’s tricky © Translating an existing architecture to Kubernetes External access to brokers and other components Persistent Storage options on prem and clouds Security Configuration and Upgrades @gamussa | #devnexus | @ConfluentINc

Slide 13

13 kafkaesque world of Kafka on Kubernetes @gamussa | #devnexus | @ConfluentINc

Slide 14

14 Dude, you said «it doesn’t have to be hard way» @gamussa | #devnexus | @ConfluentINc

Slide 15

DO KAFKA ON KUBERNETES DEMO AND EVERYONE LOOSES THEIR MINDS @gamussa | #devnexus | @ConfluentINc 15

Slide 16

Tweet, yeah 🚀 Follow @gamussa 📸🖼🏋😂 Tag @gamussa With #devnexus @confluentinc

Slide 17

17 Let’s talk Kubernetes @gamussa | #devnexus | @ConfluentINc

Slide 18

18 @gamussa | #devnexus | @ConfluentINc

Slide 19

19 Kubernetes ●Schedules and allocates resources ●Networking between Pods ●Storage ●Service Discovery @gamussa | #devnexus | @ConfluentINc

Slide 20

20 Kubernetes Architecture kubectl https://thenewstack.io/kubernetes-an-overview/ @gamussa | #devnexus | @ConfluentINc

Slide 21

21 @gamussa | #devnexus | @ConfluentINc

Slide 22

22 Pod • Basic Unit of Deployment in Kubernetes • A collection of containers sharing: • Namespace • Network • Volumes @gamussa | #devnexus | @ConfluentINc

Slide 23

23 Storage • Persistent Volume (PV) & Persistent Volume Claim (PVC) • PV is a piece of storage that is provisioned dynamic or static of any individual pod that uses the PV @gamussa | #devnexus | @ConfluentINc

Slide 24

24 Kubernetes Storage classes Fast Slow Admin @gamussa | #devnexus | @ConfluentINc

Slide 25

25 Kubernetes Storage classes Fast Slow Admin I feel the need for speed © Claim Developer @gamussa | #devnexus | @ConfluentINc

Slide 26

26 Kubernetes Storage classes Fast Slow Admin Fast PV I feel the need for speed © Pod Claim Developer @gamussa | #devnexus | @ConfluentINc

Slide 27

27 Kubernetes Storage classes Fast Slow Admin Fast PV I feel the need for speed © Pod Claim Claim Developer @gamussa | #devnexus | @ConfluentINc

Slide 28

28 Provisioning storage in Kubernetes Kubernetes controller Storage System Kubernetes node @gamussa | #devnexus | @ConfluentINc

Slide 29

29 Provisioning storage in Kubernetes The controller requests storage system to attach volume Kubernetes controller Storage System Kubernetes node @gamussa | #devnexus | @ConfluentINc

Slide 30

30 Provisioning storage in Kubernetes The storage system attaches the volume Kubernetes controller Storage System Kubernetes node @gamussa | #devnexus | @ConfluentINc

Slide 31

31 Provisioning storage in Kubernetes Kubelet requests storage system to mount Volume Kubernetes controller Storage System Kubernetes node @gamussa | #devnexus | @ConfluentINc

Slide 32

32 Provisioning storage in Kubernetes Kubelet can now request container runtime to bind mount the volume into the requested path in the container. Kubernetes controller Storage System Kubernetes node /var/lib @gamussa | #devnexus | @ConfluentINc

Slide 33

33 But I just want to deploy kafka @gamussa | #devnexus | @ConfluentINc

Slide 34

Slide 35

35 Confluent Operator: Apache Kafka on Kubernetes made simple Confluent Platform Docker Images Run Apache Kafka and Confluent Confluent Operator Platform as a cloud-native application on Kubernetes to minimize operating complexity and Kubernetes AWS Azure GCP RH OpenShift Mesosphere Pivotal increase developer agility On-Premises @gamussa | #devnexus | @ConfluentINc Cloud

Slide 36

Custom resource controller

Slide 37

37 a custom Kubernetes controller PODS API Server StatefulSets PVs Controllers & Custom Controllers Custom Resources Master Node Worker Node | Applications run on Kubernetes ● Applications use objects like StatefulSets, Configmaps, PVs ● Custom Controllers create custom resources that provide unique application functionality: ConfigMaps Scheduler @gamussa ● Nodes and pods are where #devnexus | @ConfluentINc ○ Upgrades, elasticity, Kafka Operational Logic

Slide 38

38 @gamussa | #devnexus | @ConfluentINc

Slide 39

39 Workloads Deployment @gamussa | #devnexus | @ConfluentINc

Slide 40

40 Usual suspects Certified container images @gamussa Operator Helm Charts | #devnexus | @ConfluentINc

Slide 41

41 We just need to deploy Kafka on Kubernetes @gamussa | #devnexus | @ConfluentINc

Slide 42

42 We will use confluentinc/cp-helmcharts @gamussa | #devnexus | @ConfluentINc

Slide 43

43 Helm Charts @gamussa | #devnexus | @ConfluentINc

Slide 44

44 Helm Charts @gamussa | #devnexus | @ConfluentINc

Slide 45

45 Helm Charts @gamussa | #devnexus | @ConfluentINc

Slide 46

46 https://cnfl.io/helm_video @gamussa | #devnexus | @ConfluentINc

Slide 47

47 Helm Charts is just a GO Templates. How Charts help with rolling update? @gamussa | #devnexus | @ConfluentINc

Slide 48

48 We will use StatefulSets with OrderedReady @gamussa | #devnexus | @ConfluentINc

Slide 49

49 We need SRE / Operator knowledge to manage the platform. You need Operator! @gamussa | #devnexus | @ConfluentINc

Slide 50

50 @gamussa | #devnexus | @ConfluentINc

Slide 51

Rolling Upgrade Kafka Broker Upgrades: 1. Stop the broker, upgrade Kafka 2. Wait for Partition Leader reassignment 3. Start the upgraded broker 4. Wait for zero under-replicated partitions 5. Upgrade the next broker

Slide 52

52 Will it Scale Spin up new brokers, connect workers easily Manual Rebalance required in v1.0 Determine balancing plan Execute balancing plan Monitor Resources @gamussa | #devnexus | @ConfluentINc

Slide 53

53 One More Thing…

Slide 54

54 Usual suspects Certified container images @gamussa | #devnexus | @ConfluentINc

Slide 55

55 @gamussa | #devnexus | @ConfluentINc

Slide 56

Thanks! @gamussa viktor@confluent.io https://slackpass.io/confluentcommunity #kubernetes @gamussa | @ #devnexus | @ConfluentINc