Kafka on Kubernetes: Does it really have to be “The Hard Way”?

Kafka on Kubernetes: Does it really have to be «The Hard Way»? April, 2019 / New York, 2019 @gamussa | #kafkasummit | @ConfluentINc

Raffle, yeah 🚀 Follow @gamussa @confluentinc 📸🖼👬 Tag @gamussa With #kafkasummit

3 Evolution of #devkafkaops ansible/chef Docker Kubernetes Shell scripts @gamussa | #kafkasummit | @ConfluentINc

4 @gamussa | #kafkasummit | @ConfluentINc

5 🙋 @gamussa | #kafkasummit | @ConfluentINc

6 🙋 Who run stateless workloads in Kubernetes? Who thinks it’s a good idea? Who run stateful workloads in Kubernetes? Who thinks it’s a good idea? @gamussa | #kafkasummit | @ConfluentINc

7 kafkaesque world of Kafka on Kubernetes @gamussa | #kafkasummit | @ConfluentINc

8 #devkafkaops Well, it’s tricky © Translating an existing architecture to Kubernetes External access to brokers and other components Persistent Storage options on prem and clouds Security Configuration and Upgrades @gamussa | #kafkasummit | @ConfluentINc

9 We just need to deploy Kafka on Kubernetes @gamussa | #kafkasummit | @ConfluentINc

10 We will use confluentinc/cp-helmcharts @gamussa | #kafkasummit | @ConfluentINc

11 Helm Charts is just a GO Templates. How Charts help with rolling restart? @gamussa | #kafkasummit | @ConfluentINc

12 We will use StatefulSets with OrderedReady @gamussa | #kafkasummit | @ConfluentINc

13 We need SRE / Operator knowledge to manage the platform. You need Operator! @gamussa | #kafkasummit | @ConfluentINc

14 @gamussa | #kafkasummit | @ConfluentINc

15 Show me your Operator @gamussa | #kafkasummit | @ConfluentINc

16 Demo @gamussa | #kafkasummit | @ConfluentINc

17 DO KAFKA ON KUBERNETES DEMO AND EVERYONE LOOSES THEIR MIND @gamussa | #kafkasummit | @ConfluentINc

18 What just happened? ZK and Kafka deployed Security with TLS is configured External access is configured Monitoring is enabled @gamussa | #kafkasummit | @ConfluentINc

19 Confluent Operator - Automated Security Configuration SASL PLAIN and Mutual TLS Authentication Automate configuration of truststores and keystores with secret objects Automate configuration of Kafka and all Confluent Platform Components @gamussa | #kafkasummit | @ConfluentINc

20 Confluent Operator - Scale Automate Scaling: Spin up new brokers, connect workers easily Distribute partitions to new brokers: Determine balancing plan Execute balancing plan Monitor Resources @gamussa | #kafkasummit | @ConfluentINc

21 Be like Justin! @gamussa | #kafkasummit | @ConfluentINc

Rolling Upgrade Kafka Broker Upgrades: 1. Stop the broker, upgrade Kafka 2. Wait for Partition Leader reassignment 3. Start the upgraded broker 4. Wait for zero underreplicated partitions 5. Upgrade the next broker @gamussa | #kafkasummit | @ConfluentINc 22

23 Will it fly? vs. @gamussa | #kafkasummit | @ConfluentINc

24 ● We are in private Preview GA Plans Release now ● 24 customers testing the Operator in Preview: ● Global customers ● Banks, Fin Tech, Retailers, Consumer Tech ● We are in the final stages of Preview and about to launch soon @gamussa | #kafkasummit | @ConfluentINc

Thanks! @gamussa viktor@confluent.io michael.ng@confluent.io https://slackpass.io/confluentcommunity #kubernetes @gamussa | @ #kafkasummit | @ConfluentINc

26