Kafka on Kubernetes: Does It Really Have To Be “The Hard Way”?
A presentation at Percona Live 2019 in in Austin, TX, USA by Viktor Gamov
Kafka on Kubernetes: Does it really have to be «The Hard Way»? May, 2019 / Texas, 2019 @gamussa @gamussa || #PerconaLive #PerconaLive || @ConfluentINc @ConfluentINc
Raffle, yeah 🚀 Follow @gamussa @confluentinc 📸🖼👬 Tag @gamussa With #PerconaLive
3 Evolution of #devkafkaops ansible/chef Docker Kubernetes Shell scripts @gamussa | #PerconaLive | @ConfluentINc
4 @gamussa | #PerconaLive | @ConfluentINc
5 🙋 @gamussa | #PerconaLive | @ConfluentINc
6 🙋 Who run stateless workloads in Kubernetes? Who thinks it’s a good idea? Who run stateful workloads in Kubernetes? Who thinks it’s a good idea? @gamussa | #PerconaLive | @ConfluentINc
7 kafkaesque world of Kafka on Kubernetes @gamussa | #PerconaLive | @ConfluentINc
8 #devkafkaops Well, it’s tricky © Translating an existing architecture to Kubernetes External access to brokers and other components Persistent Storage options on prem and clouds Security Configuration and Upgrades @gamussa | #PerconaLive | @ConfluentINc
9 But I just want to deploy kafka @gamussa | #PerconaLive | @ConfluentINc
10 Workloads Deployment @gamussa | #PerconaLive | @ConfluentINc
11 https://cnfl.io/helm_video @gamussa | #PerconaLive | @ConfluentINc
12 We will use StatefulSets with OrderedReady @gamussa | #PerconaLive | @ConfluentINc
13 We need SRE / Operator knowledge to manage the platform. You need Operator! @gamussa | #PerconaLive | @ConfluentINc
14 @gamussa | #PerconaLive | @ConfluentINc
15 Show me your Operator @gamussa | #PerconaLive | @ConfluentINc
16 KLyfft - Kafka app on Kubernetes 2 Web Apps powered by a microservice each Apps are a ride sharing driver and rider location tracking Rider hails a ride and the driver drives towards the rider Real time location events powered by Kafka on Kubernetes Rider @gamussa | #PerconaLive | @ConfluentINc Driver
17 Demo @gamussa | #PerconaLive | @ConfluentINc
18 DO KAFKA ON KUBERNETES DEMO AND EVERYONE LOOSES THEIR MIND @gamussa | #PerconaLive | @ConfluentINc
19 Confluent Operator - Automated Security Configuration SASL PLAIN and Mutual TLS Authentication Automate configuration of truststores and keystores with secret objects Automate configuration of Kafka and all Confluent Platform Components @gamussa | #PerconaLive | @ConfluentINc
20 Be like Justin! @gamussa | #PerconaLive | @ConfluentINc
Rolling Upgrade Kafka Broker Upgrades: 1. Stop the broker, upgrade Kafka 2. Wait for Partition Leader reassignment 3. Start the upgraded broker 4. Wait for zero underreplicated partitions 5. Upgrade the next broker @gamussa | #PerconaLive | @ConfluentINc 21
22 Want to see advanced deployment? vs. @gamussa | #PerconaLive | @ConfluentINc
23 http://gamov.dev/ksummit_nyc19_k8s @gamussa | #PerconaLive | @ConfluentINc
24 ● We are in private Preview GA Plans Release now ● 24 customers testing the Operator in Preview: ● Global customers ● Banks, Fin Tech, Retailers, Consumer Tech ● We are in the final stages of Preview and about to launch soon @gamussa | #PerconaLive | @ConfluentINc
Thanks! @gamussa viktor@confluent.io https://slackpass.io/confluentcommunity #kubernetes @gamussa | @ #PerconaLive | @ConfluentINc
26
When it comes to choosing a distributed streaming platform for real-time data pipelines, everyone knows the answer: Apache Kafka! And when it comes to deploying applications at scale without needing to integrate different pieces of infrastructure yourself, the answer nowadays is increasingly Kubernetes. However, with all great things, the devil is truly in the details. While Kubernetes does provide all the building blocks that are needed, a lot of thought is required to truly create an enterprise-grade Kafka platform that can be used in production. In this technical deep dive, Viktor will go through challenges and pitfalls of managing Kafka on Kubernetes as well as the goals and lessons learned from the development of the Confluent Operator for Kubernetes.
