Curious Case of Streaming Platform on Kubernetes @gamussa #devoops @confluentinc
Slide 1
Slide 2
Slide 3
Slide 4
#devkafkaops @gamussa #devoops @ @confluentinc
Slide 5
https://twitter.com/kelseyhightower/status/963413508300812295 @gamussa #devoops @ @confluentinc
Slide 6
https://twitter.com/kelseyhightower/status/963414038603427840 @gamussa #devoops @ @confluentinc
Slide 7
@gamussa #devoops @ @confluentinc
Slide 8
Why streams? Tao of the streaming platform
Slide 9
@gamussa #devoops @ @confluentinc
Slide 10
@
Slide 11
@gamussa #devoops @ @confluentinc
Slide 12
@
Slide 13
Streaming Platform Architecture Application Application Application Native Client library Kafka Streams Load Balancer * REST Proxy Schema Registry Kafka Brokers @gamussa Kafka Connect Zookeeper Nodes #devoops @ @confluentinc
Slide 14
Bare minimum Kafka Brokers @gamussa Zookeeper Nodes #devoops @ @confluentinc
Slide 15
Shard data to get scalability Producer (1) Producer (2) Producer (3) Messages are sent to different partitions Cluster of machines Partitions live on different machines @gamussa #devoops @confluentinc
Slide 16
Linearly Scalable Architecture Producers Single topic: - Many producers machines - Many consumer machines - Many Broker machines No Bottleneck!! Consumers @gamussa #devoops @confluentinc
Slide 17
Replicate to get fault tolerance leader msg Machine A @gamussa Machine B replicate #devoops msg @confluentinc
Slide 18
Replication provides resiliency A ‘replica’ takes over on machine failure @gamussa #devoops @confluentinc
Slide 19
Microservices Docker Kubernetes Monolith @gamussa #devoops @ @confluentinc
Slide 20
https://twitter.com/sahrizv/status/1018184792611827712 @gamussa #devoops @ @confluentinc
Slide 21
Kafka as Microservices Backbone HTTP etc Online Offline Billing Inventory @gamussa Fulfillment Finance #devoops @confluentinc @ Fraud
Slide 22
Stateful Workloads Key Concepts
Slide 23
Refresher - Kubernetes Architecture kubectl https://thenewstack.io/kubernetes-an-overview/ @gamussa #devoops @ @confluentinc
Slide 24
Controller Brain behind Kubernetes resources e.g. replication controller, namespace controller etc. @gamussa #devoops @ @confluentinc
Slide 25
Pod Basic Unit of Deployment in Kubernetes A collection of containers sharing: Namespace Network Volumes @gamussa #devoops @ @confluentinc
Slide 26
Custom Resource Definition(CRD) Extend existing Kubernetes API API StatefulSet ReplicaSet ... CRD Controller StatefulSet Controller ReplicaSet Controller ... Custom Controller ReplicaSet ... Custom Resource Instance @gamussa #devoops @ StatefulSet @confluentinc
Slide 27
Custom Resource Definition(CRD) Usually works together Custom Controller (operator pattern) API StatefulSet ReplicaSet ... CRD Controller StatefulSet Controller ReplicaSet Controller ... Custom Controller ReplicaSet ... Custom Resource Instance @gamussa #devoops @ StatefulSet @confluentinc
Slide 28
Custom Resource Definition(CRD) Users can create and access Customer Resources with kubectl, just as they do for built-in API StatefulSet ReplicaSet ... CRD Controller StatefulSet Controller ReplicaSet Controller ... Custom Controller ReplicaSet ... Custom Resource Instance StatefulSet resources like pods. @gamussa #devoops @ @confluentinc
Slide 29
Storage Persistent Volume (PV) & Persistent Volume Claim (PVC) Both PV and PVC are ‘resources’ @gamussa #devoops @ @confluentinc
Slide 30
Storage Persistent Volume (PV) & Persistent Volume Claim (PVC) PV is a piece of storage that is provisioned dynamic or static of any individual pod that uses the PV @gamussa #devoops @ @confluentinc
Slide 31
Storage Persistent Volume (PV) & Persistent Volume Claim (PVC) PVC is a request for storage by a User @gamussa #devoops @ @confluentinc
Slide 32
Storage Persistent Volume (PV) & Persistent Volume Claim (PVC) PVCs consume PV @gamussa #devoops @ @confluentinc
Slide 33
StatefulSet Rely on Headless Service to provide network identity Headless Service Ideal for highly available stateful workloads @gamussa Pod-0 #devoops @ Pod-1 Pod-2 Containers Containers Containers Volumes Volumes Volumes @confluentinc
Slide 34
StatefulSet Rely on Headless Service to provide network identity @gamussa Headless Service Pod-0 #devoops @ Pod-1 Pod-2 Containers Containers Containers Volumes Volumes Volumes @confluentinc
Slide 35
StatefulSet Ideal for highly available stateful workloads @gamussa Headless Service Pod-0 #devoops @ Pod-1 Pod-2 Containers Containers Containers Volumes Volumes Volumes @confluentinc
Slide 36
Workloads Deployment Enter Helm #ihateyaml
Slide 37
Helm Charts @gamussa #devoops @ @confluentinc
Slide 38
Helm Charts @gamussa #devoops @ @confluentinc
Slide 39
Helm Charts @gamussa #devoops @ @confluentinc
Slide 40
Kubernetes Operator Custom Resource Definition (CRDs) Extending K8s Controller @gamussa #devoops @ @confluentinc
Slide 41
Kubernetes Operator Embedded with operational knowledge of both data software and Kubernetes Backup/restore Scale up/down Rebalance data Regular health checks @gamussa #devoops @ @confluentinc
Slide 42
Bare minimum Kafka Brokers @gamussa Zookeeper Nodes #devoops @ @confluentinc
Slide 43
ZK & Kafka install PVC for Storage Uses ZK Headless Svc StatefulSet for 3-node zk PVC for Storage Optional Pod Anti-Affinity to spread the ZK ensemble across nodes. StatefulSet for n-node Kafka Headless Service A group of NodePort Services for external traffic ConfigMap for Prometheus JMX exporter ConfigMap for Prometheus JMX exporter @gamussa #devoops @ @confluentinc
Slide 44
Kafka Kubernetes journey: Step 1 https://cnfl.io/cp-helm @gamussa #devoops @ @confluentinc
Slide 45
Let’s see some YAML! @gamussa #devoops @confluentinc
Slide 46
Step 2: Operator Deploy and Manage your production streaming platform with Confluent Operator. Automated Provisioning Platform Operations Resiliency Monitoring @gamussa #devoops @ @confluentinc
Slide 47
Confluent Platform Reference Architecture Each Confluent Platform component has specific characteristics: Security (SSL certificates) DNS names and zones Application Application Application Native Client library Kafka Streams Load Balancer * Schema Registry REST Proxy Kafka Connect Host selection Fault tolerance Kafka Brokers Scaling @gamussa #devoops @ @confluentinc Zookeeper Nodes
Slide 48
Confluent Operator: Automated Provisioning Load Balancer Kafka Pod Kafka Pod Kafka Pod Storage @gamussa #devoops @ @confluentinc
Slide 49
Confluent Operator: Scale Horizontally Automate scaling: Spin up new broker pod(s) Distribute partitions to the new broker(s) Determine balancing plan Execute balancing plan Monitor resources @gamussa #devoops @ @confluentinc
Slide 50
Confluent Operator: Rolling Upgrade Automated rolling upgrade with no downtime for Kafka. Stop broker Wait for leader election to complete Start broker with new version Wait for zero under-replicatedpartitions Repeat @gamussa #devoops @ @confluentinc
Slide 51
Will it fly? Let’s see @gamussa #devoops @confluentinc
Slide 52
Confluent Operator Automate provisioning Scale your Kafkas and CP clusters elastically Monitor SLAs through Confluent Control Center or Prometheus Operate at scale with enterprise support from Confluent @gamussa #devoops @ @confluentinc
Slide 53
Future
Slide 54
Vision of the future vs. @gamussa #devoops @confluentinc
Slide 55
Resources and Next Steps https://cnfl.io/cp-helm https://cnfl.io/k8s https://slackpass.io/confluentcommunity #kubernetes @gamussa #devoops @confluentinc
Slide 56
Thanks! @gamussa viktor@confluent.io We are hiring! https://www.confluent.io/careers/ @gamussa #devoops @ @confluentinc
