Curious Case of Streaming Platform on Kubernetes @gamussa #devoops @confluentinc

#devkafkaops @gamussa #devoops @ @confluentinc

https://twitter.com/kelseyhightower/status/963413508300812295 @gamussa #devoops @ @confluentinc

https://twitter.com/kelseyhightower/status/963414038603427840 @gamussa #devoops @ @confluentinc

@gamussa #devoops @ @confluentinc

Why streams? Tao of the streaming platform

@gamussa #devoops @ @confluentinc

@

@gamussa #devoops @ @confluentinc

@

Streaming Platform Architecture Application Application Application Native Client library Kafka Streams Load Balancer * REST Proxy Schema Registry Kafka Brokers @gamussa Kafka Connect Zookeeper Nodes #devoops @ @confluentinc

Bare minimum Kafka Brokers @gamussa Zookeeper Nodes #devoops @ @confluentinc

Shard data to get scalability Producer (1) Producer (2) Producer (3) Messages are sent to different partitions Cluster of machines Partitions live on different machines @gamussa #devoops @confluentinc

Linearly Scalable Architecture Producers Single topic: - Many producers machines - Many consumer machines - Many Broker machines No Bottleneck!! Consumers @gamussa #devoops @confluentinc

Replicate to get fault tolerance leader msg Machine A @gamussa Machine B replicate #devoops msg @confluentinc

Replication provides resiliency A ‘replica’ takes over on machine failure @gamussa #devoops @confluentinc

Microservices Docker Kubernetes Monolith @gamussa #devoops @ @confluentinc

https://twitter.com/sahrizv/status/1018184792611827712 @gamussa #devoops @ @confluentinc

Kafka as Microservices Backbone HTTP etc Online Offline Billing Inventory @gamussa Fulfillment Finance #devoops @confluentinc @ Fraud

Stateful Workloads Key Concepts

Refresher - Kubernetes Architecture kubectl https://thenewstack.io/kubernetes-an-overview/ @gamussa #devoops @ @confluentinc

Controller Brain behind Kubernetes resources e.g. replication controller, namespace controller etc. @gamussa #devoops @ @confluentinc

Pod Basic Unit of Deployment in Kubernetes A collection of containers sharing: Namespace Network Volumes @gamussa #devoops @ @confluentinc

Custom Resource Definition(CRD) Extend existing Kubernetes API API StatefulSet ReplicaSet ... CRD Controller StatefulSet Controller ReplicaSet Controller ... Custom Controller ReplicaSet ... Custom Resource Instance @gamussa #devoops @ StatefulSet @confluentinc

Custom Resource Definition(CRD) Usually works together Custom Controller (operator pattern) API StatefulSet ReplicaSet ... CRD Controller StatefulSet Controller ReplicaSet Controller ... Custom Controller ReplicaSet ... Custom Resource Instance @gamussa #devoops @ StatefulSet @confluentinc

Custom Resource Definition(CRD) Users can create and access Customer Resources with kubectl, just as they do for built-in API StatefulSet ReplicaSet ... CRD Controller StatefulSet Controller ReplicaSet Controller ... Custom Controller ReplicaSet ... Custom Resource Instance StatefulSet resources like pods. @gamussa #devoops @ @confluentinc

Storage Persistent Volume (PV) & Persistent Volume Claim (PVC) Both PV and PVC are ‘resources’ @gamussa #devoops @ @confluentinc

Storage Persistent Volume (PV) & Persistent Volume Claim (PVC) PV is a piece of storage that is provisioned dynamic or static of any individual pod that uses the PV @gamussa #devoops @ @confluentinc

Storage Persistent Volume (PV) & Persistent Volume Claim (PVC) PVC is a request for storage by a User @gamussa #devoops @ @confluentinc

Storage Persistent Volume (PV) & Persistent Volume Claim (PVC) PVCs consume PV @gamussa #devoops @ @confluentinc

StatefulSet Rely on Headless Service to provide network identity Headless Service Ideal for highly available stateful workloads @gamussa Pod-0 #devoops @ Pod-1 Pod-2 Containers Containers Containers Volumes Volumes Volumes @confluentinc

StatefulSet Rely on Headless Service to provide network identity @gamussa Headless Service Pod-0 #devoops @ Pod-1 Pod-2 Containers Containers Containers Volumes Volumes Volumes @confluentinc

StatefulSet Ideal for highly available stateful workloads @gamussa Headless Service Pod-0 #devoops @ Pod-1 Pod-2 Containers Containers Containers Volumes Volumes Volumes @confluentinc

Workloads Deployment Enter Helm #ihateyaml

Helm Charts @gamussa #devoops @ @confluentinc

Helm Charts @gamussa #devoops @ @confluentinc

Helm Charts @gamussa #devoops @ @confluentinc

Kubernetes Operator Custom Resource Definition (CRDs) Extending K8s Controller @gamussa #devoops @ @confluentinc

Kubernetes Operator Embedded with operational knowledge of both data software and Kubernetes Backup/restore Scale up/down Rebalance data Regular health checks @gamussa #devoops @ @confluentinc

Bare minimum Kafka Brokers @gamussa Zookeeper Nodes #devoops @ @confluentinc

ZK & Kafka install PVC for Storage Uses ZK Headless Svc StatefulSet for 3-node zk PVC for Storage Optional Pod Anti-Affinity to spread the ZK ensemble across nodes. StatefulSet for n-node Kafka Headless Service A group of NodePort Services for external traffic ConfigMap for Prometheus JMX exporter ConfigMap for Prometheus JMX exporter @gamussa #devoops @ @confluentinc

Kafka Kubernetes journey: Step 1 https://cnfl.io/cp-helm @gamussa #devoops @ @confluentinc

Let’s see some YAML! @gamussa #devoops @confluentinc

Step 2: Operator Deploy and Manage your production streaming platform with Confluent Operator. Automated Provisioning Platform Operations Resiliency Monitoring @gamussa #devoops @ @confluentinc

Confluent Platform Reference Architecture Each Confluent Platform component has specific characteristics: Security (SSL certificates) DNS names and zones Application Application Application Native Client library Kafka Streams Load Balancer * Schema Registry REST Proxy Kafka Connect Host selection Fault tolerance Kafka Brokers Scaling @gamussa #devoops @ @confluentinc Zookeeper Nodes

Confluent Operator: Automated Provisioning Load Balancer Kafka Pod Kafka Pod Kafka Pod Storage @gamussa #devoops @ @confluentinc

Confluent Operator: Scale Horizontally Automate scaling: Spin up new broker pod(s) Distribute partitions to the new broker(s) Determine balancing plan Execute balancing plan Monitor resources @gamussa #devoops @ @confluentinc

Confluent Operator: Rolling Upgrade Automated rolling upgrade with no downtime for Kafka. Stop broker Wait for leader election to complete Start broker with new version Wait for zero under-replicatedpartitions Repeat @gamussa #devoops @ @confluentinc

Will it fly? Let’s see @gamussa #devoops @confluentinc

Confluent Operator Automate provisioning Scale your Kafkas and CP clusters elastically Monitor SLAs through Confluent Control Center or Prometheus Operate at scale with enterprise support from Confluent @gamussa #devoops @ @confluentinc

Future

Vision of the future vs. @gamussa #devoops @confluentinc

Resources and Next Steps https://cnfl.io/cp-helm https://cnfl.io/k8s https://slackpass.io/confluentcommunity #kubernetes @gamussa #devoops @confluentinc

Thanks! @gamussa viktor@confluent.io We are hiring! https://www.confluent.io/careers/ @gamussa #devoops @ @confluentinc