Cloud-Native Streaming Platform: Apache Kafka Meets Kubernetes
A presentation at Oracle Code One 2018 in in San Francisco, CA, USA by Viktor Gamov
Cloud-Native Streaming Platform: Apache Kafka Meets Kubernetes @gamussa #OracleCodeOne @confluentinc
#devkafkaops @gamussa #OracleCodeOne @ @confluentinc
@gamussa #OracleCodeOne @confluentinc
https://twitter.com/kelseyhightower/status/963413508300812295 @gamussa #OracleCodeOne @ @confluentinc
https://twitter.com/kelseyhightower/status/963414038603427840 @gamussa #OracleCodeOne @ @confluentinc
Don’t despair Not even over the fact that you don’t despair -= Franz Kafka =@gamussa #OracleCodeOne @ @confluentinc
Kafka Streaming Architecture Fundamentals
@gamussa #OracleCodeOne @ @confluentinc
@gamussa #OracleCodeOne @ @confluentinc
@
Kafka as Microservices Backbone HTTP etc Online Offline Billing Inventory @gamussa Fulfillment #OracleCodeOne @ Finance @confluentinc Fraud
High-level Architecture Application Application Application Native Client library Kafka Streams Load Balancer * REST Proxy Schema Registry Kafka Brokers @gamussa Kafka Connect Zookeeper Nodes #OracleCodeOne @ @confluentinc
Bare minimum Kafka Brokers @gamussa Zookeeper Nodes #OracleCodeOne @ @confluentinc
Shard data to get scalability Producer (1) Producer (2) Producer (3) Messages are sent to different partitions Cluster of machines Partitions live on different machines @gamussa #OracleCodeOne @confluentinc
Linearly Scalable Architecture Producers Single topic: - Many producers machines - Many consumer machines - Many Broker machines No Bottleneck!! Consumers @gamussa #OracleCodeOne @confluentinc
Replicate to get fault tolerance leader msg Machine A @gamussa Machine B replicate msg #OracleCodeOne @confluentinc
Replication provides resiliency A ‘replica’ takes over on machine failure @gamussa #OracleCodeOne @confluentinc
Kubernetes Fundamentals
Microservices Docker Kubernetes Monolith @gamussa #OracleCodeOne @ @confluentinc
https://twitter.com/sahrizv/status/1018184792611827712 @gamussa #OracleCodeOne @ @confluentinc
@gamussa #OracleCodeOne @ @confluentinc
Orchestration Compute Networking Storage Service Discovery @gamussa #OracleCodeOne @ @confluentinc
Kubernetes Schedules and allocates resources Networking between Pods Storage Service Discovery @gamussa #OracleCodeOne @ @confluentinc
Refresher - Kubernetes Architecture kubectl https://thenewstack.io/kubernetes-an-overview/ @gamussa #OracleCodeOne @ @confluentinc
Pod Basic Unit of Deployment in Kubernetes A collection of containers sharing: Namespace Network Volumes @gamussa #OracleCodeOne @ @confluentinc
Storage Persistent Volume (PV) & Persistent Volume Claim (PVC) Both PV and PVC are ‘resources’ @gamussa #OracleCodeOne @ @confluentinc
Storage Persistent Volume (PV) & Persistent Volume Claim (PVC) PV is a piece of storage that is provisioned dynamic or static of any individual pod that uses the PV @gamussa #OracleCodeOne @ @confluentinc
Storage Persistent Volume (PV) & Persistent Volume Claim (PVC) PVC is a request for storage by a User @gamussa #OracleCodeOne @ @confluentinc
Storage Persistent Volume (PV) & Persistent Volume Claim (PVC) PVCs consume PV @gamussa #OracleCodeOne @ @confluentinc
Stateful Workloads
StatefulSet Rely on Headless Service to provide network identity Headless Service Ideal for highly available stateful workloads @gamussa Pod-0 Pod-2 Containers Containers Containers Volumes Volumes Volumes #OracleCodeOne @ Pod-1 @confluentinc
StatefulSet Rely on Headless Service to provide network identity @gamussa Headless Service Pod-0 Pod-2 Containers Containers Containers Volumes Volumes Volumes #OracleCodeOne @ Pod-1 @confluentinc
StatefulSet Ideal for highly available stateful workloads @gamussa Headless Service Pod-0 Pod-2 Containers Containers Containers Volumes Volumes Volumes #OracleCodeOne @ Pod-1 @confluentinc
Workloads Deployment @gamussa #OracleCodeOne @confluentinc
Helm Charts @gamussa #OracleCodeOne @ @confluentinc
Helm Charts @gamussa #OracleCodeOne @ @confluentinc
Helm Charts @gamussa #OracleCodeOne @ @confluentinc
https://cnfl.io/helm_video @gamussa #OracleCodeOne @confluentinc
Basic components are not enough @gamussa #OracleCodeOne @confluentinc
Kafka deployment checklist PVC for Storage Uses ZK Headless Svc StatefulSet for 3-node zk PVC for Storage Optional Pod Anti-Affinity to spread the ZK ensemble across nodes StatefulSet for n-node Kafka Headless Service A group of NodePort Services for external traffic ConfigMap for Prometheus JMX exporter ConfigMap for Prometheus JMX exporter @gamussa #OracleCodeOne @ @confluentinc
Kafka Kubernetes journey: Step 1 https://cnfl.io/cp-helm @gamussa #OracleCodeOne @ @confluentinc
Let’s see some YAML!
Meet Kubernetes Operator @gamussa #OracleCodeOne @ @confluentinc
Kubernetes Operator Embedded with operational knowledge of both data software and Kubernetes Backup/restore Scale up/down Rebalance data Regular health checks @gamussa #OracleCodeOne @ @confluentinc
Controller Brain behind Kubernetes resources e.g. replication controller, namespace controller etc. @gamussa #OracleCodeOne @ @confluentinc
Custom Resource Definition(CRD) Extend existing Kubernetes API API StatefulSet ReplicaSet ... CRD Controller StatefulSet Controller ReplicaSet Controller ... Custom Controller ReplicaSet ... Custom Resource Instance @gamussa StatefulSet #OracleCodeOne @ @confluentinc
Custom Resource Definition(CRD) Usually works together Custom Controller API StatefulSet ReplicaSet ... CRD Controller StatefulSet Controller ReplicaSet Controller ... Custom Controller ReplicaSet ... Custom Resource Instance @gamussa StatefulSet #OracleCodeOne @ @confluentinc
Custom Resource Definition(CRD) Users can create and access Customer Resources with kubectl, just as they do for built-in API StatefulSet ReplicaSet ... CRD Controller StatefulSet Controller ReplicaSet Controller ... Custom Controller ReplicaSet ... Custom Resource Instance StatefulSet resources like pods. @gamussa #OracleCodeOne @ @confluentinc
Operator Deploy and Manage your production streaming platform with Confluent Operator. Automated Provisioning Platform Operations Resiliency Monitoring @gamussa #OracleCodeOne @ @confluentinc
Confluent Platform Reference Architecture Each Confluent Platform component has specific characteristics: Security (SSL certificates) DNS names and zones Application Application Application Native Client library Kafka Streams Load Balancer * Schema Registry REST Proxy Kafka Connect Host selection Fault tolerance Zookeeper Nodes Kafka Brokers Scaling @gamussa #OracleCodeOne @ @confluentinc
Confluent Operator: Automated Provisioning Load Balancer Kafka Pod Kafka Pod Kafka Pod Storage @gamussa #OracleCodeOne @ @confluentinc
Confluent Operator: Scale Horizontally Automate scaling: Spin up new broker pod(s) Distribute partitions to the new broker(s) Determine balancing plan Execute balancing plan Monitor resources @gamussa #OracleCodeOne @ @confluentinc
Confluent Operator: Rolling Upgrade Automated rolling upgrade with no downtime for Kafka. Stop broker Wait for leader election to complete Start broker with new version Wait for zero under-replicatedpartitions Repeat @gamussa #OracleCodeOne @ @confluentinc
Will it fly? Let’s see @gamussa #OracleCodeOne @confluentinc
Confluent Operator Automate provisioning Scale your Kafkas and CP clusters elastically Monitor SLAs through Confluent Control Center or Prometheus Operate at scale with enterprise support from Confluent @gamussa #OracleCodeOne @ @confluentinc
Advanced use cases vs. @gamussa #OracleCodeOne @confluentinc
Don’t despair! @gamussa #OracleCodeOne @ @confluentinc
Resources and Next Steps https://cnfl.io/helm_video https://cnfl.io/cp-helm https://cnfl.io/k8s https://slackpass.io/confluentcommunity #kubernetes @gamussa #OracleCodeOne @confluentinc
Thanks! @gamussa viktor@confluent.io We are hiring! https://cnfl.io/open_house @gamussa #OracleCodeOne @ @confluentinc
When the time comes to choose a distributed streaming platform for real-time data pipelines, everyone knows the answer: Apache Kafka. And when it comes to deploying real-time stream processing applications at scale without having to integrate some different pieces of infrastructure yourself? The answer is Kubernetes. This session discusses best practices for running Apache Kafka and other components of a streaming platform such as Kafka Connect and Schema Registry as well as stream processing apps on Kubernetes. It covers the challenges and lessons learned from developing the Confluent Operator for Kubernetes as well as different custom deployments on various Kubernetes installations.
Video
Buzz and feedback
Here’s what was said about this presentation on social media.
