Cloud-Native Streaming Platform: Apache Kafka Meets Kubernetes

Cloud-Native Streaming Platform: Apache Kafka Meets Kubernetes @gamussa #OracleCodeOne @confluentinc

#devkafkaops @gamussa #OracleCodeOne @ @confluentinc

@gamussa #OracleCodeOne @confluentinc

https://twitter.com/kelseyhightower/status/963413508300812295 @gamussa #OracleCodeOne @ @confluentinc

https://twitter.com/kelseyhightower/status/963414038603427840 @gamussa #OracleCodeOne @ @confluentinc

Don’t despair Not even over the fact that you don’t despair -= Franz Kafka =@gamussa #OracleCodeOne @ @confluentinc

Kafka Streaming Architecture Fundamentals

@gamussa #OracleCodeOne @ @confluentinc

@gamussa #OracleCodeOne @ @confluentinc

@

Kafka as Microservices Backbone HTTP etc Online Offline Billing Inventory @gamussa Fulfillment #OracleCodeOne @ Finance @confluentinc Fraud

High-level Architecture Application Application Application Native Client library Kafka Streams Load Balancer * REST Proxy Schema Registry Kafka Brokers @gamussa Kafka Connect Zookeeper Nodes #OracleCodeOne @ @confluentinc

Bare minimum Kafka Brokers @gamussa Zookeeper Nodes #OracleCodeOne @ @confluentinc

Shard data to get scalability Producer (1) Producer (2) Producer (3) Messages are sent to different partitions Cluster of machines Partitions live on different machines @gamussa #OracleCodeOne @confluentinc

Linearly Scalable Architecture Producers Single topic: - Many producers machines - Many consumer machines - Many Broker machines No Bottleneck!! Consumers @gamussa #OracleCodeOne @confluentinc

Replicate to get fault tolerance leader msg Machine A @gamussa Machine B replicate msg #OracleCodeOne @confluentinc

Replication provides resiliency A ‘replica’ takes over on machine failure @gamussa #OracleCodeOne @confluentinc

Kubernetes Fundamentals

Microservices Docker Kubernetes Monolith @gamussa #OracleCodeOne @ @confluentinc

https://twitter.com/sahrizv/status/1018184792611827712 @gamussa #OracleCodeOne @ @confluentinc

@gamussa #OracleCodeOne @ @confluentinc

Orchestration Compute Networking Storage Service Discovery @gamussa #OracleCodeOne @ @confluentinc

Kubernetes Schedules and allocates resources Networking between Pods Storage Service Discovery @gamussa #OracleCodeOne @ @confluentinc

Refresher - Kubernetes Architecture kubectl https://thenewstack.io/kubernetes-an-overview/ @gamussa #OracleCodeOne @ @confluentinc

Pod Basic Unit of Deployment in Kubernetes A collection of containers sharing: Namespace Network Volumes @gamussa #OracleCodeOne @ @confluentinc

Storage Persistent Volume (PV) & Persistent Volume Claim (PVC) Both PV and PVC are ‘resources’ @gamussa #OracleCodeOne @ @confluentinc

Storage Persistent Volume (PV) & Persistent Volume Claim (PVC) PV is a piece of storage that is provisioned dynamic or static of any individual pod that uses the PV @gamussa #OracleCodeOne @ @confluentinc

Storage Persistent Volume (PV) & Persistent Volume Claim (PVC) PVC is a request for storage by a User @gamussa #OracleCodeOne @ @confluentinc

Storage Persistent Volume (PV) & Persistent Volume Claim (PVC) PVCs consume PV @gamussa #OracleCodeOne @ @confluentinc

Stateful Workloads

StatefulSet Rely on Headless Service to provide network identity Headless Service Ideal for highly available stateful workloads @gamussa Pod-0 Pod-2 Containers Containers Containers Volumes Volumes Volumes #OracleCodeOne @ Pod-1 @confluentinc

StatefulSet Rely on Headless Service to provide network identity @gamussa Headless Service Pod-0 Pod-2 Containers Containers Containers Volumes Volumes Volumes #OracleCodeOne @ Pod-1 @confluentinc

StatefulSet Ideal for highly available stateful workloads @gamussa Headless Service Pod-0 Pod-2 Containers Containers Containers Volumes Volumes Volumes #OracleCodeOne @ Pod-1 @confluentinc

Workloads Deployment @gamussa #OracleCodeOne @confluentinc

Helm Charts @gamussa #OracleCodeOne @ @confluentinc

Helm Charts @gamussa #OracleCodeOne @ @confluentinc

Helm Charts @gamussa #OracleCodeOne @ @confluentinc

https://cnfl.io/helm_video @gamussa #OracleCodeOne @confluentinc

Basic components are not enough @gamussa #OracleCodeOne @confluentinc

Kafka deployment checklist PVC for Storage Uses ZK Headless Svc StatefulSet for 3-node zk PVC for Storage Optional Pod Anti-Affinity to spread the ZK ensemble across nodes StatefulSet for n-node Kafka Headless Service A group of NodePort Services for external traffic ConfigMap for Prometheus JMX exporter ConfigMap for Prometheus JMX exporter @gamussa #OracleCodeOne @ @confluentinc

Kafka Kubernetes journey: Step 1 https://cnfl.io/cp-helm @gamussa #OracleCodeOne @ @confluentinc

Let’s see some YAML!

Meet Kubernetes Operator @gamussa #OracleCodeOne @ @confluentinc

Kubernetes Operator Embedded with operational knowledge of both data software and Kubernetes Backup/restore Scale up/down Rebalance data Regular health checks @gamussa #OracleCodeOne @ @confluentinc

Controller Brain behind Kubernetes resources e.g. replication controller, namespace controller etc. @gamussa #OracleCodeOne @ @confluentinc

Custom Resource Definition(CRD) Extend existing Kubernetes API API StatefulSet ReplicaSet ... CRD Controller StatefulSet Controller ReplicaSet Controller ... Custom Controller ReplicaSet ... Custom Resource Instance @gamussa StatefulSet #OracleCodeOne @ @confluentinc

Custom Resource Definition(CRD) Usually works together Custom Controller API StatefulSet ReplicaSet ... CRD Controller StatefulSet Controller ReplicaSet Controller ... Custom Controller ReplicaSet ... Custom Resource Instance @gamussa StatefulSet #OracleCodeOne @ @confluentinc

Custom Resource Definition(CRD) Users can create and access Customer Resources with kubectl, just as they do for built-in API StatefulSet ReplicaSet ... CRD Controller StatefulSet Controller ReplicaSet Controller ... Custom Controller ReplicaSet ... Custom Resource Instance StatefulSet resources like pods. @gamussa #OracleCodeOne @ @confluentinc

Operator Deploy and Manage your production streaming platform with Confluent Operator. Automated Provisioning Platform Operations Resiliency Monitoring @gamussa #OracleCodeOne @ @confluentinc

Confluent Platform Reference Architecture Each Confluent Platform component has specific characteristics: Security (SSL certificates) DNS names and zones Application Application Application Native Client library Kafka Streams Load Balancer * Schema Registry REST Proxy Kafka Connect Host selection Fault tolerance Zookeeper Nodes Kafka Brokers Scaling @gamussa #OracleCodeOne @ @confluentinc

Confluent Operator: Automated Provisioning Load Balancer Kafka Pod Kafka Pod Kafka Pod Storage @gamussa #OracleCodeOne @ @confluentinc

Confluent Operator: Scale Horizontally Automate scaling: Spin up new broker pod(s) Distribute partitions to the new broker(s) Determine balancing plan Execute balancing plan Monitor resources @gamussa #OracleCodeOne @ @confluentinc

Confluent Operator: Rolling Upgrade Automated rolling upgrade with no downtime for Kafka. Stop broker Wait for leader election to complete Start broker with new version Wait for zero under-replicatedpartitions Repeat @gamussa #OracleCodeOne @ @confluentinc

Will it fly? Let’s see @gamussa #OracleCodeOne @confluentinc

Confluent Operator Automate provisioning Scale your Kafkas and CP clusters elastically Monitor SLAs through Confluent Control Center or Prometheus Operate at scale with enterprise support from Confluent @gamussa #OracleCodeOne @ @confluentinc

Advanced use cases vs. @gamussa #OracleCodeOne @confluentinc

Don’t despair! @gamussa #OracleCodeOne @ @confluentinc

Resources and Next Steps https://cnfl.io/helm_video https://cnfl.io/cp-helm https://cnfl.io/k8s https://slackpass.io/confluentcommunity #kubernetes @gamussa #OracleCodeOne @confluentinc

Thanks! @gamussa viktor@confluent.io We are hiring! https://cnfl.io/open_house @gamussa #OracleCodeOne @ @confluentinc