Cloud-Native Streaming Platform: Apache Kafka Meets Kubernetes
A presentation at Cloud-Native Data Day in December 2018 in Boston, MA, USA by Viktor Gamov
Abstract
When the time comes to choose a distributed streaming platform for real-time data pipelines, everyone knows the answer: Apache Kafka. And when it comes to deploying real-time stream processing applications at scale without having to integrate some different pieces of infrastructure yourself? The answer is Kubernetes. This session discusses best practices for running Apache Kafka and other components of a streaming platform such as Kafka Connect and Schema Registry as well as stream processing apps on Kubernetes. It covers the challenges and lessons learned from developing the Confluent Operator for Kubernetes as well as different custom deployments on various Kubernetes installations.