Cloud-Native Streaming Platform: Running Apache Kafka on PKS (Pivotal Container Service)

Abstract

When it comes time to choose a distributed streaming platform of choice for real-time data pipelines, everyone knows the answer: Apache Kafka. And when it comes to deploying real-time stream processing applications at scale without having to integrate some different pieces of infrastructure yourself? The answer is Kubernetes. In this talk, Viktor Gamov, Solutions Architect at Confluent and Prasad Radhakrishnan, Head of Platform Architecture for Data at Pivotal discuss the best practices on running Apache Kafka and other components of a streaming platform such as Kafka Connect, Schema Registry as well as stream processing apps on PKS (Pivotal Container Service). In this talk, the presenters will cover the challenges and lessons learned from development of Confluent Operator for Kubernetes as well as various custom deployments on PKS.