You can go your way: Kafka without Zookeeper
A presentation at JPoint in in Moscow, Russia by Viktor Gamov
Ka ka without Zookeeper You can go your way | @con luentinc f f @gamussa
@gamussa | #JPoint | @confluentinc
★gamov.dev/twitter ★gamov.dev/telegram ★gamov.dev/youtube @gamussa | #JPoint | @confluentinc
Agenda What we are going to talk about @gamussa | #JPoint | @confluentinc
Agenda What we are going to talk about • Brief history of Ka ka f @gamussa | #JPoint | @confluentinc
Agenda What we are going to talk about • Brief history of Ka @gamussa | #JPoint | @confluentinc
f ka Past and Future
replication @gamussa | #JPoint | @confluentinc
replication @gamussa | #JPoint | @confluentinc
@gamussa | #JPoint | @confluentinc
streams @gamussa | #JPoint | @confluentinc
@gamussa | #JPoint | @confluentinc
exactly once @gamussa | #JPoint | @confluentinc
exactly once @gamussa | #JPoint | @confluentinc
ksqlDB @gamussa | #JPoint | @confluentinc
KIP 500 @gamussa | #JPoint | @confluentinc
KIP 500 @gamussa | #JPoint | @confluentinc
KIP-455: Create an Administrative API for Replica Reassignment KIP-497: Add inter-broker API to alter ISR KIP-543: Expand Con igCommand’s non-ZK functionality KIP-555: Deprecate Direct Zookeeper access in Ka ka Administrative Tools KIP-589: Add API to update Replica state in Controller KIP-590: Redirect Zookeeper Mutation Protocols to The Controller KIP-595: A Raft Protocol for the Metadata Quorum KIP-631: The Quorum-based Ka ka Controller f f f @gamussa | #JPoint | @confluentinc
How Ka ka uses ZooKeeper f @gamussa | #JPoint | @confluentinc
Clients @gamussa | #JPoint | @confluentinc
@gamussa | #JPoint | @confluentinc
@gamussa | #JPoint | @confluentinc
producer consumer streams connect @gamussa | #JPoint | @confluentinc
admin client @gamussa | #JPoint | @confluentinc
admin client /bin tools @gamussa | #JPoint | @confluentinc
quota management replica reassignment admin client /bin tools @gamussa | #JPoint | @confluentinc
@gamussa | #JPoint | @confluentinc
@gamussa | #JPoint | @confluentinc
@gamussa | #JPoint | @confluentinc
Brokers @gamussa | #JPoint | @confluentinc
@gamussa | #JPoint | @confluentinc
@gamussa | #JPoint | @confluentinc
@gamussa | #JPoint | @confluentinc
@gamussa | #JPoint | @confluentinc
@gamussa | #JPoint | @confluentinc
ISR changes event notifications broker registration @gamussa | #JPoint | @confluentinc
@gamussa | #JPoint | @confluentinc
@gamussa | #JPoint | @confluentinc
Client Compatibility @gamussa | #JPoint | @confluentinc
@gamussa | #JPoint | @confluentinc
@gamussa | #JPoint | @confluentinc
@gamussa | #JPoint | @confluentinc
CreateTopic @gamussa | #JPoint | @confluentinc
CreateTopic @gamussa | #JPoint | @confluentinc
CreateTopic @gamussa | #JPoint | @confluentinc
@gamussa | #JPoint | @confluentinc
AlterConfig @gamussa | #JPoint | @confluentinc
AlterConfig @gamussa | #JPoint | @confluentinc
AlterConfig @gamussa | #JPoint | @confluentinc
35 @gamussa | #JPoint | @confluentinc
@gamussa | #JPoint | @confluentinc
AlterConfig @gamussa | #JPoint | @confluentinc
AlterConfig @gamussa | #JPoint | @confluentinc
AlterConfig @gamussa | #JPoint | @confluentinc
AlterConfig @gamussa | #JPoint | @confluentinc
KIP-500: Current State @gamussa | #JPoint | @confluentinc
@gamussa | #JPoint | @confluentinc
Metadata Quorum @gamussa | #JPoint | @confluentinc
@gamussa | #JPoint | @confluentinc
@gamussa | #JPoint | @confluentinc
Leader (Controller) @gamussa | #JPoint | @confluentinc
Follower Leader (Controller) Follower @gamussa | #JPoint | @confluentinc
Follower Leader (Controller) Follower Observers @gamussa | #JPoint | @confluentinc
@gamussa | #JPoint | @confluentinc
Data Nodes Metadata Nodes @gamussa | #JPoint | @confluentinc
Dedicated Deployment Data Nodes Metadata Nodes @gamussa | #JPoint | @confluentinc
Data Nodes Metadata Nodes @gamussa | #JPoint | @confluentinc
Data Nodes @gamussa | #JPoint | @confluentinc
Data/Metadata Nodes @gamussa | #JPoint | @confluentinc
Mixed Deployment Data/Metadata Nodes @gamussa | #JPoint | @confluentinc
Mixed Deployment @gamussa | #JPoint | @confluentinc
Mixed Deployment @gamussa | #JPoint | @confluentinc
Mixed Deployment Observer Promotion @gamussa | #JPoint | @confluentinc
Mixed Deployment Observer Promotion @gamussa | #JPoint | @confluentinc
@gamussa | #JPoint | @confluentinc
Single Node Deployment @gamussa | #JPoint | @confluentinc
Single Node Deployment @gamussa | #JPoint | @confluentinc
Single Node Deployment MM MM MM @gamussa | #JPoint | @confluentinc
KRaft: Ka kaesque Raft f @gamussa | #JPoint | @confluentinc
https://www.kafka-summit.org/sessions/a-kafkaesque-raft-protocol @gamussa | #JPoint | @confluentinc
Writes Fencing Ka ka Raft Single Leader Single Leader Monotonically increasing epoch Monotonically increasing term Log reconciliation Offset and epoch Term and index Push/Pull Pull Push Commit Semantics ISR Majority Leader Election From ISR through Zookeeper Majority Fault Tolerance F+1 2F+1 f @gamussa | #JPoint | @confluentinc
@gamussa | #JPoint | @confluentinc
STOP! Demo time! @gamussa | #JPoint | @confluentinc
Want to learn more about Ka ka? f @gamussa | #JPoint | @confluentinc
Learn Ka ka. Watch full version https://gamov.dev/developer Start building with Apache Ka ka at Con luent Developer. f f f f f Con luent Developer developer.con luent.io
As Always Have A Nice Day @gamussa | @confluentinc
Very soon, Apache Kafka® will no longer need ZooKeeper! With KIP-500, Kafka will include its built-in consensus layer, removing the ZooKeeper dependency altogether. The next significant milestone in this effort is coming in Apache Kafka 2.8.0. You will have early access to the new code and spin up a development version of Kafka without ZooKeeper’s opportunity to play with the Raft implementation as the distributed consensus algorithm.
Ultimately, removing ZooKeeper simplifies overall infrastructure design and operational workflows for your Kafka deployments. In this talk, Viktor will provide a brief overview of what limitations Zookeeper imposed on Kafka architecture, what will be changed, and the current status of Zookeeper removal.
Video
Buzz and feedback
Here’s what was said about this presentation on social media.
