Streams Must Flow, Deploying fault-tolerant stream processing applications to Kubernetes
A presentation at NYC Kubernetes and Cloud Native Meetups in in New York, NY, USA by Viktor Gamov
Streams Must Flow: fault-tolerant stream processing apps on Kubernetes April, 2019 / New York, NY @gamussa | #cloudnativenyc | @ConfluentINc
2 Special thanks! @gwenshap @gamussa @MatthiasJSax | #cloudnativenyc | @ConfluentINc
3 Agenda Kafka Streams 101 How do Kafka Streams applications scale? Stateful Workloads Recommendations for Kafka Streams @gamussa | #cloudnativenyc | @ConfluentINc
4 Kafka Streams – 101 Your App @gamussa | #cloudnativenyc | @ConfluentINc Other Systems Kafka Connect Kafka Connect Other Systems Kafka Streams
5 Stock Trade Stats Example KStream<String, Trade> source = builder.stream(STOCK_TOPIC); KStream<Windowed<String>, TradeStats> stats = source .groupByKey() .windowedBy(TimeWindows.of(5000).advanceBy(1000)) .aggregate(TradeStats::new, (k, v, tradestats) -> tradestats.add(v), Materialized.<~>as(“trade-aggregates”) .withValueSerde(new TradeStatsSerde())) .toStream() .mapValues(TradeStats::computeAvgPrice); stats.to(STATS_OUT_TOPIC, Produced.keySerde(WindowedSerdes.timeWindowedSerdeFrom(String.class))); @gamussa | #cloudnativenyc | @ConfluentINc
6 Stock Trade Stats Example KStream<String, Trade> source = builder.stream(STOCK_TOPIC); KStream<Windowed<String>, TradeStats> stats = source .groupByKey() .windowedBy(TimeWindows.of(5000).advanceBy(1000)) .aggregate(TradeStats::new, (k, v, tradestats) -> tradestats.add(v), Materialized.<~>as(“trade-aggregates”) .withValueSerde(new TradeStatsSerde())) .toStream() .mapValues(TradeStats::computeAvgPrice); stats.to(STATS_OUT_TOPIC, Produced.keySerde(WindowedSerdes.timeWindowedSerdeFrom(String.class))); @gamussa | #cloudnativenyc | @ConfluentINc
7 Stock Trade Stats Example KStream<String, Trade> source = builder.stream(STOCK_TOPIC); KStream<Windowed<String>, TradeStats> stats = source .groupByKey() .windowedBy(TimeWindows.of(5000).advanceBy(1000)) .aggregate(TradeStats::new, (k, v, tradestats) -> tradestats.add(v), Materialized.<~>as(“trade-aggregates”) .withValueSerde(new TradeStatsSerde())) .toStream() .mapValues(TradeStats::computeAvgPrice); stats.to(STATS_OUT_TOPIC, Produced.keySerde(WindowedSerdes.timeWindowedSerdeFrom(String.class))); @gamussa | #cloudnativenyc | @ConfluentINc
8 Stock Trade Stats Example KStream<String, Trade> source = builder.stream(STOCK_TOPIC); KStream<Windowed<String>, TradeStats> stats = source .groupByKey() .windowedBy(TimeWindows.of(5000).advanceBy(1000)) .aggregate(TradeStats::new, (k, v, tradestats) -> tradestats.add(v), Materialized.<~>as(“trade-aggregates”) .withValueSerde(new TradeStatsSerde())) .toStream() .mapValues(TradeStats::computeAvgPrice); stats.to(STATS_OUT_TOPIC, Produced.keySerde(WindowedSerdes.timeWindowedSerdeFrom(String.class))); @gamussa | #cloudnativenyc | @ConfluentINc
9 Topologies builder.stream Source Node state stores source.groupByKey .windowedBy(…) .aggregate(…) Processor Node mapValues() Processor Node streams Sink Node to(…) @gamussa | #cloudnativenyc | @ConfluentINc Processor Topology
How Do Kafka Streams Application Scale? @gamussa | #cloudnativenyc | @ConfluentINc
11 Partitions, Tasks, and Consumer Groups input topic Task executes processor topology One consumer group: can be executed with 1 - 4 threads on 1 - 4 machines 4 input topic partitions => 4 tasks result topic @gamussa | #cloudnativenyc | @ConfluentINc
12 Scaling with State “no state” Trade Stats App Instance 1 @gamussa | #cloudnativenyc | @ConfluentINc
Scaling with State “no state” Trade Stats App Trade Stats App Instance 1 @gamussa Instance 2 | #cloudnativenyc | @ConfluentINc 13
14 Scaling with State “no state” Trade Stats App Trade Stats App Instance 1 @gamussa Instance 2 | #cloudnativenyc | Trade Stats App Instance 3 @ConfluentINc
15 Scaling and FaultTolerance Two Sides of Same Coin @gamussa | #cloudnativenyc | @ConfluentINc
16 Fault-Tolerance Trade Stats App Trade Stats App Instance 1 @gamussa | Instance 2 #cloudnativenyc | Trade Stats App Instance 3 @ConfluentINc
17 Fault-Tolerant State State Updates Input Topic Changelog Topic Result Topic @gamussa | #cloudnativenyc | @ConfluentINc
18 Migrate State Trade Stats App Instance 1 Trade Stats App Instance 2 restore Changelog Topic @gamussa | #cloudnativenyc | @ConfluentINc
19 Recovery Time • Changelog topics are log compacted • Size of changelog topic linear in size of state • Large state implies high recovery times @gamussa | #cloudnativenyc | @ConfluentINc
20 Stateful Workloads @gamussa | #cloudnativenyc | @ConfluentINc
21 StatefulSet ● Rely on Headless Headless Service Service to provide network identity Pod-0 ● Ideal for highly Pod-1 Pod-2 Containers Containers Containers Volumes Volumes Volumes available stateful workloads @gamussa | #cloudnativenyc | @ConfluentINc
Recommendations for Kafka Streams @gamussa | #cloudnativenyc | @ConfluentINc
23 Stock Stats App Stock Stats App Stock Stats App Kafka Streams Kafka Streams Kafka Streams Instance 1 Instance 2 Instance 3 @gamussa | #cloudnativenyc | @ConfluentINc
24 WordCount App WordCount App WordCount App Kafka Streams Kafka Streams Kafka Streams Instance 1 Instance 2 Instance 3 @gamussa | #cloudnativenyc | @ConfluentINc
25 StatefulSets are new and complicated. We don’t need them. @gamussa | #cloudnativenyc | @ConfluentINc
26 Recovering state takes time. Statelful is faster. @gamussa | #cloudnativenyc | @ConfluentINc
27 But I’ll want to scale-out and back anyway. @gamussa | #cloudnativenyc | @ConfluentINc
28 @gamussa | #cloudnativenyc | @ConfluentINc
29 I don’t really trust my storage admin anyway @gamussa | #cloudnativenyc | @ConfluentINc
30 Recommendations ● Keep changelog shards small ● If you trust your storage: Use StatefulSets ● Use anti-affinity when possible ● Use “parallel” pod management
31 🛑 Stop! Demo time! @gamussa | #cloudnativenyc | @ConfluentINc
32 Summary Kafka Streams has recoverable state, that gives streams apps easy elasticity and high availability Kubernetes makes it easy to scale applications It also has StatefulSets for applications with state @gamussa | #cloudnativenyc | @ConfluentINc
33 Summary Now you know how to deploy Kafka Streams on Kubernetes and take advantage on all the scalability and highavailability capabilities @gamussa | #cloudnativenyc | @ConfluentINc
34 But what about Kafka itself? @gamussa | #cloudnativenyc | @ConfluentINc
35 Confluent Operator Automate provisioning Scale your Kafkas and CP clusters elastically Monitoring with Confluent Control Center or Prometheus Operate at scale with enterprise support from Confluent @gamussa | #cloudnativenyc | @ConfluentINc
36 Resources and Next Steps https://cnfl.io/helm_video https://cnfl.io/cp-helm https://cnfl.io/k8s https://slackpass.io/confluentcommunity #kubernetes @gamussa | #cloudnativenyc | @ConfluentINc
Thanks! @gamussa viktor@confluent.io We are hiring! https://www.confluent.io/careers/ @gamussa | @ #cloudnativenyc | @ConfluentINc
38
All things change constantly, and we need to get on board with streams! Kafka Streams, Apache Kafka’s stream processing library, allows developers to build sophisticated stateful stream processing applications which you can deploy in an environment of your choice. In this talk, Viktor explains the essentials of dynamic scaling and state migration in Kafka Streams.
Resources
The following resources were mentioned during the presentation or are useful additional information.
Buzz and feedback
Here’s what was said about this presentation on social media.
