Event-Driven Architectures for Spring Developers
A presentation at SpringOne Platform by Pivotal in in Austin, TX, USA by Viktor Gamov
1 #SpringOne @s1p Event-Driven Architectures for Spring Developers Viktor Gamov Developer Advocate, Confluent @gamussa Gary Russell Engineer, Pivotal @gprussell
2 Why Event Streaming ? @gamussa | #SpringOne | @gprussell
3 The world is changing.
4 The New Business Reality Past Today Technology was a support function Technology is the business Innovation required for growth Innovation required for survival Running the business on yesterday’s data Yesterday’s data = failure. was “good enough” Modern, real-time data infrastructure is required. @gamussa | #SpringOne | @gprussell
5 Taxis become Software 2 min @gamussa | #SpringOne | @gprussell
6 Transportation Then Hardware product Up-front purchase Opaque No data Now Hardware, Software, and Global Internet Service The world is changing. On-demand Real-time visibility Built on a foundation of data
7 What enables this transformation? @gamussa | #SpringOne | @gprussell
8 Cloud Machine Learning Mobile Event Streaming Rethink Data Centers Rethink Decision Making Rethink User Experience Rethink Data @gamussa | #SpringOne | @gprussell
9 Do you see me? Or: Would you blindly cross the street with traffic information that is 5 minutes old?
10 Transportation Banking Retail ETA Fraud detection Real-time inventory Real-time sensor diagnostics Trading and risk systems Real-time POS reporting Driver-rider match Mobile applications / customer experience Personalization @gamussa | #SpringOne Entertainment Real-time recommendations Personalized news feed In-app purchases | @gprussell
11 This is a fundamental paradigm shift… Cloud Future of the datacenter Infrastructure as code Event Streaming Future of data Data as continuous stream of events @gamussa | #SpringOne | @gprussell
17 12 The Event Streaming Paradigm @gamussa | #SpringOne | @gprussell
13 Two Problems in Application Infrastructure What’s the state of the world? What’s happening in the world? Solution: Solution: Databases Messaging, RPC, ETL, etc. @gamussa | #SpringOne | @gprussell
14 ETL/Data Integration Batch Messaging High Throughput Expensive Time Consuming Fast (Low Latency) Difficult to Scale Durable No Persistence Persistent Data Loss Maintains Order No Replay @gamussa | #SpringOne | @gprussell
15 ETL/Data Integration Batch Messaging High Throughput Expensive Time Consuming Fast (Low Latency) Difficult to Scale Durable No Persistence Persistent Data Loss Maintains Order No Replay Transient Messages Stored records @gamussa | #SpringOne | @gprussell
16 @gamussa | #SpringOne | @gprussell
17 ETL/Data Integration Batch Event Streaming Paradigm High High Throughput Throughput Expensive Time Consuming Durable Durable Fast Fast (Low (Low Latency) Latency) Messaging Difficult to Scale No Persistence Replay Persistent Persistent Data Loss Maintains Maintains Order Order No Replay Stored records Transient Messages @gamussa | #SpringOne | @gprussell
18 Event Streaming Paradigm To rethink data as neither stored records nor transient messages, but instead as a continuously updating Stream of Events @gamussa | #SpringOne | @gprussell
19 24 An Event records the fact that something happened A good was sold An invoice was issued @gamussa | A payment was made #SpringOne | @gprussell A new customer registered
20 25 A Stream represents history as a sequence of Events @gamussa | #SpringOne | @gprussell
21 26 Events change the way we think Monolithic Approach Event-First Approach ● a database ● an event ● a stream ● a variable ● a singleton ● an RPC ● a ‘data’ flow ● a stream processor Orders Service Payments Service Orders Service Order Validation Service Customers Service request response event streams Email Notification Service DB @gamussa | #SpringOne | @gprussell Tax Service
22 27 An Event Streaming Platform gives you three key functionalities Publish & Subscribe to Events @gamussa Store Events | #SpringOne Process & Analyze Events | @gprussell
30 23 Event-Driven Apps, with Historical Context Real-Time Inventory Real-Time Fraud Detection Real-Time Customer 360 Machine Learning Models Real-Time Data Transformation Event Streaming Platform … ✓ Real-time but also persistent ✓ Elastic, scalable, reliable ✓ High throughput, low latency ✓ All apps and systems can now speak to each other for a complete view of data Universal Event Pipeline Data Stores Logs @gamussa 3rd Party Apps | #SpringOne Custom Apps/Microservices | @gprussell
24 Why Combine Real-time With Historical Context? Contextual Event-Driven App Event-Driven App (Location Tracking) (ETA) 2 min Only Real-time Events Messaging Queues and Event Streaming Platforms can do this Whereisismy mydriver? driver? Where @gamussa | #SpringOne Real-time combined with stored data Only Event Streaming Platforms can do this When When will will my my driver driver get get here? here? | @gprussell
33 25 How to Build Event Streaming Architectures With Kafka @gamussa | #SpringOne | @gprussell
26 is a distributed event streaming platform Publish & Subscribe to Events @gamussa Store Events | #SpringOne Process & Analyze Events | @gprussell
35 27 01 Stream your data in real-time as Events 02 03 Store your Event Streams @gamussa | #SpringOne Process & Analyze your Events Streams | @gprussell
36 28 From apps, microservices Use a Kafka producer client from your favorite language … and many more 01 https://gamov.dev/kafka-clients-demo Stream your data in real-time as Events From/to other systems Use Kafka Connect plus a Connector for your system … and many more confluent.io/hub @gamussa | #SpringOne @gamussa | | @gprussell #SpringOne | @gprussell
29 From apps, microservices: producer example network write @Service class Producer @Autowired constructor(private val kafkaTemplate: KafkaTemplate<String, User>) { @Value(“${topic.name}”) lateinit var TOPIC: String; internal fun sendMessage(user: User) { this.kafkaTemplate.send(this.TOPIC, user.getName(), user) println(String.format(“Produced user !-> %s”, user)) } } … and more
30 From/to other systems: Kafka Connect and more Tip: Great option to gradually move workloads to Kafka while keeping production running!
31 Kafka Connect ● Deployed standalone (development) or as a distributed cluster (production) ● Elastic service that works on bare-metal, VMs, containers, Kubernetes, … ● The individual ‘Connector’ determines delivery guarantees, e.g., exactly-once VM VM
32 Single Message Transforms for real-time ETL Ingress: modify an Event before storing Egress: modify an Event on its way out ● Obfuscate sensitive information, e.g. PII ● Route high-priority events to faster stores ● Add origin of event for lineage tracking ● Direct events to different Elasticsearch ● Remove unnecessary data fields ● … and more { user: ab123, gender: female, ip: 1.2.3.95 } indexes ● Cast data types to match destination ● … and more { user: ab123, ip: 1.2.3.XXX }
33 Confluent Hub Discover Connectors, SMTs, and converters Documentation, support, etc. Easy installation confluent.io/hub @gamussa | #SpringOne | @gprussell
34 43 Kafka Cluster Storage is Distributed 02 Scalable Store your Event Streams Reliable Durable Performant VM @gamussa | #SpringOne | @gprussell
35 Kafka scales from S to XXL Messages / sec Topics 1 10,000,000 5 250,000 25,000 500 Partitions Brokers 1,000,000 25,000 300 1,500 25 3
36 https://www.confluent.io/blog/publishing-apache-kafka-new-york-times/ Store your Events as long as you want Event Streaming Paradigm Kafka = Source of Truth, stores every articleHighly since Scalable 1851 Fast (Low Latency) Durable Persistent Maintains Order Normalized assets (images, articles, bylines, etc.) @gamussa Denormalized into “Content View” | #SpringOne | @gprussell
37 Achievement Data Unlocked: All Your Data Now Available as Streams of Events @gamussa | #SpringOne | @gprussell
49 38 With Streaming SQL KSQL 03 With apps, microservices Process & Analyze your Events Streams streams or … and more Kafka consumer clients With separate frameworks … and more @gamussa | #SpringOne | @gprussell
39 Stream Processing with KSQL Process event streams to create new, continuously updated streams or tables Stream 03 Stream 01 Streaming Query Query Query Stream 02 Table CREATE TABLE OrderTotals AS SELECT * FROM … EMIT CHANGES
Up Stream Processing with KSQL com ing 40 fea tur e Query tables in Kafka from other apps, similar to a relational database Result Table Other Applications (Java, Go, Python, etc.) can directly query tables request-response via network (KSQL REST API) SELECT * FROM OrderTotals WHERE region = ‘Europe’ (KL IP8 )
41 An example use case Creating an event-driven dashboard from a currencies database Maxwell is streaming change events Results are continuously updating Elasticsearch Currencies Aggregations are computed in real-time https://github.com/garyrussell/s1p-2019
42 @StreamListener fun processCurrency(input: KStream<String, Double>) { val groupByKey: KGroupedStream<String, Double> = input.groupByKey() val countKTable = groupByKey.count() val sumKTable = groupByKey.reduce { value1, value2 !-> value1 + value2 } val avgRate = sumKTable.join(countKTable) { sum, count !-> sum / count } avgRate.toStream().to(“avg-rates”) } @gamussa | #SpringOne | @gprussell
43 @StreamListener fun processCurrency(input: KStream<String, Double>) { val groupByKey: KGroupedStream<String, Double> = input.groupByKey() val countKTable = groupByKey.count() val sumKTable = groupByKey.reduce { value1, value2 !-> value1 + value2 } val avgRate = sumKTable.join(countKTable) { sum, count !-> sum / count } avgRate.toStream().to(“avg-rates”) } @gamussa | #SpringOne | @gprussell
44 @StreamListener fun processCurrency(input: KStream<String, Double>) { val groupByKey: KGroupedStream<String, Double> = input.groupByKey() val countKTable = groupByKey.count() val sumKTable = groupByKey.reduce { value1, value2 !-> value1 + value2 } val avgRate = sumKTable.join(countKTable) { sum, count !-> sum / count } avgRate.toStream().to(“avg-rates”) } @gamussa | #SpringOne | @gprussell
45 @StreamListener fun processCurrency(input: KStream<String, Double>) { val groupByKey: KGroupedStream<String, Double> = input.groupByKey() val countKTable = groupByKey.count() val sumKTable = groupByKey.reduce { value1, value2 !-> value1 + value2 } val avgRate = sumKTable.join(countKTable) { sum, count !-> sum / count } avgRate.toStream().to(“avg-rates”) } @gamussa | #SpringOne | @gprussell
46 Where your Kafka Streams apps live KStreams Application App instance 1 network read/write fun processCurrency(input: KStream<String, Double>) { val groupByKey: KGroupedStream<String, Double> = input.groupByKey() val countKTable = groupByKey.count() val sumKTable = groupByKey.reduce { value1, value2 !-> value1 + value2 } VM val avgRate = sumKTable.join(countKTable) { sum, count !-> sum / count } avgRate.toStream().to(“avg-rates”) }
47 Where your Kafka Streams apps live KStreams Application App instance 1 … network read/write App instance n VM Elastic & Scalable Fault-tolerant Exactly-once Kafka security Aggregations Windowing Streams & Tables
48 Stream Processing with Kafka Streams apps Process event streams to create new, continuously updated streams or tables Inventory Event-driven apps and services communicate through Kafka Reporting Orders Frontend New apps can easily be added by tapping into existing event streams Shipping
49 Stream Processing with Kafka Streams apps Query your application’s tables and state from other apps Reporting App Table Result App instance 1 … App instance n request-response via network (e.g. REST API) Other Applications (Java, Go, Python, etc.) can directly query tables
50 ● Spring For Apache Kafka 2.3 What’s New? @gamussa | #SpringOne | @gprussell Producers • Option for Producer per thread • AggregatingReplyingKafkaTemplate
51 ● Spring For Apache Kafka 2.3 What’s New? @gamussa | #SpringOne | @gprussell Consumers • Consumer RecordInterceptor • Relative Seeks in ConsumerSeekAware • Configurable delay between poll()s • Micrometer Timers • Backoff between redeliveries after delivery failures • RetryingDeserializer
52 ● Spring For Apache Kafka 2.3 What’s New? @gamussa | #SpringOne | @gprussell Streams • RecoveringDeserializationExc eptionHandler • Transformers • Add headers (SpEL) • Invoke Spring Integration flows
53 ● Spring For Apache Kafka 2.3 What’s New? @gamussa | #SpringOne | @gprussell Miscellaneous • Delegating Serializer/ Deserializer
54 is a distributed event streaming platform Publish & Subscribe to Events @gamussa Store Events | #SpringOne Process & Analyze Events | @gprussell
64 55 Where to go from here for more details on event-driven architectures with Kafka @gamussa | #SpringOne | @gprussell
56 THANK YOU @gamussa viktor@confluent.io @gprussell grussell@pivotal.io cnfl.io/meetups cnfl.io/blog cnfl.io/slack
Developers have long employed message queues to decouple subsystems and provide an approximation of asynchronous processing. However, these queuing systems don’t adequately deliver on the promise of event-driven architectures and often lead to contrived integration patterns.
The events carry both notification and state and form a powerful primitive on which to build systems for developers and data engineers alike. Developers benefit from the asynchronous communication that events enable between services, and data engineers benefit from the integration capabilities.
In this talk, Gary and Viktor discuss the concepts of events, their relevance to software and data engineers, and their powers for effectively unifying architectures.
You’ll learn how stream processing makes sense in microservices. The talk concludes with a hands-on demonstration of these concepts in practice using modern toolchain: Kotlin, Spring Boot, and Apache Kafka!
Video
Code
The following code examples from the presentation can be tried out live.
Resources
The following resources were mentioned during the presentation or are useful additional information.
Buzz and feedback
Here’s what was said about this presentation on social media.
