The Awakening of the New Event-Driven (Beast)
A presentation at Kafka Summit London 2019 in in London, UK by Viktor Gamov
The Awakening of the New Event-Driven (beast) May, 2019 / London, UK @gamussa @gamussa | | #KafkaSummit #KafkaSummit || @ConfluentINc @ConfluentINc
2 @gamussa | #KafkaSummit | @ConfluentINc
3 I build highly scalable Hello World apps @gamussa | #KafkaSummit | @kennybastani @ConfluentINc
Raffle, yeah 🚀 Follow @gamussa 📸🖼🏋 @confluentinc Tag @gamussa With #KafkaSummit
5 @gamussa | #KafkaSummit | @ConfluentINc
https://gamov.dev/streamingratings https://gamov.dev/klyfft
Photo by Khai Sze Ong on Unsplash
Photo by Rainier Ridao on Unsplash \
Photo by Rohit Tandon on Unsplash
Photo by Theodore Moore on Unsplash
11 @gamussa | #KafkaSummit | @ConfluentINc
Refactoring to microservices
It used to be so simple @gamussa | #KafkaSummit | @ConfluentINc
14 @gamussa | #KafkaSummit | @ConfluentINc
15 Normal Applications (i.e., monoliths) @gamussa | #KafkaSummit | @ConfluentINc
16 Monoliths are hard to think about @gamussa | #KafkaSummit | @ConfluentINc
17 Monoliths are hard to change @gamussa | #KafkaSummit | @ConfluentINc
18 Reintegration? @gamussa | #KafkaSummit | @ConfluentINc
19 Reintegration @gamussa | #KafkaSummit | @ConfluentINc
Photo by Deva Darshan from Pexels
21 There are no good ways to integrate microservices @gamussa | #KafkaSummit | @ConfluentINc
22 There are no-good ways to integrate microservices @gamussa | #KafkaSummit | @ConfluentINc
23 Filesystem @gamussa | #KafkaSummit | @ConfluentINc
24 @gamussa | #KafkaSummit | @ConfluentINc
25 Database @gamussa | #KafkaSummit | @ConfluentINc
26 @gamussa | #KafkaSummit | @ConfluentINc
27 Integrating Microservices through the database ● “I have a database and I know how to use it.” ● Eventually causes services to co-mingle. ● Violation the «bounded context» ● Great to use inside a service boundary! ● Terrible for sharing data or negotiating change. @gamussa | #KafkaSummit | @ConfluentINc
28 RPC @gamussa | #KafkaSummit | @ConfluentINc
29 @gamussa | #KafkaSummit | @ConfluentINc
30 Integrating microservices via RPC ● Avoids problems of database integration ● Feels natural ● Aligns with the request/response paradigm ● Problem: cascading failures ● Question: how do you debug this system? 🤔 ● Answer: you build a log. @gamussa | #KafkaSummit | @ConfluentINc
31 Log4j type of log? 🤔 @gamussa | #KafkaSummit | @ConfluentINc
32 Events? @gamussa | #KafkaSummit | @ConfluentINc
33 What’s an event? @gamussa | #KafkaSummit | @ConfluentINc
34 A shared narrative describing the evolution of the business over time @gamussa | #KafkaSummit | @ConfluentINc
35 A combination of: Notification State transfer @gamussa | #KafkaSummit | @ConfluentINc
Also, events are immutable. @gamussa | #KafkaSummit | @ConfluentINc 36
37 Core abstraction ● DB - table ● Hadoop - file ● Kafka - ? @gamussa | #KafkaSummit | @ConfluentINc
38 LOG @gamussa | #KafkaSummit | @ConfluentINc
40 Stop! Demo time @gamussa | #KafkaSummit | @ConfluentINc
41 kLyfft - lyft-like eventdriven application ● Two Microservices ● Rider appears on Driver’s screen ● Rider hails a ride and the driver drives towards the rider ● Real time location events powered by Kafka @gamussa | #KafkaSummit | @ConfluentINc
42 A Richer Microservices Application @gamussa | #KafkaSummit | @ConfluentINc
43 @gamussa | #KafkaSummit | @ConfluentINc
44 Ratings Service operating on the event stream Browser ratings Ratings Service KStreams API used to count and sum ratings as they arrive Webserver Order raw-rating Requested ratings ratings counts ratings sums KAFKA @gamussa | #KafkaSummit | @ConfluentINc
45 Schema Management Browser ratings Ratings Service KStreams API used to count and sum ratings as they arrive Webserver Order raw-rating Requested ratings ratings counts ratings sums KAFKA @gamussa | #KafkaSummit | Schema Registry @ConfluentINc
46 Access legacy relation data Browser ratings Ratings Service KStreams API used to count and sum ratings as they arrive Webserver Order raw-rating Requested ratings Connect ratings counts ratings sums KAFKA @gamussa | #KafkaSummit | Movies Schema Registry @ConfluentINc
47 Materialize tables inside the app Browser ratings Ratings Service Movies Lookup table created inside the Rating Service Webserver Order raw-rating Requested ratings Connect ratings counts ratings sums KAFKA @gamussa | #KafkaSummit | Movies Schema Registry @ConfluentINc
48 Create a new table, persist it to the log Browser ratings rated-movies Ratings Service Writable table created for rated movies Movies Webserver Order raw-rating Requested ratings Connect ratings counts ratings sums KAFKA @gamussa | #KafkaSummit | Movies rated-movies Schema Registry @ConfluentINc
49 Create a new table, persist it to the log Browser ratings rated-movies Ratings Service Movies TRANSACTION Webserver Order raw-rating Requested ratings Connect ratings counts ratings sums KAFKA @gamussa | #KafkaSummit | Movies rated-movies Schema Registry @ConfluentINc
50 Hydrate a materialized view Browser ratings Ratings Service an embedded database CDC Webserver Order raw-rating Requested ratings Connect ratings counts ratings sums KAFKA @gamussa | #KafkaSummit | Movies rated-movies Schema Registry @ConfluentINc
51 @gamussa | #KafkaSummit | @ConfluentINc
52 What’s a database anyway? @gamussa | #KafkaSummit | @ConfluentINc
53 SQL @gamussa | #KafkaSummit | @ConfluentINc
54 Tabular Model @gamussa | #KafkaSummit | @ConfluentINc
55 Storage Engine @gamussa | #KafkaSummit | @ConfluentINc
56 Commit Log @gamussa | #KafkaSummit | @ConfluentINc
57 What are these things? Inventory INVENTORY ORDERS Orders Orders Service Order Created C is CQRS Fraud Service ORDERS Inventory Service (see previous figure) Order Validated Orders View Q in CQRS KAFKA @gamussa Order Details Service OV TOPIC GET Load Balancer POST | Order Validations #KafkaSummit | @ConfluentINc
58 You are not just writing microserviceS @gamussa | #KafkaSummit | @ConfluentINc
59 You are building an inside-out database. @gamussa | #KafkaSummit | @ConfluentINc
60 And that is a good thing… @gamussa | #KafkaSummit | @ConfluentINc
61
Developers have long employed message queues to decouple subsystems and provide an approximation of asynchronous processing. However, these queuing systems don’t adequately deliver on the promise of event-driven architectures and often lead to the usage anti-patterns. The events are carrying both notification and state. This allows for developers and data engineers to event-driven systems. Developers benefit from the asynchronous communication that events enable between services, and data engineers benefit from the integration capabilities. In this talk, Viktor will discuss the concepts of events, their relevance to software and data engineers, as well as its power for effectively unifying architectures. You learn how stream processing makes sense in microservices and data integration projects. The talk concludes with a hands-on demonstration of these concepts in practice, using modern toolchain – Kotlin, Spring Boot and Apache Kafka!
Video
Resources
The following resources were mentioned during the presentation or are useful additional information.
Buzz and feedback
Here’s what was said about this presentation on social media.
