The Awakening of the New Event-Driven (Beast)

A presentation at Kafka Summit London 2019 in May 2019 in London, UK by Viktor Gamov

Slide 1

The Awakening of the New Event-Driven (beast) May, 2019 / London, UK @gamussa @gamussa | | #KafkaSummit #KafkaSummit || @ConfluentINc @ConfluentINc

Slide 2

2 @gamussa | #KafkaSummit | @ConfluentINc

Slide 3

3 I build highly scalable Hello World apps @gamussa | #KafkaSummit | @kennybastani @ConfluentINc

Slide 4

Raffle, yeah 🚀 Follow @gamussa 📸🖼🏋 @confluentinc Tag @gamussa With #KafkaSummit

Slide 5

5 @gamussa | #KafkaSummit | @ConfluentINc

Slide 6

https://gamov.dev/streamingratings https://gamov.dev/klyfft

Slide 7

Photo by Khai Sze Ong on Unsplash

Slide 8

Photo by Rainier Ridao on Unsplash \

Slide 9

Photo by Rohit Tandon on Unsplash

Slide 10

Photo by Theodore Moore on Unsplash

Slide 11

11 @gamussa | #KafkaSummit | @ConfluentINc

Slide 12

Refactoring to microservices

Slide 13

It used to be so simple @gamussa | #KafkaSummit | @ConfluentINc

Slide 14

14 @gamussa | #KafkaSummit | @ConfluentINc

Slide 15

15 Normal Applications (i.e., monoliths) @gamussa | #KafkaSummit | @ConfluentINc

Slide 16

16 Monoliths are hard to think about @gamussa | #KafkaSummit | @ConfluentINc

Slide 17

17 Monoliths are hard to change @gamussa | #KafkaSummit | @ConfluentINc

Slide 18

18 Reintegration? @gamussa | #KafkaSummit | @ConfluentINc

Slide 19

19 Reintegration @gamussa | #KafkaSummit | @ConfluentINc

Slide 20

Photo by Deva Darshan from Pexels

Slide 21

21 There are no good ways to integrate microservices @gamussa | #KafkaSummit | @ConfluentINc

Slide 22

22 There are no-good ways to integrate microservices @gamussa | #KafkaSummit | @ConfluentINc

Slide 23

23 Filesystem @gamussa | #KafkaSummit | @ConfluentINc

Slide 24

24 @gamussa | #KafkaSummit | @ConfluentINc

Slide 25

25 Database @gamussa | #KafkaSummit | @ConfluentINc

Slide 26

26 @gamussa | #KafkaSummit | @ConfluentINc

Slide 27

27 Integrating Microservices through the database ● “I have a database and I know how to use it.” ● Eventually causes services to co-mingle. ● Violation the «bounded context» ● Great to use inside a service boundary! ● Terrible for sharing data or negotiating change. @gamussa | #KafkaSummit | @ConfluentINc

Slide 28

28 RPC @gamussa | #KafkaSummit | @ConfluentINc

Slide 29

29 @gamussa | #KafkaSummit | @ConfluentINc

Slide 30

30 Integrating microservices via RPC ● Avoids problems of database integration ● Feels natural ● Aligns with the request/response paradigm ● Problem: cascading failures ● Question: how do you debug this system? 🤔 ● Answer: you build a log. @gamussa | #KafkaSummit | @ConfluentINc

Slide 31

31 Log4j type of log? 🤔 @gamussa | #KafkaSummit | @ConfluentINc

Slide 32

32 Events? @gamussa | #KafkaSummit | @ConfluentINc

Slide 33

33 What’s an event? @gamussa | #KafkaSummit | @ConfluentINc

Slide 34

34 A shared narrative describing the evolution of the business over time @gamussa | #KafkaSummit | @ConfluentINc

Slide 35

35 A combination of: Notification State transfer @gamussa | #KafkaSummit | @ConfluentINc

Slide 36

Also, events are immutable. @gamussa | #KafkaSummit | @ConfluentINc 36

Slide 37

37 Core abstraction ● DB - table ● Hadoop - file ● Kafka - ? @gamussa | #KafkaSummit | @ConfluentINc

Slide 38

38 LOG @gamussa | #KafkaSummit | @ConfluentINc

Slide 39

Slide 40

40 Stop! Demo time @gamussa | #KafkaSummit | @ConfluentINc

Slide 41

41 kLyfft - lyft-like eventdriven application ● Two Microservices ● Rider appears on Driver’s screen ● Rider hails a ride and the driver drives towards the rider ● Real time location events powered by Kafka @gamussa | #KafkaSummit | @ConfluentINc

Slide 42

42 A Richer Microservices Application @gamussa | #KafkaSummit | @ConfluentINc

Slide 43

43 @gamussa | #KafkaSummit | @ConfluentINc

Slide 44

44 Ratings Service operating on the event stream Browser ratings Ratings Service KStreams API used to count and sum ratings as they arrive Webserver Order raw-rating Requested ratings ratings counts ratings sums KAFKA @gamussa | #KafkaSummit | @ConfluentINc

Slide 45

45 Schema Management Browser ratings Ratings Service KStreams API used to count and sum ratings as they arrive Webserver Order raw-rating Requested ratings ratings counts ratings sums KAFKA @gamussa | #KafkaSummit | Schema Registry @ConfluentINc

Slide 46

46 Access legacy relation data Browser ratings Ratings Service KStreams API used to count and sum ratings as they arrive Webserver Order raw-rating Requested ratings Connect ratings counts ratings sums KAFKA @gamussa | #KafkaSummit | Movies Schema Registry @ConfluentINc

Slide 47

47 Materialize tables inside the app Browser ratings Ratings Service Movies Lookup table created inside the Rating Service Webserver Order raw-rating Requested ratings Connect ratings counts ratings sums KAFKA @gamussa | #KafkaSummit | Movies Schema Registry @ConfluentINc

Slide 48

48 Create a new table, persist it to the log Browser ratings rated-movies Ratings Service Writable table created for rated movies Movies Webserver Order raw-rating Requested ratings Connect ratings counts ratings sums KAFKA @gamussa | #KafkaSummit | Movies rated-movies Schema Registry @ConfluentINc

Slide 49

49 Create a new table, persist it to the log Browser ratings rated-movies Ratings Service Movies TRANSACTION Webserver Order raw-rating Requested ratings Connect ratings counts ratings sums KAFKA @gamussa | #KafkaSummit | Movies rated-movies Schema Registry @ConfluentINc

Slide 50

50 Hydrate a materialized view Browser ratings Ratings Service an embedded database CDC Webserver Order raw-rating Requested ratings Connect ratings counts ratings sums KAFKA @gamussa | #KafkaSummit | Movies rated-movies Schema Registry @ConfluentINc

Slide 51

51 @gamussa | #KafkaSummit | @ConfluentINc

Slide 52

52 What’s a database anyway? @gamussa | #KafkaSummit | @ConfluentINc

Slide 53

53 SQL @gamussa | #KafkaSummit | @ConfluentINc

Slide 54

54 Tabular Model @gamussa | #KafkaSummit | @ConfluentINc

Slide 55

55 Storage Engine @gamussa | #KafkaSummit | @ConfluentINc

Slide 56

56 Commit Log @gamussa | #KafkaSummit | @ConfluentINc

Slide 57

57 What are these things? Inventory INVENTORY ORDERS Orders Orders Service Order Created C is CQRS Fraud Service ORDERS Inventory Service (see previous figure) Order Validated Orders View Q in CQRS KAFKA @gamussa Order Details Service OV TOPIC GET Load Balancer POST | Order Validations #KafkaSummit | @ConfluentINc

Slide 58

58 You are not just writing microserviceS @gamussa | #KafkaSummit | @ConfluentINc

Slide 59

59 You are building an inside-out database. @gamussa | #KafkaSummit | @ConfluentINc

Slide 60

60 And that is a good thing… @gamussa | #KafkaSummit | @ConfluentINc