The Awakening of the New Event-Driven (Beast)

The Awakening of the New Event-Driven (beast) May, 2019 / London, UK @gamussa @gamussa | | #KafkaSummit #KafkaSummit || @ConfluentINc @ConfluentINc

2 @gamussa | #KafkaSummit | @ConfluentINc

3 I build highly scalable Hello World apps @gamussa | #KafkaSummit | @kennybastani @ConfluentINc

Raffle, yeah 🚀 Follow @gamussa 📸🖼🏋 @confluentinc Tag @gamussa With #KafkaSummit

5 @gamussa | #KafkaSummit | @ConfluentINc

https://gamov.dev/streamingratings https://gamov.dev/klyfft

Photo by Khai Sze Ong on Unsplash

Photo by Rainier Ridao on Unsplash \

Photo by Rohit Tandon on Unsplash

Photo by Theodore Moore on Unsplash

11 @gamussa | #KafkaSummit | @ConfluentINc

Refactoring to microservices

It used to be so simple @gamussa | #KafkaSummit | @ConfluentINc

14 @gamussa | #KafkaSummit | @ConfluentINc

15 Normal Applications (i.e., monoliths) @gamussa | #KafkaSummit | @ConfluentINc

16 Monoliths are hard to think about @gamussa | #KafkaSummit | @ConfluentINc

17 Monoliths are hard to change @gamussa | #KafkaSummit | @ConfluentINc

18 Reintegration? @gamussa | #KafkaSummit | @ConfluentINc

19 Reintegration @gamussa | #KafkaSummit | @ConfluentINc

Photo by Deva Darshan from Pexels

21 There are no good ways to integrate microservices @gamussa | #KafkaSummit | @ConfluentINc

22 There are no-good ways to integrate microservices @gamussa | #KafkaSummit | @ConfluentINc

23 Filesystem @gamussa | #KafkaSummit | @ConfluentINc

24 @gamussa | #KafkaSummit | @ConfluentINc

25 Database @gamussa | #KafkaSummit | @ConfluentINc

26 @gamussa | #KafkaSummit | @ConfluentINc

27 Integrating Microservices through the database ● “I have a database and I know how to use it.” ● Eventually causes services to co-mingle. ● Violation the «bounded context» ● Great to use inside a service boundary! ● Terrible for sharing data or negotiating change. @gamussa | #KafkaSummit | @ConfluentINc

28 RPC @gamussa | #KafkaSummit | @ConfluentINc

29 @gamussa | #KafkaSummit | @ConfluentINc

30 Integrating microservices via RPC ● Avoids problems of database integration ● Feels natural ● Aligns with the request/response paradigm ● Problem: cascading failures ● Question: how do you debug this system? 🤔 ● Answer: you build a log. @gamussa | #KafkaSummit | @ConfluentINc

31 Log4j type of log? 🤔 @gamussa | #KafkaSummit | @ConfluentINc

32 Events? @gamussa | #KafkaSummit | @ConfluentINc

33 What’s an event? @gamussa | #KafkaSummit | @ConfluentINc

34 A shared narrative describing the evolution of the business over time @gamussa | #KafkaSummit | @ConfluentINc

35 A combination of: Notification State transfer @gamussa | #KafkaSummit | @ConfluentINc

Also, events are immutable. @gamussa | #KafkaSummit | @ConfluentINc 36

37 Core abstraction ● DB - table ● Hadoop - file ● Kafka - ? @gamussa | #KafkaSummit | @ConfluentINc

38 LOG @gamussa | #KafkaSummit | @ConfluentINc

40 Stop! Demo time @gamussa | #KafkaSummit | @ConfluentINc

41 kLyfft - lyft-like eventdriven application ● Two Microservices ● Rider appears on Driver’s screen ● Rider hails a ride and the driver drives towards the rider ● Real time location events powered by Kafka @gamussa | #KafkaSummit | @ConfluentINc

42 A Richer Microservices Application @gamussa | #KafkaSummit | @ConfluentINc

43 @gamussa | #KafkaSummit | @ConfluentINc

44 Ratings Service operating on the event stream Browser ratings Ratings Service KStreams API used to count and sum ratings as they arrive Webserver Order raw-rating Requested ratings ratings counts ratings sums KAFKA @gamussa | #KafkaSummit | @ConfluentINc

45 Schema Management Browser ratings Ratings Service KStreams API used to count and sum ratings as they arrive Webserver Order raw-rating Requested ratings ratings counts ratings sums KAFKA @gamussa | #KafkaSummit | Schema Registry @ConfluentINc

46 Access legacy relation data Browser ratings Ratings Service KStreams API used to count and sum ratings as they arrive Webserver Order raw-rating Requested ratings Connect ratings counts ratings sums KAFKA @gamussa | #KafkaSummit | Movies Schema Registry @ConfluentINc

47 Materialize tables inside the app Browser ratings Ratings Service Movies Lookup table created inside the Rating Service Webserver Order raw-rating Requested ratings Connect ratings counts ratings sums KAFKA @gamussa | #KafkaSummit | Movies Schema Registry @ConfluentINc

48 Create a new table, persist it to the log Browser ratings rated-movies Ratings Service Writable table created for rated movies Movies Webserver Order raw-rating Requested ratings Connect ratings counts ratings sums KAFKA @gamussa | #KafkaSummit | Movies rated-movies Schema Registry @ConfluentINc

49 Create a new table, persist it to the log Browser ratings rated-movies Ratings Service Movies TRANSACTION Webserver Order raw-rating Requested ratings Connect ratings counts ratings sums KAFKA @gamussa | #KafkaSummit | Movies rated-movies Schema Registry @ConfluentINc

50 Hydrate a materialized view Browser ratings Ratings Service an embedded database CDC Webserver Order raw-rating Requested ratings Connect ratings counts ratings sums KAFKA @gamussa | #KafkaSummit | Movies rated-movies Schema Registry @ConfluentINc

51 @gamussa | #KafkaSummit | @ConfluentINc

52 What’s a database anyway? @gamussa | #KafkaSummit | @ConfluentINc

53 SQL @gamussa | #KafkaSummit | @ConfluentINc

54 Tabular Model @gamussa | #KafkaSummit | @ConfluentINc

55 Storage Engine @gamussa | #KafkaSummit | @ConfluentINc

56 Commit Log @gamussa | #KafkaSummit | @ConfluentINc

57 What are these things? Inventory INVENTORY ORDERS Orders Orders Service Order Created C is CQRS Fraud Service ORDERS Inventory Service (see previous figure) Order Validated Orders View Q in CQRS KAFKA @gamussa Order Details Service OV TOPIC GET Load Balancer POST | Order Validations #KafkaSummit | @ConfluentINc

58 You are not just writing microserviceS @gamussa | #KafkaSummit | @ConfluentINc

59 You are building an inside-out database. @gamussa | #KafkaSummit | @ConfluentINc

60 And that is a good thing… @gamussa | #KafkaSummit | @ConfluentINc

61