Commit Logs in n Age of Microservices

A presentation at Seattle Event Driven Meetup in March 2019 in Seattle, WA, USA by Viktor Gamov

Slide 1

Slide 1

!//Join the two streams and the table then send an email for each orders.join(payments, EmailTuple!::new, !//Join Orders and Payments streams JoinWindows.of(Duration.ofMinutes(1)), serdes) !//Next join to the GKTable of Customers Commit Logs (key1, tuple) !-> tuple.order.getCustomerId(), In an ofwe Microservices !// note Age how, because use a GKtable, we can join on any .join(customers, attribute of the Customer. EmailTuple!::setCustomer) March, 2019 / Seattle, WA @gamussa !//Now for each tuple send an email. | #SeattleEventDriven | @ConfluentINc emailTuple) .peek((key,

Slide 2

Slide 2

2 @gamussa | #SeattleEventDriven | @ConfluentINc

Slide 3

Slide 3

https://gamov.dev/seattle-microservices

Slide 4

Slide 4

Raffle, yeah 🚀 Follow @gamussa 📸🖼🏋 @confluentinc Tag @gamussa With #seattleeventdriven

Slide 5

Slide 5

5 Normal Applications (i.e., monoliths) @gamussa | #SeattleEventDriven | @ConfluentINc

Slide 6

Slide 6

6 Monoliths are hard to think about @gamussa | #SeattleEventDriven | @ConfluentINc

Slide 7

Slide 7

7 Monoliths are hard to change @gamussa | #SeattleEventDriven | @ConfluentINc

Slide 8

Slide 8

8 Reintegration? @gamussa | #SeattleEventDriven | @ConfluentINc

Slide 9

Slide 9

9 Reintegration @gamussa | #SeattleEventDriven | @ConfluentINc

Slide 10

Slide 10

10 There are no good ways to integrate microservices @gamussa | #SeattleEventDriven | @ConfluentINc

Slide 11

Slide 11

11 There are no-good ways to integrate microservices @gamussa | #SeattleEventDriven | @ConfluentINc

Slide 12

Slide 12

12 Filesystem @gamussa | #SeattleEventDriven | @ConfluentINc

Slide 13

Slide 13

13 @gamussa | #SeattleEventDriven | @ConfluentINc

Slide 14

Slide 14

14 Database @gamussa | #SeattleEventDriven | @ConfluentINc

Slide 15

Slide 15

15 Integrating Microservices through the database ● “I have a database and I know how to use it.” ● Eventually causes services to co-mingle. ● Violation the «bounded context» ● Great to use inside a service boundary! ● Terrible for sharing data or negotiating change. @gamussa | #SeattleEventDriven | @ConfluentINc

Slide 16

Slide 16

16 RPC @gamussa | #SeattleEventDriven | @ConfluentINc

Slide 17

Slide 17

17 Integrating microservices via RPC ● Avoids problems of database integration ● Feels natural ● Aligns with the request/response paradigm ● Problem: cascading failures ● Question: how do you debug this system? 🤔 ● Answer: you build a log. @gamussa | #SeattleEventDriven | @ConfluentINc

Slide 18

Slide 18

18 Events? @gamussa | #SeattleEventDriven | @ConfluentINc

Slide 19

Slide 19

19 What’s an event? @gamussa | #SeattleEventDriven | @ConfluentINc

Slide 20

Slide 20

20 A shared narrative describing the evolution of the business over time @gamussa | #SeattleEventDriven | @ConfluentINc

Slide 21

Slide 21

21 A combination of: Notification State transfer @gamussa | #SeattleEventDriven | @ConfluentINc

Slide 22

Slide 22

22 Also, events are immutable. @gamussa | #SeattleEventDriven | @ConfluentINc

Slide 23

Slide 23

23 Kafka Basics @gamussa | #SeattleEventDriven | @ConfluentINc

Slide 24

Slide 24

24 Streaming Platform Storage Pub / Sub Processing @gamussa | #SeattleEventDriven | @ConfluentINc

Slide 25

Slide 25

25 Core Abstraction Core abstraction ● DB - table ● Hadoop - file ● Kafka - ? @gamussa | #SeattleEventDriven | @ConfluentINc

Slide 26

Slide 26

26 LOG @gamussa | #SeattleEventDriven | @ConfluentINc

Slide 27

Slide 27

27 The log is a simple idea New Old Messages are added at the end of the log @gamussa | #SeattleEventDriven | @ConfluentINc

Slide 28

Slide 28

28 The log is a simple idea New Old Messages are added at the end of the log @gamussa | #SeattleEventDriven | @ConfluentINc

Slide 29

Slide 29

29 Consumers have a position all of their own Ricardo is here Scan New Old Robin is here Scan @gamussa Viktor is here | Scan #SeattleEventDriven | @ConfluentINc

Slide 30

Slide 30

30 Consumers have a position all of their own Ricardo is here Scan New Old Robin is here @gamussa | Scan #SeattleEventDriven Viktor is here | Scan @ConfluentINc

Slide 31

Slide 31

31 Consumers have a position all of their own Ricardo is here Scan New Old Robin is here @gamussa | Viktor is here Scan #SeattleEventDriven | @ConfluentINc Scan

Slide 32

Slide 32

32 Only Sequential Access Read to offset & scan Old @gamussa | #SeattleEventDriven | New @ConfluentINc

Slide 33

Slide 33

33 Linearly Scalable Architecture Producers Single topic: - Many producers machines - Many consumer machines - Many Broker machines No Bottleneck!! Consumers @gamussa | #SeattleEventDriven | @ConfluentINc

Slide 34

Slide 34

34 The Connect API Producer Consumer The Log Connectors Connectors Streaming Engine @gamussa | #SeattleEventDriven | @ConfluentINc

Slide 35

Slide 35

35 Ingest / Output to any data source Kafka Connect @gamussa | Kafka Connect Kafka #SeattleEventDriven | @ConfluentINc

Slide 36

Slide 36

36 Stream Processing in Kafka Producer Connectors Consumer The Log Connectors Streaming Engine @gamussa | #SeattleEventDriven | @ConfluentINc

Slide 37

Slide 37

37 KSQL: an engine for continuous computation SELECT card_number, count() FROM authorizations WINDOW (SIZE 5 MINUTE) GROUP BY card_number HAVING count() > 3; @gamussa | #SeattleEventDriven | @ConfluentINc

Slide 38

Slide 38

38 Kafka Streams: a Java API for the same StreamsBuilder builder = new StreamsBuilder(); builder .stream(“caterpillars”) .map(Metamorphosis!::coolTransformation) .to(“butterflies”); final Topology topology = builder.build(); new KafkaStreams(topology, config()).start(); @gamussa | #SeattleEventDriven | @ConfluentINc

Slide 39

Slide 39

39 Kafka Streams: a Java API for the same StreamsBuilder builder = new StreamsBuilder(); builder .stream(“caterpillars”) .map(Metamorphosis!::coolTransformation) .to(“butterflies”); final Topology topology = builder.build(); new KafkaStreams(topology, config()).start(); @gamussa | #SeattleEventDriven | @ConfluentINc

Slide 40

Slide 40

40 Microservices @gamussa | #SeattleEventDriven | @ConfluentINc

Slide 41

Slide 41

41 Suppose we have these services Customer Service Shipping Service @gamussa | #SeattleEventDriven | @ConfluentINc

Slide 42

Slide 42

42 @gamussa | #SeattleEventDriven | @ConfluentINc

Slide 43

Slide 43

43 services share the same core facts C rs e m o t us Or de rs Most services live in here Catalog @gamussa | #SeattleEventDriven | @ConfluentINc

Slide 44

Slide 44

44 Kafka is a Backbone for Events exchange Notification Kafka Data is replicated @gamussa | #SeattleEventDriven | @ConfluentINc

Slide 45

Slide 45

45 Two types of events Notification @gamussa | Data replication #SeattleEventDriven | @ConfluentINc

Slide 46

Slide 46

46 ECommerce Microservices (with RPC) Webserver

  • Orders Service calls Shipping Service to tell it to ship item. Submit Order Orders Service Shipping Service shipOrder() @gamussa Customer Service getCustomer() |
  • Shipping service looks up address to ship to (from Customer Service) - No Kafka 😢 #SeattleEventDriven | @ConfluentINc

Slide 47

Slide 47

47 Refactoring Orders and Shipping Webserver Submit Order Orders Service Customer Service Shipping Service Order Created

  • Orders Service no longer knows about the Shipping service (or any other service). RPC
  • Events are fire and forget. getCustomer() Message Broker (Kafka) @gamussa | KAFKA #SeattleEventDriven | @ConfluentINc

Slide 48

Slide 48

48 Refactoring Customers Webserver Submit Order Orders Service Shipping Service Order Created Customer Service Customer Updated

  • Call to Customer service is gone. - Instead data in replicated, as events, into the shipping service, where it is queried locally. KAFKA @gamussa | #SeattleEventDriven | @ConfluentINc

Slide 49

Slide 49

49 Events are the key to scalable service ecosystems Orders Service @gamussa | #SeattleEventDriven | @ConfluentINc

Slide 50

Slide 50

50 A Richer Microservices Application @gamussa | #SeattleEventDriven | @ConfluentINc

Slide 51

Slide 51

51 Orders Service operating on the event stream Browser Orders Service Webserver KStreams API used to validate orders as they are created. Order Order Order Requested Received Validated KAFKA @gamussa | #SeattleEventDriven | @ConfluentINc

Slide 52

Slide 52

52 Schema Management Browser Orders Service Webserver KStreams API used to validate orders as they are created. Order Order Order Requested Received Validated KAFKA @gamussa | #SeattleEventDriven | Schema Registry @ConfluentINc

Slide 53

Slide 53

53 Access legacy relational data Browser Orders Service KStreams API used to validate orders as they are created. Webserver Order Connect Order Order Requested Received Validated Stock KAFKA @gamussa | #SeattleEventDriven | Schema Registry Registry @ConfluentINc

Slide 54

Slide 54

54 Materialize tables inside your app Browser Orders Service Stock Lookup table created inside the Orders Service Webserver Order Connect Order Order Requested Received Validated Stock KAFKA Connect @gamussa | #SeattleEventDriven | Schema Registry Registry @ConfluentINc

Slide 55

Slide 55

55 Create a new table, persist it to the log Browser Orders Service Reserved Stocks Stock Webserver Order Connect Order Order Requested Received Validated Stock KAFKA @gamussa | #SeattleEventDriven | Writable table created for reserved stocks Reserved Stocks Schema Registry Registry @ConfluentINc

Slide 56

Slide 56

56 Transactional guarantees are yours Browser Orders Service Reserved Stocks Stock TRANSACTION Webserver Order Connect Order Order Requested Received Validated Stock KAFKA @gamussa | #SeattleEventDriven | Reserved Stocks Schema Registry Registry @ConfluentINc

Slide 57

Slide 57

57 Hydrate a materialized view Browser an embedded database Orders Service Connect Webserver Order Connect Order Order Requested Received Validated Stock KAFKA @gamussa | #SeattleEventDriven | Reserved Stocks Schema Registry Registry @ConfluentINc

Slide 58

Slide 58

58 Services in the Micro: Orders Service INVENTORY Inventory ORDERS Orders Orders Service C is CQRS Order Created GET Inventory Service ORDERS POST Load Balancer Fraud Service Order Validated Order Details Service Orders View Q in CQRS @gamussa | OV TOPIC KAFKA Order Validations #SeattleEventDriven | @ConfluentINc

Slide 59

Slide 59

59 @gamussa | #SeattleEventDriven | @ConfluentINc

Slide 60

Slide 60

60 What’s a database anyway? @gamussa | #SeattleEventDriven | @ConfluentINc

Slide 61

Slide 61

61 SQL @gamussa | #SeattleEventDriven | @ConfluentINc

Slide 62

Slide 62

62 Tabular Model @gamussa | #SeattleEventDriven | @ConfluentINc

Slide 63

Slide 63

63 Storage Engine @gamussa | #SeattleEventDriven | @ConfluentINc

Slide 64

Slide 64

64 Commit Log @gamussa | #SeattleEventDriven | @ConfluentINc

Slide 65

Slide 65

65 What are these things? Inventory INVENTORY ORDERS Orders Orders Service Order Created C is CQRS Fraud Service ORDERS (see previous figure) Order Validated Orders View Q in CQRS @gamussa Inventory Service | Order Details Service OV TOPIC GET Load Balancer POST KAFKA Order Validations #SeattleEventDriven | @ConfluentINc

Slide 66

Slide 66

66 You are not just writing microserviceS @gamussa | #SeattleEventDriven | @ConfluentINc

Slide 67

Slide 67

67 You are building an inside-out database. @gamussa | #SeattleEventDriven | @ConfluentINc

Slide 68

Slide 68

68 And that is a good thing… @gamussa | #SeattleEventDriven | @ConfluentINc

Slide 69

Slide 69

69 @gamussa | #SeattleEventDriven | @ConfluentINc

Slide 70

Slide 70

70 One more thing… @gamussa | #SeattleEventDriven | @ConfluentINc

Slide 71

Slide 71

https://kafka-summit.org Gamov30 @gamussa | @ @tlberglund | #DEVnexus

Slide 72

Slide 72

Thanks! @gamussa viktor@confluent.io We are hiring! https://www.confluent.io/careers/ @gamussa | @ #seattleeventdriven | @ConfluentINc