Commit Logs in n Age of Microservices

!//Join the two streams and the table then send an email for each orders.join(payments, EmailTuple!::new, !//Join Orders and Payments streams JoinWindows.of(Duration.ofMinutes(1)), serdes) !//Next join to the GKTable of Customers Commit Logs (key1, tuple) !-> tuple.order.getCustomerId(), In an ofwe Microservices !// note Age how, because use a GKtable, we can join on any .join(customers, attribute of the Customer. EmailTuple!::setCustomer) March, 2019 / Seattle, WA @gamussa !//Now for each tuple send an email. | #SeattleEventDriven | @ConfluentINc emailTuple) .peek((key,

2 @gamussa | #SeattleEventDriven | @ConfluentINc

https://gamov.dev/seattle-microservices

Raffle, yeah 🚀 Follow @gamussa 📸🖼🏋 @confluentinc Tag @gamussa With #seattleeventdriven

5 Normal Applications (i.e., monoliths) @gamussa | #SeattleEventDriven | @ConfluentINc

6 Monoliths are hard to think about @gamussa | #SeattleEventDriven | @ConfluentINc

7 Monoliths are hard to change @gamussa | #SeattleEventDriven | @ConfluentINc

8 Reintegration? @gamussa | #SeattleEventDriven | @ConfluentINc

9 Reintegration @gamussa | #SeattleEventDriven | @ConfluentINc

10 There are no good ways to integrate microservices @gamussa | #SeattleEventDriven | @ConfluentINc

11 There are no-good ways to integrate microservices @gamussa | #SeattleEventDriven | @ConfluentINc

12 Filesystem @gamussa | #SeattleEventDriven | @ConfluentINc

13 @gamussa | #SeattleEventDriven | @ConfluentINc

14 Database @gamussa | #SeattleEventDriven | @ConfluentINc

15 Integrating Microservices through the database ● “I have a database and I know how to use it.” ● Eventually causes services to co-mingle. ● Violation the «bounded context» ● Great to use inside a service boundary! ● Terrible for sharing data or negotiating change. @gamussa | #SeattleEventDriven | @ConfluentINc

16 RPC @gamussa | #SeattleEventDriven | @ConfluentINc

17 Integrating microservices via RPC ● Avoids problems of database integration ● Feels natural ● Aligns with the request/response paradigm ● Problem: cascading failures ● Question: how do you debug this system? 🤔 ● Answer: you build a log. @gamussa | #SeattleEventDriven | @ConfluentINc

18 Events? @gamussa | #SeattleEventDriven | @ConfluentINc

19 What’s an event? @gamussa | #SeattleEventDriven | @ConfluentINc

20 A shared narrative describing the evolution of the business over time @gamussa | #SeattleEventDriven | @ConfluentINc

21 A combination of: Notification State transfer @gamussa | #SeattleEventDriven | @ConfluentINc

22 Also, events are immutable. @gamussa | #SeattleEventDriven | @ConfluentINc

23 Kafka Basics @gamussa | #SeattleEventDriven | @ConfluentINc

24 Streaming Platform Storage Pub / Sub Processing @gamussa | #SeattleEventDriven | @ConfluentINc

25 Core Abstraction Core abstraction ● DB - table ● Hadoop - file ● Kafka - ? @gamussa | #SeattleEventDriven | @ConfluentINc

26 LOG @gamussa | #SeattleEventDriven | @ConfluentINc

27 The log is a simple idea New Old Messages are added at the end of the log @gamussa | #SeattleEventDriven | @ConfluentINc

28 The log is a simple idea New Old Messages are added at the end of the log @gamussa | #SeattleEventDriven | @ConfluentINc

29 Consumers have a position all of their own Ricardo is here Scan New Old Robin is here Scan @gamussa Viktor is here | Scan #SeattleEventDriven | @ConfluentINc

30 Consumers have a position all of their own Ricardo is here Scan New Old Robin is here @gamussa | Scan #SeattleEventDriven Viktor is here | Scan @ConfluentINc

31 Consumers have a position all of their own Ricardo is here Scan New Old Robin is here @gamussa | Viktor is here Scan #SeattleEventDriven | @ConfluentINc Scan

32 Only Sequential Access Read to offset & scan Old @gamussa | #SeattleEventDriven | New @ConfluentINc

33 Linearly Scalable Architecture Producers Single topic: - Many producers machines - Many consumer machines - Many Broker machines No Bottleneck!! Consumers @gamussa | #SeattleEventDriven | @ConfluentINc

34 The Connect API Producer Consumer The Log Connectors Connectors Streaming Engine @gamussa | #SeattleEventDriven | @ConfluentINc

35 Ingest / Output to any data source Kafka Connect @gamussa | Kafka Connect Kafka #SeattleEventDriven | @ConfluentINc

36 Stream Processing in Kafka Producer Connectors Consumer The Log Connectors Streaming Engine @gamussa | #SeattleEventDriven | @ConfluentINc

37 KSQL: an engine for continuous computation SELECT card_number, count() FROM authorizations WINDOW (SIZE 5 MINUTE) GROUP BY card_number HAVING count() > 3; @gamussa | #SeattleEventDriven | @ConfluentINc

38 Kafka Streams: a Java API for the same StreamsBuilder builder = new StreamsBuilder(); builder .stream(“caterpillars”) .map(Metamorphosis!::coolTransformation) .to(“butterflies”); final Topology topology = builder.build(); new KafkaStreams(topology, config()).start(); @gamussa | #SeattleEventDriven | @ConfluentINc

39 Kafka Streams: a Java API for the same StreamsBuilder builder = new StreamsBuilder(); builder .stream(“caterpillars”) .map(Metamorphosis!::coolTransformation) .to(“butterflies”); final Topology topology = builder.build(); new KafkaStreams(topology, config()).start(); @gamussa | #SeattleEventDriven | @ConfluentINc

40 Microservices @gamussa | #SeattleEventDriven | @ConfluentINc

41 Suppose we have these services Customer Service Shipping Service @gamussa | #SeattleEventDriven | @ConfluentINc

42 @gamussa | #SeattleEventDriven | @ConfluentINc

43 services share the same core facts C rs e m o t us Or de rs Most services live in here Catalog @gamussa | #SeattleEventDriven | @ConfluentINc

44 Kafka is a Backbone for Events exchange Notification Kafka Data is replicated @gamussa | #SeattleEventDriven | @ConfluentINc

45 Two types of events Notification @gamussa | Data replication #SeattleEventDriven | @ConfluentINc

46 ECommerce Microservices (with RPC) Webserver

Orders Service calls Shipping Service to tell it to ship item. Submit Order Orders Service Shipping Service shipOrder() @gamussa Customer Service getCustomer() |
Shipping service looks up address to ship to (from Customer Service) - No Kafka 😢 #SeattleEventDriven | @ConfluentINc

47 Refactoring Orders and Shipping Webserver Submit Order Orders Service Customer Service Shipping Service Order Created

Orders Service no longer knows about the Shipping service (or any other service). RPC
Events are fire and forget. getCustomer() Message Broker (Kafka) @gamussa | KAFKA #SeattleEventDriven | @ConfluentINc

48 Refactoring Customers Webserver Submit Order Orders Service Shipping Service Order Created Customer Service Customer Updated

Call to Customer service is gone. - Instead data in replicated, as events, into the shipping service, where it is queried locally. KAFKA @gamussa | #SeattleEventDriven | @ConfluentINc

49 Events are the key to scalable service ecosystems Orders Service @gamussa | #SeattleEventDriven | @ConfluentINc

50 A Richer Microservices Application @gamussa | #SeattleEventDriven | @ConfluentINc

51 Orders Service operating on the event stream Browser Orders Service Webserver KStreams API used to validate orders as they are created. Order Order Order Requested Received Validated KAFKA @gamussa | #SeattleEventDriven | @ConfluentINc

52 Schema Management Browser Orders Service Webserver KStreams API used to validate orders as they are created. Order Order Order Requested Received Validated KAFKA @gamussa | #SeattleEventDriven | Schema Registry @ConfluentINc

53 Access legacy relational data Browser Orders Service KStreams API used to validate orders as they are created. Webserver Order Connect Order Order Requested Received Validated Stock KAFKA @gamussa | #SeattleEventDriven | Schema Registry Registry @ConfluentINc

54 Materialize tables inside your app Browser Orders Service Stock Lookup table created inside the Orders Service Webserver Order Connect Order Order Requested Received Validated Stock KAFKA Connect @gamussa | #SeattleEventDriven | Schema Registry Registry @ConfluentINc

55 Create a new table, persist it to the log Browser Orders Service Reserved Stocks Stock Webserver Order Connect Order Order Requested Received Validated Stock KAFKA @gamussa | #SeattleEventDriven | Writable table created for reserved stocks Reserved Stocks Schema Registry Registry @ConfluentINc

56 Transactional guarantees are yours Browser Orders Service Reserved Stocks Stock TRANSACTION Webserver Order Connect Order Order Requested Received Validated Stock KAFKA @gamussa | #SeattleEventDriven | Reserved Stocks Schema Registry Registry @ConfluentINc

57 Hydrate a materialized view Browser an embedded database Orders Service Connect Webserver Order Connect Order Order Requested Received Validated Stock KAFKA @gamussa | #SeattleEventDriven | Reserved Stocks Schema Registry Registry @ConfluentINc

58 Services in the Micro: Orders Service INVENTORY Inventory ORDERS Orders Orders Service C is CQRS Order Created GET Inventory Service ORDERS POST Load Balancer Fraud Service Order Validated Order Details Service Orders View Q in CQRS @gamussa | OV TOPIC KAFKA Order Validations #SeattleEventDriven | @ConfluentINc

59 @gamussa | #SeattleEventDriven | @ConfluentINc

60 What’s a database anyway? @gamussa | #SeattleEventDriven | @ConfluentINc

61 SQL @gamussa | #SeattleEventDriven | @ConfluentINc

62 Tabular Model @gamussa | #SeattleEventDriven | @ConfluentINc

63 Storage Engine @gamussa | #SeattleEventDriven | @ConfluentINc

64 Commit Log @gamussa | #SeattleEventDriven | @ConfluentINc

65 What are these things? Inventory INVENTORY ORDERS Orders Orders Service Order Created C is CQRS Fraud Service ORDERS (see previous figure) Order Validated Orders View Q in CQRS @gamussa Inventory Service | Order Details Service OV TOPIC GET Load Balancer POST KAFKA Order Validations #SeattleEventDriven | @ConfluentINc

66 You are not just writing microserviceS @gamussa | #SeattleEventDriven | @ConfluentINc

67 You are building an inside-out database. @gamussa | #SeattleEventDriven | @ConfluentINc

68 And that is a good thing… @gamussa | #SeattleEventDriven | @ConfluentINc

69 @gamussa | #SeattleEventDriven | @ConfluentINc

70 One more thing… @gamussa | #SeattleEventDriven | @ConfluentINc

https://kafka-summit.org Gamov30 @gamussa | @ @tlberglund | #DEVnexus

Thanks! @gamussa viktor@confluent.io We are hiring! https://www.confluent.io/careers/ @gamussa | @ #seattleeventdriven | @ConfluentINc