Commit Logs in n Age of Microservices
A presentation at Seattle Event Driven Meetup in in Seattle, WA, USA by Viktor Gamov
!//Join the two streams and the table then send an email for each orders.join(payments, EmailTuple!::new, !//Join Orders and Payments streams JoinWindows.of(Duration.ofMinutes(1)), serdes) !//Next join to the GKTable of Customers Commit Logs (key1, tuple) !-> tuple.order.getCustomerId(), In an ofwe Microservices !// note Age how, because use a GKtable, we can join on any .join(customers, attribute of the Customer. EmailTuple!::setCustomer) March, 2019 / Seattle, WA @gamussa !//Now for each tuple send an email. | #SeattleEventDriven | @ConfluentINc emailTuple) .peek((key,
2 @gamussa | #SeattleEventDriven | @ConfluentINc
https://gamov.dev/seattle-microservices
Raffle, yeah 🚀 Follow @gamussa 📸🖼🏋 @confluentinc Tag @gamussa With #seattleeventdriven
5 Normal Applications (i.e., monoliths) @gamussa | #SeattleEventDriven | @ConfluentINc
6 Monoliths are hard to think about @gamussa | #SeattleEventDriven | @ConfluentINc
7 Monoliths are hard to change @gamussa | #SeattleEventDriven | @ConfluentINc
8 Reintegration? @gamussa | #SeattleEventDriven | @ConfluentINc
9 Reintegration @gamussa | #SeattleEventDriven | @ConfluentINc
10 There are no good ways to integrate microservices @gamussa | #SeattleEventDriven | @ConfluentINc
11 There are no-good ways to integrate microservices @gamussa | #SeattleEventDriven | @ConfluentINc
12 Filesystem @gamussa | #SeattleEventDriven | @ConfluentINc
13 @gamussa | #SeattleEventDriven | @ConfluentINc
14 Database @gamussa | #SeattleEventDriven | @ConfluentINc
15 Integrating Microservices through the database ● “I have a database and I know how to use it.” ● Eventually causes services to co-mingle. ● Violation the «bounded context» ● Great to use inside a service boundary! ● Terrible for sharing data or negotiating change. @gamussa | #SeattleEventDriven | @ConfluentINc
16 RPC @gamussa | #SeattleEventDriven | @ConfluentINc
17 Integrating microservices via RPC ● Avoids problems of database integration ● Feels natural ● Aligns with the request/response paradigm ● Problem: cascading failures ● Question: how do you debug this system? 🤔 ● Answer: you build a log. @gamussa | #SeattleEventDriven | @ConfluentINc
18 Events? @gamussa | #SeattleEventDriven | @ConfluentINc
19 What’s an event? @gamussa | #SeattleEventDriven | @ConfluentINc
20 A shared narrative describing the evolution of the business over time @gamussa | #SeattleEventDriven | @ConfluentINc
21 A combination of: Notification State transfer @gamussa | #SeattleEventDriven | @ConfluentINc
22 Also, events are immutable. @gamussa | #SeattleEventDriven | @ConfluentINc
23 Kafka Basics @gamussa | #SeattleEventDriven | @ConfluentINc
24 Streaming Platform Storage Pub / Sub Processing @gamussa | #SeattleEventDriven | @ConfluentINc
25 Core Abstraction Core abstraction ● DB - table ● Hadoop - file ● Kafka - ? @gamussa | #SeattleEventDriven | @ConfluentINc
26 LOG @gamussa | #SeattleEventDriven | @ConfluentINc
27 The log is a simple idea New Old Messages are added at the end of the log @gamussa | #SeattleEventDriven | @ConfluentINc
28 The log is a simple idea New Old Messages are added at the end of the log @gamussa | #SeattleEventDriven | @ConfluentINc
29 Consumers have a position all of their own Ricardo is here Scan New Old Robin is here Scan @gamussa Viktor is here | Scan #SeattleEventDriven | @ConfluentINc
30 Consumers have a position all of their own Ricardo is here Scan New Old Robin is here @gamussa | Scan #SeattleEventDriven Viktor is here | Scan @ConfluentINc
31 Consumers have a position all of their own Ricardo is here Scan New Old Robin is here @gamussa | Viktor is here Scan #SeattleEventDriven | @ConfluentINc Scan
32 Only Sequential Access Read to offset & scan Old @gamussa | #SeattleEventDriven | New @ConfluentINc
33 Linearly Scalable Architecture Producers Single topic: - Many producers machines - Many consumer machines - Many Broker machines No Bottleneck!! Consumers @gamussa | #SeattleEventDriven | @ConfluentINc
34 The Connect API Producer Consumer The Log Connectors Connectors Streaming Engine @gamussa | #SeattleEventDriven | @ConfluentINc
35 Ingest / Output to any data source Kafka Connect @gamussa | Kafka Connect Kafka #SeattleEventDriven | @ConfluentINc
36 Stream Processing in Kafka Producer Connectors Consumer The Log Connectors Streaming Engine @gamussa | #SeattleEventDriven | @ConfluentINc
37 KSQL: an engine for continuous computation SELECT card_number, count() FROM authorizations WINDOW (SIZE 5 MINUTE) GROUP BY card_number HAVING count() > 3; @gamussa | #SeattleEventDriven | @ConfluentINc
38 Kafka Streams: a Java API for the same StreamsBuilder builder = new StreamsBuilder(); builder .stream(“caterpillars”) .map(Metamorphosis!::coolTransformation) .to(“butterflies”); final Topology topology = builder.build(); new KafkaStreams(topology, config()).start(); @gamussa | #SeattleEventDriven | @ConfluentINc
39 Kafka Streams: a Java API for the same StreamsBuilder builder = new StreamsBuilder(); builder .stream(“caterpillars”) .map(Metamorphosis!::coolTransformation) .to(“butterflies”); final Topology topology = builder.build(); new KafkaStreams(topology, config()).start(); @gamussa | #SeattleEventDriven | @ConfluentINc
40 Microservices @gamussa | #SeattleEventDriven | @ConfluentINc
41 Suppose we have these services Customer Service Shipping Service @gamussa | #SeattleEventDriven | @ConfluentINc
42 @gamussa | #SeattleEventDriven | @ConfluentINc
43 services share the same core facts C rs e m o t us Or de rs Most services live in here Catalog @gamussa | #SeattleEventDriven | @ConfluentINc
44 Kafka is a Backbone for Events exchange Notification Kafka Data is replicated @gamussa | #SeattleEventDriven | @ConfluentINc
45 Two types of events Notification @gamussa | Data replication #SeattleEventDriven | @ConfluentINc
46 ECommerce Microservices (with RPC) Webserver
- Orders Service calls Shipping Service to tell it to ship item. Submit Order Orders Service Shipping Service shipOrder() @gamussa Customer Service getCustomer() |
- Shipping service looks up address to ship to (from Customer Service) - No Kafka 😢 #SeattleEventDriven | @ConfluentINc
47 Refactoring Orders and Shipping Webserver Submit Order Orders Service Customer Service Shipping Service Order Created
- Orders Service no longer knows about the Shipping service (or any other service). RPC
- Events are fire and forget. getCustomer() Message Broker (Kafka) @gamussa | KAFKA #SeattleEventDriven | @ConfluentINc
48 Refactoring Customers Webserver Submit Order Orders Service Shipping Service Order Created Customer Service Customer Updated
- Call to Customer service is gone. - Instead data in replicated, as events, into the shipping service, where it is queried locally. KAFKA @gamussa | #SeattleEventDriven | @ConfluentINc
49 Events are the key to scalable service ecosystems Orders Service @gamussa | #SeattleEventDriven | @ConfluentINc
50 A Richer Microservices Application @gamussa | #SeattleEventDriven | @ConfluentINc
51 Orders Service operating on the event stream Browser Orders Service Webserver KStreams API used to validate orders as they are created. Order Order Order Requested Received Validated KAFKA @gamussa | #SeattleEventDriven | @ConfluentINc
52 Schema Management Browser Orders Service Webserver KStreams API used to validate orders as they are created. Order Order Order Requested Received Validated KAFKA @gamussa | #SeattleEventDriven | Schema Registry @ConfluentINc
53 Access legacy relational data Browser Orders Service KStreams API used to validate orders as they are created. Webserver Order Connect Order Order Requested Received Validated Stock KAFKA @gamussa | #SeattleEventDriven | Schema Registry Registry @ConfluentINc
54 Materialize tables inside your app Browser Orders Service Stock Lookup table created inside the Orders Service Webserver Order Connect Order Order Requested Received Validated Stock KAFKA Connect @gamussa | #SeattleEventDriven | Schema Registry Registry @ConfluentINc
55 Create a new table, persist it to the log Browser Orders Service Reserved Stocks Stock Webserver Order Connect Order Order Requested Received Validated Stock KAFKA @gamussa | #SeattleEventDriven | Writable table created for reserved stocks Reserved Stocks Schema Registry Registry @ConfluentINc
56 Transactional guarantees are yours Browser Orders Service Reserved Stocks Stock TRANSACTION Webserver Order Connect Order Order Requested Received Validated Stock KAFKA @gamussa | #SeattleEventDriven | Reserved Stocks Schema Registry Registry @ConfluentINc
57 Hydrate a materialized view Browser an embedded database Orders Service Connect Webserver Order Connect Order Order Requested Received Validated Stock KAFKA @gamussa | #SeattleEventDriven | Reserved Stocks Schema Registry Registry @ConfluentINc
58 Services in the Micro: Orders Service INVENTORY Inventory ORDERS Orders Orders Service C is CQRS Order Created GET Inventory Service ORDERS POST Load Balancer Fraud Service Order Validated Order Details Service Orders View Q in CQRS @gamussa | OV TOPIC KAFKA Order Validations #SeattleEventDriven | @ConfluentINc
59 @gamussa | #SeattleEventDriven | @ConfluentINc
60 What’s a database anyway? @gamussa | #SeattleEventDriven | @ConfluentINc
61 SQL @gamussa | #SeattleEventDriven | @ConfluentINc
62 Tabular Model @gamussa | #SeattleEventDriven | @ConfluentINc
63 Storage Engine @gamussa | #SeattleEventDriven | @ConfluentINc
64 Commit Log @gamussa | #SeattleEventDriven | @ConfluentINc
65 What are these things? Inventory INVENTORY ORDERS Orders Orders Service Order Created C is CQRS Fraud Service ORDERS (see previous figure) Order Validated Orders View Q in CQRS @gamussa Inventory Service | Order Details Service OV TOPIC GET Load Balancer POST KAFKA Order Validations #SeattleEventDriven | @ConfluentINc
66 You are not just writing microserviceS @gamussa | #SeattleEventDriven | @ConfluentINc
67 You are building an inside-out database. @gamussa | #SeattleEventDriven | @ConfluentINc
68 And that is a good thing… @gamussa | #SeattleEventDriven | @ConfluentINc
69 @gamussa | #SeattleEventDriven | @ConfluentINc
70 One more thing… @gamussa | #SeattleEventDriven | @ConfluentINc
https://kafka-summit.org Gamov30 @gamussa | @ @tlberglund | #DEVnexus
Thanks! @gamussa viktor@confluent.io We are hiring! https://www.confluent.io/careers/ @gamussa | @ #seattleeventdriven | @ConfluentINc
Microservice architectures provide a robust challenge to the traditional centralized database we have come to understand. In this talk, we’ll explore the notion of unbundling that database, and putting a distributed commit log at the center of our information architecture. As events impinge on our system, we store them in a durable, immutable log (happily provided by Apache Kafka), allowing each microservice to create a derived view of the data according to the needs of its clients. Event-based integration avoids the now-well-known problems of RPC and database-based service integration, and allows the information architecture of the future to take advantage of the growing functionality of stream processing systems like Kafka, allowing us to create systems that can more easily adapt to the changing needs of the enterprise and provide the real-time results we are increasingly being asked to provide.
Video
Buzz and feedback
Here’s what was said about this presentation on social media.
