Building a Real-time Alerting, Analytics and Reporting System at Scale
A presentation at NYC In-Memory Computing Meetup in in New York, NY, USA by Viktor Gamov
real-time streaming and analytics at scale with Apache Kafka and Apache Ignite June, 2019 / New York, NY @denismagda | @gamussa | #NYCInMemory
2 Hello 👋 @gamussa @denismagda @denismagda | @gamussa | #NYCInMemory
Digital transformation challenges @denismagda | @gamussa | #NYCInMemory
4 Digital Transformations Challenges Application Layer Web-Scale Apps IoT 10-100x Queries and Transactions (per sec) Mobile Apps Social Media 10-1000x Faster Analytics (Hours to Sec) 50x Data Storage (Big Data) Data Layer NoSQL RDBMS @denismagda | @gamussa Hadoop | #NYCInMemory
4 Digital Transformations Challenges Application Layer ● 10-100x more queries and transactions Web-Scale Apps IoT 10-100x Queries and Transactions (per sec) Mobile Apps Social Media 10-1000x Faster Analytics (Hours to Sec) 50x Data Storage (Big Data) Data Layer NoSQL RDBMS @denismagda | @gamussa Hadoop | #NYCInMemory
4 Digital Transformations Challenges Application Layer ● 10-100x more queries and transactions ● 50x more data today as a decade ago Web-Scale Apps IoT 10-100x Queries and Transactions (per sec) Mobile Apps Social Media 10-1000x Faster Analytics (Hours to Sec) 50x Data Storage (Big Data) Data Layer NoSQL RDBMS @denismagda | @gamussa Hadoop | #NYCInMemory
4 Digital Transformations Challenges Application Layer ● 10-100x more queries and transactions ● 50x more data today as a decade ago ● Overnight analytics become real-time Web-Scale Apps IoT 10-100x Queries and Transactions (per sec) Mobile Apps Social Media 10-1000x Faster Analytics (Hours to Sec) 50x Data Storage (Big Data) Data Layer NoSQL RDBMS @denismagda | @gamussa Hadoop | #NYCInMemory
5 @denismagda | @gamussa | #NYCInMemory
5 @denismagda | @gamussa | #NYCInMemory
5 @denismagda | @gamussa | #NYCInMemory
5 @denismagda | @gamussa | #NYCInMemory
5 @denismagda | @gamussa | #NYCInMemory
5 @denismagda | @gamussa | #NYCInMemory
5 @denismagda | @gamussa | #NYCInMemory
In-Memory Computing and Stream processing Application Layer Web-Scale Apps IoT Mobile Apps Social Media Confluent Platform GridGain In-Memory Computing Platform Event Streaming Transactional Persistence @denismagda | @gamussa | #NYCInMemory
In-Memory Computing and Stream processing • Performance and velocity increases Application Layer Web-Scale Apps IoT Mobile Apps Social Media Confluent Platform GridGain In-Memory Computing Platform Event Streaming Transactional Persistence @denismagda | @gamussa | #NYCInMemory
In-Memory Computing and Stream processing • Performance and velocity increases Application Layer Web-Scale Apps IoT Mobile Apps Social Media • Scalability up to petabytes of data Confluent Platform GridGain In-Memory Computing Platform Event Streaming Transactional Persistence @denismagda | @gamussa | #NYCInMemory
In-Memory Computing and Stream processing • Performance and velocity increases Application Layer Web-Scale Apps IoT Mobile Apps Social Media • Scalability up to petabytes of data • Act faster by analyzing streams of data Confluent Platform GridGain In-Memory Computing Platform Event Streaming Transactional Persistence using SQL language @denismagda | @gamussa | #NYCInMemory
8 Streaming-First Workd @denismagda | @gamussa | #NYCInMemory
9 Kappa Architecture: GridGain and Kafka Connect 💵 @denismagda | @gamussa | #NYCInMemory
Demo @denismagda | @gamussa | #NYCInMemory
Enter Kafka Connect @denismagda | @gamussa | #NYCInMemory
13 @denismagda | @gamussa | #NYCInMemory
13 @denismagda PRODUCER Producer Application | @gamussa | #NYCInMemory
13 CONSUMER @denismagda PRODUCER Producer Application | @gamussa | Consumer Application #NYCInMemory
14 KAFKA CONNECT KAFKA CONNECT CONSUMER PRODUCER @denismagda | @gamussa | #NYCInMemory
14 KAFKA CONNECT KAFKA CONNECT CONSUMER PRODUCER Source Connector SMTs Converter @denismagda | @gamussa | #NYCInMemory
14 KAFKA CONNECT KAFKA CONNECT CONSUMER PRODUCER Source Connector SMTs Converter SMTs Converter @denismagda | @gamussa | Sink Connector #NYCInMemory
15 Discover connectors, SMTs, and converters @denismagda | @gamussa | #NYCInMemory
16 Discover connectors, SMTs, and converters Descriptions, licensing, support, and more @denismagda | @gamussa | #NYCInMemory
17 Lower the Bar to Enter the World Coding Sophistication Core developers who use Java/Scala streams Core developers who don’t use Java/Scala Data engineers, architects, DevOps/SRE BI analysts User Population @denismagda | @gamussa | #NYCInMemory
17 Lower the Bar to Enter the World Coding Sophistication Core developers who use Java/Scala streams Core developers who don’t use Java/Scala Data engineers, architects, DevOps/SRE BI analysts User Population @denismagda | @gamussa | #NYCInMemory
Store and process with GridGain @denismagda | @gamussa | #NYCInMemory
GridGain: Real-time Streaming and Analytics @denismagda | @gamussa | #NYCInMemory 19
20 Essential GridGain APIs Distributed memory-centric storage Co-located Computations Combines the performance and scale of inmemory computing together with the disk durability and strong consistency in one system Brings the computations to the servers where the data actually resides, eliminating need to move data over the network Distributed SQL Horizontally, fault-tolerant distributed SQL database that treats memory and disk as active storage tiers Distributed Key-Value Read, write and transact with fast key-value APIs ACID Transactions Machine and Deep Learning Supports distributed ACID transactions for key-value as well as SQL operations Set of simple, scalable and efficient tools that allow building predictive machine learning models without costly data transfers (ETL) @denismagda | @gamussa | #NYCInMemory
21 GridGain SQL For Real-Time Analytics Ignite Node Toronto 2 Canada Montreal Ottawa Calgary 1 Ignite Node 3 2 India New Delhi
- Initial Query 2. Query execution over local data 3. Reduce multiple results in one @denismagda Mumbai | @gamussa | #NYCInMemory
Thanks! @denismagda dmagda@gridgain.com @gamussa viktor@confluent.io @denismagda | @ @gamussa | #NYCInMemory
Q&A @denismagda | @gamussa | #NYCInMemory
Are you tasked to build a system or upgrade an existing architecture to a solution capable of handling unbound streams of data, do real-time alerting, storing always growing terabytes and petabytes of data and, finally, act on the data within milliseconds SLAs?
By integrating Apache Kafka with Apache Ignite you’ll solve all your requirements faster and easier. A battle-tested recipe is simple — take Kafka Connect and have your data stream through Kafka pipelines, add a pinch of KSQL to act on the streams with SQL in real-time with zero delays, rinse and flush the pre-processed data in Ignite as in-memory databases and get further insights by analyzing your hot and cold datasets.
Denis and Viktor will demonstrate how to implement the solution in practice. They’ll explain architectural reasoning and the benefits of real-time integration and share common usage patterns.
Video
Resources
The following resources were mentioned during the presentation or are useful additional information.
Buzz and feedback
Here’s what was said about this presentation on social media.
