Real-Time Dashboards in the Corporate World: How Apache Pinot Powers Industry Leaders
A presentation at DEVNEXUS in in Atlanta, GA, USA by Viktor Gamov
Real-Time a in the Corporate World How Apache Pinot® Powers Industry Leaders
Who Much Pizza We Sold? @gamussa | @devnexus | @apachepinot
Viktor GAMOV Head of Developer Advocacy | StarTree f THE CLOUD CONNECTIVITY COMPANY Twitter X: @gamussa Kong Con idential
«I Build Highly Scalable and Highly Available, (sometimes) Over-Engineered Hello World Applications» @gamussa
A Taxonomy of Analytics @gamussa | @devnexus | @apachepinot
REAL-TIME OBSERVABILITY/ MONITORING DASHBOARDS REPORTING EXTERNAL BATCH INTERNAL USER-FACING ANALYTICS @gamussa | @devnexus | @apachepinot REPORTING FEATURES
REAL-TIME DATADOG TRINO, PRESTO, BIGQUERY SNOWFLAKE HADOOP LEGACY DWH @gamussa EXTERNAL BATCH INTERNAL PINOT CLICKHOUSE ROCKSET DRUID | @devnexus | @apachepinot SEE ALSO (BUT WITH CACHING)
Real-Time Analytics Summit 2024 May 7-9, 2024 | San Jose, CA Use code DELIGHT35 to register at rtasummit.com
Who Does Real-Time Analytics? @gamussa | @devnexus | @apachepinot
Who Viewed My Pro ile? Total users 700 Million+ QPS 100,000s Latency SLA < 100 ms p99th Freshness Chinmay Soman Founding Engineer @gamussa | @devnexus | @apachepinot f Seunghyun Lee Senior Software Engineer Seconds
LinkedIn Feed Total users 700 Million QPS 100,000 Latency SLA < 100 ms p99th Freshness @gamussa | @devnexus | @apachepinot Seconds
Uber Eats Restaurant Manager ● Identify surge in realtime ● Detect missed/inaccurate orders in realtime Total users 500000+ QPS 100s Latency SLA < 100 ms p99th Freshness @gamussa | @devnexus | @apachepinot Seconds - Minutes
Who Much Pizza We Sold? @gamussa | @devnexus | @apachepinot
Let’s Break It All Down @gamussa | @devnexus | @apachepinot
@gamussa | @devnexus | @apachepinot
@gamussa | @devnexus | @apachepinot
@gamussa | @devnexus | @apachepinot
@gamussa | @devnexus | @apachepinot
@gamussa | @devnexus | @apachepinot
Let’s See It in Action https://stree.ai/pizza-demo @gamussa | @devnexus | @apachepinot
Pinot Architecture @gamussa | @devnexus | @apachepinot
Batch Ingestion [ , { “ip”: “111.173.165.103”, “userid”: 10, “remote_user”: “-“, “time”: “3271”, “_time”: 3271, “request”: “GET /site/user_status.html HTTP/1.1”, “status”: “406”, “bytes”: “1289”, “referrer”: “-“, “agent”: “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/ 59.0.3071.115 Safari/537.36” }, ] . . . . . . @gamussa | @devnexus | @apachepinot
Batch Ingestion [ ] { { { { }, }, }, } TABLE . . . . . . . . . . . . @gamussa | @devnexus | @apachepinot
Batch Ingestion segments [ ] { { { { }, }, }, } 1 2 3 4 5 6 7 8 TABLE . . . . . . . . . . . . @gamussa | @devnexus | @apachepinot
, { “ip”: “111.173.165.103”, “userid”: 10, “remote_user”: “-“, “time”: “3271”, “_time”: 3271, “request”: “GET”, “status”: “406”, “bytes”: “1289”, “referrer”: “-“, “agent”: “Mozilla/5.0” }, , ip userid remote_user time range index bytes time _time userid inverted index time range index request status bytes range index referrer agent referrer/agent/status star tree index . . . @gamussa | @devnexus | @apachepinot . . . Segment Structure
Segments @gamussa | @devnexus | @apachepinot
Segments @gamussa | @devnexus | @apachepinot
Segments 1 2 3 4 5 6 7 8 9 10 11 12 @gamussa | @devnexus | @apachepinot
Servers 1 2 3 4 5 6 7 8 9 10 11 12 SERVER @gamussa | @devnexus | @apachepinot
Servers 1 2 3 4 13 14 15 5 6 7 8 17 18 19 9 10 11 12 SERVER 1 SERVER 2 @gamussa | @devnexus | @apachepinot 16
Brokers SELECT * FROM SOMETHING BROKER (query needs segments 2 and 8) 1 3 4 6 1 2 3 5 2 4 5 7 8 9 10 7 9 11 12 8 10 12 6 11 SERVER 1 SERVER 2 DEEP STORAGE @gamussa | @devnexus | @apachepinot SERVER 3
All of Pinot BROKER ZOOKEEPE R BROKER CLIENTS CONTROLL ER scatter/gather [ { { { { }, }, }, } constant ingest 1 3 4 6 1 2 3 5 2 4 5 7 8 9 10 7 9 11 12 8 10 12 11 SERVER 1 SERVER 2 ] DEEP STORAGE . . . . . . . . . . . @gamussa | @devnexus | @apachepinot . 6 SERVER 3
, { “ip”: “111.173.165.103”, “userid”: 10, “remote_user”: “-“, “time”: “3271”, “_time”: 3271, “request”: “GET”, “status”: “406”, “bytes”: “1289”, “referrer”: “-“, “agent”: “Mozilla/5.0” }, , 1 2 3 4 5 6 7 8 OFFLINE TABLE . . . @gamussa | @devnexus | @apachepinot . . . Batch Ingestion
Realtime Ingestion CONSUMING SEGMENT 1 REALTIME TABLE @gamussa | @devnexus | @apachepinot
Realtime Ingestion CONSUMING SEGMENT 1 2 REALTIME TABLE @gamussa | @devnexus | @apachepinot
Realtime Ingestion CONSUMING SEGMENT 1 2 3 REALTIME TABLE @gamussa | @devnexus | @apachepinot
Realtime Ingestion CONSUMING SEGMENT 1 2 3 4 REALTIME TABLE @gamussa | @devnexus | @apachepinot
Realtime Ingestion CONSUMING SEGMENT 1 2 3 4 REALTIME TABLE @gamussa | @devnexus | @apachepinot
Slack http://stree.ai/slack https://stree.ai/pinot-slack @gamussa | @devnexus | @apachepinot
This isn’t your typical tech talk—it’s a story of the journey into how big players like LinkedIn, Uber Eats, and Stripe are mastering the art of real-time data. Viktor is here to demystify Apache Pinot’s superpowers, showing you how it instantly transforms mountains of data into actionable insights. Business decision-makers have long had access to dashboards and reports, but now analytics can be made available to users as features of stickier, more engaging applications. With stories, insights, and a touch of humor, Viktor will unpack the cool features of Apache Pinot, including the Star-Tree Index, and show you why it’s a game-changer in data strategy. This session is for everyone, whether you’re a data geek, a business guru, or just curious about the future of tech. Viktor’s dynamic style will keep you on the edge of your seat, eager to put these insights into action. So, buckle up and get ready to be wowed by the power of real-time analytics with Apache Pinot – it will be a blast!
Video
Code
The following code examples from the presentation can be tried out live.
Buzz and feedback
Here’s what was said about this presentation on social media.
