Real-Time a in the Corporate World How Apache Pinot® Powers Industry Leaders
Slide 1
Slide 2
Who Much Pizza We Sold? @gamussa | @devnexus | @apachepinot
Slide 3
Viktor GAMOV Head of Developer Advocacy | StarTree f THE CLOUD CONNECTIVITY COMPANY Twitter X: @gamussa Kong Con idential
Slide 4
«I Build Highly Scalable and Highly Available, (sometimes) Over-Engineered Hello World Applications» @gamussa
Slide 5
A Taxonomy of Analytics @gamussa | @devnexus | @apachepinot
Slide 6
REAL-TIME OBSERVABILITY/ MONITORING DASHBOARDS REPORTING EXTERNAL BATCH INTERNAL USER-FACING ANALYTICS @gamussa | @devnexus | @apachepinot REPORTING FEATURES
Slide 7
REAL-TIME DATADOG TRINO, PRESTO, BIGQUERY SNOWFLAKE HADOOP LEGACY DWH @gamussa EXTERNAL BATCH INTERNAL PINOT CLICKHOUSE ROCKSET DRUID | @devnexus | @apachepinot SEE ALSO (BUT WITH CACHING)
Slide 8
Real-Time Analytics Summit 2024 May 7-9, 2024 | San Jose, CA Use code DELIGHT35 to register at rtasummit.com
Slide 9
Who Does Real-Time Analytics? @gamussa | @devnexus | @apachepinot
Slide 10
Who Viewed My Pro ile? Total users 700 Million+ QPS 100,000s Latency SLA < 100 ms p99th Freshness Chinmay Soman Founding Engineer @gamussa | @devnexus | @apachepinot f Seunghyun Lee Senior Software Engineer Seconds
Slide 11
LinkedIn Feed Total users 700 Million QPS 100,000 Latency SLA < 100 ms p99th Freshness @gamussa | @devnexus | @apachepinot Seconds
Slide 12
Uber Eats Restaurant Manager ● Identify surge in realtime ● Detect missed/inaccurate orders in realtime Total users 500000+ QPS 100s Latency SLA < 100 ms p99th Freshness @gamussa | @devnexus | @apachepinot Seconds - Minutes
Slide 13
Who Much Pizza We Sold? @gamussa | @devnexus | @apachepinot
Slide 14
Let’s Break It All Down @gamussa | @devnexus | @apachepinot
Slide 15
@gamussa | @devnexus | @apachepinot
Slide 16
@gamussa | @devnexus | @apachepinot
Slide 17
@gamussa | @devnexus | @apachepinot
Slide 18
@gamussa | @devnexus | @apachepinot
Slide 19
@gamussa | @devnexus | @apachepinot
Slide 20
Let’s See It in Action https://stree.ai/pizza-demo @gamussa | @devnexus | @apachepinot
Slide 21
Pinot Architecture @gamussa | @devnexus | @apachepinot
Slide 22
Batch Ingestion [ , { “ip”: “111.173.165.103”, “userid”: 10, “remote_user”: “-“, “time”: “3271”, “_time”: 3271, “request”: “GET /site/user_status.html HTTP/1.1”, “status”: “406”, “bytes”: “1289”, “referrer”: “-“, “agent”: “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/ 59.0.3071.115 Safari/537.36” }, ] . . . . . . @gamussa | @devnexus | @apachepinot
Slide 23
Batch Ingestion [ ] { { { { }, }, }, } TABLE . . . . . . . . . . . . @gamussa | @devnexus | @apachepinot
Slide 24
Batch Ingestion segments [ ] { { { { }, }, }, } 1 2 3 4 5 6 7 8 TABLE . . . . . . . . . . . . @gamussa | @devnexus | @apachepinot
Slide 25
, { “ip”: “111.173.165.103”, “userid”: 10, “remote_user”: “-“, “time”: “3271”, “_time”: 3271, “request”: “GET”, “status”: “406”, “bytes”: “1289”, “referrer”: “-“, “agent”: “Mozilla/5.0” }, , ip userid remote_user time range index bytes time _time userid inverted index time range index request status bytes range index referrer agent referrer/agent/status star tree index . . . @gamussa | @devnexus | @apachepinot . . . Segment Structure
Slide 26
Segments @gamussa | @devnexus | @apachepinot
Slide 27
Segments @gamussa | @devnexus | @apachepinot
Slide 28
Segments 1 2 3 4 5 6 7 8 9 10 11 12 @gamussa | @devnexus | @apachepinot
Slide 29
Servers 1 2 3 4 5 6 7 8 9 10 11 12 SERVER @gamussa | @devnexus | @apachepinot
Slide 30
Servers 1 2 3 4 13 14 15 5 6 7 8 17 18 19 9 10 11 12 SERVER 1 SERVER 2 @gamussa | @devnexus | @apachepinot 16
Slide 31
Brokers SELECT * FROM SOMETHING BROKER (query needs segments 2 and 8) 1 3 4 6 1 2 3 5 2 4 5 7 8 9 10 7 9 11 12 8 10 12 6 11 SERVER 1 SERVER 2 DEEP STORAGE @gamussa | @devnexus | @apachepinot SERVER 3
Slide 32
All of Pinot BROKER ZOOKEEPE R BROKER CLIENTS CONTROLL ER scatter/gather [ { { { { }, }, }, } constant ingest 1 3 4 6 1 2 3 5 2 4 5 7 8 9 10 7 9 11 12 8 10 12 11 SERVER 1 SERVER 2 ] DEEP STORAGE . . . . . . . . . . . @gamussa | @devnexus | @apachepinot . 6 SERVER 3
Slide 33
, { “ip”: “111.173.165.103”, “userid”: 10, “remote_user”: “-“, “time”: “3271”, “_time”: 3271, “request”: “GET”, “status”: “406”, “bytes”: “1289”, “referrer”: “-“, “agent”: “Mozilla/5.0” }, , 1 2 3 4 5 6 7 8 OFFLINE TABLE . . . @gamussa | @devnexus | @apachepinot . . . Batch Ingestion
Slide 34
Realtime Ingestion CONSUMING SEGMENT 1 REALTIME TABLE @gamussa | @devnexus | @apachepinot
Slide 35
Realtime Ingestion CONSUMING SEGMENT 1 2 REALTIME TABLE @gamussa | @devnexus | @apachepinot
Slide 36
Realtime Ingestion CONSUMING SEGMENT 1 2 3 REALTIME TABLE @gamussa | @devnexus | @apachepinot
Slide 37
Realtime Ingestion CONSUMING SEGMENT 1 2 3 4 REALTIME TABLE @gamussa | @devnexus | @apachepinot
Slide 38
Realtime Ingestion CONSUMING SEGMENT 1 2 3 4 REALTIME TABLE @gamussa | @devnexus | @apachepinot
Slide 39
Slack http://stree.ai/slack https://stree.ai/pinot-slack @gamussa | @devnexus | @apachepinot
