Fight Crime with Kafka Streams and Bintray Firehose API
Slide 1
Slide 2
Consuming liquid software from firehose like...
Slide 3
Slide 4
Hackers? WAT? × Baruch Sadogursky × Head of DevRel @jfrog × @jbaruch × Viktor Gamov × Developer Advocate @confluentinc × @gamussa
Slide 5
WTF is firehose?!
Slide 6
Slide 7
Firehose api × Twitter has it, AWS has it, Bitntray has it × Streaming events from the socket × A lot of information
Slide 8
Bintray firehose api × × × × × Login success Login failure File uploaded File downloaded File deleted
Slide 9
Consumption × Jfrog cli (piping etc) × Rest api × Java api (soon)
Slide 10
Our process is easy ka Kaf ams e r St ka Kaf r cto e n Con L Q S K
Slide 11
Kafka Basics
Slide 12
What is a Streaming Pl atform? Producer Connectors Consumer The Log Streaming Engine Connectors
Slide 13
Kafka’s Distributed L og Producer Connectors Consumer The Log Streaming Engine Connectors
Slide 14
The log is a simple ide a Old New Messages are added at the end of the log
Slide 15
Consumers have a pos ition all of their own George Scan is here Old Fred Scan is here New Sally Scan is here
Slide 16
Only Sequential Acces s Old Read to offset & scan New
Slide 17
Shard data to get sca lability Producer (1) Producer (2) Producer (3) Cluster of machines Messages are sent to different partitions Partitions live on different machines
Slide 18
Replicate to get faul t tolerance leader Machine A msg Machine B replicate msg
Slide 19
Replication provides r esiliency A ‘replica’ takes over on machine failure
Slide 20
The Connect API Producer Connectors Consumer The Log Streaming Engine Connectors
Slide 21
Ingest / Output to pra ctically any data sou rce Kafka Connect Kafka Kafka Connect
Slide 22
What is a Streaming Pl atform? Producer Connectors Consumer The Log Streaming Engine Connectors
Slide 23
KSQL: continuous com putation SELECT card_number, count() FROM authorization_attempts WINDOW (SIZE 5 MINUTE) GROUP BY card_number HAVING count() > 3;
Slide 24
Join Streams and Tabl es Kafka Kafka Streams / KSQL Topic Stream Join Table Compacted Topic
Slide 25
USE CASE LOOKING FOR HACKERS
Slide 26
Bintray firehose api
Slide 27
Bintray firehose api
Slide 28
Use case: looking for hackers! 1. Honeypot: secret file downloads attempts 2. Brute Force: login attempts 3. * leaked passwords: usage of the same password in multiple places
Slide 29
Code is on github × https://github.com/russianhackers × Kafka Connect for bintry × Docker compose and stuff
Slide 30
Thank ya’ll! × @Gamussa × @jbaruch × #OracleCodeOne
