Fight Crime with Kafka Streams and the Bintray Firehose API
A presentation at Oracle Code One 2018 in in San Francisco, CA, USA by Viktor Gamov
Fight Crime with Kafka Streams and Bintray Firehose API
Consuming liquid software from firehose like...
Hackers? WAT? × Baruch Sadogursky × Head of DevRel @jfrog × @jbaruch × Viktor Gamov × Developer Advocate @confluentinc × @gamussa
WTF is firehose?!
Firehose api × Twitter has it, AWS has it, Bitntray has it × Streaming events from the socket × A lot of information
Bintray firehose api × × × × × Login success Login failure File uploaded File downloaded File deleted
Consumption × Jfrog cli (piping etc) × Rest api × Java api (soon)
Our process is easy ka Kaf ams e r St ka Kaf r cto e n Con L Q S K
Kafka Basics
What is a Streaming Pl atform? Producer Connectors Consumer The Log Streaming Engine Connectors
Kafka’s Distributed L og Producer Connectors Consumer The Log Streaming Engine Connectors
The log is a simple ide a Old New Messages are added at the end of the log
Consumers have a pos ition all of their own George Scan is here Old Fred Scan is here New Sally Scan is here
Only Sequential Acces s Old Read to offset & scan New
Shard data to get sca lability Producer (1) Producer (2) Producer (3) Cluster of machines Messages are sent to different partitions Partitions live on different machines
Replicate to get faul t tolerance leader Machine A msg Machine B replicate msg
Replication provides r esiliency A ‘replica’ takes over on machine failure
The Connect API Producer Connectors Consumer The Log Streaming Engine Connectors
Ingest / Output to pra ctically any data sou rce Kafka Connect Kafka Kafka Connect
What is a Streaming Pl atform? Producer Connectors Consumer The Log Streaming Engine Connectors
KSQL: continuous com putation SELECT card_number, count() FROM authorization_attempts WINDOW (SIZE 5 MINUTE) GROUP BY card_number HAVING count() > 3;
Join Streams and Tabl es Kafka Kafka Streams / KSQL Topic Stream Join Table Compacted Topic
USE CASE LOOKING FOR HACKERS
Bintray firehose api
Bintray firehose api
Use case: looking for hackers! 1. Honeypot: secret file downloads attempts 2. Brute Force: login attempts 3. * leaked passwords: usage of the same password in multiple places
Code is on github × https://github.com/russianhackers × Kafka Connect for bintry × Docker compose and stuff
Thank ya’ll! × @Gamussa × @jbaruch × #OracleCodeOne
Can you find a malicious activity needle in the haystack of events on one of the busiest distribution hubs in the world? Processing the streaming events from the Bintray Firehose API with Kafka Streams can give you the superpower to do that. This session shows a real-life example of using Kafka KSQL to process and parse huge amounts of data to identify a worrying trend that might be a sign of a malicious activity.
Code
The following code examples from the presentation can be tried out live.
