I don’t always test my streams, but when I do I do it in production

Slide 1

I don’t always test my streams, but when I do I do it in production August 2020 / Kafka Summit @gamussa @gamussa || @jconf.dev @jconf.dev || @confluentinc @confluentinc

Slide 2

@gamussa | @jconf.dev | @confluentinc

Slide 3

Special thanks @inponomarev @gamussa @bsideup | @jconf.dev | @confluentinc

Slide 4

https://gamov.dev/jdevconf2020 @gamussa | @jconf.dev | @confluentinc

Slide 5

Agenda 01 04 Kafka Apps Testing Streams 02 05 Testing basics Testing ksqlDB 03 06 Can we better? Conclusion What are those? TopologyTestDriver vs TestContainers Run Kafka and CP locally Ksql-test-runner vs TestContainers Always Be Testing Test with TestContainers @gamussa | @jconf.dev | @confluentinc

Slide 6

Preface @gamussa | @jconf.dev | @confluentinc

Slide 7

Senior Developer: - Where did you get this code? Junior Developer: - From Stackoverflow Senior Developer: - from question or answer? @gamussa | @jconf.dev | @confluentinc

Slide 8

An example from my life: Kafka Tutorials website @gamussa | @jconf.dev | @confluentinc

Slide 9

Literate Programming… Do you speak it… https://en.wikipedia.org/wiki/Literate_programming @gamussa | @jconf.dev | @confluentinc

Slide 10

@gamussa | @jconf.dev | @confluentinc

Slide 11

@gamussa | @jconf.dev | @confluentinc

Slide 12

@gamussa | @jconf.dev | @confluentinc

Slide 13

@gamussa | @jconf.dev | @confluentinc

Slide 14

@gamussa | @jconf.dev | @confluentinc

Slide 15

Kafka Apps aka stream processing @gamussa | @jconf.dev | @confluentinc

Slide 16

Stream Processing is the toolset for dealing with events as they move! @gamussa | @jconf.dev | @confluentinc

Slide 17

Producers Kafka Consumers @gamussa | @jconf.dev | @confluentinc

Slide 18

What are Kafka Apps? My App @gamussa | @jconf.dev | @confluentinc

Slide 19

Schema, WUT? @gamussa | @jconf.dev | @confluentinc

Slide 20

Let’s see some code @gamussa | @jconf.dev | @confluentinc

Slide 21

Kafka Streams for stream processing @gamussa | @jconf.dev | @confluentinc

Slide 22

the KAFKA STREAMS API is a JAVA API to BUILD REAL-TIME APPLICATIONS @gamussa | @jconf.dev | @confluentinc

Slide 23

https://kafka.apache.org/documentation/streams/developer-guide/testing.html @gamussa | @jconf.dev | @confluentinc

Slide 24

Topology Test driver @gamussa | @jconf.dev | @confluentinc

Slide 25

• TopologyTestDriver • Unit test your Kafka Streams Topology • Doesn’t Require Kafka infra to run • Convenience methods to define input and output • Convenience methods to provide fixtures and validate the results @gamussa | @jconf.dev | @confluentinc

Slide 26

Let’s see some code @gamussa | @jconf.dev | @confluentinc

Slide 27

@gamussa | @jconf.dev | @confluentinc

Slide 28

@gamussa | @jconf.dev | @confluentinc

Slide 29

When TTD doesn’t help? https://gamov.dev/livestreams-s01e05 @gamussa | @jconf.dev | @confluentinc

Slide 30

Ksqldb for Stream processing @gamussa | @jconf.dev | @confluentinc

Slide 31

@gamussa | @jconf.dev | @confluentinc

Slide 32

@gamussa | @jconf.dev | @confluentinc

Slide 33

Let’s see some code @gamussa | @jconf.dev | @confluentinc

Slide 34

Conclusion @gamussa | @jconf.dev | @confluentinc

Slide 35

What did you learn today? • What’s Kafka apps apps and how to test them • Producer / Consumer API • Kafka Streams • ksqlDB • How containers and TestContainer help a lot @gamussa | @jconf.dev | @confluentinc

Slide 36

@gamussa | @jconf.dev | @confluentinc

Slide 37

Want to learn more? @gamussa | #springone | @confluentinc

Slide 38

Learn Kafka. Start building with Apache Kafka at Confluent Developer. developer.confluent.io Watch full version https://gamov.dev/developer

Slide 39

Confluent Community Slack A vibrant community of over 16,000 members Come along and discuss Apache Kafka and Confluent Platform on dedicated channels including #ksqlDB, #connect, #clients, and more http://cnfl.io/slack @gamussa | @jconf.dev | @confluentinc

Slide 40

@gamussa | @jconf.dev | @confluentinc