Apache Kafka DevOps with Kubernetes and GitOps

A presentation at Devnexus 2021 in February 2021 in by Viktor Gamov

Slide 1

Slide 1

Apache Kafka DevOps with Kubernetes and GitOps @gamussa | @devnexus | @confluentinc

Slide 2

Slide 2

What are we going to talk about? @gamussa | @devnexus | @confluentinc

Slide 3

Slide 3

★gamov.dev/twitter ★gamov.dev/telegram ★gamov.dev/youtube @gamussa | @devnexus | @confluentinc

Slide 4

Slide 4

https://gamov.dev/kafka-devops

Slide 5

Slide 5

Preface

Slide 6

Slide 6

A Minimal Microservice Implementation (RPC) Order Web UI Orders Service Payment Gateway Warehouse Control Shipping Service Payments Service @gamussa | @devnexus | @confluentinc Users Web UI Users Service

Slide 7

Slide 7

Event Streaming Paradigm To rethink data as neither stored records nor transient messages, but instead as a continuously updating Stream of Events

Slide 8

Slide 8

A Stream represents history as a sequence of Events @gamussa | @devnexus | @confluentinc

Slide 9

Slide 9

Events change the way we think Event-First Approach ●an event ●a stream ●a ‘data’ flow ●a stream processor Monolithic Approach ●a database ●a variable ●a singleton ●an RPC Orders Service Payments Service Orders Service Order Validation Service Customers Service request response event streams Email Notification Service DB @gamussa | @devnexus | @confluentinc Tax Service

Slide 10

Slide 10

Refactoring to Messages Order Web UI Orders Service Payment Gateway Warehouse Control Shipping Service Payments Service @gamussa | @devnexus | @confluentinc Users Web UI Users Service

Slide 11

Slide 11

Refactoring to Messages Order Web UI Orders Service Payment Gateway Warehouse Control Shipping Service Payments Service @gamussa | @devnexus | @confluentinc Users Web UI Users Service

Slide 12

Slide 12

Refactoring to Messages Order Web UI Orders Service Payment Gateway Warehouse Control Shipping Service Payments Service @gamussa | @devnexus | @confluentinc Users Web UI Users Service

Slide 13

Slide 13

Refactoring to Messages Order Web UI Orders Service Payment Gateway Warehouse Control Shipping Service Payments Service @gamussa | @devnexus | @confluentinc Users Web UI Users Service

Slide 14

Slide 14

Kafka Connect Broker Broker Broker Cluster Data Source Kafka Connect @gamussa Kafka Connect | @devnexus | @confluentinc Data Sink

Slide 15

Slide 15

Gitops

Slide 16

Slide 16

• The Git repo represents the desired state of the system. • Use existing development workflows and tools, like PRs and code reviews. • A clear history of the changes to your system in a Git. @gamussa | @devnexus | @confluentinc

Slide 17

Slide 17

Gitops with Flux @gamussa | @devnexus | @confluentinc

Slide 18

Slide 18

Stop! Demo time @gamussa | @devnexus | @confluentinc

Slide 19

Slide 19

• ccloud • kubectl • helm • fluxctl • k3d (for local testing) • kubeseal • jq • yq Prerequisites @gamussa | @devnexus | @confluentinc

Slide 20

Slide 20

Operator pattern

Slide 21

Slide 21

Slide 22

Slide 22

Slide 23

Slide 23

Gitops with Flux @gamussa | @devnexus | @confluentinc

Slide 24

Slide 24

cCloud-operator @gamussa | @devnexus | @confluentinc

Slide 25

Slide 25

Connect operator @gamussa | @devnexus | @confluentinc

Slide 26

Slide 26

Stop! Demo time @gamussa | @devnexus | @confluentinc

Slide 27

Slide 27

Let’s scale connect

Slide 28

Slide 28

Want to learn more? @gamussa | @devnexus | @confluentinc

Slide 29

Slide 29

Learn Kafka. Watch full version https://gamov.dev/developer Start building with Apache Kafka at Confluent Developer. Confluent Developer developer.confluent.io

Slide 30

Slide 30

@gamussa | @devnexus | @confluentinc

Slide 31

Slide 31

@gamussa | @devnexus | @confluentinc