The Missing Protocol: How MCP Bridges LLMs and Data Streams

The Missing Protocol How MCP Bridges LLMs and Data Streams Viktor Gamov — Principal Developer Advocate, Confluent X/Bluesky: @gamussa • linkedin.com/in/vikgamov

If you see this slide, the projector is working. (and my clicker too)

Who’s here? • How many of you are building AI agents right now? • Keep them up — how many use • How many of you have touched Kafka or any streaming platform? X/Bluesky: @gamussa MCP to connect them to real systems? ai-assisted.enginnering 4 / 53

Three words changed AI this year. Can you name all three? Raise your hand if you can name the blue one. Keep your guesses. I’ll reveal the labels in about 4 minutes.

The Hype Everyone is talking about MCP. I have no idea why. (just kidding, I totally know why) Let’s look at the data… X/Bluesky: @gamussa ai-assisted.enginnering 5 / 53

The Reveal: Three Search Terms As promised. The three lines from four minutes ago — now with labels. Red: RAG X/Bluesky: @gamussa • Blue: Model Context Protocol • Yellow: Agent Skills The blue line is the reason we’re here. That’s what I’m going to talk about. ai-assisted.enginnering 6 / 53

Who Am I? Viktor Gamov Principal Developer Advocate at Confluent Java Champion • book author I’ve been streaming data since before streaming was cool. (and definitely before it was on the LinkedIn influencer curriculum) X/Bluesky: @gamussa ai-assisted.enginnering 7 / 53

Slides and Resources Everything from this talk — slides, links, resources gamov.dev/rel → speaking.gamov.io Scan the QR or type the short URL. X/Bluesky: @gamussa ai-assisted.enginnering 8 / 53

Part 1: The Problem X/Bluesky: @gamussa ai-assisted.enginnering 9 / 53

Your Agents Are Impressive. And Blind. Your agents can: • Query databases (stale by the time you ask) • Search the web (everyone else’s data) • Read your docs (written last quarter) • Call APIs (request/response, point-in-time) But they cannot see X/Bluesky: @gamussa what’s happening in your systems ai-assisted.enginnering right now. 10 / 53

The Evolution (and the Gap) LLM

• Context
• RAG
• Agents
• MCP Every layer solves one problem. But the API is still stateless. Each call starts fresh. RAG gives context. Agents give actions. MCP gives a standard way to connect both. X/Bluesky: @gamussa ai-assisted.enginnering 11 / 53

Why Streaming Data? What’s streaming? Where does it live? • Orders placed • Apache Kafka (most common) • Sensors reporting • Apache Flink (processing) • Users clicking • Schema Registry • Payments processing • Topics, not tables This is the data your agents are X/Bluesky: @gamussa ai-assisted.enginnering (contracts) missing. 12 / 53

The Integration Nightmare Every AI tool vendor says: “Just use our SDK!” So you write custom code for: Claude API Your vector DB OpenAI API Your Kafka cluster Gemini API Your Flink jobs Llama via Ollama Your monitoring Mistral API Your internal APIs That’s N × M X/Bluesky: @gamussa integrations. Does it make sense? No. It’s a nightmare. ai-assisted.enginnering 13 / 53

Remember USB-A? Have you ever tried to put a USB-A cable in correctly on the first try? I’ll give you a moment. You turn it one way. Doesn’t fit. You turn it the other way. Still doesn’t fit. You turn it back. Now it fits. Thank you, European Union. We finally have USB-C. MCP is USB-C for AI integrations. X/Bluesky: @gamussa ai-assisted.enginnering 14 / 53

Part 2: Enter MCP X/Bluesky: @gamussa ai-assisted.enginnering 15 / 53

What is MCP? Model Context Protocol • A • JSON-RPC based — an open standard by Anthropic protocol, not a product (I know, I know… JSON-RPC in 2026… but hear me out) • Client-Server architecture • Language agnostic • One protocol X/Bluesky: @gamussa to connect LLMs to everything ai-assisted.enginnering 16 / 53

MCP Architecture Host (Claude / IDE / Custom App) MCP Client MCP Server JSON-RPC → Tools Resources Prompts X/Bluesky: @gamussa ai-assisted.enginnering 17 / 53

The Three Primitives Tools Functions the LLM can call Data the LLM can query_topic() topic: send_message() schema: create_pipeline() metrics: / / / / / X/Bluesky: @gamussa / Resources orders avro/user lag ai-assisted.enginnering read Prompts Templates the LLM can use debug_consumer analyze_throughput explain_schema 18 / 53

Part 3: MCP Meets Data Streaming X/Bluesky: @gamussa ai-assisted.enginnering 19 / 53

What If… What if your AI agent could: • Subscribe • Query • Understand • Monitor • Create X/Bluesky: @gamussa to a Kafka topic and react to events a Flink materialized view in real time your Avro schemas without you explaining them consumer lag and alert you before things break new streaming pipelines from natural language ai-assisted.enginnering 20 / 53

MCP + Kafka Architecture AI Application (Agent + MCP Client) ↓ JSON-RPC Kafka MCP Server ↓ Streaming Platform: X/Bluesky: @gamussa Kafka • Schema Registry ai-assisted.enginnering • Flink 21 / 53

Real Example: Anomaly Detection You: “Hey Claude, anything weird happening in the orders topic?” Claude (via MCP): I checked the orders topic. In the last 5 minutes: • Order volume dropped 73% compared to the hourly average • Consumer group payment-processor has lag of • Last successful commit was 45,000 messages 8 minutes ago This looks like the payment service is down. Want me to check the payment-service-health topic? X/Bluesky: @gamussa ai-assisted.enginnering 22 / 53

The Streaming Lakehouse Pipeline Kafka + Flink + Tableflow + Iceberg Sources → Kafka → Flink SQL + Tableflow → Iceberg Tables → Query Engines (Spark / Trino / DuckDB) MCP doesn’t replace any of these. It X/Bluesky: @gamussa orchestrates them. ai-assisted.enginnering 23 / 53

Part 4: Let’s Build It X/Bluesky: @gamussa ai-assisted.enginnering 24 / 53

Kafka MCP Server Config { “mcpServers”: { “confluent-kafka”: { “command”: “npx”, “args”: [ “-y”, “@confluentinc/mcp-confluent”, “-e”, “.env” ] } } } X/Bluesky: @gamussa ai-assisted.enginnering 25 / 53

Two Flavors, Same API Confluent Cloud OSS / Self-managed @confluentinc/mcp-confluent mcp-kafka • Managed Kafka, Flink, SR • Any Kafka cluster • Confluent Cloud auth • Open source • Full platform integration • Same tools, same API github.com/confluentinc/mcp-confluent They’re X/Bluesky: @gamussa tool-compatible github.com/gamussa/mcp-kafka — swap the server, same agent code works. ai-assisted.enginnering 26 / 53

What Tools Does It Expose? list-topics → See all topics in the cluster consume-messages → Read from any topic produce-message → Write to any topic list-schemas → Browse Schema Registry create-flink-statement → Run Flink SQL get-consumer-group-lag → Monitor consumer health create-connector → Deploy connectors diagnostics → Troubleshoot errors X/Bluesky: @gamussa ai-assisted.enginnering 27 / 53

Flink SQL via MCP Ask your agent: “Create a real-time dashboard for flight delays” The agent writes and submits: INSERT INTO flight_dashboard SELECT airline_name, CASE WHEN dep_delay > 0 THEN ‘DELAYED’ ELSE ‘ON_TIME’ END AS status, CASE WHEN HOUR(dep_time) < 12 THEN ‘morning’ WHEN HOUR(dep_time) < 17 THEN ‘afternoon’ ELSE ‘evening’ END AS time_of_day, . . . X/Bluesky: @gamussa ai-assisted.enginnering 28 / 53

Meet the Brewmaster An AI agent that manages a craft brewery’s streaming platform. The setup The agent • Confluent Cloud cluster • LangChain4J + TUI4J + Anthropic • Topics for sensors, alerts, metrics • @confluentinc/mcp-confluent • Flink SQL for real-time analytics • Natural language only • Schema Registry for Avro schemas • No code, no dashboards Let’s see what happens when you talk to your data platform like a person. X/Bluesky: @gamussa ai-assisted.enginnering 29 / 53

Demo: The Brewmaster Agent X/Bluesky: @gamussa ai-assisted.enginnering 30 / 53

Part 5: Honest Trade-offs X/Bluesky: @gamussa ai-assisted.enginnering 37 / 53

What Works ✓ Schema-aware queries — agent understands Avro/Protobuf automatically ✓ Ad-hoc exploration — “show me the last 10 messages from topic X” ✓ Pipeline creation — natural language to Flink SQL ✓ Cross-system correlation — “why is consumer lag growing?” ✓ Debugging X/Bluesky: @gamussa — agent reads logs, schemas, configs in context ai-assisted.enginnering 38 / 53

What Doesn’t (Yet) ✗ High-throughput consumption — MCP is request/response, not streaming ✗ Latency guarantees — LLM reasoning adds seconds, not milliseconds ✗ Autonomous production writes — do you really want an LLM producing to orders? ✗ Complex stateful processing — Flink is better at this than any agent MCP is for orchestration and exploration, not replacing your streaming pipeline. X/Bluesky: @gamussa ai-assisted.enginnering 39 / 53

Use Case A: 3 AM Anomaly Investigation Scenario: Pager goes off. “Orders topic looks weird.” You want to: understand what’s happening, fast, without writing queries Best tool: MCP + LLM Why: ad-hoc, exploratory, cross-system correlation, nobody wants to write kafka-console-consumer at 3 AM X/Bluesky: @gamussa ai-assisted.enginnering 40 / 53

Use Case B: Continuous Temperature Alerts Scenario: Alert every time fermenter temperature drifts >2° in 10s window You want to: reliable, low-latency, stateful, deterministic detection Best tool: Flink SQL Why: sub-second latency, deterministic, well-understood, no LLM in the hot path X/Bluesky: @gamussa (the agent wrote it, but Flink runs it) ai-assisted.enginnering 41 / 53

Use Case C: “Daily operational summary” Scenario: “Every morning, summarize yesterday’s activity in a Slack message” Best tool: It depends — • If you need structured reliable output • If you want rich narrative analysis The real answer: X/Bluesky: @gamussa → Flink aggregates, agent narrates → MCP + LLM reads live data both, with each doing what it’s good at ai-assisted.enginnering 42 / 53

The Elephant in the Room: Governance Before you deploy this on Monday, three questions: 1. Who can the agent talk to? (RBAC on your MCP server is not optional) 2. What can it write? (default to read-only in prod; writes need human-in-loop) 3. Who owns the audit trail? (every tool call is an event — log it like one) These aren’t MCP’s job. They’re X/Bluesky: @gamussa your job. ai-assisted.enginnering 43 / 53

When to Use What Use Case Tool Real-time event processing Flink Persistent streaming pipelines Kafka + Flink Ad-hoc data exploration MCP + LLM Pipeline creation & debugging MCP + LLM Anomaly investigation MCP + LLM Automated alerting Flink (not an agent) “What happened at 3 AM?” MCP + LLM The right tool for the right job. Not everything needs AI. But some things really do. X/Bluesky: @gamussa ai-assisted.enginnering 45 / 53

Why This Matters We’re at an inflection point. Before MCP — AI and streaming are separate worlds. Your dashboards show data. Your agents answer questions. They don’t talk to each other. After MCP — AI agents become first-class citizens of your data platform. They can read streams, understand schemas, create pipelines, investigate incidents. The streaming platform becomes the MCP becomes the X/Bluesky: @gamussa nervous system. interface between human intent and machine execution. ai-assisted.enginnering 46 / 53

What to Do Monday Morning 1. Install the Confluent MCP server npx @confluentinc/mcp-confluent 2. Connect Claude Desktop to your dev cluster (not prod… yet) 3. Ask it questions “What topics do I have? What’s the schema for X?” 4. Build an agent 5. Come talk to me X/Bluesky: @gamussa for your own streaming use case — I’ll be around after ai-assisted.enginnering 47 / 53

Resources • MCP Spec modelcontextprotocol.io • Confluent MCP github.com/confluentinc/mcp-confluent • Brewmaster demo github.com/gAmUssA/tui-kafka-agent • Confluent Cloud ask me for $400 credit code gamov.dev/rel X/Bluesky: @gamussa ai-assisted.enginnering 48 / 53

The Real Question I Want to Leave You With Would you let your LLM write to a production Kafka topic? I’ve told you I want to hear X/Bluesky: @gamussa Who’s first? my answer: no, not without human-in-loop. yours. ai-assisted.enginnering 49 / 53

As always, have a nice day. Viktor Gamov — X/Bluesky: @gamussa

X/Bluesky: @gamussa ai-assisted.enginnering 52 / 53

Questions? I’ll be around after. Come find me. X/Bluesky: @gamussa LinkedIn: linkedin.com/in/vikgamov X/Bluesky: @gamussa ai-assisted.enginnering 51 / 53

THE MISSING PROTOCOL How MCP Bridges LLMs and Data Streams Viktor Gamov Part 1: The Problem → → Part 2: Enter MCP Part 4: Let’s Build It → → | @gamussa | Confluent Part 3: Meets Streaming Part 5: Honest Trade-offs speaking.gamov.io ai-assisted.enginnering | AI in Practice | Thu 2:15 PM