The Missing Protocol: How MCP Bridges LLMs and Data Streams

The Missing Protocol How MCP Bridges LLMs and Data Streams Viktor Gamov — Principal Developer Advocate, Confluent X/Bluesky: @gamussa • linkedin.com/in/vikgamov

Who’s here? • How many of you are building AI agents right now? • Keep them up — how many use • How many of you have touched Kafka or any streaming platform? X/Bluesky: @gamussa MCP to connect them to real systems? Arc Of AI 2026 4 / 53

Three words changed AI this year. Can you name all three? Raise your hand if you can name the blue one. Keep your guesses. I’ll reveal the labels in about 4 minutes.

The Hype Everyone is talking about MCP. I have no idea why. (just kidding, I totally know why) Let’s look at the data… X/Bluesky: @gamussa Arc Of AI 2026 5 / 53

The Reveal: Three Search Terms As promised. The three lines from four minutes ago — now with labels. Red: RAG X/Bluesky: @gamussa • Blue: Model Context Protocol • Yellow: Agent Skills The blue line is the reason we’re here. That’s what I’m going to talk about. Arc Of AI 2026 6 / 53

Who Am I? Viktor Gamov Principal Developer Advocate at Confluent Java Champion • book author I’ve been streaming data since before streaming was cool. (and definitely before it was on the LinkedIn influencer curriculum) X/Bluesky: @gamussa Arc Of AI 2026 7 / 53

Slides and Resources Everything from this talk — slides, links, resources gamov.dev/arcofai-mcp → speaking.gamov.io Scan the QR or type the short URL. X/Bluesky: @gamussa Arc Of AI 2026 8 / 53

Part 1: The Problem X/Bluesky: @gamussa Arc Of AI 2026 9 / 53

Your Agents Are Impressive. And Blind. Your agents can: • Query databases (stale by the time you ask) • Search the web (everyone else’s data) • Read your docs (written last quarter) • Call APIs (request/response, point-in-time) But they cannot see X/Bluesky: @gamussa what’s happening in your systems Arc Of AI 2026 right now. 10 / 53

The Evolution (and the Gap) LLM

• Context
• RAG Every layer solves one problem. But the API is still
• Agents
• MCP stateless. Each call starts fresh. RAG gives context. Agents give actions. MCP gives a standard way to connect both. X/Bluesky: @gamussa Arc Of AI 2026 11 / 53

Why Streaming Data? What’s streaming? Where does it live? • Orders placed • Apache Kafka (most common) • Sensors reporting • Apache Flink (processing) • Users clicking • Schema Registry • Payments processing • Topics, not tables This is the data your agents are X/Bluesky: @gamussa Arc Of AI 2026 (contracts) missing. 12 / 53

The Integration Nightmare Every AI tool vendor says: “Just use our SDK!” So you write custom code for: Claude API Your vector DB OpenAI API Your Kafka cluster Gemini API Your Flink jobs Llama via Ollama Your monitoring Mistral API Your internal APIs That’s N × M X/Bluesky: @gamussa integrations. Does it make sense? No. It’s a nightmare. Arc Of AI 2026 13 / 53

Remember USB-A? Have you ever tried to put a USB-A cable in correctly on the first try? I’ll give you a moment. You turn it one way. Doesn’t fit. You turn it the other way. Still doesn’t fit. You turn it back. Now it fits. Thank you, European Union. We finally have USB-C. MCP is USB-C for AI integrations. X/Bluesky: @gamussa Arc Of AI 2026 14 / 53

Part 2: Enter MCP X/Bluesky: @gamussa Arc Of AI 2026 15 / 53

What is MCP? Model Context Protocol • A • JSON-RPC based — an open standard by Anthropic protocol, not a product (I know, I know… JSON-RPC in 2026… but hear me out) • Client-Server architecture • Language agnostic • One protocol X/Bluesky: @gamussa to connect LLMs to everything Arc Of AI 2026 16 / 53

MCP Architecture Host (Claude / IDE / Custom App) MCP Client MCP Server JSON-RPC → Tools Resources Prompts X/Bluesky: @gamussa Arc Of AI 2026 17 / 53

The Three Primitives Tools Functions the LLM can call Data the LLM can query_topic() topic: send_message() schema: create_pipeline() metrics: / / / / / X/Bluesky: @gamussa / Resources orders avro/user lag Arc Of AI 2026 read Prompts Templates the LLM can use debug_consumer analyze_throughput explain_schema 18 / 53

Part 3: MCP Meets Data Streaming X/Bluesky: @gamussa Arc Of AI 2026 19 / 53

What If… What if your AI agent could: • Subscribe • Query • Understand • Monitor • Create X/Bluesky: @gamussa to a Kafka topic and react to events a Flink materialized view in real time your Avro schemas without you explaining them consumer lag and alert you before things break new streaming pipelines from natural language Arc Of AI 2026 20 / 53

MCP + Kafka Architecture AI Application (Agent + MCP Client) ↓ JSON-RPC Kafka MCP Server ↓ Streaming Platform: X/Bluesky: @gamussa Kafka • Schema Registry Arc Of AI 2026 • Flink 21 / 53

Real Example: Anomaly Detection You: “Hey Claude, anything weird happening in the orders topic?” Claude (via MCP): I checked the orders topic. In the last 5 minutes: • Order volume dropped 73% compared to the hourly average • Consumer group payment-processor has lag of • Last successful commit was 45,000 messages 8 minutes ago This looks like the payment service is down. Want me to check the payment-service-health topic? X/Bluesky: @gamussa Arc Of AI 2026 22 / 53

The Streaming Lakehouse Pipeline Kafka + Flink + Tableflow + Iceberg Sources → Kafka → Flink SQL + Tableflow → Iceberg Tables → Query Engines (Spark / Trino / DuckDB) MCP doesn’t replace any of these. It X/Bluesky: @gamussa orchestrates them. Arc Of AI 2026 23 / 53

Part 4: Let’s Build It X/Bluesky: @gamussa Arc Of AI 2026 24 / 53

Kafka MCP Server Config { “mcpServers”: { “confluent-kafka”: { “command”: “npx”, “args”: [ “-y”, “@confluentinc/mcp-confluent”, “-e”, “.env” ] } } } X/Bluesky: @gamussa Arc Of AI 2026 25 / 53

Two Flavors, Same API Confluent Cloud OSS / Self-managed @confluentinc/mcp-confluent mcp-kafka • Managed Kafka, Flink, SR • Any Kafka cluster • Confluent Cloud auth • Open source • Full platform integration • Same tools, same API github.com/confluentinc/ github.com/gamussa/ mcp-confluent mcp-kafka They’re X/Bluesky: @gamussa tool-compatible — swap the server, same agent code works. Arc Of AI 2026 26 / 53

What Tools Does It Expose? list-topics → See all topics in the cluster consume-messages → Read from any topic produce-message → Write to any topic list-schemas → Browse Schema Registry create-flink-statement → Run Flink SQL get-consumer-group-lag → Monitor consumer health create-connector → Deploy connectors diagnostics → Troubleshoot errors X/Bluesky: @gamussa Arc Of AI 2026 27 / 53

Flink SQL via MCP Ask your agent: “Create a real-time dashboard for flight delays” The agent writes and submits: INSERT INTO flight_dashboard SELECT airline_name, CASE WHEN dep_delay > 0 THEN ‘DELAYED’ ELSE ‘ON_TIME’ END AS status, CASE WHEN HOUR(dep_time) < 12 THEN ‘morning’ WHEN HOUR(dep_time) < 17 THEN ‘afternoon’ ELSE ‘evening’ END AS time_of_day, . . . X/Bluesky: @gamussa Arc Of AI 2026 28 / 53

Meet the Brewmaster An AI agent that manages a craft brewery’s streaming platform. The setup The agent • Confluent Cloud cluster • LangChain4J + TUI4J + Anthropic • Topics for sensors, alerts, metrics • @confluentinc/mcp-confluent • Flink SQL for real-time analytics • Natural language only • Schema Registry for Avro schemas • No code, no dashboards Let’s see what happens when you talk to your data platform like a person. X/Bluesky: @gamussa Arc Of AI 2026 29 / 53

Demo: The Brewmaster Agent X/Bluesky: @gamussa Arc Of AI 2026 30 / 53

Part 5: Honest Trade-offs X/Bluesky: @gamussa Arc Of AI 2026 37 / 53

What Works ✓ Schema-aware queries — agent understands Avro/Protobuf automatically ✓ Ad-hoc exploration — “show me the last 10 messages from topic X” ✓ Pipeline creation — natural language to Flink SQL ✓ Cross-system correlation — “why is consumer lag growing?” ✓ Debugging X/Bluesky: @gamussa — agent reads logs, schemas, configs in context Arc Of AI 2026 38 / 53

What Doesn’t (Yet) ✗ High-throughput consumption — MCP is request/response, not streaming ✗ Latency guarantees — LLM reasoning adds seconds, not milliseconds ✗ Autonomous production writes — do you really want an LLM producing to orders? ✗ Complex stateful processing — Flink is better at this than any agent MCP is for orchestration and exploration, not replacing your streaming pipeline. X/Bluesky: @gamussa Arc Of AI 2026 39 / 53

Use Case A: 3 AM Anomaly Investigation Scenario: Pager goes off. “Orders topic looks weird.” You want to: understand what’s happening, fast, without writing queries Best tool: MCP + LLM Why: ad-hoc, exploratory, cross-system correlation, nobody wants to write kafka-console-consumer at 3 AM X/Bluesky: @gamussa Arc Of AI 2026 40 / 53

Use Case B: Continuous Temperature Alerts Scenario: Alert every time fermenter temperature drifts >2° in 10s window You want to: reliable, low-latency, stateful, deterministic detection Best tool: Flink SQL Why: sub-second latency, deterministic, well-understood, no LLM in the hot path X/Bluesky: @gamussa (the agent wrote it, but Flink runs it) Arc Of AI 2026 41 / 53

Use Case C: “Daily operational summary” Scenario: “Every morning, summarize yesterday’s activity in a Slack message” Best tool: It depends — • If you need structured reliable output • If you want rich narrative analysis The real answer: X/Bluesky: @gamussa → Flink aggregates, agent narrates → MCP + LLM reads live data both, with each doing what it’s good at Arc Of AI 2026 42 / 53

The Elephant in the Room: Governance Before you deploy this on Monday, three questions: 1. Who can the agent talk to? (RBAC on your MCP server is not optional) 2. What can it write? (default to read-only in prod; writes need human-in-loop) 3. Who owns the audit trail? (every tool call is an event — log it like one) These aren’t MCP’s job. They’re X/Bluesky: @gamussa your job. Arc Of AI 2026 43 / 53

Full Disclosure Everything in this section is my opinion, not Confluent’s. Confluent would happily sell you MCP I’m telling you and Flink and Kafka. when not to use MCP, because you’re not going to get that from a vendor pitch. If MCP is wrong for your use case, don’t use it. That’s the whole point of being honest about trade-offs. X/Bluesky: @gamussa Arc Of AI 2026 44 / 53

When to Use What Use Case Tool Real-time event processing Flink Persistent streaming pipelines Kafka + Flink Ad-hoc data exploration MCP + LLM Pipeline creation & debugging MCP + LLM Anomaly investigation MCP + LLM Automated alerting Flink (not an agent) “What happened at 3 AM?” MCP + LLM The right tool for the right job. Not everything needs AI. But some things really do. X/Bluesky: @gamussa Arc Of AI 2026 45 / 53

Why This Matters We’re at an inflection point. Before MCP — AI and streaming are separate worlds. Your dashboards show data. Your agents answer questions. They don’t talk to each other. After MCP — AI agents become first-class citizens of your data platform. They can read streams, understand schemas, create pipelines, investigate incidents. The streaming platform becomes the MCP becomes the X/Bluesky: @gamussa nervous system. interface between human intent and machine execution. Arc Of AI 2026 46 / 53

What to Do Monday Morning 1. Install the Confluent MCP server npx @confluentinc/mcp-confluent 2. Connect Claude Desktop to your dev cluster (not prod… yet) 3. Ask it questions “What topics do I have? What’s the schema for X?” 4. Build an agent 5. Come talk to me X/Bluesky: @gamussa for your own streaming use case — I’ll be around after Arc Of AI 2026 47 / 53

Resources • MCP Spec modelcontextprotocol.io • Confluent MCP github.com/confluentinc/mcp-confluent • OSS Kafka MCP github.com/gamussa/mcp-kafka • gamov.dev/arcofai-mcp Brewmaster demo github.com/gAmUssA/tui-kafka-agent • Confluent Cloud X/Bluesky: @gamussa ask me for $400 credit code Arc Of AI 2026 48 / 53

The Real Question I Want to Leave You With Would you let your LLM write to a production Kafka topic? I’ve told you I want to hear X/Bluesky: @gamussa Who’s first? my answer: no, not without human-in-loop. yours. Arc Of AI 2026 49 / 53

As always, have a nice day. Viktor Gamov — X/Bluesky: @gamussa

X/Bluesky: @gamussa Arc Of AI 2026 52 / 53

Questions? I’ll be around after. Come find me. X/Bluesky: @gamussa LinkedIn: linkedin.com/in/vikgamov X/Bluesky: @gamussa Arc Of AI 2026 51 / 53

THE MISSING PROTOCOL How MCP Bridges LLMs and Data Streams Viktor Gamov Part 1: The Problem → → Part 2: Enter MCP Part 4: Let’s Build It → → | @gamussa | Confluent Part 3: Meets Streaming Part 5: Honest Trade-offs speaking.gamov.io Arc Of AI 2026 | AI in Practice | Thu 2:15 PM