The Missing Protocol: How MCP Bridges LLMs and Data Streams

The Missing Protocol How MCP Bridges LLMs and Data Streams Viktor Gamov — Principal Developer Advocate, Confluent, an IBM Company X/Bluesky: @gamussa • linkedin.com/in/vikgamov

3:07 AM. You got paged. You asked the chatbot. It read the dashboards and the runbook: “Production is healthy.” The orders topic had been dead for 47 minutes. A chatbot answers when you ask. An agent reacts when the data moves. X/Bluesky: @gamussa Amsterdam JUG, Sept 2026 2 / 47

Who’s here? • Who has shipped something with an AI agent in the last six months? • Keep them up — who wired that agent to a real system with MCP? • Who works with Kafka, Flink, or anything event-driven? • And who has shipped agent-written code that turned out subtly wrong? X/Bluesky: @gamussa Amsterdam JUG, Sept 2026 4 / 47

Who Am I? Viktor Gamov Principal Developer Advocate at Confluent, an IBM Company Java Champion • book author Some of you were here in June, when I taught you to write streaming SQL by hand. X/Bluesky: @gamussa Amsterdam JUG, Sept 2026 5 / 47

Slides and Resources Everything from this talk — slides, links, resources gamov.dev/rel → speaking.gamov.io Scan the QR or type the short URL. X/Bluesky: @gamussa Amsterdam JUG, Sept 2026 6 / 47

Part 1: The Problem X/Bluesky: @gamussa Amsterdam JUG, Sept 2026 7 / 47

Your Agents Are Impressive. And Blind. Your agents can: • Query databases (stale by the time you ask) • Search the web (everyone else’s data) • Read your docs (written last quarter) • Call APIs (request/response, point-in-time) But they cannot see what’s happening in your systems right now. X/Bluesky: @gamussa Amsterdam JUG, Sept 2026 8 / 47

The Evolution (and the Gap) LLM

• Context
• RAG
• Agents
• MCP Every layer solves one problem. But the API is still stateless. Each call starts fresh. RAG gives context. Agents give actions. MCP gives a standard way to connect both. X/Bluesky: @gamussa Amsterdam JUG, Sept 2026 9 / 47

The Integration Nightmare Every AI tool vendor says: “Just use our SDK!” So you write custom code for: Claude API Your vector DB OpenAI API Your Kafka cluster Gemini API Your Flink jobs Llama via Ollama Your monitoring Mistral API Your internal APIs That’s N × M integrations. Does it make sense? No. It’s a nightmare. X/Bluesky: @gamussa Amsterdam JUG, Sept 2026 10 / 47

N x M Becomes One Protocol Every vendor SDK, times every system, times every model. One protocol that your platform implements once, and every agent gets it. MCP is that protocol. X/Bluesky: @gamussa Amsterdam JUG, Sept 2026 11 / 47

Part 2: Enter MCP X/Bluesky: @gamussa Amsterdam JUG, Sept 2026 12 / 47

What is MCP? Model Context Protocol — an open standard by Anthropic • A protocol, not a product • JSON-RPC based (I know, I know… JSON-RPC in 2026… but hear me out) • Client-Ser ver architecture • Language agnostic • One protocol to connect LLMs to everything X/Bluesky: @gamussa Amsterdam JUG, Sept 2026 13 / 47

MCP Architecture Host (Claude / IDE / Custom App) MCP Client MCP Server JSON-RPC → Tools Resources Prompts X/Bluesky: @gamussa Amsterdam JUG, Sept 2026 14 / 47

The Three Primitives Tools Resources Prompts Functions the LLM can call Data the LLM can read Templates the LLM can use query_topic() topic://orders debug_consumer send_message() schema://avro/user analyze_throughput create_pipeline() metrics://lag explain_schema X/Bluesky: @gamussa Amsterdam JUG, Sept 2026 15 / 47

Part 3: MCP Meets Data Streaming X/Bluesky: @gamussa Amsterdam JUG, Sept 2026 16 / 47

What If… What if your AI agent could: • Subscribe to a Kafka topic and react to events • Query a Flink materialized view in real time • Understand your Avro schemas without you explaining them • Monitor consumer lag and alert you before things break • Create new streaming pipelines from natural language X/Bluesky: @gamussa Amsterdam JUG, Sept 2026 17 / 47

MCP + Kafka Architecture AI Application (Agent + MCP Client) ↓ JSON-RPC Kafka MCP Server ↓ Streaming Platform: Kafka • Schema Registry • Flink X/Bluesky: @gamussa Amsterdam JUG, Sept 2026 18 / 47

Real Example: Anomaly Detection You: “Hey Claude, anything weird happening in the orders topic?” Claude (via MCP): I checked the orders topic. In the last 5 minutes: • Order volume dropped 73% compared to the hourly average • Consumer group payment-processor has lag of 45,000 messages • Last successful commit was 8 minutes ago This looks like the payment service is down. Want me to check the payment-service-health topic? X/Bluesky: @gamussa Amsterdam JUG, Sept 2026 19 / 47

Part 4: Let’s Build It X/Bluesky: @gamussa Amsterdam JUG, Sept 2026 20 / 47

Kafka MCP Server Config { “mcpServers”: { “confluent-kafka”: { “command”: “npx”, “args”: [ “-y”, “@confluentinc/mcp-confluent”, “-e”, “.env” ] } } } X/Bluesky: @gamussa Amsterdam JUG, Sept 2026 21 / 47

What Tools Does It Expose? list-topics → See all topics in the cluster consume-messages → Read from any topic produce-message → Write to any topic list-schemas → Browse Schema Registry create-flink-statement → Run Flink SQL get-consumer-group-lag → Monitor consumer health create-connector → Deploy connectors diagnostics → Troubleshoot errors X/Bluesky: @gamussa Amsterdam JUG, Sept 2026 22 / 47

Meet the Brewmaster An AI agent that manages a craft brewery’s streaming platform. The setup The agent • Confluent Cloud cluster • Topics for sensors, alerts, metrics • LangChain4j agentic + skills, TamboUI, Anthropic • Flink SQL for real-time analytics • @confluentinc/mcp-confluent • Schema Registry for Avro schemas • A supervisor and four specialists, each with scoped tools • No code, no dashboards Let’s see what happens when you talk to your data platform like a person. X/Bluesky: @gamussa Amsterdam JUG, Sept 2026 23 / 47

Demo: The Brewmaster Agent X/Bluesky: @gamussa Amsterdam JUG, Sept 2026 24 / 47

Part 5: Access Is Not Competence X/Bluesky: @gamussa Amsterdam JUG, Sept 2026 31 / 47

Before There Was a Critic Same model, same MCP connection, one agent with every tool. No skill, no critic. I asked for the Act 3 job. First draft: INSERT INTO brewery-alerts SELECT fermenter_id, style, AVG(temperature) AS avg_temp, TUMBLE_START(event_time, INTERVAL ‘10’ SECOND) AS window_start FROM brewery-sensors GROUP BY TUMBLE(event_time, INTERVAL ‘10’ SECOND), fermenter_id, style HAVING AVG(temperature) > 22 X/Bluesky: @gamussa Amsterdam JUG, Sept 2026 32 / 47

Find the Traps — five in six lines 1. GROUP BY TUMBLE(…) — Apache Flink syntax. Confluent Cloud only takes the window TVF: TUMBLE(TABLE t, DESCRIPTOR(…)) 2. brewery-sensors and brewery-alerts unquoted — the dash is a parse error without backticks 3. event_time, fermenter_id, style, temperature — none exist. DESCRIBE says $rowtime, sensor_id, beer_style, temperature_c 4. AVG(temperature) > 22 — one threshold for three styles. A Pilsner at 22 is already ruined 5. Four columns into a nine-column sink; window_start is a STRING there. The critic failed a real draft on this X/Bluesky: @gamussa Amsterdam JUG, Sept 2026 Five traps. It never ran DESCRIBE. I paid for the good model. 33 / 47

Twenty-Two Documented Ways to Get This Wrong Habits Confluent Cloud rejects: Habits that cost you data: • GROUP BY TUMBLE(…) instead of the window TVF • DROP TABLE deletes the Kafka topic and its data • CREATE TABLE … WITH (‘c onnector’ = ‘kafka’) • ‘value.format’ = ‘json’ — needs json-registry • No savepoints — delete a statement, lose its state • PROCTIME() — not supported • CREATE CATALOG — the catalog is your env • SET ‘e xecution.checkpointing.*’ — not yours to set X/Bluesky: @gamussa I did not write this list. Confluent ships it inside a Skill. Amsterdam JUG, Sept 2026 34 / 47

Three Things That Do Not Fix This “Use a bigger model.” The model that knew Pilsner temperatures wrote that statement. “Give it more context.” Paste the Apache Flink docs in and you just fed it the wrong dialect. “Write a better prompt.” Now it lives in one clipboard; nobody can version or review it. The agent did not need more knowledge. It needed a procedure. And a critic. MCP gave the agent access. Nobody gave it judgment. X/Bluesky: @gamussa Amsterdam JUG, Sept 2026 35 / 47

What Is an Agent Skill? And the Critic A folder with a markdown file in it: SKILL.md (the rules) • references/ (dialect traps, reserved words, troubleshooting) One line per skill in the prompt. activate_skill and read_skill_resource fetch the rest. In the demo: sqlAuthor activates the skill, runs DESCRIBE, drafts a typed FlinkStatement. sqlCritic re-reads both and returns a typed CriticResult: successful, feedback, statement. The loop exits on a boolean, not a parsed paragraph. Three rounds, max. The submitter holds the write tools and only ever sees an approved statement. In rehearsal the critic failed the first draft. That is the feature. X/Bluesky: @gamussa Amsterdam JUG, Sept 2026 36 / 47

MCP vs Skills MCP Agent Skills @confluentinc/mcp-confluent know-how, in markdown • Gives the agent access • Gives the agent know-how • A ser ver speaking JSON-RPC • A folder, vendored and pinned like a dependency • Answers “what can I touch?” • Answers “how do I do this right?” github.com/confluentinc/mcp-confluent github.com/confluentinc/agent-skills MCP is the API. A Skill is the code review. X/Bluesky: @gamussa Amsterdam JUG, Sept 2026 37 / 47

The Agent Moves Into the Stream Yes, MCP is request/response, so the agent is moving into the stream. Streaming Agents (Confluent Cloud, GA Q2 2026) run event-driven agents on Flink and Kafka and call tools over MCP. Apache Flink Agents 0.3 (June 2026) is the open-source Flink sub-project, with Agent Skills support in Java and Python. The Real-Time Context Engine (GA) serves fresh, governed context to any agent through MCP. Access keeps getting easier, and the judgment is still yours to write. X/Bluesky: @gamussa Amsterdam JUG, Sept 2026 38 / 47

Part 6: Honest Trade-offs X/Bluesky: @gamussa Amsterdam JUG, Sept 2026 39 / 47

What Works ✓ Schema-aware queries ✓ Ad-hoc exploration ✓ Pipeline creation — agent understands Avro/Protobuf automatically — “show me the last 10 messages from topic X” — natural language to Flink SQL ✓ Cross-system correlation — “why is consumer lag growing?” ✓ Debugging X/Bluesky: @gamussa — agent reads logs, schemas, configs in context Amsterdam JUG, Sept 2026 40 / 47

What Doesn’t (Yet) ✗ High-throughput consumption — MCP is request/response, not streaming ✗ Latency guarantees — LLM reasoning adds seconds, not milliseconds ✗ Autonomous production writes — do you really want an LLM producing to orders? ✗ Complex stateful processing — Flink is better at this than any agent Skills are nondeterministic (Confluent’s own README). A skill raises the floor; it does not remove the review. X/Bluesky: @gamussa Amsterdam JUG, Sept 2026 41 / 47

When to Use What Use Case Real-time event processing Persistent streaming pipelines Ad-hoc data exploration Tool Flink Kafka + Flink MCP + LLM Pipeline creation & debugging Anomaly investigation Automated alerting “What happened at 3 AM?” MCP + LLM MCP + LLM Flink (not an agent) MCP + LLM The right tool for the right job. Not everything needs AI. But some things really do. X/Bluesky: @gamussa Amsterdam JUG, Sept 2026 42 / 47

Why This Matters We’re at an inflection point. Before MCP — AI and streaming are separate worlds. Your dashboards show data. Your agents answer questions. They don’t talk to each other. After MCP — AI agents become first-class citizens of your data platform. They can read streams, understand schemas, create pipelines, investigate incidents. The streaming platform becomes the nervous system. MCP is the interface, and Skills are how you train what comes through it. X/Bluesky: @gamussa Amsterdam JUG, Sept 2026 43 / 47

What to Do Monday Morning 1. Install the Confluent MCP server npx @confluentinc/mcp-confluent 2. Point it at your dev cluster (not prod… yet) 3. Ask it questions “What topics do I have? What’s the schema for X?” 4. Add langchain4j-skills and confluent-cloud-flink-sql to your agent. Put a critic in front of create-flink-statement. 5. Write down the thing your team gets wrong, as a skill — then come find me X/Bluesky: @gamussa Amsterdam JUG, Sept 2026 44 / 47

Resources • MCP Spec modelcontextprotocol.io • Confluent MCP github.com/confluentinc/mcp-confluent • Agent Skills github.com/confluentinc/agent-skills • Skills spec agentskills.io/specification gamov.dev/rel X/Bluesky: @gamussa Amsterdam JUG, Sept 2026 45 / 47

The Real Question I Want to Leave You With Would you let your LLM write to a production Kafka topic? I want to hear yours. Who’s first? X/Bluesky: @gamussa Amsterdam JUG, Sept 2026 46 / 47

As always, have a nice day. Viktor Gamov — X/Bluesky: @gamussa