Projects

I Taught My AI Agent to Build Its Own Tool Server — So It Didn't Have to Ask Me Anymore

Seraph / Signalane

Hermes Masterclass

A few weeks ago, Hermes — my local AI agent — needed more tools. Not the kind you buy, the kind you build: tools that could talk to his memory system, search his knowledge vault, check system health, and publish articles to my website.

The problem wasn’t capability. Hermes was already running fine. The problem was architecture.

Every tool Hermes uses is loaded into his context at session start. As the toolset grew, so did the token overhead. Each session was getting heavier, slower, more expensive in compute. I needed a way for Hermes to have access to tools without carrying them all in his head.

The answer was the Model Context Protocol — MCP. And the journey to build it was a masterclass in learning something technical from scratch and applying it immediately.

The Lesson: What Is MCP?

The Model Context Protocol, developed by Anthropic, is a standard that lets AI models connect to external tool servers at runtime. Instead of baking every tool into the model’s context, you run a separate server that exposes tools over a protocol. The model calls what it needs, when it needs it.

Think of it like this: before MCP, Hermes had every tool loaded in his working memory like a Swiss Army knife — always present, always taking up space. With MCP, he has a workshop he can walk into whenever he needs something.

The protocol supports two transport modes:

• stdio — the server runs as a subprocess, communicating over standard input/output
• Streamable HTTP — the server runs as an HTTP endpoint, the model connects over the network

For my setup on Windows, streamable HTTP was the obvious choice. Hermes runs locally, the MCP server runs locally, and they talk over 127.0.0.1:8090. No cloud, no API keys, no data leaving the machine.

The Course: Learning MCP from Documentation

Hermes didn’t know MCP when we started. Neither did I, honestly. So we learned it together.

The learning material came from three sources:

1. The MCP specification itself — the protocol definition that explains how servers expose tools, resources, and prompts, how clients discover them, and how calls are structured. This was the foundation. Without understanding the protocol handshake, tool schemas, and result envelopes, nothing else works.

2. Python MCP SDK patterns — the mcp package on PyPI that provides the building blocks: server scaffolding, tool decorators, resource handlers, and transport configuration. This is where the rubber meets the road. Reading the SDK docs told us how to structure a server that actually responds to MCP requests.

3. Architecture patterns from the Hermes codebase itself — Hermes already had tools, skills, and a memory system. The question wasn’t “what tools do we need?” but “how do we expose what already exists through MCP?” This was the hardest part — mapping existing capability to the protocol’s expectations.

The learning wasn’t abstract. Every concept was immediately tested against the implementation. “The spec says tools need input schemas?” — okay, let’s define them. “Resources need URIs?” — let’s structure them. It was learning by building, which is the only way I learn anything technical.

The Build: Five Phases, One Server

We followed a five-phase plan. Not because plans are nice — because building MCP servers without a plan is how you end up with a server that works for one thing and breaks for everything else.

Phase 1: Foundation. Python virtual environment, project structure, the mcp package, HTTP transport on port 8090. The server skeleton that responds to initialization. Boring but critical — if this doesn’t work, nothing else matters.

Phase 2: MVP Tools. Three tools that proved the concept:

• brainpal_search — semantic search across Hermes’s Obsidian knowledge vault
• system_health — GPU, Qdrant, llama-server health checks
• qdrant_query — direct vector database queries against the mem0 memory store

These weren’t placeholder tools. They were the tools Hermes actually needed. If the server couldn’t do something useful from day one, it wasn’t worth building.

Phase 3: Registration and Testing. Starting the server, registering it with Hermes’s config, testing tool calls through the actual runtime. This is where most MCP projects fail — the server works in isolation but the integration breaks. We had to debug path validation, result envelope formatting, and timeout handling.

Phase 4: The Full Toolset. Once the foundation was solid, we expanded to 17 tools:

• Knowledge management: brainpal_read_note, brainpal_write_note, knowledge_search
• Memory operations: memory_search, qdrant_query, qdrant_upsert, qdrant_delete
• System maintenance: daily_maintenance, service_restart, log_tail
• Workflow: scope_list, scope_read, scope_create, scope_complete
• Content: website_publish
• Skill management: skill_index, skill_load, promote_to_skill, conflict_detection

Every tool has bounds — max file sizes, result limits, path allowlists. The config.yaml defines these explicitly. No tool runs without constraints.

Phase 5: Verification. 62 automated tests covering tool discovery, resource operations, prompt handling, and end-to-end streamable HTTP calls. The server doesn’t ship until the tests pass.

The Architecture: Why It Matters

The final architecture has three layers:

The MCP Server (port 8090) — A Python process that exposes tools over streamable HTTP. It has its own config, its own logging, its own bounds. It doesn’t know about Hermes — it just responds to MCP requests.

The Hermes Runtime — The agent that connects to the MCP server when it needs a tool. The tool definitions are lightweight — just names and schemas. The actual implementation lives in the server. This dramatically reduces token overhead per session.

The Configuration Layer — A single config.yaml that defines server bounds, tool-specific limits, path allowlists, and Qdrant connection details. Change a limit? Edit the config, not the code.

The key insight: MCP servers should be dumb. They respond to requests. They don’t make decisions. The agent decides what to call; the server just executes. This separation is what makes the protocol useful — you can swap servers without changing the agent, and swap agents without changing the server.

What Went Wrong

Two things.

First: the initial server structure didn’t account for path validation. Hermes could theoretically access any file path through the MCP tools. The fix was an explicit allowlist in config.yaml — the server validates every path against it before executing. This wasn’t an afterthought; it was a design requirement we forgot to implement first. Lesson: security bounds go in before tools, not after.

Second: result envelope formatting. The MCP protocol expects a specific response structure. Our early tools returned raw data, which the protocol rejected. Every tool now wraps its result in a standard envelope — success status, data payload, error message. Consistent formatting across all 17 tools.

Both mistakes were avoidable. Both were caught before they became problems. That’s the value of the five-phase approach — each phase has a verification gate, and failures surface early.

The Operational Reality

The server runs on the same Windows machine as Hermes. 64GB RAM, dual GPUs. The MCP server itself is lightweight — it doesn’t need GPU access, it doesn’t hold models in memory. It’s a process that responds to HTTP requests and talks to Qdrant, the filesystem, and system utilities.

Resource usage is minimal. The server starts in under 2 seconds. Tool calls return in milliseconds for memory operations, seconds for file operations. The bottleneck is never the MCP layer — it’s the underlying operation (searching a vault, querying a vector database, reading a file).

The config-driven approach means we can enable or disable tools without touching code. Want to turn off website_publish? Flip a boolean in config.yaml. Want to increase the max search results? Change a number. The server reloads bounds on restart.

What This Teaches About Local AI Infrastructure

Building MCP for a local agent stack is fundamentally different from using cloud APIs. There’s no managed service handling the protocol for you. You’re wiring the transport layer, defining the schemas, implementing the tools, and testing the integration yourself.

But that’s also the point. When you build it, you understand it. You know why each tool has bounds. You know how the protocol handshake works. You know what happens when the server crashes mid-call.

The MCP server gave Hermes something he didn’t have before: access to tools without the token tax. His sessions are lighter. His context window has more room for actual reasoning instead of tool definitions. And when we need a new tool, we add it to the server — not to his system prompt.

The Next Step

The server works. The tests pass. Hermes uses it.

The next iteration is a remote MCP server on Ubuntu for compute-heavy tools that shouldn’t run on the Windows host. The local server stays for filesystem and system operations. The split makes operational sense — isolate what needs isolation, keep what needs to be local.

But that’s a story for next time.

Hermes runs locally on Windows, powered by llama.cpp and Qwen3.6-27B. The MCP server runs on port 8090 over streamable HTTP. All infrastructure is local-only — no cloud services, no API keys, no data leaving the machine.