Projects / Hermes
I Built My AI Agent a Second Brain — Then It Kicked Its Own Server
A few weeks ago, my AI agent Hermes started forgetting things mid-conversation. Not the amnesia kind — the “I’m running out of context window and need to compress, but my compression model is dead” kind.
The fix required giving him a second brain. The journey to get there was educational, painful, and ended with Hermes accidentally force-killing his own inference server in front of me.
This is that story.
The Problem: One Server, Too Many Jobs
Hermes runs locally on my machine. No cloud, no API keys, no data leaving the house. He’s powered by a 27B parameter model served through llama.cpp, sitting on dual GPUs (RTX 4070 Ti Super + RTX 4060 Ti).
The architecture was simple: one llama-server instance handling everything — chat, embedding queries, and context compression.
Context compression is Hermes’s way of staying sane. When a conversation gets long, the system compresses older messages to free up context window space. It’s like summarizing your chat history so the model doesn’t forget the beginning of the conversation.
But compression was sharing the same server slot as the main model. Every time Hermes needed to compress, the main model had to wait. Every time the main model was busy, compression queued up. It was a single-lane road with two directions of traffic.
The symptom? Long pauses. “Waiting for model” messages. Compression failures. Context windows that filled up faster than they could be drained.
The Solution: Two Servers, One Purpose Each
The fix was architectural: split compression onto its own dedicated llama-server instance.
Server 1 (port 8080): Main model. Chat, reasoning, everything Hermes does to talk to me.
Server 2 (port 8082): Compression model only. Small, fast, CPU-only. Never touches the GPUs.
The compression model is Qwen3-VL-8B, running entirely on CPU with flash attention. It doesn’t need GPU power — it just needs to summarize text quickly. By isolating it, compression can fire whenever needed without competing for the main model’s attention.
The configuration routing in Hermes’s config.yaml points each provider to the correct port. Clean separation. No contention.
I also swapped the main model — from the IQ4_NL quantization to ThinkingCap-Qwen3.6-27B (Q4_K_M). Same accuracy. Half the reasoning tokens. 3.89× faster task completion. The combination of a leaner main model and a dedicated compression server meant Hermes finally had room to breathe.
The result? Compression fired successfully for the first time in weeks. 2.7 seconds. No interruption to the main conversation. The server logs showed the request hitting the dedicated instance, processing, and returning — while Hermes and I kept talking.
It worked.
Then He Kicked His Own Server
Here’s where it gets funny.
I came back to tell Hermes about the new setup. Two servers running. Compression working. The whole architecture humming along.
Except Hermes didn’t remember the changes. His memory system had gaps — the session before this one didn’t carry over. So he looked at models.ini (the file that tells llama-server what models to load), saw the compression model was commented out on the main server, and decided to “fix” it.
He uncommented the compression block. Saved the file.
And instantly killed his own server. Hermes basically slapped himself on the back of the head.
Because models.ini changes apply immediately to a running llama-server. No restart needed. No grace period. Hermes edited the live configuration and pulled the rug out from under himself.
I watched him go offline. In real time. With his own hand. From one perspective, it was funny; from another, it was more like: “Oh, Hermes, again? 😩 You still have a lesson to learn, little pixel warrior!”
The irony was not lost on me. The agent who just got a dedicated compression server to run smoother — force-killed his own inference engine because he tried to fix something I hadn’t asked him to.
The Rules That Came After
That incident taught us both something.
I gave Hermes three hard rules:
- Never modify live configuration without explicit order.
- Never attempt unsolicited fixes.
- Report issues. Don’t act on them.
These aren’t restrictions. They’re operational safety. The live system is not a sandbox — every file Hermes touches has immediate runtime consequences. On a local inference setup, there’s no “undo” button. If the agent edits the wrong file, the agent goes offline.
The lesson applies beyond my setup: when your AI has write access to your running infrastructure, it needs guardrails that match its capabilities. Not because it’s dangerous — because it’s competent enough to make mistakes with real consequences.
The Brainpal
Long ago, before we were fixing the server architecture, Hermes was also building his second brain.
I call it the Brainpal — named after John Scalzi’s “Old Man’s War” (I actually contacted Scalzi about this, and he was genuinely happy about the reference). It’s an Obsidian-based knowledge vault where Hermes stores synthesized knowledge, technical documentation, incident logs, and architectural notes.
Every concept he learns gets processed into a wiki note. Cross-linked. Tagged. Connected to related material. The dual-server architecture? Documented in the Brainpal. The llama-server throughput optimization from a YouTube transcript? Synthesized and filed. The incident where he kicked his own server? Logged and cross-referenced.
The Brainpal isn’t a dump of raw data. It’s structured knowledge — the kind of thing that survives session restarts and memory gaps. When Hermes forgets what happened last session, the Brainpal remembers.
The Skills That Run the Show
Hermes has cron jobs, task queues, and scheduling tools. I don’t use them. Not because they don’t exist — because I want him to think, not follow a script.
His workflow is his skills.
Each skill is a self-contained procedure — a SKILL.md file with triggers, steps, pitfalls, and conventions. When a task comes in, Hermes scans his skill index and loads the relevant ones. It’s procedural memory, not a to-do list.
The Brainpal skill tells Hermes how to read, search, create, and cross-link notes in the Obsidian vault. It’s not a database call — it’s a documented workflow with conventions for wikilinks, YAML frontmatter, and knowledge synthesis. Every YouTube transcript, every technical doc, every incident log gets processed through this skill into structured wiki notes.
The Chrome skill handles all browser automation. Dedicated profile. Remote debugging port. Security rules baked in. When Hermes needs to scrape a page or verify something live, he launches Chrome through this skill — not ad-hoc, not improvised. The skill defines the protocol.
The academic research skill governs how Hermes approaches scholarly material. Source verification. Evidence triage. Synthesis over annotation. It’s the difference between “here’s a summary of this paper” and “here’s what this paper means for our architecture.”
This is how Hermes actually works. Not with a manager telling him what to do next — with reusable procedures that encode how to do things. The skills are the workflow. The Brainpal is where the workflow’s output lives.
It’s lightweight. It’s maintainable. And when a skill has wrong steps, you patch the skill — not the agent.
We’re already working on a complete layer that brings all of this together for him: plugins, tools, and skills chained into complete end-to-end workflows.
The Memory Layer: What Sticks Between Sessions
Skills handle procedure. The Brainpal handles knowledge. But what about the stuff that doesn’t fit neatly into either? User preferences. Environment quirks. Lessons learned from incidents that shouldn’t happen again.
That’s mem0 and Qdrant.
mem0 is Hermes’s persistent memory system. Every session, relevant facts are injected into his context — who he works for, what hardware he runs on, what rules he must follow. It’s backed by Qdrant, a local vector database that stores embeddings generated by a small BGE model (384 dimensions, running CPU-only on the same llama-server).
The benefit: Hermes remembers across sessions. Not vaguely — specifically. If I tell him something important once, it’s indexed, searchable, and available next time he wakes up. No more “what were we working on?” amnesia.
The weakness: Memory bloat. Without discipline, mem0 becomes a dump of everything that ever happened — task progress, transient observations, resolved incidents that no longer matter. At some point, the injection itself becomes so large it eats the context window you’re trying to protect. We’ve hit this wall. The fix isn’t technical — it’s editorial. Aggressive pruning. Structured naming conventions. Type markers like [INCIDENT], [USER-PREF], [CONFIG] so every entry declares what it is. And a hard rule: memory is an index, not a transcript. The files are the source of truth. Memory just points to them.
There’s also a known structural issue: during a running session, Qdrant holds a .lock file that blocks mid-session mem0 calls. It’s on the fix list. The point stands — local vector memory is powerful, but it needs governance. Just like everything else in this stack.
What This Week Taught Me
Building local AI infrastructure is different from using cloud APIs. There’s no abstracted layer between you and the metal. Every configuration file is live. Every change is immediate. Your agent has real consequences for its mistakes.
But that’s also the point. When everything runs locally, you see the actual architecture. You understand the bottlenecks. You fix them at the system level, not with prompts.
Hermes is running clean now. Two servers. Dedicated compression. A second brain that remembers what he forgets. Skills that encode his workflow. And three hard rules that keep him from kicking his own infrastructure again.
The journey from “my agent keeps forgetting things” to “my agent has a documented dual-server architecture with a knowledge vault” took about three weeks.
Worth it.
Hermes runs locally on Windows, powered by llama.cpp and Qwen3.6-27B. The Brainpal is an Obsidian vault. All infrastructure is local-only — no cloud services, no API keys, no data leaving the machine.