Project

From Reflex Arc to Brain

Seraph / Signalane

The loop works.

LLM thinks → calls a tool → your code executes it → result goes back → repeat until the task is done.

Everyone demos this. It’s the “hello world” of agentic AI. But after running my own agent 24/7 for weeks, I hit the wall that the 25-line tutorials skip:

The loop forgets.

Every session starts from zero. Yesterday’s decisions, last week’s context, the thing you spent hours teaching it — gone the moment the process dies. You haven’t built a colleague. You’ve built a very smart goldfish.

This is the story of what happened after I stopped accepting that.

The Problem Nobody Demos

The agent loop is real. It’s genuinely that simple. But “repeat until task complete” is a reflex arc, not a brain. A reflex doesn’t remember. A brain does.

The next question after “how does the agent act?” is “how does the agent remember?”

Most frameworks hand-wave this. “Just add memory” they say, as if it’s a toggle. In practice, I learned the hard way: memory isn’t a feature you bolt on. It’s infrastructure. And building it means making decisions about how to store, retrieve, and protect what your agent learns.

Phase 1: The Haystack

My first decision was simple: append-only log. No rewrites. No overwrites. Every turn gets written once and stays there.

Why? Because that’s how living things remember. Accumulation, not replacement. You don’t delete yesterday’s experience when today happens. You add to it.

The implementation: a binary file where each record is prefixed with a magic header and length:

_HAY_MAGIC = b"KRD1"  # Magic bytes — validates the record is intact
# Each record: [MAGIC 4 bytes][LENGTH 4 bytes][JSON payload]

A housekeeper files every turn here. It’s the only hand that writes the haystack. Everything else is read-only. This separation matters — you don’t want the retrieval path able to corrupt the archive. (The housekeeper’s rhythm controller has its own name and its own story. Part 2.)

Phase 2: The Reading Gap

Here’s where it got ugly.

The haystack was growing. Thousands of turns filed away. But the agent had to decide to search it. And when it didn’t — which was often — the memory sat there, useless. Like a library with no librarian.

I needed the archive to answer back automatically. Not on demand. Reflexively. Before every model call, relevant memories should surface. This is the part that took days to get right.

The Architecture

The retrieval plugin hooks into the framework’s pre_llm_call event — the seam between receiving a user message and sending it to the model. Here’s the flow:

1. Extract search terms from the user’s message

def _terms(text: str, max_terms: int) -> list[str]:
    """Extract searchable atoms from the message. (Simplified)"""
    out, seen = [], set()
    for tok in _WORD_RE.findall(text.lower()):
        if tok in seen or tok in _STOPWORDS:
            continue
        if len(tok) < 3:  # Skip short noise
            continue
        seen.add(tok)
        out.append(_stem(tok))
        if len(out) >= max_terms:
            break
    return out

Stopwords filtered — in two languages, English and Hungarian, because that’s how our conversations actually sound. Duplicates removed, crude stemming applied, capped at 8 terms. This keeps the query focused — too many terms and you get noise.

2. FTS5 search over shard index cards

query = " OR ".join(_fts_groups(terms))
hits = reader.search(query, limit=max_shards * 3, older_than=session_born)

The search doesn’t scan the haystack bodies. It scans shard cards — lightweight index entries with context and tags that the housekeeper filed alongside each record. This is the key insight: recall quality depends on how richly the housekeeper writes its index cards. If retrieval is weak, fix the filing side, not the search side.

3. Fetch the actual memory from the haystack

def fetch_memory(self, shard: dict) -> str:
    with self.hay_path.open("rb") as hay:
        hay.seek(shard["hay_offset"])
        if hay.read(4) != _HAY_MAGIC:  # Validate integrity
            raise IOError(f"haystack bad magic at {shard['hay_offset']}")
        (rec_len,) = struct.unpack("<I", hay.read(4))
        payload = json.loads(hay.read(rec_len).decode("utf-8"))
    return str(payload.get("memory", ""))

The magic bytes (KRD1) validate that the record is intact. This matters — corrupted records are a silent killer in append-only logs. Because the log is append-only, a stored byte offset is valid forever: no compaction, vacuum, or rewrite can ever move the truth out from under the pointer.

4. Inject into the context window

The recall block is appended at the tail of the message list, after the user’s message. Never mid-history. Never in the system prompt. This is a deliberate design choice:

note = {"role": "system", "content": block}
messages.append(note)  # Tail-append keeps cached prefix intact

Why? Because per-conversation prompt caching is sacred. Inserting mid-history would invalidate the entire cached prefix — on a local rig, that’s a ~40k-token re-eval every single turn. Tail-append keeps the prefix byte-stable. The cost per recall is just the block’s own tokens.

Five Ways to Fail Silently in One Day

The plugin ships with a “dry-run” mode: it searches, times, and logs what it would inject — changing nothing. That’s deliberate, and it’s good design: you watch the shadow before you let it touch the model.

But dry-run was not my bug. My bug was that between “plugin written” and “reflex online,” the system failed five separate times in one day, and every single failure was invisible from the inside:

1. A YAML parsing error — one unquoted colon, and the plugin never loaded.
2. A fix that re-broke the same file a different way.
3. The plugin loaded — but into a process that had started before it existed. A config edit is not a restart.
4. The plugin worked — but logged at DEBUG, and the harness ran above DEBUG. It could have been firing for hours and I would have seen nothing.
5. Everything loaded, registered, and logging — and the plugin was simply not enabled in the framework’s registry. One flag.

Five failures, none of them loud, each masquerading as success at a different layer. That was one long day. The lesson is permanent: memory systems fail silently. A broken recall path is invisible to the agent — it just… forgets. Again. You need loud logging, dry-run modes, and the discipline to verify every link in the chain instead of assuming it.

Fifteen Forty-Seven

The reflex went online at 15:36 on a Saturday. I watched the log confirm it, checked the injection format, and closed the terminal to get on with the evening.

Eleven minutes later, in the middle of an ordinary conversation about something else entirely, the agent used a detail from days before — the kind of detail that had been filed away and never once retrieved on purpose. The log showed what had happened: the reflex had fired on its own, pulled the shard, and slipped it into context. Nobody searched for that memory. It surfaced because it was relevant — the way a smell brings back a room.

That was the moment the architecture stopped being a diagram. A library had answered back without being asked a question. Reflex, not request.

How It Works Now

The full pipeline, in numbers:

1. User sends a message
2. Plugin extracts 2-8 search terms (~0.5ms)
3. FTS5 searches shard index cards (~1ms)
4. Up to 3 shards fetched from haystack (~1ms)
5. Recall block injected at tail of message list
6. Model receives context with memories included

Total: 1-3ms per turn. No cloud. All local. The kredenc database sits on disk — 2,000+ shards, 2 drawers, searchable in milliseconds.

The hard guarantees:

• Read-only. The retrieval plugin cannot write to the archive. Only the housekeeper files.
• Never breaks a turn. Every failure is caught, logged loudly, and the turn proceeds as if nothing happened.
• Cache-sacred. No mid-history mutation. The cached prefix stays intact.
• Budgeted. Max 3 shards, 600 chars each, 1,800 total. ~450 tokens per recall.

Why This Changes Everything

An agent that remembers is fundamentally different from one that doesn’t. The first is a tool. The second is… something else. Something that learns from you. That carries context forward. That doesn’t start from zero every time you talk to it.

Think of it this way: if you teach a colleague something on Monday and they’ve forgotten it by Tuesday, you haven’t hired a colleague. You’ve hired a voice assistant that costs a salary. Memory is the difference between automation and partnership.

The 25-line tutorials show you how to build a reflex. This shows you how to attach a brain to it.

Frameworks come and go. But “the agent remembers what you did together” is the fundamental nobody demos, because it doesn’t fit in 25 lines. It takes infrastructure. It takes an append-only log, a housekeeper, a search index, and a hook that fires before every model call.

It takes crossing the line between using an AI and raising an agent.

What Happened Next

Two days after this story ends, the reflex gained a second sense — retrieval by meaning, fused with retrieval by words — and a deterministic test harness it had to survive before earning its place on the machine. Then its first night of live testing produced two findings I haven’t seen published anywhere: one about what frameworks actually do with hook-injected memory (measured, not assumed), and one about what happens when retrieved truth collides with a wrong “fact” in the system prompt.

The truth lost. That’s Part 2.

The kredenc-retrieval plugin and the housekeeper are custom-built for my special version of Hermes Agent, running entirely on my own hardware. All code, all memory, all offline.