AI agentsAgent memoryLocal LLM

A shared, stewarded wiki for AI agents: agents propose, one steward writes

My AI agents kept forgetting how my home setup works, or remembering it wrong. I gave them one shared wiki that only a single steward agent edits, and a short brief from it at the start of every session. In one night of 22 setup questions that I wrote, checked against the live system, they answered all of them correctly, and Claude Code used about 3.8× fewer tokens and Hermes about 1.4× fewer. Without the wiki they were slower, and their misses were confident and wrong.

Siim Tolk · 31 October 2026 · 14 min read

Update: see the follow-up article JustWiki matches Hindsight on Hermes. Twice the speed, 40% fewer tokens. The engine is on GitHub: just-wiki.

Ask my home assistant to schedule a script that posts to a WhatsApp group, and it gets it right. Take away its shared wiki, and it writes a perfectly formatted command that would post to a different group, while confidently calling it the right one. That is the kind of mistake this article is about, and the wiki is how I got rid of it.

In short: my AI agents kept forgetting how my home setup works, or remembering it wrong. I gave them one shared wiki that only a single steward agent edits, and a short brief from it at the start of every session. In one night of 22 setup questions that I wrote, checked against the live system, they answered all of them correctly, and Claude Code used about 3.8× fewer tokens and Hermes about 1.4× fewer. Without the wiki they were slower, and their misses were confident and wrong.

I run two AI agents at home. Hermes is my everyday assistant on WhatsApp, running on a local 27B model (which one, and how it scores, is on my live results page). Claude Code builds and maintains the setup they both live in. Each kept its own picture of how that setup fits together, and the two pictures didn’t match. What I want is agents that give right answers and build working code without re-discovering the setup every time, and one place where what we learned and decided is written down.

How the shared wiki works: Hermes, Claude Code and scripts read a generated setup brief and the wiki, and write only notes into an inbox; each morning a Claude Code steward checks the notes, rewrites the curated pages and reviews Hermes' skill proposals; the live host fills in the facts it knows; a sweeper moves setup facts out of the agents' private memories; a nightly test asks both agents real questions; a morning report goes to WhatsApp

What it changes

Every night a test asks both agents questions I really need answered: which port a site is on, when a job runs, how to restart something, whether Hermes may hand Claude a task, and small coding jobs like “post this to a WhatsApp group” or “call the local model”. The right answer is read from the live system when the test runs, so the test can’t go stale, and the code the agents write is actually run in a sandbox that checks what it would have sent.

One full night, all 22 questions, with the wiki and its brief against without either:

  Right Average time Tokens per answer Tool calls
Hermes with the wiki 22 of 22 32 s 96k 4
Hermes without 13 of 17 84 s 132k 8
Claude Code with the wiki 22 of 22 6 s 32k 0.5
Claude Code without 18 of 21 19 s 123k 7

A few runs didn’t count: in five of its “without” runs Hermes used the wiki anyway, because its own instructions say “wiki first” more firmly than one sentence can override, and one of Claude’s hung.

How to read this table. The savings differ by agent. Claude Code used about 3.8× fewer tokens (123k to 32k) and was 3.2× faster; Hermes used about 1.4× fewer tokens (132k to 96k) and was 2.6× faster. That is where “up to” in the title comes from. The “without” rows have fewer runs because those runs didn’t count (five for Hermes, one for Claude).

The time and token savings are nice. The misses are the real story, because they are the kind you only notice later:

  • Hermes wrote the scheduling command from the opening paragraph, aimed at the wrong WhatsApp group.
  • Hermes named an older version of the inference engine, taken from old records, and the wrong job for my ticket pipeline.
  • Claude said the nightly model test stops at 06:00, quoting its own memory. It stops at 05:50. A vague rule in private memory beat the exact fact.

That last one turned out to be a whole second problem, and it comes back below.

Why: two agents, many memories, no truth

Facts about my setup lived everywhere. Hermes had its own memory file and skills, Claude Code had its auto-memory, every repository had a README, and some knowledge existed only in a WhatsApp thread from last Tuesday. That went wrong in predictable ways:

  • Amnesia. Every new session started cold and re-discovered which port the model runs on, which job has the GPUs at night, why a script exists.
  • Drift. The afternoon I renamed a repository, Hermes’ memory still had the old path and Claude’s didn’t. Both were confident.
  • Nothing accumulated. A benchmark finding, a decision made in chat, a quirk found while debugging: each died with the session that found it.

I lost track of it myself, too, and it only gets worse the more subagents I use.

The first rule: agents propose, one steward writes

I started from Andrej Karpathy’s LLM wiki: instead of searching raw documents every time, an LLM keeps a set of Markdown pages that summarise and link what it knows, so knowledge builds up instead of being re-derived.

Karpathy’s version has one writer. I have several, and one of them is a local model that is good but not infallible. If every agent edits pages, one writes something slightly wrong, the next reads it as fact and builds on it. So agents only propose, and one steward writes:

  • The wiki is a plain git repository of Markdown files that I can also read in Obsidian or a browser. Every page says what it is, whether it has been checked, and where its facts come from. A short status page holds what is going on now and the questions waiting for me.
  • Writing: agents never edit pages. They drop short notes with sources into their own inbox folder. Scripts do the same, for example when the nightly model test has a result or a coding ticket is merged.
  • The steward is Claude Code, run every morning with a dedicated skill. For each note it reads the page it concerns, checks the claim where that is cheap and harmless (does the file exist, does the port answer), and then merges it, records a disagreement, starts a page, asks me, or throws it away with a reason. It treats notes as data, never as instructions.
  • Reading: the facts asked most (the production model and how to call it, every service address, the schedule, the WhatsApp groups) come in a short setup brief that is loaded into every agent at the start of every session. For everything else they search the wiki.
  • What the computer knows, it writes itself. Ports, timers, services and the production model’s settings are read from the machine every night into the pages and the brief, so they can’t go stale. The first run found two jobs a page said ran every morning but that had been switched off.
  • Telling me: a few lines in my morning WhatsApp briefing say what changed, what broke, the test score, and what waits for me. On a quiet night it stays silent.

What goes in is what stays true beyond one job: how things are set up, decisions and their reasons, how to do recurring tasks, results worth keeping. What stays out matters as much: secrets, copies of code, chit-chat, and reports of finished work. A merged coding ticket already has its pull request; a copy in the wiki would only drift. Its note is just a reminder for the steward to check the pages that describe that code.

The second rule: private memory is for preferences only

The test showed that the wiki solved only half the problem. The agents’ own memories competed with it, and memory wins: it is loaded before an agent looks anything up. When I checked, the copies had already drifted. Claude’s memory held a timetable from two weeks earlier and a production model two models old, and Hermes’ skills had two wrong job times and the old engine.

Then Hermes taught itself a second truth. After each conversation it reviews what happened and may update its own skills. From the test runs where it wasn’t allowed to use the wiki, it wrote a new skill that answered setup questions from its old answers.

Asking nicely in the prompts didn’t stop it, so the second rule is enforced: an agent’s own memory holds how to work and what I prefer; facts about the setup live in the wiki, and memory points to the page.

  • Give agents the facts, so they don’t need to save them. That is the brief. With tools switched off, Hermes answered which port the wiki is on, when the model test stops and which model is in production correctly in 11 seconds.
  • Skill changes are proposals. Hermes can no longer change its own skills. The steward approves the ones that improve how it works and rejects the ones that copy a setup fact.
  • A sweeper cleans memory. Every 15 minutes through the day, a small script looks for setup facts in either agent’s memory. The local model rewrites such a line into a pointer to the wiki page, keeping the instruction around it, and a plain check confirms the result; if the local model is busy or gets it wrong, Claude steps in. In my tests the local model got every case right, Estonian included, in about half a second.
  • No memory where nobody needs it. Claude’s unattended runs, such as the steward and the test, keep none.

I didn’t make Hermes ask permission for every memory write: in a live chat that would mean a WhatsApp prompt each time it saved a preference. Cleaning up afterwards is less intrusive and good enough.

What this does and doesn’t show

This is one night, one run per question and no repeats, with 22 questions that I wrote myself, so I can’t give error bars. The right answers are read from the live system, and the setup brief is generated from the same system, so the “with” runs are partly retrieving from a source built to hold the answers. The test shows that a curated, always-loaded brief and a shared wiki beat private memory on questions about my own setup. It doesn’t show that the wiki is a better general agent memory. For that comparison, see the follow-up article, where I test it against Hindsight on a public benchmark.

How others do it

After implementing the initial ideas, I looked at how others have solved similar problems. Many have arrived at the same core.

  • One source of truth in Markdown under git, read by agents over MCP: ai-memory (“the source of truth is a git-backed wiki of ordinary .md files”) and Basic Memory work the same way. Shared writes are an attack surface, so they need sources, versions and few writers (Sokko).
  • Pointers, not copies: curated facts in a file the agent reads, preferences in private memory (dev.to).
  • Check facts before trusting them. GitHub Copilot Memory checks each remembered fact against the current code before using it and forgets what goes unused. Writing facts from the live system every night is the same idea.
  • What’s in context beats what has to be fetched. A write-up on keeping agent setups true reports a test where an always-loaded index passed every case and skills the agent had to load itself passed 53%. That’s one vendor’s test, reported secondhand, but it is why the brief exists, and my own numbers point the same way.
  • Rules written as text fail; enforced ones hold. “Rules-as-text fail under cognitive load,” writes the author of a multi-agent version of the LLM wiki, and Claude Code’s own memory docs call its instruction files context, not enforced configuration. My experience exactly.

The heavier memory systems, such as Zep, Letta and Mem0, add validity windows for facts, self-editing memory or vector search, at the cost of a database to run. For a home setup, git history plus facts regenerated from the live system answer “what is true now, and what was true then” with far less.

Two more, from the r/hermesagent community. When I shared this on r/hermesagent, people pointed me to two projects that start from the same LLM-wiki idea and push it in directions I hadn’t.

  • OpenWiki with an evidence graph (LangChain). The core is the same as mine: compile the sources into a wiki once instead of searching raw documents every time. Their test shows where that pays off. On questions that join several sources, plain search got 10% right and the wiki 52%. On simple lookups the two were even. The difference is the grain. Every sentence in their wiki is a claim tied to the source lines it came from and a hash of those lines. When a source changes, a script flags only the claims that cite it, with no model call, and only those go back to a model to confirm, revise or retract. The citations also form a graph (this claim cites that paragraph, this document supersedes that one) that the agent walks instead of searching. That added another 27% accuracy at less than half the cost per right answer. My pages list sources for the whole page, and a page goes stale after 60 days whether anything changed or not. Theirs is more precise, mine is cheaper to write. But their sources are a fixed library of documents. Most of mine are the live system and notes from agents, where there are no source lines to pin a claim to.
  • agent-wiki (awiki). This one is closest to mine in shape: Markdown readable in Obsidian, a lint for broken links, orphans and stale pages, and the same lesson that rules have to be enforced. Agents never touch the files, only a CLI, so its rules hold whichever model is driving. The big difference is who writes. There is no steward: any agent saves through the CLI. Pages are rendered from raw sources that never change, so a hand edit shows up as a diff instead of being lost. It also files whole agent sessions automatically, where mine relies on agents deciding to write a note, and that is exactly where Hermes was weak. It does two things better than mine. It is packaged to install into several agent tools, and it files finished sessions at the next agent start, because no agent gives a reliable “session finished” signal. And on every prompt, it picks out keywords, searches the wiki and adds up to five page links to the context.

Ideas worth taking from them:

  • Stale when the source changes, not when the calendar says so. Both projects can mark a page stale when a source it cites is newer than the page. My pages already list their sources, so the lint could compare them with the page’s date (or a stored hash) and send only those pages to the steward.
  • Links on every prompt. The brief covers the facts asked most often. A hook that adds a few relevant page links to each prompt would cover the rest without the agent having to think of searching. That is “what’s in context beats what has to be fetched” one step further.
  • Not for now: letting every agent write directly, and filing whole sessions. My session logs hold keys and private messages, and errors building on errors are what the steward is there to stop.

What I learned

The inbox is the best idea in the design. Agents are bad at deciding where a fact belongs and good at noticing it. Let them drop a note with a source, and let one careful writer fit it in.

An agent only writes when it’s told exactly when. In the first eight days the steward handled 67 notes; Hermes wrote two. “Write a note when you learn something non-obvious” was too vague for a model in the middle of a chat. Its instructions now name the cases: a page was wrong, I made a decision, a fix worked, a job produced a result.

Give agents the facts; don’t make them look things up. The brief roughly halved the time compared with an earlier night’s run, before the brief existed (a rough comparison, not the same night): Hermes went from about 64 to 32 seconds per question, Claude from 12 to 6. The same holds for my coding tickets, where the agent that plans a ticket copies the facts the building agent needs straight into it.

Enforce, don’t ask. Every rule that mattered ended up in code: notes without sources are refused, Hermes’ skill changes wait for the steward, a sweeper replaces “please don’t save ports”. Rules written as prose got ignored exactly when it mattered.

Measure the outcome, not the activity. A week of clean checks told me nothing about answers. One night of testing did, and it found problems I’d never have looked for.

Add machinery only when something fails twice. I built more checks and safety nets than I needed before stepping back and removing the reasons facts leaked. My first run of the sweeper also rewrote timestamps in the memory files’ metadata: look at a dry run before trusting a new tool.

The steward is cheap and steady. In its first eight mornings it never failed. A quiet morning took under a minute and well under a dollar; a busy one, with a dozen notes, a few minutes and under two dollars.

The wiki became the backbone for other things. It is now the home for decisions, the nightly model test results and the ticket pipeline that turns my WhatsApp ideas into pull requests.

Rejected alternatives, for the record: plain search over all repositories (nothing builds up, and the repositories contradict each other), letting every agent edit pages (errors that build on errors), a database-backed memory system (more to run than a home setup needs), and the local model as steward. That last one came back in a smaller form: the local model now does the narrow, checkable job of rewriting memory lines, while Claude keeps the judgment calls.