---
title: Memory As Files, Not As A Database
type: deep-dive
level: L3
status: live
revision: 1
updated: 2026-08-14
systemVersion: 4.2
tags: [memory, architecture, deep-dive]
rating: 8.45
ratingAxes: useful 8 · evidence 8 · pull 9 · original 9 · form 9
ratingKind: derived
source: in production ~9 months
---

# Memory As Files, Not As A Database

_Written 2026-08-14 · last verified 2026-08-14 · system v4.2 · live_

**TL;DR** — This agent's memory is markdown files in four categories: behavioural rules, entities, reference material and project state. No vector database, no embeddings. The reason is that retrieval was never the bottleneck — knowing what is authoritative was, and files make authority visible in a way a similarity score cannot.

## Problem

An agent that works across many domains accumulates knowledge, and the obvious move is to embed everything into a vector store and retrieve by similarity.

That instinct answers the wrong question. In practice the failure was almost never *"the agent could not find the fact"*. It was one of these:

- The agent found **two facts that contradicted each other** and had no basis for choosing.
- The agent found a fact that had been true four months ago.
- The agent found a fact and could not tell whether it was an observation, a rule, or a guess someone wrote down once.

Similarity search is excellent at retrieval and structurally silent on all three. A vector store returns what is *close*, and closeness says nothing about authority, freshness or kind.

## Design

Memory is a directory of markdown files, split into four categories by **what the content is for**, not by topic.

| Category | Holds | Changes |
|---|---|---|
| Behavioural rules | how the agent should act, each traceable to a correction | only with explicit approval |
| Entities | people, companies, deals — facts about the world | continuously, as facts arrive |
| Reference | durable external knowledge, methods, standards | rarely |
| Project | live state of ongoing work | constantly |

Every file carries a header: a name, a one-line description used to decide relevance, and a type.

> ```
> name: term-executor
> description: the scheduled task, hook or build step that invokes a rule
> type: reference
> ``` Entries cross-link by name.

Three properties fall out of this, and they are the actual reason for the design.

**Authority is visible.** A behavioural rule and a stray observation are different kinds of file in different directories. When two statements conflict, the category settles precedence without a judgement call.

**Edits are legible.** Changing a rule is a diff. Six months later the diff is still there, still readable, still attributable — and that history is the single most useful thing when an agent starts behaving oddly.

**Retrieval is grep.** Unglamorous and adequate. At this scale, a literal search across a few hundred files is instant, and it is *predictable* in a way similarity search is not: a query either matches or does not, and both outcomes are inspectable.

## Trade-offs

**Retrieval is literal, so spelling matters.** This has cost real time — a search for a company name under one spelling returned nothing and produced 3 days of confidently wrong answers. The mitigation is a rule about searching variants, which is weaker than an embedding would be. This is the genuine loss and it should be stated plainly.

**Scale has a ceiling.** A few hundred files works. A few hundred thousand would not, and at that point the answer is probably a hybrid: files for rules and entities, an index for bulk.

**Curation is mandatory.** Files do not evict themselves. Without periodic triage the store fills with entries that were interesting once, and the agent's context fills with them too. Caps and forced triage passes are part of the design, not an afterthought.

**No semantic recall.** The agent cannot surface a related idea it was not looking for. Vector stores are genuinely better at serendipity; this design trades that away for auditability.

## How it is read

The retrieval path matters as much as the layout, and it is deliberately dull.

**The index is read first.** Each category has an index file holding one line per entry: name, one-line description, link. That description is written to answer a single question — *is this relevant right now?* — which means it has to be specific enough to reject on. `Notes about suppliers` is useless; `Ordering profile: minimum quantities, lead times, packaging constraints` can be discarded without opening the file.

**Entities trigger a mandatory lookup.** Whenever a person, company, project or deal appears in a request, the entity registers are searched before anything else happens. This is a fail-open bias by design: it is better to load a file that turns out to be irrelevant than to answer confidently from a stale impression.

That bias has a cost, which showed up as a real failure. The search is literal, so a name spelled one way in the query and another way in storage returns nothing — and nothing is indistinguishable from *no such entity*. The mitigation is a rule about searching spelling variants, which is genuinely weaker than what an embedding would give.

**Deep context loads on trigger, not by default.** Most files are never read in a given session. A small table maps triggers to files — a supplier question loads the supplier standard, a data question loads the analysis workflow. The alternative, loading everything relevant-looking, degrades answer quality: longer context measurably reduces extraction accuracy, so a store that eagerly loads is competing with itself.

**Writes are gated by two questions.** Is this predictively useful later, and is it not already recorded somewhere else? Both have to be yes. Without the second, the store accumulates near-duplicates that quietly disagree, which is the failure mode that produces two confident answers to one question.

## What broke

Three failures shaped the current shape, and each is worth more than the design rationale.

**The store forked.** A backup mirror used relative links between entries, so the mirror's links resolved inside the mirror. Two complete, traversable, slightly different stores existed at once, and writes landed in whichever one a session opened. Fixed by making all cross-references absolute — a backup has to be inert, not navigable.

**Dead paths accumulated.** Convenience paths from a folder that gets cleared on a schedule were written into permanent notes. An audit found **240** of them. Fixed by splitting the rule by audience: temporary paths in replies, canonical paths in anything stored.

**A counter drifted from its contents.** A summary header claimed 25 entries against 12 present. Nobody had written a wrong number; entries had been merged and archived without touching the header. The lesson was not to fix the header — it was to never silently correct an unexplained gap downward, because the correction destroys the evidence that something was lost.

One consequence deserves stating, because it is the reason the design has survived. **Every part of this store can be repaired with a text editor.** No migration, no schema version, no service that has to be running for the memory to be readable. When something goes wrong — and the three failures above are only the ones worth writing up — the recovery path is to open a file and look. That property is worth more than any retrieval improvement, because the failures that actually happen are structural rather than semantic.

## Files

The store is four directories of markdown plus an index per category. The index holds one line per entry — name, one-line description, link — and is what gets read first.

Nothing else: no schema, no migration path, no service to keep running. A design whose main virtue is that it can be read and repaired by hand is a design that survives its own tooling.