OnSiteReliability.com

SRE Monitoring: A Complete Checklist

Sat, 18 Jul 2026 00:00:00 +0000

Most monitoring setups grow organically. A team adds a dashboard here, an alert there, and over time the system accumulates gaps nobody notices until an outage reveals them. The Google SRE books offer a structured alternative: a layered approach to monitoring that starts with what users experience and works inward toward root causes.

This post synthesizes the monitoring guidance from three Google books: Site Reliability Engineering (2016), The Site Reliability Workbook (2018), and Building Secure and Reliable Systems (2020). The result is a single checklist you can use to audit your monitoring coverage.

The Four Golden Signals

Google’s SRE book defines four signals that every service should monitor. These are the foundation.

Signal	What to Monitor	Alert Threshold Guidance
Latency	Request duration (split success vs error), P50/P95/P99	Page on sustained P99 breach of SLO
Traffic	Requests/sec, connections/sec, sessions/sec	Alert on sudden drops (more than increases)
Errors	Error rate %, explicit (5xx), implicit (wrong content), policy (too slow)	Page when error budget burn rate is high
Saturation	CPU, memory, disk, connections, queue depth: the resource that will exhaust first	Page at 80%+ of hard capacity

Traffic drops are often more informative than spikes. A sudden drop in requests frequently means users cannot reach you at all. Spikes might just mean you are popular.

The Full Monitoring Checklist

The four golden signals tell you something is wrong. A complete monitoring stack tells you why, and warns you before users notice. The checklist below is organized in tiers from user-facing symptoms down to operational meta-signals.

Tier 1: User-Facing SLIs (Symptoms)

This is where your monitoring should be strongest. Everything else exists to explain anomalies in these signals.

End-to-end latency from the user perspective (synthetic probes)
Success rate of user-visible operations
Availability measured from outside the system (black-box)
Error rate differentiated by severity (client vs server errors)
Traffic volume with anomaly detection (sudden drop equals outage signal)

Tier 2: Infrastructure Health (Causes)

Once you know something is wrong at Tier 1, these metrics help you find the cause.

Compute saturation (CPU, memory per pod/node)
Network saturation (bandwidth, connection limits, packet loss)
Storage saturation (disk IOPS, capacity)
Database health (connections, replication lag, query latency)
Cache effectiveness (hit rate, eviction rate, memory usage)
Queue depth and processing lag

Tier 3: Dependency Health

Modern services fail at their boundaries. Monitor what you depend on.

Upstream dependency latency and error rates
Downstream dependency availability
Cloud provider service health (region-level signals)
DNS resolution success and latency
Certificate validity and expiration tracking

Tier 4: Capacity and Scaling

Saturation problems are predictable if you track the trends.

Current utilization vs headroom for each constrained resource
Autoscaler health (desired vs actual replicas)
IP address pool exhaustion
Session/connection capacity vs current load
Rate limit proximity
Time-to-exhaustion predictions for growing resources

Tier 5: Configuration and Deployment

Many outages correlate with recent changes. Make those changes visible.

Deployment health (rollout status, rollback signals)
Configuration drift detection
Feature flag and config change correlation with errors
Canary/rollout monitoring

Tier 6: Security and Compliance

Security failures are reliability failures. Monitor them the same way.

Authentication/authorization failure rates
Certificate expiration (30-day, 7-day, 1-day warnings)
Anomalous access patterns
Compliance-relevant audit trail gaps
Rate limiting and abuse detection

Tier 7: Data Correctness and Freshness

For services that process or serve data, correctness is an SLI.

Data pipeline lag (freshness SLI)
Data completeness checks
Cross-system consistency validation

Tier 8: Incident Detection Meta-Signals

Your monitoring system itself can fail silently. Watch the watchers.

Customer-reported issues (manual escalation)
Correlated multi-signal alerts (not just individual thresholds)
Maintenance window awareness
Operational workflow health (runbook automation success)
Monitoring system health (dead alerts, unexercised rules)

SLO-Based Alerting

The SRE Workbook advocates multi-window, multi-burn-rate alerting over simple threshold-based alerts. The idea: alert not on a raw error count, but on the rate at which you are consuming your error budget.

Severity	Long Window	Short Window	Burn Rate	Budget Consumed	Action
Page	1 hour	5 minutes	14.4x	2%	Immediate
Page	6 hours	30 minutes	6x	5%	Immediate
Ticket	24 hours	2 hours	3x	10%	Non-urgent
Ticket	3 days	6 hours	1x	10%	Next business day

A 14.4x burn rate over one hour means you will exhaust your entire monthly error budget in about two days if the condition persists. That warrants a page. A 1x burn rate sustained over three days means you will barely miss your SLO target by month’s end. That warrants a ticket, not a 3am wake-up.

The dual-window approach (long window plus short window) prevents both slow-burn blindness and flappy alerting. The long window catches sustained issues. The short window confirms the problem is happening right now, not just an artifact of earlier errors still in the window.

Five Questions Before Creating Any Alert

Before you add any new alert rule, run it through these five questions from the SRE book.

Does this rule detect an otherwise undetected condition that is urgent, actionable, and actively or imminently user-visible?
Will I ever be able to ignore this alert, knowing it is benign?
Does this alert definitely indicate that users are being negatively affected?
Can I take action in response? Is that action urgent, or could it wait until morning?
Are other people getting paged for this issue, rendering at least one page unnecessary?

If the answer to question 2 is yes, the alert will train your team to ignore pages. If the answer to question 4 is “it can wait,” it should be a ticket, not a page. These questions are a forcing function for alert hygiene.

Key Principles

A few principles tie the checklist together.

Heavy use of white-box monitoring with modest but critical uses of black-box monitoring. White-box (metrics, logs, traces from inside the system) tells you why. Black-box (synthetic probes from outside) tells you what users actually experience. You need both, but white-box carries most of the diagnostic weight.

Symptoms over causes. Spend more effort catching symptoms than causes. A user does not care whether your database is slow or your cache is cold. They care that the page did not load. Alert on what users experience, and use cause-level metrics for diagnosis.

Every page should be actionable, require intelligence, and represent a novel problem. If a page requires no thought, automate the response. If it requires no action, it should not page.

Alert on burn rate, not raw threshold. A single error is not an incident. A sustained rate of errors that threatens your SLO target is.

SLIs are always ratios: good events divided by total events, expressed as a percentage. Error budget equals one minus the SLO target. Track on a four-week rolling window.

References

Site Reliability Engineering (2016)
The Site Reliability Workbook (2018)
Building Secure and Reliable Systems (2020)

Engineering Confidence: A Book About qmail and Earning Trust Through Architecture

Mon, 22 Jun 2026 00:00:00 +0000

I wrote a book called Engineering Confidence. It is about qmail, the 1990s mail server that shipped with a public security guarantee and a cult following among Unix administrators. I wrote this book because qmail is worth remembering. Imagine the confidence needed to thrown open your source code to the whole world, and dare it to find a security hole. On top of it, the developer of qmail, D.J. Bernstein offered $500 to anyone who found a security hole in the latest version of qmail. That confidence is rarely seen in software today. That confidence was based on principles, it was based on structure. This structure is worth learning, worth considering when we make decisions about software today.

The book is not an installation manual. It uses qmail as a case study in how software earns trust: through architecture, boundaries, constrained privilege, careful parsing, explicit claims, and long-term maintenance decisions. Modern engineering teams make claims all the time: this system is secure, reliable, simple, maintainable, zero trust, cloud native, production ready. qmail is a reminder that confidence has to be earned in the design.

What it covers

The book follows qmail from the hostile mail environment of the 1990s, through Bernstein’s architectural response, to the long-term consequences of a frozen upstream. It examines the security guarantee, the disputes around it, the patch culture that grew in the absence of releases, and what modern engineering teams can still learn from the design.

Who it is for

Site reliability engineers, security engineers, infrastructure engineers, engineering managers, and anyone interested in how technical confidence is built, challenged, and maintained. No prior qmail experience required.

Status

The book is nearing completion. Formats planned: EPUB, PDF, Kindle, and paperback.

The Morris Worm: When the Internet First Broke

Sun, 21 Jun 2026 00:00:00 +0000

On the evening of November 2, 1988, around 6 PM EST, a program was launched onto the Internet from one or more hosts. Within hours, thousands of machines running BSD UNIX were so overloaded with rogue processes that they could not function. Systems ran out of swap space. Process tables filled up. Administrators disconnected entire networks in a panic. The Internet had experienced its first major self-inflicted wound.

The program responsible was a worm written by Robert T. Morris, a graduate student at Cornell. Eugene Spafford’s technical analysis of the decompiled code, published just weeks after the incident, remains one of the most thorough postmortems in computing history. What it reveals is not a work of genius, but something more instructive: a mediocre program that succeeded because the systems it attacked were unprepared for even a mediocre adversary.

How the Worm Spread

The worm targeted Sun 3 and VAX machines running 4 BSD UNIX. It used three attack vectors to break into new hosts.

The first was a buffer overflow in fingerd. The finger daemon used the C library function gets to read network input into a fixed-size buffer without bounds checking. The worm sent a carefully constructed 536-byte string that overflowed the buffer, overwrote the return address on the stack, and caused the program to execute /bin/sh. On VAX machines, this gave the worm a remote shell. On Suns, it produced a core dump instead, because the shellcode was VAX-specific.

The second was a debug mode in sendmail. The worm issued the DEBUG command to sendmail and then specified a set of shell commands as the recipient of a message. This debug feature was not supposed to be enabled in production, but it was present in the code for testing mail delivery, and many vendors and administrators left it compiled in. The worm exploited this to pipe commands directly to /bin/sh.

The third was password guessing combined with trust relationships. The worm read /etc/passwd, attempted to crack passwords using the account name, variations of the user’s real name, and a built-in dictionary of 432 words. When it broke a password, it used rsh and rexec to reach other machines where that user had accounts, exploiting the common practice of reusing passwords across hosts.

The Infection Sequence

The worm had two components: a main program and a bootstrap called the vector program. Once the worm obtained a shell on a target machine through any of the three attack methods, it followed a precise sequence.

It compiled and ran the vector program on the target. The vector connected back to the infecting machine on a predetermined TCP port, authenticated with a challenge number, and downloaded three files: a Sun 3 binary of the worm, a VAX binary, and the vector source code. The infecting worm then sent shell commands to try each binary. If one ran successfully, the new host was marked as infected.

The newly running worm immediately hid itself. It changed its process name, unlinked its binary from disk, killed its parent process, loaded the worm binaries into memory, encrypted them, and deleted the files. From that point on, it existed only in memory with no obvious filesystem footprint.

Why It Got Out of Control

The worm included a mechanism to avoid reinfecting machines. Running copies would listen on TCP port 23357 on localhost. A new worm arriving on an already-infected host would connect to that port, exchange random numbers, and based on the result, one of the two would exit.

This mechanism was fatally flawed in two ways. First, one out of every seven worms would skip the check entirely and become “immortal,” ignoring any other worms on the same machine. Spafford concludes this was deliberate, designed to prevent administrators from running a fake listener on the worm port to kill incoming copies. Second, even worms marked for self-destruction would continue running through at least one full password-cracking cycle before exiting. The result was exponential growth: each pass through the main loop forced the worm to infect at least one local host, and multiple copies on a single machine would each do the same.

The worm’s author apparently did not understand the propagation dynamics of his own program. The code contained mechanisms to limit growth, which suggests awareness that uncontrolled replication was undesirable. But those mechanisms were insufficient, producing the catastrophic load that made the worm so visible and so disruptive.

The Response

By late Wednesday night, personnel at UC Berkeley and MIT had captured copies of the program and begun analyzing it. By 5 AM Thursday, less than 12 hours after the worm appeared, the Berkeley Computer Systems Research Group had published an interim set of defenses: a patch to sendmail and a suggestion to rename the C compiler to prevent the worm from building itself on new hosts.

By 7 PM Thursday, a simpler defense was discovered at Purdue: creating a directory named sh in /usr/tmp caused the worm’s shell test (if [ -f sh ]) to fail, blocking further infection without renaming any system utilities. This worked because the worm used the -f flag, which tests for a regular file but returns false for directories. The author had used [ instead of the more portable test command, and -f instead of -e, both signs of limited experience with shell scripting.

On November 8, the National Computer Security Center convened a hastily-organized workshop in Baltimore. The attendees agreed not to distribute their reverse-engineered source code publicly. By that date, Spafford was aware of at least eleven independent decompilations of the worm. The genie was already out of the bottle.

What the Code Revealed

Spafford’s analysis of the reverse-engineered code is ruthless in its assessment. The worm was not the work of a genius. The code quality was mediocre. Local variables went uninitialized. Functions were called with wrong numbers of arguments. Return codes from system calls were never checked. Data structures were uniformly linked lists with linear searches, where hash tables would have been trivial to implement and far more efficient.

The one section that showed genuine sophistication was the password-cracking routine, which ran nine times faster than the standard Berkeley crypt function. Spafford notes that this code does not appear to have been written by the same person who wrote the rest of the worm. It included support for both encryption and decryption, even though the worm only needed encryption.

The worm did not write to the filesystem except during infection. It did not transmit any information from infected systems to any external location. It did not exploit root access even when it cracked root passwords. Spafford finds this last omission difficult to attribute to deliberate restraint, suggesting instead that the author simply did not think of it.

The program could have been far more dangerous. The fingerd exploit worked only on VAX, not Sun, likely because the author never built a Sun version of the shellcode. Three Purdue graduate students replicated the Sun exploit in under three hours when asked. The worm’s password-cracking dictionary was only 432 words. It did not gather hostnames from .rhosts files early in its execution. It failed to infect local network hosts due to a bug in the routing logic.

The Larger Lessons

The worm exposed problems that the UNIX community had long known about but never fixed. The gets function had no bounds checking and was documented as dangerous, yet it remained in the standard C library and in production daemons. Sendmail’s debug mode was a known risk that administrators tolerated for convenience. Password hashes were stored in world-readable files, allowing offline cracking. Trust relationships between hosts via .rhosts and hosts.equiv created lateral movement paths that one compromised account could traverse.

Spafford’s recommendations read like a modern security checklist written decades early: replace unbounded string functions with bounded alternatives, audit all network-facing code for their use, run each daemon under its own user ID (least privilege), implement shadow password files, enforce password complexity, and establish coordinated vulnerability disclosure mechanisms.

The response to the incident was ad hoc and nearly failed. Patches were distributed via Usenet and mailing lists, but those communication channels depended on the same Internet that was under attack. Sites that disconnected for self-protection could not receive the fixes. Three weeks after the incident, some sites had still not reconnected.

The worm demonstrated that the internet’s openness was simultaneously its greatest strength and its greatest vulnerability. The same properties that allowed researchers across the country to collaborate on a fix in under 12 hours also allowed a single program to propagate across thousands of machines in the same timeframe.

References

Eugene H. Spafford, “The Internet Worm Program: An Analysis,” Purdue Technical Report CSD-TR-823, 1988

Boris Cherny on Building Claude Code

Sat, 25 Apr 2026 00:00:00 +0000

Boris Cherney is the creator and engineering lead behind Claude Code at Anthropic. Before that, he spent seven years at Meta leading code quality across Instagram, Facebook, WhatsApp, and Messenger. In a recent conversation on the Pragmatic Engineer podcast, Boris shared how Claude Code went from a solo side project to one of the fastest-growing developer tools, how it writes roughly 80% of all code at Anthropic, and what his daily workflow looks like when shipping 20 to 30 pull requests a day with zero handwritten code.

The whole conversation is worth listening to. But the parts that stuck with me most were the practical lessons: things Boris does every day, principles that shaped the tool, and shifts in how he thinks about engineering. Here are the ones I keep coming back to.

Plan First, Then Let It One-Shot

Boris almost always starts in plan mode. He iterates on the plan with Claude before letting it implement anything. This is the highest-leverage workflow pattern he describes.

“Once there is a good plan, it just will one-shot the implementation almost every time,” he says. “The most important thing is to go back and forth a little bit to get the plan right.”

The instinct most people have is to jump straight into implementation. That works when you are the one writing the code and can course-correct on the fly. When an agent is doing the work, a bad plan means a bad implementation that you have to throw away. A few extra minutes refining the plan saves significantly more time than fixing a misdirected implementation.

Parallelize Your Agents

Boris runs five terminal tabs, each with a separate checkout of the repository, with Claude Code running in parallel across all of them. He round-robins between tabs: starts Claude in plan mode on tab one, moves to tab two, starts another, and keeps cycling. When he runs out of tabs, he overflows to the desktop app and the iOS app.

“If you told me six months ago I’d be writing, I don’t know, a third, maybe like a third, half, something like this of my code on a phone. That’s crazy. But that’s what I’m doing today.”

The key insight is that the bottleneck is no longer typing speed or even thinking speed. It is how well you manage multiple concurrent streams of work. Boris frames this directly: “The work has become jumping between Claudes. It’s not so much about deep work, it’s about how good am I at context-switching.”

Automate Recurring Review Comments

At Meta, Boris was one of the most prolific code reviewers at the company. He tracked every recurring comment in a spreadsheet. When a particular issue appeared three or four times, he wrote a lint rule for it.

“Every time that I would have to comment about something, I would drop it in a spreadsheet,” he explains. “Anytime that a particular row had more than three or four instances, I would write a lint rule for it. Just automate it with an op. This is one of our superpowers as engineers: we are able to automate all of the tedious work.”

The modern version is even faster. “Now what I do is when a coworker puts up a pull request and I’m like, this is lintable, I’ll just at Claude, please write a lint rule for this, in that PR on their PR.” The feedback loop from noticing a pattern to encoding it as an automated check has collapsed from days to minutes.

Understand the Layer Below

Boris spent part of his early time at Anthropic working on reinforcement learning, not because it was his job, but because he wanted to understand the layer underneath what he was building.

“Always understand the layer under. It’s really important because that just gives you the depth and you have a little bit more levers to work at the layer that you actually work at.” He notes that this advice has not changed in a decade. What has changed is which layer matters. “Before it was like understand the JavaScript VM and frameworks and stuff. Now it’s: understand the model.”

This is a practical call to action. If you are building on top of language models, you should have at least a working understanding of how they reason, where they fail, and what makes them perform better. That knowledge directly improves the quality of the prompts you write, the architectures you design, and the failure modes you anticipate.

Prototype Aggressively

When building the todo list feature for Claude Code, Boris created roughly twenty working prototypes in about a day and a half. Not mockups, not wireframes. Working, interactive implementations. He tried each one, got a feel for it, and moved on.

“On our team the culture is we don’t really write stuff. We just show. Prototyping everything is so baked into the way that we build.” He contrasts this with the old approach of starting with static mocks in Figma or writing a PRD. “There’s just no way we could have shipped this if we started with static mocks or a PRD. It’s a thing that you have to build and you have to feel.”

The economic argument is simple: the cost of building has dropped dramatically. When implementation is cheap, the right move is to explore broadly rather than speculate narrowly. “Personally I’m wrong like half the time. At least half of my ideas are bad. And I don’t know which half until I try it.”

Embrace Generalism

At Anthropic, everyone holds the same title: Member of Technical Staff. Boris sees this as more than a cultural choice. It is a signal about how work gets done.

“Without this title, the default would have been: I see your name on Slack and under your name it says software engineer. And then I’m like, okay, I guess you’re the coding person. I’m not going to ask you product questions. But when everyone’s title is Member of Technical Staff, by default you assume everyone does everything.”

He believes this is a preview of where the broader industry is heading. “People are going to become more and more multi-discipline and this will become more and more rewarded. In some ways, I think this will be the year of the generalist.” When AI tools lower the cost of execution across disciplines, the people who can work across boundaries become disproportionately valuable.

Revisit Old Assumptions

The pace of model improvement means that ideas which failed months ago may now work. Boris describes this as one of the hardest adjustments.

“The model is improving so quickly that the ideas that worked with the old model might not work with a new model. The things that didn’t work with the old model might work with a new model. You just always have to bring this beginner mindset.”

In a traditional engineering culture, re-trying a failed idea is often met with skepticism. Why are you doing this again? We already tried it. Boris argues that this instinct is now counterproductive. “It’s the first time ever where it’s actually not crazy to just try the same idea every few months because the model improves and it just works.”

He also points out that this dynamic inverts the usual seniority advantage. “New people that are newer to the team, people that are newer to engineering, sometimes do things in a better way than I do. And I just have to look at them and I have to learn and I have to adjust my expectations.”

Keep a Clean Codebase

At Meta, Boris led the Better Engineering program, a Zuckerberg mandate requiring every engineer to spend 20% of their time fixing tech debt. His team used causal inference to measure the impact. The finding: code quality contributes double-digit percentage points to engineering productivity.

That finding now extends to models. “When you start a migration, finish the migration,” Boris says. “As an engineer, you’re going to have a bad time. As a new hire, you’re going to have a bad time. As a model, you might just pick the wrong thing and then the user has to course-correct you. So actually the better thing to do is just always have a clean codebase. This is great for engineers and nowadays it’s great for models too.”

A codebase with two half-migrated frameworks is confusing for humans and actively harmful for AI agents that cannot tell which pattern is the intended one. If code quality had a double-digit impact on human productivity, the impact on model-assisted productivity is likely even larger.

Layer Your Safety Measures

Claude Code’s security model uses what Boris calls the Swiss cheese model: multiple overlapping layers, none of which is perfect on its own, but whose combined probability of catching problems increases with each additional layer.

“For things like safety and security, there’s no one perfect answer. You just need a bunch of layers. With enough layers, the probability of catching anything goes up. You just have to count the number of nines in that probability and pick the threshold that you want.”

For prompt injection specifically, there are at least three layers: model alignment (training the model to resist injection), runtime classifiers that detect suspicious requests, and sub-agent summarization that processes external content through a separate context before returning results to the main agent. “This isn’t just one mechanism. It’s a layer, and by having a bunch of these different layers, it just reduces the probability a lot.”

This principle applies broadly. Any system that interacts with untrusted input needs defense in depth. A single mechanism, no matter how sophisticated, has failure modes. Layered defenses make those failure modes independent, which makes the system far more robust.

The Printing Press Moment

Boris compares this moment in software engineering to the invention of the printing press. Before the press, less than 1% of the European population was literate. Writing was a niche skill performed by scribes employed by lords and kings, many of whom were themselves illiterate.

After the press, the cost of printed material dropped 100x in a few decades, and the quantity of printed material increased 10,000x over the following century. “If you think about what happened to the scribes, they ceased to become scribes, but now there’s a category of writers and authors. These people now exist. And the reason they exist is because the market for literature just expanded a ton.”

The parallel to coding is direct. Programming has been a niche skill performed by a small fraction of the population. As AI tools make coding accessible to everyone, the market for software will expand in ways that are difficult to predict. “The economy as we know it would not have existed without it. So what’s next? What is the thing that we can’t even predict today that will exist because anyone can do this?”

It is an optimistic framing, and one grounded in historical precedent. The scribes did not disappear. The scope of what they could do expanded beyond anything they could have imagined.

References

Building a Knowledge Management System with Structured Memory

Fri, 24 Apr 2026 00:00:00 +0000

Operational knowledge about complex systems lives across dozens of chat threads, ticketing systems, wikis, and people’s heads. When a question comes up (“Which customers were affected by recent incidents?”), answering it requires searching multiple systems and reconstructing context from scratch every time. The information exists, but it is scattered and unstructured, which means it might as well not exist when you need it fast.

I built a file-based knowledge management system using Claude Code’s persistent memory to solve this problem. The system stores knowledge as individual Markdown files with structured YAML frontmatter. Each file represents one discrete unit of knowledge: an incident, a service architecture, a customer relationship, or operational feedback. The approach is simple, but the structure is what makes it work.

How a Memory File Is Structured

Every memory file has three parts, and each part serves a specific purpose.

The frontmatter is YAML at the top of the file. It contains a name, a one-line description, and a type classification. The description matters more than you might think. It is written to support relevance matching in future lookups, so it needs to be specific enough that the system can decide whether this file is worth reading for a given question. A description like “Q4 incident” is almost useless. A description like “Database connection pool exhaustion caused 45-minute outage affecting billing service, October 2025” is immediately useful.

The body is the actual knowledge, structured consistently by type. Incident files follow a template: timeline, root cause, resolution, customer impact, retro findings. Architecture files capture service relationships, failure modes, and operational procedures. The consistency matters because it makes cross-cutting queries possible. If every incident file has a Customer Impact section in the same format, you can ask “which customers were affected across all incidents” and get a reliable answer.

The last section is a “Why / How to apply” footer. This is what distinguishes the system from a simple note archive. It encodes judgment: not just what happened, but when this knowledge should influence a decision. A memory file about a database failover incident might end with “Apply when: evaluating connection pool sizing for new services, or when diagnosing intermittent timeout errors in services that share the same database cluster.” This lets the system surface relevant context proactively rather than only responding to exact keyword matches.

The Index

A single file called MEMORY.md serves as the table of contents. Each entry is one line, organized by topic rather than chronologically. This file is always loaded into conversation context, so it acts as a retrieval hint. When a question comes in, the system scans the index to decide which detailed files to read. It does not need to open every file on every question. It reads the index, identifies the relevant files by their one-line descriptions, and pulls only what is needed.

This is the same principle behind the AGENTS.md approach described in OpenAI’s harness engineering experiment: give the system a map, not a thousand-page instruction manual. A short index with pointers to deeper sources of truth scales far better than a monolithic document that tries to contain everything.

The Type System

Four memory types serve different purposes, and keeping them separate matters for retrieval quality.

Project memories capture incidents, architecture decisions, and ongoing initiatives. These are the most frequently created and the most likely to go stale. An incident memory is useful for weeks or months after the event. An architecture memory stays relevant until the architecture changes.

User memories record who the person is, their role, and their preferences. These help the system tailor its responses. An engineer asking about a service gets different context than an executive asking about the same service.

Feedback memories capture corrections and confirmed approaches. When someone says “don’t mock the database in these tests” or “yes, the single bundled PR was the right call here,” that guidance gets recorded so it carries forward. Corrections are easy to notice. Confirmations are quieter but equally important. If you only save corrections, the system avoids past mistakes but drifts away from approaches that have already been validated.

Reference memories are pointers to external systems. They record that bugs are tracked in a specific Linear project, or that a particular Grafana dashboard is what on-call watches. The system cannot access those external systems directly, but knowing where to look is often enough to be useful.

Why Separation and Structure Matter

The key design choices are separation and structure. Each topic gets its own file rather than appending to a single document. This means updates do not risk corrupting unrelated knowledge. If you need to update the customer impact section of an incident memory, you are editing one file, not navigating a sprawling document where a misplaced edit could break something else.

Consistent internal structure makes the system queryable in ways that unstructured notes never are. When every incident file follows the same template, questions that span multiple incidents become straightforward. “What patterns appear across recent incidents?” is answerable because the data is organized the same way every time.

Where This Approach Fits

This works well for teams managing dozens of services with recurring incidents, complex customer relationships, and institutional knowledge that would otherwise be lost to staff turnover or buried in chat history. The structured format makes it possible to ask questions that span multiple knowledge domains without manually correlating information from separate systems.

The limitation is staleness. Memory files are snapshots. Tickets close, customers resolve issues, architecture changes. The system mitigates this by treating memories as starting context to verify against current state, not as authoritative truth. Before acting on a memory that names a specific function or file path, the system checks whether it still exists. A memory that says “X exists” is not the same as “X exists now.”

Active maintenance is still required. Outdated files need to be updated or removed. But the cost of that maintenance is low compared to the alternative: rebuilding context from scratch every time someone asks a question that the team answered three months ago in a chat thread that nobody can find.

References

Claude Code Memory Documentation covers CLAUDE.md files, auto memory, and the MEMORY.md index system
Claude Code Overview for the broader context on how Claude Code works as an agentic coding tool
Harness Engineering (OpenAI) describes the AGENTS.md approach to repository knowledge management referenced in this post