Short answer
#A release nobody described
A release goes out on a Thursday. The notes say what release notes usually say: improvements to the billing flow, fixes, a faster export. Nothing in them is false. Three weeks later a partner writes in to ask why the notifications they receive from you now arrive from a different address, signed in a different way, and whether that was intentional.
It was intentional, as it turns out. Somebody moved the outgoing notifications into a separate worker during the billing work, because it made the code tidier, and the new worker signs requests with a new key. It was a reasonable decision made by a competent person. It simply was not written down anywhere an owner would look, because from the point of view of the person making it, it was a detail of the billing work and not a change to how the system talks to the outside world.
This is the ordinary way a system drifts from what its owner believes about it. Not through mistakes and not through secrecy, but through decisions that were too small to mention and that, added together, change what the system is.
What this article is about
Not about writing better release notes. About the fact that the most useful question an owner can ask about their system is a question about a period of time, and that a single description of the system, however good, cannot answer it.
#One picture ages at once
Suppose you did the hard part already. You have a description of your system that you trust: the components, the integrations, where data is kept, what runs on a schedule, and which statements are supported by evidence. That is a real achievement, and most owners never get there.
It is also a photograph. On the day it was taken it was accurate. From the next release it describes a system that no longer exists in exactly that form, and there is no moment at which it announces that it has become wrong. A written architecture document has two states, current and quietly wrong, and it moves from one to the other without any event anybody would notice.
The usual answer is to redo it. Commission the picture again in a year, or before the next big conversation. That gives you a second photograph, which is better than an old one, but on its own it still answers only the question of what is there now. It does not tell you what moved, and the movement is what you needed.
-
A picture answers what is there
Which components exist, which outside services they talk to, where personal data sits. Useful on the day you get it, and the basis for everything after.
It is a statement about a moment.
-
A second picture answers what is there now
The same questions, a few months later. On its own it is just a newer photograph, and reading two photographs side by side by eye is slow and unreliable.
Two statements about two moments are not yet a statement about the time between them.
-
A comparison answers what changed
What arrived, what changed, what went away, and what the earlier picture said that can no longer be supported. This is the question an owner actually has before a negotiation, an audit or a change of team.
It is a statement about a period, and it does not age the way a picture does.
#What a difference contains
Comparing two analyses of the same system gives four kinds of answer. The first three are the ones you would expect. The fourth is the one that makes keeping a picture different from writing one.
The difference is computed from what each analysis recorded, not from memory, so it can be checked. Removal is the one answer that needs coverage as well as absence.
- Arrived. Something the earlier picture did not contain: a new service, a new integration, a new place where data is written.
- Changed. The same thing, now doing something else: a different address, a new dependency, a purpose that moved.
- Gone. Something that was there and is not, stated only where the second analysis looked at the place it used to be.
- No longer confirmed. A statement the earlier picture made and supported with evidence, which nothing in the current picture supports any more. It is not deleted and it is not left standing. It is marked, with the date it was last supported.
The fourth category deserves a moment, because it is the one people rely on without knowing. “Payment retries happen three times, an hour apart” is the kind of sentence that gets repeated in support replies and in contracts. If the retry logic moved and the evidence for the sentence went with it, no document will tell you, because no document knows it has become wrong. A kept picture can, because it knows what it said before and what it can still stand behind.
#Three things that look like change
The first comparison most people make is between two documents, and it goes badly for a predictable reason. Almost every sentence is different, so everything looks like change, and an owner either panics or stops reading. A useful comparison separates three things that show up as a difference and mean very different things.
| Kind | What happened | What to do with it |
|---|---|---|
| A change in the system | The system itself is different. A route moved, an integration was added, a store of data appeared. | Read it, and ask whether it was meant. This is the list that belongs on the agenda. |
| Something newly learned | The system did not change, but the second analysis found out something the first did not: a timeout that was always there, a dependency nobody had recorded. | Add it to what you know. It is not news about the system, it is news about your picture of it. |
| A reworded description | The same fact, described in different words. | Nothing. A comparison that reports rewording as change is training you to ignore it. |
The distinction is not pedantry. Treat something newly learned as a change and you will ask a developer why they added a timeout that has been in the code for two years, which costs you a little credibility each time. Treat a real change as rewording and you miss the one thing the comparison was for. 1ADK keeps these apart by comparing structured facts recorded in each analysis rather than the text written about them, so a sentence that was rewritten does not appear as a change to the system, and a fact supplied for the first time appears as knowledge rather than as something that moved.
A billing service is analysed twice, a month apart. The raw difference between the two descriptions has dozens of changed sentences. Sorted, it reads like this. One change in the system: the endpoint that receives payment events moved from one address to a versioned one. Two things newly learned: the retry interval for failed charges, and the fact that exports are written to object storage, both of which were true a month ago and simply not recorded. Everything else is rewording. One line deserves a conversation, and it is now easy to see which.
#Silence is not removal
Reporting that something was added is easy, because it is in the new picture and not in the old one. Reporting that something was removed is much harder, and it is where careless comparisons do the most damage.
If the second analysis does not mention the nightly export, there are two very different explanations. Either the export is gone, or the second analysis did not look at the part of the system where it lives. From the text alone you cannot tell which. A comparison that reports it as gone in the second case has invented a fact, and that fact will be repeated to a customer, an auditor or a buyer as if it were established.
The rule
Absence is evidence only where somebody looked. A thing is reported as gone only when the analysis that no longer sees it covered the place it used to be. Everywhere else it becomes a question: this was here before, and this analysis did not look where it was.
This is why every analysis in 1ADK records its coverage, meaning what was examined completely, what was examined in part and what was not examined at all. It is less satisfying than a confident list of removals. It is also the only version of the list you can repeat to somebody outside the company without checking it first.
#Thirty minutes with the developer
A comparison is worth something only when it changes a conversation. The most useful conversation it can change is the regular one between the owner and whoever builds the system, which in most small companies is either a status call about tickets or does not happen at all.
A sorted comparison gives that conversation an agenda, and the agenda is short enough to get through in half an hour.
-
What changed in the system
Only the structural changes, one by one. For each, one question: was this meant, and does anything outside the company depend on the old behaviour? Most will be fine. The one that is not is why the meeting exists.
Ask about intent, not about quality. You are checking the picture, not marking the work.
-
What we learned that was already there
The newly learned facts. These are not anybody doing anything, so they need no explanation, only a quick confirmation that the picture now has them right.
This part usually takes two minutes and quietly removes a dozen future surprises.
-
What is still unknown
The open questions, in the order of how much they matter to a decision coming up. Some have an answer the developer knows. Write it down where the next analysis can find it, rather than in the meeting notes.
An answer given in a meeting and not recorded is an answer you will have to ask for again.
-
Where to look next
One area of the system to cover better in the next analysis, chosen because a question about it is open or because coverage there is thin. Agree on it, and the next comparison will be more useful than this one.
The owner chooses. The reason for the choice is written down.
1ADK assembles exactly this agenda from an accepted analysis and the one before it, with links to the evidence for each line and a copy that can be pasted into meeting notes. You do not need the product to hold the meeting, though. You need the four headings, a sorted comparison and the discipline to keep rewording off the agenda.
#How often to look again
The value of a comparison falls as the period it covers grows. A month of changes can be read in one sitting and attributed to the work that caused it. Six months of changes is a pile in which nobody remembers why anything moved, and the meeting turns into an archaeology session.
There is no universal interval, but there are moments when not looking costs more than looking.
- After a release that touched money, personal data or an integration. These are the changes that become a question from somebody else. Get the comparison before the question arrives.
- Before any conversation with somebody outside. A security questionnaire, a due diligence request, a new enterprise customer, a change of supplier. Arrive with a picture that is current and a list of what changed recently, rather than with the document from last spring.
- When the people change. A developer leaves, an agency is replaced, a technical lead joins. A comparison across that moment shows what the new people did in their first weeks, which is when misunderstandings are cheapest to correct.
- At a steady rhythm otherwise. Monthly is a reasonable default for a product that ships every week or two. The exact number matters less than the period never growing so long that the comparison stops being readable.
#A test on your last release
You can find out where you stand with the release that went out most recently. Answer these without asking the team, and notice how often the honest answer is that you would have to check.
Five questions about the last release
If more than two need somebody else, you have pictures at best, not a history.
- Name anything the last release added that talks to the outside world: a service, a webhook, a new address. These are the changes partners and customers notice before you do.
- Say whether the last release changed where any personal data is written or copied. This is the first thing a privacy question will ask, and the one most often answered from an old document.
- Name something that existed three months ago and does not exist now, and say how you know it is gone. If the answer is that nobody mentions it any more, you are taking silence for removal.
- Name one statement about your system that you repeat to customers, and say when it was last checked. This is the category that expires without anybody noticing.
- Say what the next release is expected to change in the structure of the system, not in its features. A team that cannot say this in advance will not be able to say it afterwards either.
#What people do instead
Treat release notes as the change history of the system.
Instead Release notes describe features for users. They are written by people who decide what is worth mentioning, and structural changes are exactly what they leave out. Keep them, and keep a comparison of the system next to them.
Read the code diff, or ask somebody to summarise it.
Instead A code diff is the right tool for reviewing a change and the wrong one for understanding a system. It is too large, it mixes refactoring with behaviour, and a summary of it is a fresh piece of prose each time, with nothing to compare it against next month.
Ask an AI assistant to describe the system again and compare the two answers by eye.
Instead Two free-text descriptions differ in almost every sentence, so everything looks like change. Compare recorded facts, not prose, or the comparison will train you to stop reading it.
Report anything that is no longer mentioned as removed.
Instead Report it as removed only where the place it lived was examined. Elsewhere, ask. A false removal repeated to a buyer costs more than an honest question.
Do one thorough analysis a year and nothing in between.
Instead A year of change cannot be attributed to anything. Shorter periods produce comparisons that a person can read in one sitting and act on in one meeting.
#What a comparison cannot tell you
A comparison is a strong tool with a narrow job. Plainly, what it does not do.
What this does not do
- It does not tell you why something changed. It tells you what to ask about, and the reason lives with the person who made the change.
- It does not judge whether a change was good. A moved endpoint can be an improvement or a mistake, and the comparison reads the same either way.
- It cannot see what neither analysis covered. A part of the system nobody looked at twice stays outside the history, and an honest comparison says so.
- It describes the system as recorded, not as observed running. Traffic, load and what actually happens at three in the morning are a different kind of evidence.
- It is only as fair as the two analyses are comparable. Two analyses of very different depth will show the difference in depth as well as the difference in the system, which is why coverage is recorded and shown.
What it does give you is the one thing a single picture never can: a record of how your system moved while you were busy with everything else, sorted into what needs a conversation and what does not.
When did your system last change in a way that somebody outside the company noticed before you did, and what would you have needed to know a week earlier?