Duplication is signal

Whenever an agent writes to Wire, or data gets synced in from a connector, it gets ingested. We don’t skip it and we don’t replace what came before, because we think it’s signal.

When considering multiple instances or duplicates of the same content, one common strategy is last write wins. This method optimizes for storage and tries to protect its retrieval mechanism. Storage is the wrong thing to optimize for. The cost of storage hasn’t mattered in over a decade.

Other systems, like markdown based context systems, will just add it again. They have no clue that it was already saved. This method dumps the mess for the retrieval process. Now when you search for context, you find multiple instances that your agent has to sift through in its context window.

Both common strategies don’t consider why the same context shows up multiple times.

Where multiple instances come from

An agent retries a write. A response timed out or some other error occurred. Keeping each allows you to audit and diagnose issues instead of never realizing they exist.

A connector re-pulls an unchanged source. A sync runs every morning and the source material hasn’t changed. The record of each pull indicates it was freshly pulled and that nothing changed. Your agent can reason over the changes occurring across pulls, all while being confident a sync was never missed.

An event fires repeatedly. Log lines, alerts, webhook payloads. The same line arriving 400 times with metadata is how your agent can prioritize and diagnose why the event occurred.

Two people upload the same document. Each upload brings provenance. Who had this file, and when, could matter for questions about how knowledge moved through a team.

An agent restates something it already knows. Persistent-memory agents write down what they learn, and they re-learn things. Continual re-learning is a good signal that the right context isn’t reaching your agent at the right time.

Most systems disregard all that signal. It’s too valuable for us to ignore.

What Wire does instead

Every write stands on its own. Each instance of the same content is a complete entry, with its own timestamp, its own source, its own metadata, and its own place in the graph, exactly like the first. You can navigate to any instance directly. The context is the same but the metadata around it like where it came from, when it was ingested, and any other properties you or your agent share are unique to the write. An agent walking the container sees every occurrence in order. Specifically for event-shaped data, the occurrences are the data.

When content is duplicated, the instances get grouped. The group tracks how many instances exist, when the first was seen, and when the most recent arrived. Nothing is promoted, nothing is replaced, and deleting one instance never disturbs the others. Remove one occurrence or the whole group in one call, and either way the remaining entries are already whole.

The two stages where duplication actually costs something are each handled at that stage:

Ingestion. Identical bytes produce an identical embedding, so a repeated write copies the existing vector value instead of calling the embedding model, and inherits the analysis that already ran.

Retrieval. Search is the surface where repetition is noise. Five copies of the same context would crowd out others. Our solution was to collapse upon search. Results return one representative per group. Deterministically the most recent instance. It’s returned with the group’s metadata attached, including the instance count, first and last seen, and a reference the agent can follow to walk every occurrence. An agent that doesn’t care about duplicates sees one clean result.

The principle

We want to remember what happened. Happening again is part of that story. Deduplication that deletes is destroying data to solve a presentation problem that belongs to one read path. Keep every instance, group them, make each one navigable, and collapse exactly where repetition stops being signal.

This is live on Wire’s Beta runtime today, and it’s built into the container runtime we plan to open-source once the migration completes.