Good enough
is not good enough

This way of collecting data is very wasteful.

That is how an Akamai specialist described the way many AI crawlers gather material from the web.^[1] The report around the quotation gives the concrete mechanism: some botnets revisit an entire site every day merely to discover what changed, transferring the same material again and again. This is not new: search engines have been doing it for decades, in the very same wasteful way, but the scale of "scraping" is now such that what we have known as the "open web" is now closing down, an API at a time^[2], a "bot filter" at a time^[3], effectively reducing the global value for everybody, while increasing the waste of resources.

The striking part is not that the method is wasteful. Everybody running a sizeable website knows that by now. The striking part is that we speak as though this were a law of nature.

It is not. We knew how to avoid most of this waste in the 1990s, and it turns out I actively tried to push forward a concrete solution, when the problem was still manageable. We'll get back to this.

For nearly three decades, I have repeatedly encountered the same pattern: a system that works well enough for every actor taken separately, while imposing a growing cost on everybody taken together. Each local decision is reasonable. Their aggregate result is damaging. Since nobody owns the aggregate problem, the better solution remains nobody's job.

It is time to call out this state of affairs, and act upon it... again: I personally tried many times, and now I can build on that experience. The first step is to clearly understand what is happening and why: which actor does what and why, who gains, who loses. In the coming weeks I will reconstruct one instance of the pattern from dated documents. It begins with web crawlers and ends with the institutions we build—or fail to build—for shared digital infrastructure.

Let me point at the destination from the outset: I direct Software Heritage, a non-profit organisation that collects public source code once so that others do not all have to collect it again. This is my latest effort to contribute to systematically address this long standing problem.

It is a problem that starts with two simple words: good enough.

One of my first encounters with the "good enough" curse came in the mid-nineties, during the ferocious battle to control the rapidly expanding cyberspace.

WebCrawler, AltaVista, Alexa, Yahoo… Google, the eventual winner, had not even been born yet.

What stunned me was that all of them did essentially the same thing: download the Web again and again to discover what had changed.

Sounds familiar? It should: it is exactly what AI crawlers do now.

The Web was much smaller then, but bandwidth, compute and storage were scarcer too, so the waste was already real.

And there was an obvious alternative.

In January 1998, with Pablo E. Martínez López — aka Fidel — I wrote Distributed Robots: a Technology for Fast Web Indexing. It entered the IETF record as an individual Internet-Draft on 4 March 1998,^[4] carrying the wonderfully explicit line: "EXPIRES SEPT 1998."

The idea was elementary, familiar to any systems programmer: interrupt-driven beats busy-wait.

A web server knows, cheaply, when its files change. Let it say so, and let crawlers fetch only what is new — instead of asking every server, over and over, "Anything changed?"

In May 1998 an IETF Area Director sent us a generous, detailed critique. By then I had moved on: that March I had put Piège dans le Cyberespace online — CyberSnare in English — which went viral, started thirty years of work on free and open source software, and became a book with Dominique Nora.^[5]

It took me seven months to answer point by point. I first asked what he thought of the revised structure.

The answer was an automatic out-of-office message. The thread ended there.

We produced the revised draft anyway, dated 20 April 1999, incorporating the review.^[6] Fidel sent it to the RFC Editor. No answer came.

But the problem had not disappeared. In 2001 the IETF's own WEBI group independently produced Requirements for a Resource Update Protocol.

That expired too.^[7]

Years later, looking at my web server logs, I found that in 2009 Googlebot had fetched one unchanged page twenty-four times.^[8]

Students of mine implemented the Remote Update Protocol twice, in Java and OCaml.^[9] The technical idea was not the hard part.

And this is the point: a better technical solution does not deploy itself.

Someone must maintain it, persuade others to adopt it, integrate it into existing systems, and keep pushing after the prototype works. Advocacy on the side of a full-time job is not an institution.

The alternative was written down almost thirty years ago, then rediscovered independently inside the IETF.

It was not defeated by a better idea.
It was not shown to be technically impossible.
It simply never acquired enough organised support to become infrastructure.

Meanwhile, the "good enough" solution kept scaling.

Why?

So why did it survive?

Not because anyone defended it. Because it was made liveable, one patch at a time.

Let's be precise about what "good enough" means: it does not mean bad. A good enough system does not do the right thing, but it does get the thing done. It is a hack, a wooden leg that gets you across the room… And because it gets you across the room, you stop looking for a better leg.

Downloading the whole web to find out what changed was never sensible. It was patched until it was bearable.

In 1994 Martijn Koster proposed robots.txt, so a site could say: not this, not here. It worked well enough that nobody standardised it until 2022.^[10]

In 2006 came sitemaps, so a site could say: here is what I have, and here is when it last changed.

Read the specification closely. The freshness field is "a hint and not a command", and crawlers may ignore it.^[11] So a site can say exactly what changed, and you are free not to believe it and download everything anyway. Which is what happened.

Then the crawlers got cleverer about where to spend. Google's own documentation says it plainly: "URLs that are more popular on the Internet tend to be crawled more often."^[12] And if your server struggles, the crawler backs off.

Add it up and you have a system that works. Big sites get visited often, small sites rarely, a struggling server gets a break. Nobody is delighted, but nobody is ruined.

That is what good enough looks like from the inside: not a catastrophe, but a tolerable arrangement everyone has adapted to.

And then the second thing happens, which is worse: it becomes normal.

You are almost certainly reading this on a QWERTY keyboard. Nobody chose it this morning. Everybody knows it, every keyboard has it, and whoever switches first pays the whole cost alone. So it stays: not because anyone re-decided it, but because it is installed.

That is where web crawling ended up. Installed. It always worked like this.

In December 2007 I gave a talk at Inria's fortieth anniversary in Lille, written up as an article I revised into 2011. I wrote that the current system was working « assez bien ».^[13] French had no phrase for what I meant, so I borrowed one, in a footnote: « Good enough, comme on dit chez nos amis anglosaxons. »

I was not complaining. I was describing. It was accurate.

Lock-in is harmless when the stakes are low. Nobody is much hurt by a keyboard layout.

Repeated crawling is not. It spends bandwidth, electricity and machine time on a planetary scale to fetch what has not changed.

Thirty years on from robots.txt, the patches are not holding, and everyone can see it.

So the question is not why nobody noticed the waste. Everybody noticed.

The question is why a cost that large stays invisible to the people who could act on it, until it is too late to act easily.

That is not a technical question. Economists have a name for it, and it is the whole point that we need to delve into.

AI bots are redownloading the web at scale, just to see what is new. Everyone can see the waste. Almost nobody can see the full bill.

That is a large part of the problem.

It is tempting to point the finger at somebody who is greedy, careless or malicious. Sometimes that may even be justified. But it does not explain why the same pattern keeps appearing among competent people acting in good faith. My 1998 IETF draft (RUP) showed how to cut the polling; sitemaps let a site declare what changed. Yet twenty years later the repeated downloading continues.^[14]

If we want to change the outcome, we first need to understand the mechanism.

Economists have a name for part of it: an externality.^[15]

Suppose there are N sites and M organisations that want an up-to-date copy or index. In the naïve arrangement, every collector must discover and revisit every site: N × M relationships, policies, rate limits and repeated transfers.

The cost of each crawl may look small to the crawler, and merely irritating to each site. But the aggregate cost can be enormous, while appearing on no single balance sheet.^[16]

We could do much better. Mutualise the collection: one shared infrastructure talks to the N sites. The M organisations obtain the data from it. We move from N × M interactions to N + M.

With a thousand sites and ten collectors, that is 1,010 relationships instead of 10,000.

More importantly, no page is fetched ten times over just to learn it has not changed.

That is more than my 1998 draft did: it cut the polling, not the connections.

This does not mean one machine, one copy, or one organisation controlling everything. The data can be replicated, held by independent institutions under different jurisdictions.

"Collect once" is not "store once".

Why don't we do that? The technique is not missing, and people are not villains. It is what economists call a market failure.^[17] Whoever operates the shared path pays for it, while the benefit goes to everyone: exactly the kind of good that markets under-provide, a public good nobody funds. A sound protocol does not create the institution whose job is to run it.

There is another trap. Universities, companies and research infrastructures respond by building their own partial catalogues, caches or archives, reproducing the problem one level higher. Every new silo becomes another place that others must discover, query and reconcile.

So the answer is not to punish people for using an open resource, nor to put a tollbooth in front of it. It is to make the efficient shared path exist, and easier to use than rebuilding the same machinery again and again.

That requires engineering, of course.

But it also requires resources, governance and a mandate: a market failure is not fixed by moral injunctions, but by making someone responsible for the shared path.

And this is where "good enough" stops being merely a technical problem.

Almost thirty years ago the externality was known, and seen as easy to treat as an elegant systems problem. We made it bearable a patch at a time. Some search engines did eventually get half of the fix: since 2021 a protocol called IndexNow lets a site announce a change once, to everyone^[18]. It moves announcements, not bytes, and the crawlers that arrived in 2024 use none of it: the bill started to explode, the pain grew, and other patches arrived.

That summer Software Heritage (the archive I direct) was hit by waves of automated collection associated with AI training.^[19] Some public services became unresponsive. Engineers who should have been improving the archive spent their time identifying traffic, blocking addresses and restoring service.

We were not an isolated case, far from it. Small and medium-sized web operators described the same sequence: repeated retrieval of unchanged material, rapidly growing bandwidth and compute bills, degraded service for human users, then emergency blocking. Large platforms tightened unauthenticated access. Smaller ones put computational challenges in front of their websites or shut expensive paths altogether.

These reactions are rational. A service under pressure has to defend itself with the means available that week.

Their aggregate effect is damning. Defensive restrictions do not only stop abusive collectors. They also make legitimate research, preservation, accessibility tools, independent search and ordinary automation harder. The open web becomes less open one emergency measure at a time.^[20]

This is how an externality turns into institutional damage. The first-order waste is duplicated traffic. The second-order cost is that every site is pushed to make access more brittle for everybody.

Let's not be naive: some actors act abusively, regardless of efficiency costs. They may disregard declared restrictions, conceal identity or support uses the producer rejects.

A shared collector does not make those disputes disappear. It does make them tractable.

A producer can interact with a small number of identifiable collection services instead of an adversarial cloud of indistinguishable bots. Terms, provenance and update information can be recorded once. Disagreements can be attached to a stable process rather than fought through IP-address whack-a-mole.

A global shared infrastructure has its challenges: will it slow down innovation? Who owns it? Who governs it? But the current "far west" alternative to shared infrastructure is not freedom from governance. It is thousands of hurried, uncoordinated and incompatible governance decisions embedded in firewalls.

This is not the first time that I am confronted first-hand with this pattern: in academia we faced a similar challenge when internet and the web changed scientific publishing, and I had a seat in the front row.

The problem is not confined to commercial crawlers. The research community reproduced the same structure for decades, dressed in the same costume: a law of nature.

In 2005–2006 I wrote about scientific publishing and listed five things a researcher needs from publication^[21]: a date that proves who was first; integrity, no change after it appears; circulation without barriers; very-long-term preservation; one reference that never breaks. Judging quality is not on the list; it belongs to editorial boards. The five are infrastructure, and arXiv with its overlay journals showed how to deliver them. I saw the forced transfer of copyright as the blocker, not the infrastructure. When in October 2015 a French bill arrived carrying an article titled "free access to the scientific publications of public research"^[22], I filed an amendment. One line: forbid the exclusive transfer of rights. Simple, clear, and it would have worked.

Instead, the law that passed in October 2016^[23] grants authors the right to deposit the manuscript their publisher had accepted (never the published version), never for commercial republication, and only after an embargo: six months in the sciences, twelve in the humanities. A law granting researchers permission to circulate their own work, as though the right were the publisher's to concede. Notice which solution won: not the simple one that removes the problem, the complicated one that preserves it.

The simple fix had a vested interest against it; the complicated one had only the unorganised research community against it, which is why it won.

Meanwhile the infrastructure, which I had considered solved in 2005 — just put everything on arXiv — became a problem of its own. Universities and research institutions, footing the bill, bet on Open Access repositories. They built them. Thousands of them.^[24] Each local decision was rationally defensible: reporting obligations, local curation, independence from any single service.

The global result was dismal: every aggregator must now find, poll and reconcile thousands of endpoints, where one article lives in many copies with different metadata, different timestamps and, by the law's own design, different texts. By the 2006 list, integrity and unique reference both fail. Local autonomy, implemented as disorganised global duplication.

The lesson, hard earned: keep the local functions where they belong, mutualise aggressively what is universal — durable identification, preservation, deduplication, discovery — and replicate that layer so it depends on no one operator.

It was never a technical failure; repository teams built what their institutions commissioned. The absence was above their level, and above mine: nobody had a mandate strong enough to provide the common layer.

Post 06 ended with a rule: keep local functions where they belong, mutualise what is universal.

What does that look like in practice?

A concrete example. In May, the Open Invention Network, a sponsor of Software Heritage, announced that the archivable source code behind its OIN 2.0 Linux System — more than 5,100 open-source packages, around 900 million lines of code — is preserved in Software Heritage.^[25]

OIN did not have to build and operate another universal crawler and archive. It could build on a collection layer that already existed.

The important point is not the name of the collector. It is the architecture.

Collect once into one uniform model. Deduplicate globally. Give every captured artefact an intrinsic identifier derived from the artefact itself, not assigned by a registry. Then let many services build on the result.

But one logical collection does not mean one physical copy.

A serious archive must have complete replicas held by independent institutions, in different locations and jurisdictions.

One logical collection, plural custody.

Europe can anchor such an infrastructure, but an archive of the world's source code cannot credibly remain a purely European asset. A universal archive needs genuinely international custody and governance.

Intrinsic identifiers are easy to oversell, so let me be precise about what they buy.

Precision: we can establish whether we are talking about the same file, directory, revision or release.

Independent verification: for a file, anybody holding the bytes can recompute the identifier, without consulting a registry.

Portability: the reference survives movement between servers and organisations.

What they do not buy is just as important.

An identifier does not preserve the object it names. It does not guarantee neutrality or good governance. And it does not settle law, consent or authorship.

A hash establishes identity, not legitimacy.

Since April 2025, SWHID is ISO/IEC 18670^[26]. That matters because a procurement rule, policy or compliance process can now refer to an international standard, not to one organisation's convention.

Software Heritage is one implementation of this architecture.

The point of the architecture is precisely that you should not have to trust Software Heritage to verify what it says.

If the case for the shared layer depended on trusting us, we would have designed it wrong.

“Collect once” may sound like a slogan. It is not. There is quite a bit of engineering behind those two words.

Take Software Heritage.

We do not download every repository again and again just to discover whether something changed.

When a forge provides the necessary information, we first retrieve a compact listing of its projects and their recent activity^[27]. We compare that with what we already know. If nothing changed, there is nothing to do.^[28]

And when a repository did change, Git gives us another important advantage: we negotiate what is missing and transfer the new objects^[29], instead of downloading the whole repository again.

This is still polling, and I would prefer notification. But it is already very different from blindly asking millions of repositories, over and over, “anything new?”

Of course, reality is messier.

Not every forge tells us reliably what changed. Not everything is Git. Projects move, disappear, become private or rewrite their history. Supporting all of this is a permanent engineering job.

And notification, by itself, only solves half of the problem.

This is something I had missed in 1998. My proposal let a server notify crawlers when something changed, but every crawler still had to register with every server.

Today IndexNow goes further: a site can announce a change once, and participating search engines share that notification^[30].

That is better.

But every search engine still goes back and fetches the page for itself.

Announcing once is not collecting once.

A shared collector removes that second multiplication too: detect the change once, retrieve it once, verify it once, preserve it once — then let everybody reuse the result.

There is another kind of duplication we should avoid as well.

Every organisation using collected material needs to know where it came from, when it was seen, and what terms and metadata accompanied it. Today that evidence is repeatedly reconstructed, imperfectly, by everybody downstream.

A shared archive cannot decide every question of copyright, consent or jurisdiction. Nor should it try.

But it can preserve the evidence: where and when something was found, what public information accompanied it, and what corrections or claims came later.

So the old problem has two parts.

Avoid repeatedly discovering and downloading the same thing.

And avoid repeatedly reconstructing what we already knew about it.

Both are infrastructure problems. And infrastructure has an owner, a budget and a horizon — or it has none of them.

The open paths are being walled off. The awkward part is that the clients easiest to recognise as recurrent visitors are often the ones behaving best.

A preservation service, research crawler or webhook normally does what we ask responsible automated clients to do: it comes from predictable infrastructure, identifies itself, and accepts a rate limit.

That also makes it very easy to block.

A large abusive collector can do the opposite: spread requests over many addresses, rotate identities, and run full browser engines^[31] so that each request looks much more like ordinary human traffic.

So a wall based on repeated or easily identifiable access has a perverse effect: the honest client presents a stable target; the abusive one can disappear into the crowd.

That does not make the wall irrational. When a site is being overwhelmed, the operator’s first duty is to keep it running.

SourceHut faced exactly this in 2025. After days of disruption from LLM crawlers, it put a proof-of-work challenge in front of the web interface, while leaving API access and Git operations untouched.^[32]

That is the right distinction.

Open access does not mean that every interface must accept unlimited anonymous traffic. Authentication, published quotas and dedicated archival feeds are perfectly compatible with openness. What matters is that accountable machine access has an efficient path, instead of forcing every serious client to pretend to be a browser.

And a legitimate large-scale user should not need to crawl the world in the first place. If a documented, versioned, provenance-carrying corpus is available through an efficient bulk interface, using it should be easier than launching yet another crawler. Those terms should be published and the same for everyone — an efficient path in, not a private arrangement with the largest users.

This is where a shared layer changes more than bandwidth. Its access rules can be published, argued about and improved. And somebody has to be accountable for them. When collection stays fragmented, the same decisions are made thousands of times, invisibly, in firewall rules and emergency patches.

We have solved this kind of coordination problem before.

Early versions of the European copyright directive would have put software-development platforms in scope of the upload-filtering debate. The final text explicitly excludes “open source software development and sharing platforms^[33]”. Many people worked for that exclusion; I spent part of 2018 and 2019 on it too^[34].

Collective action can change the default.

Disclosure: GitHub and Microsoft support Software Heritage. That support does not make our interests identical.

I would like to say that Software Heritage consistently followed the principle I have just stated.

We did not.