Good enough
is not good enough

This way of collecting data is very wasteful.

That is how an Akamai specialist described the way many AI crawlers gather material from the web.^[1] The report around the quotation gives the concrete mechanism: some botnets revisit an entire site every day merely to discover what changed, transferring the same material again and again. This is not new: search engines have been doing it for decades, in the very same wasteful way, but the scale of "scraping" is now such that what we have known as the "open web" is now closing down, an API at a time^[2], a "bot filter" at a time^[3], effectively reducing the global value for everybody, while increasing the waste of resources.

The striking part is not that the method is wasteful. Everybody running a sizeable website knows that by now. The striking part is that we speak as though this were a law of nature.

It is not. We knew how to avoid most of this waste in the 1990s, and it turns out I actively tried to push forward a concrete solution, when the problem was still manageable. We'll get back to this.

For nearly three decades, I have repeatedly encountered the same pattern: a system that works well enough for every actor taken separately, while imposing a growing cost on everybody taken together. Each local decision is reasonable. Their aggregate result is damaging. Since nobody owns the aggregate problem, the better solution remains nobody's job.

It is time to call out this state of affairs, and act upon it... again: I personally tried many times, and now I can build on that experience. The first step is to clearly understand what is happening and why: which actor does what and why, who gains, who loses. In the coming weeks I will reconstruct one instance of the pattern from dated documents. It begins with web crawlers and ends with the institutions we build—or fail to build—for shared digital infrastructure.

Let me point at the destination from the outset: I direct Software Heritage, a non-profit organisation that collects public source code once so that others do not all have to collect it again. This is my latest effort to contribute to systematically address this long standing problem.

It is a problem that starts with two simple words: good enough.

One of my first encounters with the "good enough" curse came in the mid-nineties, during the ferocious battle to control the rapidly expanding cyberspace.

WebCrawler, AltaVista, Alexa, Yahoo… Google, the eventual winner, had not even been born yet.

What stunned me was that all of them did essentially the same thing: download the Web again and again to discover what had changed.

Sounds familiar? It should: it is exactly what AI crawlers do now.

The Web was much smaller then, but bandwidth, compute and storage were scarcer too, so the waste was already real.

And there was an obvious alternative.

In January 1998, with Pablo E. Martínez López — aka Fidel — I wrote Distributed Robots: a Technology for Fast Web Indexing. It entered the IETF record as an individual Internet-Draft on 4 March 1998,^[4] carrying the wonderfully explicit line: "EXPIRES SEPT 1998."

The idea was elementary, familiar to any systems programmer: interrupt-driven beats busy-wait.

A web server knows, cheaply, when its files change. Let it say so, and let crawlers fetch only what is new — instead of asking every server, over and over, "Anything changed?"

In May 1998 an IETF Area Director sent us a generous, detailed critique. By then I had moved on: that March I had put Piège dans le Cyberespace online — CyberSnare in English — which went viral, started thirty years of work on free and open source software, and became a book with Dominique Nora.^[5]

It took me seven months to answer point by point. I first asked what he thought of the revised structure.

The answer was an automatic out-of-office message. The thread ended there.

We produced the revised draft anyway, dated 20 April 1999, incorporating the review.^[6] Fidel sent it to the RFC Editor. No answer came.

But the problem had not disappeared. In 2001 the IETF's own WEBI group independently produced Requirements for a Resource Update Protocol.

That expired too.^[7]

Years later, looking at my web server logs, I found that in 2009 Googlebot had fetched one unchanged page twenty-four times.^[8]

Students of mine implemented the Remote Update Protocol twice, in Java and OCaml.^[9] The technical idea was not the hard part.

And this is the point: a better technical solution does not deploy itself.

Someone must maintain it, persuade others to adopt it, integrate it into existing systems, and keep pushing after the prototype works. Advocacy on the side of a full-time job is not an institution.

The alternative was written down almost thirty years ago, then rediscovered independently inside the IETF.

It was not defeated by a better idea.
It was not shown to be technically impossible.
It simply never acquired enough organised support to become infrastructure.

Meanwhile, the "good enough" solution kept scaling.

Why?

So why did it survive?

Not because anyone defended it. Because it was made liveable, one patch at a time.

Let's be precise about what "good enough" means: it does not mean bad. A good enough system does not do the right thing, but it does get the thing done. It is a hack, a wooden leg that gets you across the room… And because it gets you across the room, you stop looking for a better leg.

Downloading the whole web to find out what changed was never sensible. It was patched until it was bearable.

In 1994 Martijn Koster proposed robots.txt, so a site could say: not this, not here. It worked well enough that nobody standardised it until 2022.^[10]

In 2006 came sitemaps, so a site could say: here is what I have, and here is when it last changed.

Read the specification closely. The freshness field is "a hint and not a command", and crawlers may ignore it.^[11] So a site can say exactly what changed, and you are free not to believe it and download everything anyway. Which is what happened.

Then the crawlers got cleverer about where to spend. Google's own documentation says it plainly: "URLs that are more popular on the Internet tend to be crawled more often."^[12] And if your server struggles, the crawler backs off.

Add it up and you have a system that works. Big sites get visited often, small sites rarely, a struggling server gets a break. Nobody is delighted, but nobody is ruined.

That is what good enough looks like from the inside: not a catastrophe, but a tolerable arrangement everyone has adapted to.

And then the second thing happens, which is worse: it becomes normal.

You are almost certainly reading this on a QWERTY keyboard. Nobody chose it this morning. Everybody knows it, every keyboard has it, and whoever switches first pays the whole cost alone. So it stays: not because anyone re-decided it, but because it is installed.

That is where web crawling ended up. Installed. It always worked like this.

In December 2007 I gave a talk at Inria's fortieth anniversary in Lille, written up as an article I revised into 2011. I wrote that the current system was working « assez bien ».^[13] French had no phrase for what I meant, so I borrowed one, in a footnote: « Good enough, comme on dit chez nos amis anglosaxons. »

I was not complaining. I was describing. It was accurate.

Lock-in is harmless when the stakes are low. Nobody is much hurt by a keyboard layout.

Repeated crawling is not. It spends bandwidth, electricity and machine time on a planetary scale to fetch what has not changed.

Thirty years on from robots.txt, the patches are not holding, and everyone can see it.

So the question is not why nobody noticed the waste. Everybody noticed.

The question is why a cost that large stays invisible to the people who could act on it, until it is too late to act easily.

That is not a technical question. Economists have a name for it, and it is the whole point that we need to delve into.

AI bots are redownloading the web at scale, just to see what is new. Everyone can see the waste. Almost nobody can see the full bill.

That is a large part of the problem.

It is tempting to point the finger at somebody who is greedy, careless or malicious. Sometimes that may even be justified. But it does not explain why the same pattern keeps appearing among competent people acting in good faith. My 1998 IETF draft (RUP) showed how to cut the polling; sitemaps let a site declare what changed. Yet twenty years later the repeated downloading continues.^[14]

If we want to change the outcome, we first need to understand the mechanism.

Economists have a name for part of it: an externality.^[15]

Suppose there are N sites and M organisations that want an up-to-date copy or index. In the naïve arrangement, every collector must discover and revisit every site: N × M relationships, policies, rate limits and repeated transfers.

The cost of each crawl may look small to the crawler, and merely irritating to each site. But the aggregate cost can be enormous, while appearing on no single balance sheet.^[16]

We could do much better. Mutualise the collection: one shared infrastructure talks to the N sites. The M organisations obtain the data from it. We move from N × M interactions to N + M.

With a thousand sites and ten collectors, that is 1,010 relationships instead of 10,000.

More importantly, no page is fetched ten times over just to learn it has not changed.

That is more than my 1998 draft did: it cut the polling, not the connections.

This does not mean one machine, one copy, or one organisation controlling everything. The data can be replicated, held by independent institutions under different jurisdictions.

"Collect once" is not "store once".

Why don't we do that? The technique is not missing, and people are not villains. It is what economists call a market failure.^[17] Whoever operates the shared path pays for it, while the benefit goes to everyone: exactly the kind of good that markets under-provide, a public good nobody funds. A sound protocol does not create the institution whose job is to run it.

There is another trap. Universities, companies and research infrastructures respond by building their own partial catalogues, caches or archives, reproducing the problem one level higher. Every new silo becomes another place that others must discover, query and reconcile.

So the answer is not to punish people for using an open resource, nor to put a tollbooth in front of it. It is to make the efficient shared path exist, and easier to use than rebuilding the same machinery again and again.

That requires engineering, of course.

But it also requires resources, governance and a mandate: a market failure is not fixed by moral injunctions, but by making someone responsible for the shared path.

And this is where "good enough" stops being merely a technical problem.