RUP: a protocol for efficient updating of web search engines
In 1998 and 1999, with Pablo E. Martínez López (Fidel), I proposed a mechanism to stop web indexers from re-downloading whole sites merely to discover what had changed. A server already knows when its own files change; it should be able to say so, and a crawler should be able to fetch only what is new. The pages below are the surviving record. They are kept online unchanged, as evidence rather than as a live proposal.
The documents
- draft-rfced-exp-cosmo-00.txt — Distributed Robots: a Technology for Fast Web Indexing. Written January 1998; entered the IETF record as an individual Internet-Draft on 4 March 1998. It prints “EXPIRES SEPT 1998” and gives no day. The Datatracker record is still there. Less formal than what follows, and the easier introduction to the idea.
- draft-RUP-01.txt — The Remote Update Protocol (RUP). Part I: RUP Architecture, dated 20 April 1999, marked “Expires 20 October 1999”. The revision written in response to the IESG review described below. It was submitted — to the RFC Editor in April 1999 — and no reply ever came; no IETF archival record of it exists.
- RUP-PartII-OBJCmd.ps — Part II, the command implementation.
What happened to it
On 10 May 1998 Patrik Fältström, then IETF Applications Area Director, reviewed the draft in the IESG and sent a detailed and constructive critique: the work overlapped the FIND working group, and rather than define a new protocol we should define objects as MIME types carried over any transport, explaining how this differed from CIP, SOIF and the W3C's RDF. He also raised URL matching, internationalisation and security at registration.
On 8 December 1998 we answered every point, accepted the MIME and SOIF encapsulation, and asked one question before resubmitting: what did he think of the new organisation we intended to adopt? The reply that came back was an automatic out-of-office message. There was no further correspondence.
The revised draft was written anyway, and it was sent in: on 16 April 1999 Pablo Martínez López submitted it to the RFC Editor, with the file attached, as an update to the 1998 draft. No reply ever came — from the same address that had answered the 1998 submission twice within hours — and there is no IETF record of it. This was not a rejection on the merits and nobody behaved badly — a volunteer reviewer was away for a fortnight, and after that carrying the work forward was simply nobody's job. In February 2001 the IETF's own WEBI working group independently opened draft-ietf-webi-rup-reqs, “Requirements for a Resource Update Protocol” — same acronym, same push-versus-poll diagnosis, aimed at caching proxies rather than crawlers. That expired too.
Implementations
Both were student projects at Paris 7 — Travaux d'Étude et de Recherche on the subject “indexation rapide du Web”, which I set for two years running. The protocol was built more than once. What never formed was anything to carry it further.
- The RUP 1.0 Java Servlet implementation, by Yerom-David Bromberg — rupjava-1.0.tar.gz. Its documentation can also be read here, a page at a time: robot in client mode, robot in server mode, API documentation. Those pages were authored as a frameset around 2000 and are served here without frames, so the navigation is gone and this site's own banner sits over them; download the archive and open main.htm to see them as they were meant to look.
- OCamlRup — ocamlrup-0.1.tar.gz — a RUP client, a generic RUP server and an Apache CGI server, with rupinfo.txt parsing and robots.txt integration, written in OCaml in 2001 by Samuel Lasry and Xavier Patourel. Sources and documentation; the 2001 binaries and compiled objects are not included.
Copyright and licence for both implementations. RupJava is copyright © Yerom-David Bromberg; OCamlRup is copyright © 2001 Samuel Lasry and Xavier Patourel. Neither carries a licence and none is asserted here.
OCamlRup in detail. Copyright © 2001 Samuel Lasry and Xavier Patourel. They attached no licence to their code and none is asserted here: copyright remains theirs, and this copy is published as an archival record of their project, with attribution and without any claim of ownership or grant of rights. If you are one of the authors and would prefer it taken down, or would like to attach a licence to it, write to me and it will be done. The mimestring, netbuffer, netencoding, netstream, netstring_str and neturl modules are not theirs: they belong to Gerd Stolpmann's netstring package and carry its own permission notice, reproduced in the archive. The same information is in the PROVENANCE file inside the tarball.
Project members
- Roberto Di Cosmo
- Pablo E. Martínez López (Fidel) — UNLP, Argentina at the time; now Universidad Nacional de Quilmes
This page is referenced from Good enough is not good enough, a series about why the open web is collected so badly and what it costs.