Network Working Group R. Di Cosmo Internet Draft ENS France P.E. Martinez Lopez 20 April 1999 UNLP Argentina Expires 20 October 1999 The Remote Update Protocol (RUP). Part I: RUP Architecture. April 20, 1999 Status of this Memo This document is an Internet-Draft and is in full conformance with all provisions of Section 10 of RFC2026. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet-Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html. Copyright Notice Copyright (C) The Internet Society (1998). All Rights Reserved. About certain keywords The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119. 1 Abstract Web Robots are centralized agents that gather data about the web in centralized databases - a process we call Web indexing. Usually a robot performs by itself all the tasks needed to discover the updates in local web spaces mantained by Web Servers, something which is clearly inefficient. We propose a protocol (the Remote Update Protocol, RUP) for cooperation between Web Servers and Web Robots in order to increase the reliability of Web indexing, and decrease the load both on the server and the robot side. If the servers conform to the RUP 1 Internet Draft RUP Architecture April 1999 protocol, the task of the Robot will appear to be distributed between the servers that it consults. This protocol uses well known techniques for encapsulation of data (MIME objects) and well known transport protocols (HTTP, SMTP, etc.), being in this way easy to be integrated in the web. In this document we discuss the architecture of RUP, its main features and notions. Subsequent documents will describe the format of data and other issues, in such a way as to obtain a first operative version of RUP. 2 Introduction 2.1 Background Web Robots are agents that automatically traverse the Web's hypertext structure in order to perform mainly indexing and/or manteinance tasks [Kos95 ]. Usually, the Robot is a centralized program that connects to the servers in order to retrieve and index the relevant documents storing the index information in a central database. Due to communication latency, exponential growth in the number of Web Servers, multi-headed servers and various other factors, the task of indexing the whole Web is a daunting one: a short back-of-the-envelope calculation, assuming a 10 seconds delay between requests to avoid overloading the servers, and an average of 100 million URLs to index, even forgetting about the bandwidth necessary to transfer the actual data, shows that we would need more than 30 years to index the Web using one centralized robot performing all the tasks needed, and several months even supposing to have hundreds of independent robots examining disjoint partitions of the Web. Considering the widely variable lifetime of URLs, this means that the snapshot taken by a web search robot is doomed to be a pretty old one, so that the probability of getting dead URLs as a result of a search on a web index is quite high, and bound to increase steadily, unless some radical change in indexing technology occurs. 2.2 Proposal The purpose of the present draft is to propose such a new technology, via a public protocol for Remote Updates. The key observation is that, as always, work should be done where it costs less: checking what is new on a local web space is best done by the Web Server, not by an external search engine. Better, checking modifications of the server's file system is a task that is already performed on their own by many Webmasters, for security and administrative reasons, on a daily basis. Hence, it is the Web Server that should notify registered robots, on a periodic basis, of relevant Di Cosmo, Martinez Lopez Experimental [Page 2] Internet Draft RUP Architecture April 1999 modifications, and provide unregistered robots with the ability to query for modifications occurred over a designated span of time, thus taking a relevant part of the workload off the robots. Also, the Web Server is the best able to know whether some URLs in its domain are not to be indexed by a given robot (like syntetic ones, or local temporary links, etc.), and this information is already available on the site through the /robots.txt file, covered in the Robot Exclusion Protocol (REP) [Kos94 , KLM96 ]. 2.3 Advantages Obtained Combining these local informations (modification logs and exclusion preferences) with a registration mechanisms for indexing robots, we can obtain the following advantages: o lower server load: registered robots will no longer crush the server with bursts of GET or HEAD http requests covering the whole server's URL addressing space. o lower robot load: registered robots will only have to retrieve modified URLs, leaving index information about unchanged URLs untouched. o lower bandwidth usage: besides drastically reducing the bandwidth abuse due to indexing bursts, the remote update protocol may further reduce bandwidth usage by sending back modification information to robots using some sort of asynchronous communication (e.g. e-mail messages). o increased index liveness: the remote update mechanism allows to maintain more up-to-date indexes, and to discover modification patterns that will allow the robot to apply sound reindexing policies (like tagging "hot" servers with higher reindexing priorities, while using lower priorities for relatively stable ones). It is worth noting that what we are proposing is the web equivalent of replacing polling of an input device with interrupt driven technology, with similar benefits. 2.4 Similar Approaches and Related Work Being such a simple idea, the proposal of Web Servers collaborating in the construction of common indexes is not a completely new one. This work has several points in common with the one performed by the FIND working group [HF ] in the CIP protocol [AM98a , AM98b ]. In those works, the CIP protocol is presented as a mechanism in Di Cosmo, Martinez Lopez Experimental [Page 3] Internet Draft RUP Architecture April 1999 which a mesh of servers that are hosts for a distributed database, interchange indexing information. This information is used by a server to reroute queries to the one that actually has the searched data. The mesh of servers should posses some hierarchical topology, and then servers that are higher in the hierarchy posses more comprehensive indexes. In that sense, our proposal can be seen as a degenerate case of CIP, where there is only one root node (the Web Robot) and zillions of leaf nodes (all the Web Servers). In this case, RUP can be defined as an instantiation of CIP where the notions of incorporation of new leaves to the mesh, and of flow of indexing information are defined as in this document. This decision is left open, having in this way the possibility to have different versions of RUP, some of them instances of CIP, and some others not. 3 Protocol Description The basic idea of the RUP protocol is that every Web Server implementing the protocol mantains information about the updates performed in its local web space, and that a Web Robot can agree with the server in the exact way this information is sent from the server to the robot. For that reason, the RUP protocol is composed of the following components: o a mechanism for robots to find the alternatives a given server offers for the remote update, o a mechanism for a robot to register with a given server and inform this server its preferences for communication, and o a mechanism for further interaction between a server and a robot, which can vary according with the robot's preferences and the server possibilities. Additionally, the data objects interchanged by robots and servers as result of RUP operations MUST be MIME objects. The exact type and syntax of these objects are left unspecified in this document, and thus the data objects are treated in an abstract way. Separate specification of these objects is needed. In the following sections we describe each of the basic mechanisms composing the RUP protocol. 3.1 Finding the alternatives offered At present, when a Web Robot wants data from a Web Server, it Di Cosmo, Martinez Lopez Experimental [Page 4] Internet Draft RUP Architecture April 1999 access the server, and retrieve the information it wants. If it is willing, it can retrieve /robots.txt file and respect the guidances provided there. But if the Robot supports the RUP procotol, it should find if the server offers RUP information, and if it does, which are the exact options offered by the server to perform the interchange of information. To acomplish that task, servers implementing the RUP protocol will have in their root WWW directory a file /rupinfo.txt, containing information about the registration procedure and the implemented features of the protocol (like version, alternatives provided for the registration, alternatives provided for the information flow, etc.). The exact contents and syntax of this file are left unspecified in this document. The only requirement is that the first line of the file should be the string "RUP Version ", where is a string indicating the version number. The version number MUST determine the exact contents and syntax of the file - they have to be described in the document specifying that version. 3.2 Registering and communicating preferences Once the Robot found that the server provides RUP information (there is a valid /rupinfo.txt in the server root directory), it can choose to register with the server to agree with it about the exact nature of further interactions between them. Tipically this registration will be an operation with the robot's preferences as arguments; it is RECOMMENDED that the robot prepares its preferences according to the alternatives provided by the server in the /rupinfo.txt file. There must be also the possibility for a robot to change its preferences in any moment after the registration, and also to unregister to stop receiving information. Tipically, these two alternatives will be operations with the corresponding information. Every one of these operations REQUIRE the server to identify the robot via a signature, for security reasons. Additional security issues for these operations can be impossed by implementations. 3.3 Flow of information for remote updates There will be mainly two ways in which Robots are notified of update information by a server: o unregistered index flow, and o registered index flow. Di Cosmo, Martinez Lopez Experimental [Page 5] Internet Draft RUP Architecture April 1999 In the first case, a robot can request information from the server without needs to follow the registration procedure described in the previous section. This alternative will tipically take the form of a single operation with the preferences for that operation as arguments. Implementations of this operation MAY require a signature to proceed, but anonymous request can be accepted also; the decision should be taken by every web administrator. In the second case, the communication will take place between the server and a registered robot. There are two main alternatives for this communication: o the update information is sent by the server directly to the robot, and o the server notifies the robot about the existence of new update information, and the robot can retrieve that information in a later moment. The first is called a push operation, and the second is called a notification operation. The server can include additional information in a notification operation - for example the number of updates available, etc. - in order for the robot to have a guideline to take the decision of when to retrieve the update information. The push operation usually contains all the update information requested by the robot in its registration. Tipically, the robot's preferences include the periodic nature of this operations (and the time between them), the transport protocol used for the operations, restrictions and filters for the update information, alternatives for error control and recovery (for example, if some periodic notification is missing), etc. 4 Security Concerns Security considerations should take into account the transmission of valuable, propietary, or highly sensitive data, the possibility for false information to be generated, and also the exposition of unwanted information. The first two issues (transmission of valuable or false data), can be resolved using the MIME nature of update information objects. For example, valuable data may be encrypted without additional effort, and also signatures can be used to avoid false information to be introduced. The second issue is a bit more delicate. Usually Web Robots index the web spaces following links from a given `root document' - in this way they have a look of the web similar to a connected graph. But if Di Cosmo, Martinez Lopez Experimental [Page 6] Internet Draft RUP Architecture April 1999 RUP technology is added to Web Servers, then all the local web space can be indexed, and not only the connected components. The lacking of references to certain information can be used as a cheap way to provide security - if nobody knows the right address, nobody can access it. There is a simple, quick, and in general, wrong solution: Web Servers should mantain their indexes following links instead of using directory information. The problem of this solution is that it is extremely difficult (if not impossible) for a server to know if some particular piece of information is addressed from somewhere in the Web. So, another solution is required in order to keep this way of protect information. We propose that servers should mantain a list of addresses that are supposed not to be indexed. In this way, the information behind them is never exposed to Web Robots. If the Web Server has a centralized publication policy - i.e. the information should be published only by the administrator - then it is easy to mantain this list. But when every user of a system can have his own local web space (tipically as a subdirectory of his home directory), this is not so easy. For this case, it is REQUIRED that every user should keep a .ruprc file in his home directory, indicating what addresses should be indexed and what of them should not. The absence of this file is enough reason for the indexing program to refuse indexing the local web space of the user. 5 Conclusions In the current state of the art for Web Robots, the fearful amount of work involved in their task is a real barrier to achieve completeness. For that reason, some form of distribution of workload is needed. In this note we have presented a protocol that can be used to distribute the task of indexing the Web. Each server cooperates with the Robots, preparing reports of changes in their contents, so that the Robot must not figure out those changes by itself. The protocol described does not impose on the servers any significant overhead, in consideration of the fact that the needed modification logs are usually maintained for administrative reasons, and if this technology spreads out, we will surely find that the information the Robots can gather and mantain will be much more accurate than at the present time. One could think of the possibility of allowing indexing tasks to be performed on the server in idle time. This is an interesting investigation area, but due to the varying nature of the indexing algorithms used by different robots, one would require robot-dependent code (or even robot supplied code) to be executed on the server. Unfortunately, this would increase the overhead and raise Di Cosmo, Martinez Lopez Experimental [Page 7] Internet Draft RUP Architecture April 1999 fundamental security issues, that would prevent easy distribution of the protocol, so we decided not to pursue this direction yet. A prototype for the program needed by a server to accept registrations and to communicate with Robots is being constructed and will be made available as a reference implementation with the final version of this note. Author's Address Roberto Di Cosmo Pablo E. Martinez Lopez DMI-LIENS (CNRS URA 1347) LIFIA, Universidad de La Plata, Ecole Normale Superieure CC.11, Correo Central, 45, Rue d'Ulm La Plata, Argentina. 75230 Paris France. E-mail:fidel@info.unlp.edu.ar E-mail:Roberto.DiCosmo@ens.fr References [AM98a] J. Allen and Michael Mealling. The architecture of the Common Indexing Protocol (CIP), November 1998. Internet-draft. URL: http://search.ietf.org/internet-drafts/, filename: draft-ietf-find-cip-arch-02.txt. [AM98b] J. Allen and Michael Mealling. MIME objects definitions for the Common Indexing Protocol (CIP), November 1998. Internet-draft. URL: http://search.ietf.org/internet-drafts/, filename: draft-ietf-find-cip-mime-03.txt. [HF] Roland Hedberg and Patrik Faltstrom. The FIND Working Group. URL: http://www.ietf.org/html.charters/ filename: find-charter.html. [KLM96] Charles P. Kollar, John R. R. Leavitt, and Michael Mauldin. Robot Exclusion standard revisited, June 1996. URL: http://www.kollar.com/ filename: robots.html. [Kos94] M. Koster. A standard for Robot Exclusion, June 1994. URL: http://info.webcrawler.com/mak/projects/robots/ filename: norobots.html. [Kos95] Martijn Koster. The Web Robots Pages, 1995. URL: http://info.webcrawler.com/mak/projects/robots/ filename: robots.html. Di Cosmo, Martinez Lopez Experimental [Page 8]