Introduction

It was inevitable that the technology under which Archie was built changed. With the introduction of faster applications and newer media on the Internet, Archie had to reshape its structure to accomodate these new factors and stay in the race.

Speed in this day and age is a big gain and Archie is thought to have achieved it with the utilization of a string indexing tree for faster string searches.

As for the World Wide Web and its many new faces, Archie also introduces its latest web crawling and indexing engine. In addition, a new web interface has been added to accomodate the web users. And with a few more newly added features and face-lifts, Archie v3.5 is ready to undergo deployment.

This manual is an attempt to clarify the many components of Archie v3.5 and its functionality. Please keep in mind that this manual is also undergoing changes, so we appreciate any comments sent to archie-group@bunyip.com.

What follows is the list of added features in Archie v3.5. You will find in the table of contents, symbols such as the one in the margin, to help you find the relevant documentation concerning the new features.

• Support for a new database module (webindex):

  • Retrieval of HTML pages through HTTP protocol.

  • Keyword extraction.

  • Controlled crawling of the WWW.

  • Site by site basis.

  • Content extraction.

  • Configurable stoplist (keyword exclusion).

• New database structure:

  • A more reliable and independent database structure.

• New strings search engine:

  • Based on a paged, tree structured strings index.

  • Faster searches.

  • Less memory required.

  • More disk space needed for construction of the index.

• New search interface:

  • New WWW (CGI compliant) front-end to the Archie database.

• A better domain filter for any database:

  • Results can be pre-configured to return in a certain order (e.g. FTP sites closest to the server first).

  • Configurable on a server basis.

• General new features:

  • Support for ls-lR.gz files.

  • ISO-Latin-1 support for searches.

Future

We hope in the very near future to be able to produce the following list of features. If you have any suggestions to add to the list please feel free to contact us at archie-group@bunyip.com:

• Search tools:

  • Add boolean searches.

  • Allow other types of searches. (For example, return a list of hosts in any database and when they were last updated.)

  • Multi-lingual Web front-end.

• Database:

  • An automated check-up program that runs periodically on the databases. It will check the consistency of the database and fix any errors it runs into. This will reduce the amount of time an Archie administrator needs to spend maintaining an Archie server.

  • Improve on filters and parsers to handle strange and unusual site listings.

  • Parallelize the parsing and updating of different databases.

  • Build multiple databses with the same access method.

• Management tools:

  • CGI web interface to allow web and FTP administrators to register their sites automatically.

  • Replace “host_manage” with a new method for managing the host_db database.

  • Statistical tools.