%
% Copyright (c) 1996 Bunyip Information Systems Inc.
% All rights reserved
%
% Archie 3.5
% August 1996
%
% db.tex
%




\chapter{The database}
\label{chap:db}
\new

\section{Overview}

The database has been completely revised. The goals of this reconstruction
were the building of a faster system and more reliable database.  The back-end
of the database is composed of


\begin{itemize}
\item Index
\item Site files
\item Start database
\end{itemize}


The index is a tree structure, allowing fast searches of the strings file.
Rebuilding the index is a relatively expensive process, so it is not updated
after every new key is inserted into the database.  Thus, in order to handle a
search request, a linear search is made of the (usually) small un-indexed
portion of the strings file, in addition to checking the fast index.  To
ensure quick response times, the index should, periodically, be brought up to
date with the strings file.


Within a catalog, files having the prefix \Path{``Stridx.''}, are part of
the index.  For example, \Path{Stridx.Strings} is the strings file, and
\Path{Stridx.Index} is the fast index. Depending on the database, the
creation of the index is done differently. The anonftp database creates an
index file for all substrings where instead the webindex database creates
an index file that contains only the left substrings. 


The site files are now independent of one another. In case of corruption, only
the site that is corrupted need be considered. In certain cases, such as
webindex, extra files may be present with each site file. The
\Path{.excerpt} files contain excerpts of the different URLs.
The \Path{.idx} files, present in both anonftp and webindex, hold extra information
for large site files, in order to speed up searches.

The directory \Path{start\_db}, present in each catalog, holds information
that ties the index with the site files.

\section{Building the index}


In order to rebuild the index, the program \Path{db\_build} must be run.

The build may take some time to run, depending on the size of
\Path{Stridx.Strings} and the amount of memory allocated to the build.
One would call \Path{db\_build} as follows:

\comm{db\_build -d anonftp -v -k 50000 -t \archie/db/tmp }

The above example builds the anonftp index using 50 megabytes of memory.
Temporary storage is used in the \Path{\archie/db/tmp} directory. The man page lists
all the other switches that can be used. One option of particular importance
is ``\Comm{-f}'', which forces the index to be rebuilt, even if the un-indexed
portion of the strings file is very small. By default, the program will not
rebuild the index if the un-indexed text is less than 1 megabyte.

\alertbox{The index file is not constructed the same for each database. In the
case of the anonftp database, the index includes information concerning every
substring present in the string file. In the case of the webindex database,
the index is built for every word prefix present in the string file, i.e.
all substring that start at the beginning of a word.}


\alertbox{You may want to rebuild the index once a week initially and then
only once a month. These estimates are speculative --- it depends
on the rate at which new strings are added to the database.}

\section{Ordering the results}

With the volume of information in the database and the amount of replication
on the network, it is often desirable to return the results in some order of
closeness. It is now possible to configure the order in which results are
returned, according to the name of the domain to which the result site
belongs. To avoid slowing down searches, this ordering is done at the time at
which the data site is inserted into the database, rather than when the
results are being returned.

The order is defined in the file \Path{\archie/etc/domain.order}. The
following is an example of such a file.

\begin{center}
\begin{tabular}{l}
\Param{\# Comment line} \\
\Param{\# Domain order for an Archie server in North America} \\
\Param{ca:edu} \\
\Param{com:gov} \\
\Param{*} \\
\Param{au:nz} \\
\end{tabular}
\end{center}


Domains on the same line have the same precedence. Hence, \Param{.edu} sites
and \Param{.ca} sites will be returned first, in no particular relative order.
The sites in the domains \Param{.com} and \Param{.gov} will be listed later.
To discourage users from making unnecessary, long distance file transfers, the
sites from New Zealand and Australia are returned last. (We assume that users
accessing a North American archie server are usually in North America.) The
``*'' represents all other sites.

One can also specify the domains using pseudo-domains defined in
\Path{\Archie/etc/ardomains.cf} as well as sub-domains such as \Param{bunyip.com}.

\alertbox{For a given keyword in the database, the list of sites in which it
belongs is ordered. The result of this is that the answer returned from a
search may seem unordered.
In fact, the list of results is composed of sub-lists, each one for a
specific keyword matching the search criteria, and each one of 
those sub-lists are ordered according to the specified order.
Note also that some clients such as xarchie reorders the results and you may
not get the results in the appropriate order.
}

\alertbox{In order for changes, in the order of domains, to take effect, you must
rebuild the precomputed orders for the existing sites. This is done with the
program \Comm{db\_reorder}, listed in section~\ref{sec:programs}. }

\alertbox{Note that there are no leading dots ``.'' in the listed domains}




\section{The other programs}
\label{sec:programs}

All programs prefixed by \Path{db\_} in \Path{\archie/bin} aid in managing the
database. They are:

\begin{center}
\begin{tabular}{|l|l|} \hline
Program & Description \\ \hline\hline

db\_build & Build the database index \\ \hline
db\_check & Verify the consistency of the database \\ \hline
db\_dump  & Dump the list of sites that are in \Path{start\_db} \\ \hline
db\_reorder & Reorder the sites according to \Path{domain.order} \\ \hline
db\_siteidx & Build the \Path{.idx} files for specific sites \\ \hline
db\_stats & Compute statistics \\ \hline
fix\_start\_db & Fix problems in the start\_db database. \\ \hline
\end{tabular}
\end{center}

Information about the command line options of these programs may be found in
their respective man pages.





