Chapter One - Center for the Study of Digital Libraries
Download
Report
Transcript Chapter One - Center for the Study of Digital Libraries
Building collections
with Greenstone
How to Build a Digital Library
Ian H. Witten and David Bainbridge
Digital Library Collections
There is a distinction between
BUILDING collections
DELIVERING information to users
Similar to ‘compile-time’ versus ‘runtime’
distinction in computer programming
Information structures should usually be
prepared in advance
Building a Collection
The Collector
A subsystem that takes you step by step through
building a simple collection
Conceals details behind the scenes
First locate information on your computer or
the Web
Plain text, HTML, Word, PDF, email file, etc.
Plug-ins
Plug-ins are software modules that handle
Format conversion
Metadata extraction
Plug-ins promote extensibility
Greenstone Archive Format
Greenstone Archive Format
XML-based file format
File format for:
Documents
Metadata
Collection Configuration File
Collection Configuration File
Defines the structure of a collection
Governs how the collection is built
Specifies how the collection will appear to users
Greenstone Extended Capabilities
Extending the Capabilities of Greenstone
Plug-ins
Classifiers
Handle different document and metadata formats
Handle different kinds of browsing structures
Format statements and Macros
Govern the user interface content and appearance
Why Greenstone?
Benefits of Greenstone
General system for constructing and presenting
digital collections
Handles millions of documents, text, images,
audio, video
User interfaces identical in Web-based and CDROM versions
Installs on Windows and Linux
Access locally or remotely using web browser
Organization of Collections
Each collection can be organized differently:
Format of source documents
Metadata
Directory structure
Document structure
Searching and browsing services
Presentation
Auxiliary services
Variation of Source Format
Source documents can be supplied in:
Plain text
HTML
PostScript
PDF
Word
E-mail
Other file types
Images
Video
Audio
Variation of Metadata
Different types of metadata
Metadata can be supplied differently
‘fields’ in MS Word
<meta> tags in HTML
Information coded into filename and directories
Spreadsheet or other data file
Explicit metadata format like MARC
Variation of Directory Structure
Collections can vary in the directory structure in
which the information is located
Variation of Document Structure
Document structure
Flat
Divided sequentially into pages
Hierarchical organization
Title or other metadata available at each level
Variation of Services
Searching
Metadata
Indexes
Hierarchical levels
Browsing
Metadata
Browser type
Variation of Presentation
Results can be presented to users in various
ways:
Format that target documents are shown in
Search results page
Metadata browsers
Interface language
Variation of Auxiliary Services
A collection may require additional services
User logging
Etc.
Collection Configuration File
Allows Variation
A digital library collection is made by
Gathering raw material
Designing the collection
Putting design information about the structure and
presentation of the collection in the Collection
Configuration File
Front Page of Collection
Statement of collection’s purpose
Statement of collection’s coverage
Explanation of how collection is organized
Searching Involves Indexes
Searching is provided by indexes built from
different parts of the documents
Entire documents
Paragraphs
Titles
Sections
Section headings
Figure captions
Indexes
Indexes can be created automatically using
Documents
Supporting files
Indexes can be rebuilt automatically
New document in the same format becomes
available
Process can awake, check for new material, and
rebuild the indexes
Plug-ins for Indexing
Source documents are converted into standard XML
form for indexing using plug-ins
Standard plug-ins process
Plain text
HTML
Word
PDF
Usenet and email messages
New plug-ins can be written for other document types
Browsing Involves Lists
Browsing involves lists that can be examined by
the user
Authors
Titles
Dates
Hierarchical classification structures
Classifier Modules
Modules called classifiers are used to create
browsers and build browsing structures from
metadata
Scrollable lists
Alphabetic selectors
Dates
Hierarchies
Programmers can write new classifiers to create
novel browsing capabilities
Search Terms
Search Terms in Greenstone:
Alphabetic characters
Digits
Separated by white space
Punctuation acts as white space
Two Types of Queries
Query for ALL of the words
Boolean AND
Query for SOME of the words
Ranked
Indexes to Search
In most collections, you can choose different
indexes to search
Examples:
Author and title indexes
Chapter and paragraph indexes
Usually the full matching document is returned
regardless of index searched
Preferences Page
Preferences Page
Allows advanced control over search operation:
Case-folding and stemming
Advanced query mode where users specify Boolean
operators
Large-query interface
Display search history
Preferences Page
Preferences Page
Specify subcollections to be included in searches
Specify presentation language
Customize interface
Textual vs. standard interface
Suppress navigation bar
Suppress alert system
Using the Collector
The Greenstone Collector
Easiest way to build a simple collection
The Collector allows you to:
Create a new collection
Modify or add to an existing collection
Delete a collection
Starting the Collector
Click the Collector link from the default
Greenstone home page
Log in
When Greenstone is installed, an account called
admin is set up with a password chosen during
installation
The Collector works through a standard web
interface
Creating a New Collection
Collector’s main purpose is to build a new
collection
Structure of a collection is determined when the
collection is set up
Simplest to copy the structure of an existing
collection and then edit
Collection Building Steps
1.
Collection Information
2.
Source Data
3.
Configuration
4.
Building
5.
Viewing
Collection Building Steps
☐ Collection Information
☐ Source Data
☐ Configuration
☐ Building
☐ Viewing
1. Collection Information
Give the collection a name and provide
associated information
Title
Short phrase used to identify the collection within the
digital library
Contact e-mail address
Brief description
Sets out the principles that govern what is included in the
collection
Collection Building Steps
☑ Collection Information
☐ Source Data
☐ Configuration
☐ Building
☐ Viewing
2. Source Data
Specify the location of the sources
Clone existing collection
Specify on a pull-down menu the existing collection
Create a completely new collection
2. Source Data
In the provided boxes, indicate where Source
Documents are located
Specification of sources
file://
http://
ftp://
file://
File name on the Greenstone server system
That file will be included in collection
Directory name on the Greenstone server
Everything in the folder and its subfolders will be
included
http://
Web page
The web page will be downloaded
All pages it links to (and all pages they link to) that
reside on the same site, below the URL, will also be
downloaded
URL that leads to a list of files
Everything in the folder and its subfolders will be
included in collection
ftp://
File to be downloaded using FTP
Directory name on the FTP server
Downloads everything in the folder and its
subfolders
Collection Building Steps
☑ Collection Information
☑ Source Data
☐ Configuration
☐ Building
☐ Viewing
3. Configuration
This step can be bypassed
Allows adjustment of configuration options
The construction and presentation of all
collections are controlled by specifications in a
special collection configuration file
Collection Building Steps
☑ Collection Information
☑ Source Data
☑ Configuration
☐ Building
☐ Viewing
4. Building
The computer does the work of the building
process
Indexes are built:
For browsing
For searching
Following specifications in the collection
configuration file
Status line shows progress
Warnings shown if files can’t be found
Collection Building Steps
☑ Collection Information
☑ Source Data
☑ Configuration
☑ Building
☐ Viewing
5. Viewing
View the collection that has just been created
E-mail can be sent to the collection’s contact
address
Must enable by editing main.cfg configuration file
Working with Existing Collections
Add more material and rebuild the collection
Edit the configuration file to modify the
collection’s structure
Delete the collection
Put the collection on CD-ROM
Adding Material to a Collection
Do not re-specify files that are already in the
collection
Files would be included twice
If the building process fails, the old version
remains unchanged
Structure of collection can be changed
Edit the configuration file
May add plug-ins or an option to a plug-in
Plug-ins & Document Formats
Plug-ins are specified in the collection
configuration file
File name determines document format
Widely used document formats:
TEXTPlug
HTMLPlug
WORDPlug
PDFPlug
PSPlug
EMAILPlug
ZIPPlug
Text Files
TEXTPlug Plug-In
*.txt
*.text
Plain text file
Title metadata based on the first line of the file
HTML Files
HTMLPlug Plug-In
*.htm
*.html
.shtml
.shm
.asp
.php
.cgi
HTML Files
HTMLPlug Plug-In
Imports HTML files
Title metadata extracted from the HTML <title> tag
Other HTML <meta> tag data can be extracted
Parses and processes any links in the file
Links to other files in the collection are trapped and
replaced by references to the document
HTML Files
file_is_url
Optional switch within the HTML plug-in
Causes URL metadata to be inserted into each
document, based on the file-name convention that is
adopted by the mirroring package. The collection
uses this metadata to allow readers to refer to the
original source material rather than a local copy
Microsoft Word Files
WORDPlug Plug-In
*.doc
Imports Microsoft Word documents
Greenstone uses independent programs to
convert Word files to HTML
Many variants on the Word format
Older Word formats use a simple text string
extraction
PDF Files
PDFPlug Plug-In
*.pdf
Imports PDF Files
Adobe’s Portable Document Format
Greenstone uses independent programs to
convert PDF files to HTML
PostScript Files
PSPlug Plug-In
*.ps
Imports PostScript Files
Works best when a standard conversion
program is already installed on the computer
Uses simple text extraction algorithm if no
conversion program is present
Email Files
EMAILPlug
Imports files containing email
Each source is checked for e-mail contents
Extracts metadata:
*.email
Subject
To
From
Date
Deals with common formats
Netscape, Eudora, Unix mail readers
Compressed & Archived Files
ZIPPlug Plug-In
*.zip
*.tar
.gz
*.z
*.tgz
*.bz
Relies on standard utility programs being present
Building Collections
Manually
Building a Collection
Building a Collection:
The process of taking a set of documents and
metadata information and creating all the indexes
and data structures that support the searching,
browsing, and viewing operations that the collection
offers
Building a Collection
Four Phases in Building a Collection
Make
Import
Import the documents and metadata, convert to a Greenstone
standard form
Build
Make a skeleton framework structure to contain the collection
Build the required indexes and data structures
Install
Make the collection operational
Building Collections Manually
☐ Getting Started
☐ Making a framework for the collection
☐ Importing the documents
☐ Building the indexes
☐ Installing the collection
Getting Started
Locate the command prompt
Go to the directory where Greenstone was
installed
cd “C:\Program Files\gsdl”
Tell system where to find Greenstone files
setup.bat
Sets the variable GSDLHOME to the Greenstone home
directory
To return later
cd “%GSDLHOME%”
Building Collections Manually
☑ Getting Started
☐ Making a framework for the collection
☐ Importing the documents
☐ Building the indexes
☐ Installing the collection
Make a framework for the collection
Use the Perl program mkcol.pl to ‘make a
collection’
Get description of usage and arguments
perl –S mkcol.pl
mkcol.pl
May leave off first part if system recognizes that .pl files
are associated with Perl
Make a framework for the collection
perl –S mkcol.pl –creator emailAddress collectionName
Make a framework for the collection
Examine the file structure
cd “%GSDLHOME%\collect\collectionName”
List directory contents
dir
Seven subdirectories are created:
archives
building
etc (contains collect.cfg file)
images
import
index
perllib
Make a framework for the collection
collect.cfg File
emailAddress placed in the creator and maintainer
lines
collectionName placed in collection-meta lines
Plug-ins are inserted
Building Collections Manually
☑ Getting Started
☑ Making a framework for the collection
☐ Importing the documents
☐ Building the indexes
☐ Installing the collection
Importing the documents
The collection’s import directory should contain
the source material
Drag the directory containing the source
material into the import directory
You may drag several source directories and
hierarchies
Importing the documents
The import process:
Brings documents into the Greenstone system
Standardizes document format
(the way that metadata is specified)
Standardizes the file structure
(that contains the documents)
Importing the documents
To get a list of options for the import program:
perl –S import.pl
The basic import command is:
perl –S import .pl collectionName
Importing the documents
You may be in any directory when the import
command is issued
The software works by knowing the collection’s
name and the Greenstone home directory
Warnings may appear
When files are found without corresponding plugins
These files will be ignored
Building Collections Manually
☑ Getting Started
☑ Making a framework for the collection
☑ Importing the documents
☐ Building the indexes
☐ Installing the collection
Building the indexes
Use the program buildcol.pl
Building the indexes
Modify collect.cfg file to customize the collection’s
appearance
collectionname
Web browsers receive this name as the title of the
collection’s front page
collectionextra
Description of the collection
Appears under “About this collection” on the collection’s
home page
Enter as a single line in the editor
Building the indexes
Modify collect.cfg file to customize the collection’s
appearance
iconcollection
Give the collection an icon image
Put the location of the image between quotes
If absent, the collection’s name will be used
Use _httpprefix_ as a shorthand way of beginning any
URL that points within the Greenstone file area
Example:
_httpprevix_/collect/collectionName/images/icon.gif
Building the indexes
To get a list of options for the build program:
perl –S buildcol.pl
The basic build command is:
perl –S buildcol .pl collectionName
Building the indexes
The building process takes about a minute on
small collections and can take much longer for
very large collections
You may ignore most warning messages
Serious problems will cause the program to
terminate
Building Collections Manually
☑ Getting Started
☑ Making a framework for the collection
☑ Importing the documents
☑ Building the indexes
☐ Installing the collection
Installing the collection
Building is done in the building directory
Collection must be moved to the index directory
before users can see it
Drag contents of the building directory to the
index directory
If index already contains files, remove them first
Forgetting to move the contents of building to
index is a common mistake
Installing the collection
To view the newly built collection:
Restart Greenstone
If using the Local Library version
Reload Greenstone Home Page
If using the Web version
Importing and
Building
General Information
Two Main Parts to Collection Building:
Importing (import.pl)
Building (buildcol.pl)
Files and Directories
Collection Specific Directories
GSDLHOME
collect – all the digital library collections
collectionName – directory of collection
import – original source material
archives – result of import process
building – temporary, contents manually moved to index
index – bulk of info served to users
(import, archives and building can be deleted)
etc – contains collect.cfg file
images – icons used for the collection
perllib – Perl programs specific to collection
Other Greenstone Directories
GSDLHOME
lib – common software for both the collection server and receptionist
bin – programs used for building process
script – Perl programs used
(mkcol.pl, import.pl, buildcol.pl)
perllib – Perl modules
plugins – Perl plugins
classify – Perl classifiers
cgi-bin – Greenstone runtime system
(absent in Local Library version)
src – source code in C++
colservr – the collection server
recpt – the receptionist
Other Greenstone Directories
GSDLHOME
packages – source code for external software packages used by Greenstone
(indexing and compression program, database manager program, etc.)
(each package is stored in a directory of its own with a readme file)
bin – executables
mappings – Unicode translation tables
etc – configuration files for the entire system, initialization and error logs, user
authorization database
images – user interface images and icons
macros – small code fragments that drive the user interface
tmp – temporary files
docs – documentation for the system
Object Identifiers
Document’s permanent name in the system
Remain the same when collection rebuilt
Assigned by the import process
Stored as an attribute in the document archive
file
Character strings starting with the letters HASH
(HASH0109d3850a6de440c4d1ca2)
Used to name directory where archive file is
stored
Plug-Ins
Plug-ins do most of the work of the import process
Operate in the order in which they are listed in the collect.cfg file
Input file is passed to each plug-in until one is found that can process it
If there is no plug-in that can process a file, a warning is printed
Plug-ins determine the traversal of the subdirectory structure in
the import directory
RecPlug - processes directories, recurses through directory
structures and passes the name through the plug-in list
GAPlug – processes Greenstone Archive Format documents (in
the archives directory structure)
ArcPlug – used during building, processes list of document OIDs
produced during import (list is stored in archives.inf file)
The Import Process
The Import Process
Brings documents and metadata into the system in a
standardized XML form
Original material placed in import directory
Import process transforms it to files in the archives directory
The original material can be deleted
New material added to collection by placing it in import directory
and re-executing the import process
Collection can be rebuilt from archive files
The new material finds it way into archives along with existing files
To keep the source form of collections
Do not delete the archives
“Source” form can be augmented and rebuilt later
The Build Process
The Build Process
Creates the indexes and data structures that
make the collection operational
Indexes for the whole collection are built all at
once
Build process does not work incrementally
Adding new material to archives requires that entire
collection be rebuilt (by issuing buildcol.pl)
Most collections can be rebuilt overnight
Options for Import and Build
Additional Options for Import
Additional Options for Build
Options for Import and Build
To see options for any Greenstone script, type its name
at the command prompt
Options for Import and Build help with debugging (see
Table 6.5 on page 310):
verbosity
archivedir
maxdocs
collectdir
out
keepold
debug
Greenstone Archive
Documents
Greenstone Archive Format
<!DOCTYPE GreenstoneArchive [
<!ELEMENT Section (Description,Content,Section*)>
<!ELEMENT Description (Metadata*)>
<!ELEMENT Content (#PCDATA)>
<!ELEMENT Metadata (#PCDATA)>
<ATTLIST Metadata name CDATA #REQUIRED>
]>
Document Metadata
Metadata – descriptive information about
author, title, date and keywords
Stored with metadata name
Stored at the beginning of the section
Example:
<Metadata name=“Title”>Freshwater Resources in
Arid Lands</Metadata>
Document Metadata
Dublin Core – a metadata standard
New metadata types can be invented
Metadata can be assigned by an automatic
process rather than manually entered
The Dublin Core
Collection
Configuration File
Collection Configuration File
Default Configuration File
Getting the Most Out
of Your Documents
Basic Plug-In Options
Document Processing Plug-ins
Document Processing Plug-ins
Document Processing Plug-ins
Assigning Metadata from a File
XML Document Type Definition (DTD)
Example XML Metadata File
Document Type Definition (DTD)
<!DOCTYPE GreenstoneDirectoryMetadata [
<!ELEMENT DirectoryMetadata (FileSet*)>
<!ELEMENT FileSet (FileName+,Description)>
<!ELEMENT FileName (#PCDATA)>
<!ELEMENT Description (Metadata*)>
<!ELEMENT Metadata (#PCDATA)>
<ATTLIST Metadata name CDATA #REQUIRED>
<ATTLIST Metadata mode (accumulate|override) "override">
]>
Example XML Metadata File
<?xml version="1.0" ?>
<!DOCTYPE GreenstoneDirectoryMetadata SYSTEM
"http://greenstone.org/dtd/GreenstoneDirectoryMetadata/1.0/GreenstoneDi
rectoryMetadata.dtd">
<DirectoryMetadata>
<FileSet>
<FileName>nugget.*</FileName>
<Description>
<Metadata name="Title">Nugget Point Lighthouse</Metadata>
<Metadata name="Place" mode="accumulate">Nugget Point</Metadata>
</Description>
</FileSet>
<FileSet>
<FileName>nugget-point-1.jpg</FileName>
<Description>
<Metadata name="Title">Nugget Point Lighthouse</Metadata>
<Metadata name="Subject">Lighthouse</Metadata>
</Description>
</FileSet>
</DirectoryMetadata>
Tagging Document Files
<!-<Section>
<Description>
<Metadata name="Title"> Realizing human rights for poor
people: Strategies for achieving the international
development targets </Metadata>
</Description>
-->
(text of section goes here)
<!-</Section>
-->
Classifiers
Format Statements
Format Statements
Examples of Format Strings