Table of Contents

  1. Introduction
  2. Implementation
    1. System Relationships & Properties
    2. User-defined Relationships
  3. Configuration
    1. Module Configuration
    2. Datastore Configuration
  4. Search Interface
  5. Resource Index Demo
  6. References

1. Introduction

The latest release of Fedora includes the Resource Index, a service which provides the infrastructure for expressing relationships among objects and their components. Examples of relationships between digital objects include well-known management relationships such as the part-whole links between individual chapters and a book, and semantic relationships useful in digital library organization such as those expressed within the Functional Requirements for Bibliographic Records (FRBR).

Fedora expresses relationships by defining a base relationship ontology [RELS-EXT] using RDFS and provides a slot in the digital object abstraction for RDF expression of relationships based on this ontology. Assertions from other ontologies may also be included along with the base Fedora relationships. All relationships are represented as a graph that can be queried using an RDF query language. The query interface to the Resource Index is exposed as a web service [risearch], providing a rich information foundation for external services [Fedora Relationships].

2. Implementation

The Fedora object model can be abstractly viewed as a directed graph, consisting of internal arcs that relate digital object nodes to their dissemination nodes and external arcs between digital objects. The Resource Index is a Fedora service that allows storage and query of this graph. The Resource Index is automatically updated whenever an object is added or modified.

The Resource Index builds on the RDF primitives build within the semantic web community. Fedora supplies a base relationship ontology [RELS-EXT](defining a core set of internal and external relationships) that can co-exist with domain-specific relationship ontologies from other namespaces. Each digital object's external relationships to other digital objects are expressing in RDF/XML within a reserved datastream in the respective object. A relationship graph over the digital objects in the repository can then be derived by merging the internal relationships implied by the Fedora object model with the external relationships explicitly stated in their relationship datastreams.

A. System Relationships & Properties

The Fedora base ontology describes such relations and properties as the behavior definition implemented, behavior mechanism used, creation date, state, and mime-type.

In the figure below, the graph (abbreviated for clarity) represents three objects in the repository. demo:SmileyStuff uses the behavior mechanism demo:dualResImageCollection, which in turn implements the behavior definition demo:Collection.

System Relationships

Dublic Core statement, as shown in the figure below, are automatically extracted from an object's DC datastream and inserted into the Resource Index. datastream, as shown in the figure below:

DC properties

B. User-defined Relationships

The Resource Index will automatically index object-to-object relationships defined in the RELS-EXT datastream. Please consult Fedora Metadata for Object-to-Object Relationships for more information.

3. Configuration

Please note that many configuration changes require a full dump-and-load of the Fedora repository to ensure consistency. For example, turning the Resource Index Module off and on again will result in an inconsistent state, as the Resource Index will know nothing about the digital objects created or modified while the module was not loaded. Similarly, enabling full-text indexing [Full-Text] after the repository has already been populated will only add new objects to the full-text model. In general, the only safe configuration changes to make on a running repository are limited to the performance-related pool, buffer and flush parameters. In all cases, configuration changes require a restart of the Fedora server before taking effect.

A utility to manage configuration changes that will not require a full dump- and-load of the repository is planned for the next release of Fedora.

The Resource Index is configured within two sections of fedora.fcfg, module and datastore.

A. Module Configuration

An example of the module configuration within fedora.fcfg: 

<module role="fedora.server.resourceIndex.ResourceIndex" 
        class="fedora.server.resourceIndex.ResourceIndexModule">
  <param name="level" value="1"/>
  <param name="datastore" value="localKowariTriplestore"/>
</module>

An explanation of the parameters and their possible values:

  • level
    Sets the operating level of the Resource Index.
    • 0
      Off: the Resource Index will not load at server startup.
    • 1
      On: the Resource Index will index system properties, inter & intra-object relationships, and user-defined relationships.
    • 2
      Same as 1, but adds indexing of parameterized methods, where the parameters have a finite domain (e.g., the method getImage with parameter size, whose domain is {small, large}). Because calculating method parameters may result in a combinatorial explosion of statements in the Resource Index (depending on the design of a particular repository's Behavior Definition Objects), this level of indexing must be explicitly set.
  • datastore
    The id of the datastore to use with the Resource Index. The referenced datastore must assert a connectorClassName parameter with a valid Trippi Connector class.

B. Datastore Configuration

The example datastore configuration below (with the path parameter modified for the installation environment) would provide a local Kowari triplestore that buffers up to 20,000 triples in memory at a time or waits for 5 seconds of buffer inactivity before flushing them to disk. Because writing triples to disk is a relatively expensive operation, the buffer takes advantage of Kowari's bulk update handler to ingest a mass of triples at a time. The performance gain is significant during a bulk ingest of objects. The size or inactivity interval of the buffer may be adjusted according to performance needs and physical memory capacity.

<datastore id="localKowariTriplestore">
  <param name="connectorClassName" value="org.trippi.impl.kowari.KowariConnector"/>
  <param name="remote" value="false"/>
  <param name="path" value="/opt/fedora/store/resourceIndex"/>
  <param name="serverName" value="fedora"/>
  <param name="modelName" value="ri"/>
  <param name="poolInitialSize" value="3"/>
  <param name="poolMaxGrowth" value="-1"/>
  <param name="readOnly" value="false"/>
  <param name="autoCreate" value="true"/>
  <param name="autoTextIndex" value="false"/>
  <param name="memoryBuffer" value="true"/>
  <param name="autoFlushDormantSeconds" value="5"/>
  <param name="autoFlushBufferSize" value="20000"/>
  <param name="bufferSafeCapacity" value="40000"/>
  <param name="bufferFlushBatchSize" value="20000"/>
</datastore>

An explanation of the parameters and their possible values follows. Certain parameters require other parameters, as indicated in the hierarchy below. Optional parameters are also indicated below. As noted previously, many of these parameters, with the exception of the pool, buffer, and flush parameters, cannot be changed on a running repository without a full dump-and-load of the repository.

  • connectorClassName
    The name of the Trippi Connector class used to communicate with the triplestore. In this initial release, the only Connector implementation is for Kowari.
    • org.trippi.impl.kowari.KowariConnector
  • remote
    Tells the connector to communicate with Kowari in remote or local mode.
    • true
      If remote is true, the host parameter must be specified.
      • host
        The hostname where Kowari is running.
      • port
        The rmi port Kowari is running on (default is 1099).
    • false
      If remote is false, path parameter must be specified.
      • path
        The local path to the main triplestore directory.
  • serverName
    The server name for rmi binding.
  • modelName
    The name of the model to use.
  • poolInitialSize
    The initial size of the session pool used for queries.
  • poolMaxGrowth
    Maximum number of additional sessions the pool may add. If specified as -1, no limit will be placed on pool growth.
  • poolSpareSessions
    The number of spare sessions to proactively make available. If unspecified, this defaults to 0, which means that additional sessions will only be created as needed. Note that if poolMaxGrowth is 0, the value of this parameter is inconsequential.
  • readOnly
    Whether the triplestore should be read-only. Most Fedora repositories will set this to false.
    • true
      No additional parameters need to be set if readOnly is true.
    • false
      The following parameters must also be set if readOnly is false.
      • autocreate
        Whether to create the model if it doesn't already exist. At startup, the model will be automatically created. In addition, an XML schema datatyped model named "xsd" will also be automatically created.
        • true
        • false
      • autoTextIndex
        Whether to propagate adds/deletes to a full-text [Full-Text] model automatically. While a very useful feature, enabling full-text indexing adds significantly to object ingest times. If true, the text model will be named modelName-fullText. Note that if this is true and autoCreate is true, the text model will also be created if it doesn't already exist.
        • true
        • false
      • memoryBuffer
        Whether to use a memory buffer or a database buffer for write operations. The buffer is where triples are stored before they're actually written. Normally, a memory buffer will be sufficient. However, in certain server environments, a database buffer is the better option, offering higher capacity and persistence.
        • true
        • false
          • dbDriver
            The JDBC driver class name. This must be in the classpath.
            • com.mysql.jdbc.Driver
            • com.mckoi.JDBCDriver
            • oracle.jdbc.driver.OracleDriver
          • dbURL
            The JDBC URL for the database. Examples: For MySQL, jdbc:mysql://localhost/mydb would use the local database named mydb. For McKoi, jdbc:mckoi://localhost:9157/ would use the local database at port 9157. For oracle, jdbc:oracle:thin:@localhost:1521:mydb would use the thin driver to connect to the local database named mydb at port 1521.
          • dbUsername
            The database username.
          • dbPassword
            The password for the db user.
          • dbTableName
            The table for buffered triple updates. This table must already exist with the following five columns: num (a large numeric type), action (char(1)), subject, predicate, and object (all large varchar or text types).
      • autoFlushDormantSeconds
        Seconds of buffer inactivity that will trigger an auto-flush. If this threshold is reached, flushing will occur in the background, during which time the buffer is still available for writing.
        • true
        • false
      • autoFlushBufferSize
        The size at which the buffer should be auto-flushed. If this threshold is reached, flushing will occur in the background, during which time the buffer is still available for writing.
        • true
        • false
      • bufferSafeCapacity
        The maximum size the buffer can reach before being forcibly flushed. If this threshold is reached, flushing will occur in the foreground and the buffer will be locked for writing until it is finished. This should be larger than autoFlushBufferSize.
        • true
        • false
      • bufferFlushBatchSize
        The number of updates to send to the triplestore at a time. This should be the same size as, or smaller than autoFlushBufferSize.
        • true
        • false

4. Search Interface

The Resource Index Search interface is exposed in a REST architectural style, providing a stateless query interface that accepts queries by value or by reference [Fedora Relationships].

The query interface to the Resource Index currently supports three RDF query languages, RDQL, iTQL, and SPO. Both RDQL and iTQL share a superficially similar syntax to SQL, with RDQL enjoying broader implementation support, but iTQL providing a richer feature set. Support for SPARQL is planned for a future release.

Please consult the Resource Index Search documentation for more information.

5. Resource Index Demo

Demonstration objects that utilize the Resource Index are included in the Fedora distribution. Please see the Demo documentation for more information.

6. References

[Fedora Relationships]
Fedora: An Architecture for Complex Objects and their Relationships, Carl Lagoze, Sandy Payette, Edwin Shin, and Chris Wilper, 2004.
[Full-Text]
Kowari Full-Text Models
[RELS-EXT]
Fedora Relationship Ontology
[risearch]
Fedora Resource Index Search Service