Fedora Generic Search Service Version 2.0 (February 2008)

• Added a plugin for the Apache Solr search server.
• Added easier configuration, so that you need only edit one file with property values, then run it with ant.
• Updated to Lucene version 2.3.0.
• Added params to indexing in the format:

  ...&indexDocXslt=[xslt-name][(paramname1=value1[,paramname2=value2[,...]])]

  Use the parameters at indexing time by putting xsl:param statement in the indexing xslt stylesheet, like this:

  <xsl:param name="someparamname" select="defaultvalue"/>

• Added optimize options for Lucene indexing:
  

  fgsindex.mergeFactor and fgsindex.maxBufferedDocs

  will affect performance, see the index.properties file in configTestOnLucene. Also added

  ...?operation=updateIndex&action=optimize

  which will perform IndexWriter.optimize() which merges all segments together into a single segment, optimizing an index for search. Removed the optimize() call after each updateIndex.
• Added untokenizedFields property to Lucene index.properties files. Adding the property with a list of all untokenized fields will ensure that they all select the appropriate analyzer.
• Added a sortFields parameter to gfindObjects for Lucene, sorting search results as specified, see the index.properties file in configTestOnLucene.
• Added properties snippetBegin and snippetEnd, making highlight code configurable, see the index.properties file in configTestOnLucene.
• Added property for custom URIResolver used by xslt transformers for basic auth and SSL, see the example dk.defxws.fedoragsearch.server.URIResolverImpl class and the index.properties file in configTestOnLucene.
• Removed encoding of special characters in indexFields. Snippets now show special characters without modification. Indexes should be reindexed.

To install the service:

• Deploy fedoragsearch.war to the webapps directory of your web server, e.g. the tomcat supplied with Fedora, or similar. You may rename the .war file, before you copy it into the webapps directory, in order to give it another webapp name.
• Edit the configuration settings.

The SOAP service operations are deployed with the .war file, and the .wsdl file is available here.

• Edit the property values in the configvalues.xml file in .../webapps/<WEBAPPNAME>/WEB-INF/classes/config/ (where <WEBAPPNAME> by default is 'fedoragsearch'):

  	o Set the property values for your environment.
  	o Select the default config in configDefault.
  	o Save this edited file outside of the web server.
  	o Run target configOnWebServer after deployment from command line:
  
  	  >ant -f configvalues.xml configOnWebServer
    

  This will set your values into fedoragsearch.properties, repository.properties, and index.properties. Read these files to make sure they are correct.
• Then you may restart <WEBAPPNAME> and call http://<HOSTPORT>/<WEBAPPNAME>/rest in order to index and search. The name "rest" may be reconfigured in .../webapps/<WEBAPPNAME>/WEB-INF/web.xml
• Try the demo command line client. Change directory to

  .../webapps/<WEBAPPNAME>/client/

  make the file executable, and run

  sh runRESTClient.sh

  then you will get the usage instruction.
• Tailor the demos for your own purpose by editing renamed copies of the demo xslt stylesheets in

  .../webapps/<WEBAPPNAME>/WEB-INF/classes/config/rest/

  Then edit fedoragsearch.properties.
• Tailor the demo Lucene indexing stylesheet for your own purpose by editing a renamed copy of the demo xslt stylesheet in

  .../webapps/<WEBAPPNAME>/WEB-INF/classes/config/index/<INDEXNAME>/demoFoxmlToLucene.xslt

  For the sake of the example, the stylesheet indexes only active Fedora objects with PID starting with "demo" The options for tailoring include fields from other metadata datastreams than DC, field types and field boosts, see the stylesheet for options. Then edit index.properties.
• For your real applications, you must carefully edit all stylesheets for your purpose.
• Inspect the Lucene index with Luke. Notice, Luke cannot open an empty Lucene index.
• You may tailor the highlight of search terms in demo.css.
• You may want to experiment with more than one configuration, in which case you may maintain them under different names in parallel to the "config" configuration, which is the default configuration. In order to activate an alternative configuration you may use the semi-secret operation configure with parameter configName, either using the demo command line client or the REST interface.

As of Fedora 2.2, it is possible to configure Fedora to send a signal to the Generic Search Service when objects are added, modified, and purged. We expect future versions of Fedora to use more sophisticated messaging techniques (like JMS) to accomplish this, but for now Fedora can be made to send simple REST calls to the service at appropriate times.

To enable this, edit your fedora.fcfg file and change the class of the fedora.server.storage.DOManager module to fedora.server.storage.GSearchDOManager. Then populate the following module parameters as needed:

• gSearchRESTURL - The REST endpoint for GSearch, for example, http://localhost:8080/fedoragsearch/rest
• gSearchUsername - If GSearch is protected by authentication, this is the username that Fedora should use to authenticate.
• gSearchPassword - The password for the above user, if applicable

Lucene

The Lucene plugin comes as the java package dk.defxws.fgslucene together with the Apache Lucene java libraries. The set of classes was inspired by the JDBC API specification.

The Lucene plugin is used by configuration as explained above, see also the DemoOnLucene example.

Lucene has a very rich functionality, and this plugin exploits a small fraction of it. As a java programmer, you may have ideas for further exploitation, which you may realize by implementing an enhanced version of the plugin. Please, share such ideas and implementations with the Fedora community.

Solr

The Solr plugin comes as the java package dk.defxws.fgssolr.

The Solr plugin is used by configuration as explained above, see also the DemoOnSolr example.

This plugin indexes documents via the HTTP POST interface of Solr. Searches may be performed via the Solr native HTTP GET to the Solr server and via gfindObjects, which accesses the Lucene index directly. Solr functionality does not include browsing, however, this is offered by the plugin via the browseIndex operation.

Zebra

The Zebra plugin comes as the java package dk.defxws.fgszebra .

The Zebra plugin is used by configuration as seen from the DemoOnZebra example, which includes a README file, which explains how to get and install Zebra, and how to configure it.

• All engine specific operations return an engine specific xml answer, which is transformed by an engine-specific xslt stylesheet into result page xml. For a SOAP request this is the answer. For a REST request this is transformed to an html answer. There may be any number of xslt stylesheets to select from, the default ones are selected in the properties file. Selecting a copy stylesheet will allow the transfer of an answer untransformed. An alternative result page format is OpenSearch, which is an RSS2.0 extension.
• Parameters allow clients to select repository, index, and xslt stylesheets by name. In a real application, these values may be determined by the developer in the code, or by the administrator in the properties file.

• Objects in the Fedora repository are exported in FOXML format, transformed into an appropriate document format by the indexing stylesheet, and indexed by the engine in question. The XML datastreams are indexed as decided in the stylesheet. One managed or external datastream may be indexed per FedoraObject (which one is configurable), assuming that they contain the same text in different mimetypes.
• The following updateIndex actions are available:
  • createEmpty - creating or emptying the index. For a new index, you have to run createEmpty once, before you can run the other actions.
  • fromFoxmlFiles ( filePath ) - indexing FOXML records; filePath may be null, in which case the configured Fedora Object Directory is used, so that the whole of the Fedora registry is indexed.
  • fromPid ( PID ) - indexing one FOXML record, as exported by Fedora API-M; in case a previous index document with the same PID exists, it is first deleted. This is the incremental update operation that shall be called after all of Fedora's API-M operations that modifies a FedoraObject.
  • deletePid ( PID ) - deleting one index document.

A typical application will index one repository in one index. However, you have the possibility to index many repositories in one or more indexes in parallel, as illustrated here:

• There are OperationsImpl classes for Zebra, Lucene and Solr. The configManyToMany example has indexes for two engines, therefore similar searches may be compared.

Multilingual Configuration

Luis Zorita had this problem and solved it (his mail to fedora-users on 24 August 2006):