Home
Categories
Dictionary
Download
Project Details
Changes Log
FAQ
License

Pipeline



Adding a document in the index will call the Pipeline for each field content in the document. A Pipeline is an ordered list of Pipeline.Functions which have two one mandatory methods:

The output of the functions in the pipeline will be passed to the next function in the pipeline. To exclude a token from entering the index the function should return null, the rest of the pipeline will not be called with this token.

General pipeline

In general (though it is by no means mandatory), a Pipeline can contain the following functions:
  • A trimmer which will trim the words in the text
  • A StopWordFilter which will remove stop words[1]
    For example words like a, the, etc... in English
  • A Stemmer which will transform words into their root form[2]
    A Stemmer usually implements the Porter Stemming Algorithm.
However extra functions can be added to the Pipeline as needed. For example it is possible to add a Metaphoning processor to the functions.

Note that the StopWordFilter and the Stemmer usually depend on the language.

Built-in language-specific pipeline

By passing a language id to the index at creation, a Pipeline configured for this language will be chosen. For the moment, the languages are supported:
  • "en": English (default) language[3]
    When creating an index without specifying a language, the English language will be used for the Pipeline
  • "fr": French language
  • "de": German language
  • "es": Spanish language

Associated built-in functions

The built-in language-specific pipelines contain the following functions customized for the specified language:
  • A generic trimmer which will trim the words in the text
  • A language-specific StopWordFilter which will remove stop words[4]
    For example words like a, the, etc... in English
  • A language-specific Stemmer which will transform words into their root form[5]
    A Stemmer usually implements the Porter Stemming Algorithm.
  • A generic Metaphone function if the index is specified to enable Metaphoning
  • A generic Remove accents function if the index is specified to enable Removing accents

Metaphone function

The Metaphone function implements a Metaphoning processor. Metaphoning index words by their pronunciation. The Metaphone processor depends on the language.

Remove accents function

The Remove accents function just remove all accentuation from the text.

Example

For example for creating a French Index:
   Index index = ElasticLunr.createIndex("fr");
Or for creating an English Index with Metaphoning:
   Index index = ElasticLunr.createIndex("en", PipelineFactory.METAPHONE_ENABLED);

Creating a custom pipeline

You can create a custom Pipeline and add functions to it by calling Pipeline.addFunction(Function function) method.

A Pipeline Function must implement the following methods: The Pipeline can be registered with a key to be used later in the index by calling the PipelineFactory.addPipeline(String lang, Pipeline pipeline) method.

For example:
   Pipeline pipeline = new Pipeline();
   pipeline.addFunction(new Trimmer());
   pipeline.addFunction(new MyStopWordFilter());
   pipeline.addFunction(new MyStemmer());      
      
   PipelineFactory factory = PipelineFactory.getInstance();
   factory.addPipeline("myKey", pipeline);
      
   Index index = ElasticLunr.createIndex("myKey");

Default pipeline

When using the Index() constructor without argument, or calling the ElasticLunr.createIndex() static method, an English-language Pipeline is created by default.

So the following code:
   Index index = ElasticLunr.createIndex();
is equivalent to:
   Index index = ElasticLunr.createIndex("en");

Notes

  1. ^ For example words like a, the, etc... in English
  2. ^ A Stemmer usually implements the Porter Stemming Algorithm.
  3. ^ When creating an index without specifying a language, the English language will be used for the Pipeline
  4. ^ For example words like a, the, etc... in English
  5. ^ A Stemmer usually implements the Porter Stemming Algorithm.

Categories: Creation | General

Copyright 2017 Wei Song. Copyright 2018 Herve Girod. All Rights Reserved. Documentation and source under the MIT licence

Project Web Hosted by SourceForge.net Copyright 1999-2010 - Geeknet, Inc., All Rights Reserved About - Legal - Help