Lucene Index in AEM - Part 3


Analyzers in Lucene Index

This post illustrates the use of analyzers in full text search with sample use case.

Apache Lucene Analyzers :
Analyzers as with the name is used to analyze the text both at the time of indexing and at the time of searching (via query execution)
  • An analyzer examines the text of fields and generates a token stream. It can be either a 
    • Single Java class or 
    • Composed of a series of Tokenizer and Filter Java classes.
  • Tokenizer breaks the data into lexical units or tokens
  • Filters then examines these tokens -> amends/discard/create new one based on the configuration
  • Series of Tokenizer + Filters => Analyzer
  • There are direct Analyzer classes, Tokenizer and Filters available OOB. Based on our requirement we can choose to use either direct Analyzer or Tokenizer + Filter combination.(Analyzer via composition)
  • Examples:
  • Analyzer - 
    • StandardAnalyzer (org.apache.lucene.analysis.standard.StandardAnalyzer)
    • Removes stop words, converts to lowercase, recognize URLs and emails - most commonly used
  • Tokenizer - 
    • Standard (org.apache.lucene.analysis.standard.StandardTokenizerFactory)
    • Splits the text field into tokens, treating whitespace and punctuation as delimiters.
  • Filters :
    • Stopwords Filter (org.apache.lucene.analysis.core.StopFilterFactory) - Removes stop words
    • Lowercase Filter (org.apache.lucene.analysis.core.LowerCaseFilterFactory) - Converts token to lowercase
    • PorterStem Filter (org.apache.lucene.analysis.en.PorterStemFilterFactory) - Creates stem words from the tokens
  • In Lucene Full Text Index defintion, 
    • If we are opting to using direct Analyzer class, fully qualified Java class name is to be mentioned (using a property called "class" - highlighted in demo video)
    • If we are using Tokenizer and Filter combination. name without Factory suffix can be used. (Standard for StandardTokenizerFactory and PorterStem for PorterStemFilterFactory - this is again highlighted in demo video) 
    • Note : If we are using Analyzer via composition, class property need to be removed. 
Use case:
We will look into the common need for a full text search - Synonym and Stemming support
To follow along with same use case/to use same Lucene full text index created in previous posts (part 1 and part 2), will consider below two scenario

In we-retail DAM assets,
  • /content/dam/we-retail/en/activities (which again has biking, climbing. hiking etc as its kind)
    • We will create stemming filter as part of analyzers in Lucene Full text Index to fetch assets related to "activities" when we search using its stem word like "activity"
  • /content/dam/we-retail/en/products/apparel (which again has gloves, coats, pants and so on under apparel cateory)
    • We will create Synonym filter as part of analyzers in Lucene Full text Index to fetch assets related to "apparel" when we search using its synonyms like "clothing or garments"
Video Demo:
Stemming:
  • Highlights the use of PorterStem Filter with Standard Tokenizer (Analyzer via composition)
  • EnglishAnalyzer class is used which has PorterStemFilter in it. (Direct Analyzer class)


Synonym:
Synonym Filter with Standard Tokenizer is used (Analyzer via composition)

Comments

  1. balarama rajuSeptember 4, 2023 at 12:27 AM

    Hi VijayaLakshmi, I am trying to include Lemmatization into lucene indexing via a custom lucene token fileter, however AEM doesn't pick-up the token filter from my core bundle. Do you know how to register a custom Lucene Filter with AEM ? so that it can be used in the composition?

    ReplyDelete

Post a Comment

Popular posts from this blog

Embedding Third party dependency/OSGi bundle in AEM application hosted in AEMasCS

Image
This post illustrate the ways of embedding third party dependency in AEM maven project. Steps are mentioned considering the project structure to be followed for AEMasCS while the same can be followed for classic versions too.  Before we begin, I would like to explain the scenario of including a dependency in our AEM maven project.  When we would like to make use of an API that is not already available in our project, we look for that in maven-central repository and specify the dependency in main pom.xml and desired module's pom.xml file. (Lets say, AWS S3 Java SDK V2 dependency) Upon updating the project in the IDE, we would see our class files (making use of those APIs) are compiled/build successfully without any errors.  Now, in the context of AEM/OSGi environment, when the same package and hence the bundle is deployed to AEM instance, dependencies that we added new might not be resolved/available in the OSGi as a bundle. Something like the below This indicates that the added dep
Read more

OSGI Factory Configuration implementation

OSGI Factory configuration is used to have many instances of an OSGI config (represented/identified via an identifier) for the single OSGI service/PID. Most often used OOB Factory configurations for example: Logger Factory/writer configuration (for log statements) Service user Mapper service Amendment (for using service resource resolver in projects) Few sample real time use case for the need of OSGI factory config: For multi tenant projects, if we have generic service used across tenants with properties/attributes specific to tenant. For single tenant, generic service targeting all locales with properties/attributes specific to locale. (where generic service -> holds logic in the lines of calling third party service for retrieving/updating information.) Implementation: ( In terms of OSGI R6 annotations ): Create Object Class Definition(OCD) defining desired properties for the OSGI service. - OSGIFactoryConfigOCD.java OSGI Service/Factory service to retrieve
Read more

Embed Third party dependency using bnd-maven-plugin

Image
This post explains the use of bnd-maven-plugin to embed third party dependency  Two main instructions of bnd-maven-plugin that serves the purpose are detailed below.  -conditionalpackage  pick packages from respective library and copy them in the bundle. Any dependencies of those copied packages will also be copied if they match the selectors. This means we need to explicitly list all the Java package names of desired dependency as part of this instruction (including its transitive dependencies) More about conditionalpackage instruction from official docs -  https://bnd.bndtools.org/instructions/conditionalpackage.html -includeresource  For including resource inline -includeresource: @artifactId-version.jar!/* For embedding as a jar with an entry in bundle class path -includeresource: artifactId-version.jar;lib:=true For several other possible combinations for including the resource, please check - https://bnd.bndtools.org/instructions/includeresource.html With complete understandin
Read more