Need for Query and Index in AEM

For functionalities like full text search, retrieve content based on certain property or conditions associated with property/to avoid iterations on huge volume of content under the root, we write query based logic in AEM.

Languages supported :
  • XPATH
  • JCR-SQL2 
XPath:
  • Created using AEM Query Builder API. - com.day.cq.search.*
  • From development point of view, we need to be aware of standard OOB predicates to arrive at XPATH query.
JCR-SQL2:
  • JCR-SQL2 queries are created using QueryManager - javax.jcr.query.QueryManager
  • QueryManager is acquired through JCR Session  - session.getWorkspace().getQueryManager()
Query Processing:
  • Before AEM 6.0/Jackrabbit 2, everything in AEM is indexed by default.
  • With Jackrabbit Oak, we can create custom indexes based on the need. OOB indexes are available under /oak:index node in the repository.
  • Oak Query Engine
    • Process queries in the form of JCR-SQL2. 
      • This means if we write queries using QueryBuilder API involving predicates, it will result in XPath query.
      • QueryEngineImpl(org.apache.jackrabbit.oak.query.QueryEngineImpl) then parses XPATH query and converts to SQL2.
    • Uses a cost based query optimizer to get the cost involved to process the query from all the available indexes.
    • All the available indexes under oak:index will estimate the cost.
    • Cost for traversal is also calculated.
    • Cost value of the index can be "Infinity". This implies that respective index cannot deal with specific condition/respective index cannot query the data.
    • Query Engine then picks the index which has lowest estimated cost.
    • Note : 
      • Cost value is an estimated worst-case value and hence need not be accurate. 
      • The above said process happens whenever a query is executed.
  • Example: 
    • If a query is written to get all pages with specific "jcr:title" value, query engine need not traverse the entire repository for jcr:title, instead looks for selected index(based on cost) and then fetch/filter results from that indexed content.
    • oak:index definition for jcr:title of page is available OOB in /oak:index/cqPageLucene/indexRules/cq:Page/properties/jcrTitle
Need for creating custom index:
  • When the query engine has to traverse the entire repository/more than allowed nodes(per configuration, its100000 - Apache Jackrabbit Query Engine Settings - In memory read limit) for what we have queried for, it will result in slow query and eventually throw UnsupportedOperationException to stop further processing. (Observing the logs, we would notice a warn level message suggesting to create an index as shown below)
*DEBUG* [0:0:0:0:0:0:0:1 [1588022611960] GET /libs/cq/search/content/querydebug.html HTTP/1.1] org.apache.jackrabbit.oak.query.QueryImpl no proper index was found for filter Filter(query=select [jcr:path], [jcr:score], * from [nt:unstructured] as a where isdescendantnode(a, '/') /* xpath: /jcr:root//element(*, nt:unstructured) */, path=//*)
*WARN* [0:0:0:0:0:0:0:1 [1588022611960] GET /libs/cq/search/content/querydebug.html HTTP/1.1] org.apache.jackrabbit.oak.query.QueryImpl Traversal query (query without index): select [jcr:path], [jcr:score], * from [nt:unstructured] as a where isdescendantnode(a, '/') /* xpath: /jcr:root//element(*, nt:unstructured) */; consider creating an index

Screenshot of OSGI config for reference:
query engine settings

 Play around in your local instance (to visualize the flow mentioned above)
  • Create a new logger entry in sling log for below highlighted APIs
  • Navigate to http://localhost:4502/libs/cq/search/content/querydebug.html and frame a sample query predicates and execute. (Can also try to intentionally induce a traversal query in local and observe the logs)
    path=/content/we-retail
    type=cq:Page
    1_property=@jcr:content/jcr:title
    1_property.value=English
    p.limit=-1
  • Observe the logs either in log file or directly in /system/console/slinglog (Have highlighted the points mentioned in the flow)

Comments

Post a Comment

Popular posts from this blog

Embedding Third party dependency/OSGi bundle in AEM application hosted in AEMasCS

Image
This post illustrate the ways of embedding third party dependency in AEM maven project. Steps are mentioned considering the project structure to be followed for AEMasCS while the same can be followed for classic versions too.  Before we begin, I would like to explain the scenario of including a dependency in our AEM maven project.  When we would like to make use of an API that is not already available in our project, we look for that in maven-central repository and specify the dependency in main pom.xml and desired module's pom.xml file. (Lets say, AWS S3 Java SDK V2 dependency) Upon updating the project in the IDE, we would see our class files (making use of those APIs) are compiled/build successfully without any errors.  Now, in the context of AEM/OSGi environment, when the same package and hence the bundle is deployed to AEM instance, dependencies that we added new might not be resolved/available in the OSGi as a bundle. Something like the below This indicates that the...
Read more

Embed Third party dependency using bnd-maven-plugin

Image
This post explains the use of bnd-maven-plugin to embed third party dependency  Two main instructions of bnd-maven-plugin that serves the purpose are detailed below.  -conditionalpackage  pick packages from respective library and copy them in the bundle. Any dependencies of those copied packages will also be copied if they match the selectors. This means we need to explicitly list all the Java package names of desired dependency as part of this instruction (including its transitive dependencies) More about conditionalpackage instruction from official docs -  https://bnd.bndtools.org/instructions/conditionalpackage.html -includeresource  For including resource inline -includeresource: @artifactId-version.jar!/* For embedding as a jar with an entry in bundle class path -includeresource: artifactId-version.jar;lib:=true For several other possible combinations for including the resource, please check - https://bnd.bndtools.org/instructions/includeresource.html Wi...
Read more

Creation of Template Types for Editable templates

Image
In this post, we will create templates types which is the base for creating editable templates. Before getting to the subject directly, have added an introduction about templates in general. Template is the base for creating pages. Starting AEM 6.2, we have categorisation for templates - Static and Editable templates. Static templates :  This has been available for several versions of AEM/CQ Created using "Create Template" option with base page component as resourceType (which is responsible for rendering the page structure).  Template ( base page component as resourceType ) -> Page This cannot be edited as such. Since it just holds a reference to base page component, any changes to the page (other than authorable components), would need a change in base page component which is ultimately a work of developer. (calls for a code change) Allowed components are configured using Design mode (list being saved to etc/designs/projectfolder/templatename/jcr:content) ...
Read more