Need for Query and Index in AEM

For functionalities like full text search, retrieve content based on certain property or conditions associated with property/to avoid iterations on huge volume of content under the root, we write query based logic in AEM.

Languages supported :
  • XPATH
  • JCR-SQL2 
XPath:
  • Created using AEM Query Builder API. - com.day.cq.search.*
  • From development point of view, we need to be aware of standard OOB predicates to arrive at XPATH query.
JCR-SQL2:
  • JCR-SQL2 queries are created using QueryManager - javax.jcr.query.QueryManager
  • QueryManager is acquired through JCR Session  - session.getWorkspace().getQueryManager()
Query Processing:
  • Before AEM 6.0/Jackrabbit 2, everything in AEM is indexed by default.
  • With Jackrabbit Oak, we can create custom indexes based on the need. OOB indexes are available under /oak:index node in the repository.
  • Oak Query Engine
    • Process queries in the form of JCR-SQL2. 
      • This means if we write queries using QueryBuilder API involving predicates, it will result in XPath query.
      • QueryEngineImpl(org.apache.jackrabbit.oak.query.QueryEngineImpl) then parses XPATH query and converts to SQL2.
    • Uses a cost based query optimizer to get the cost involved to process the query from all the available indexes.
    • All the available indexes under oak:index will estimate the cost.
    • Cost for traversal is also calculated.
    • Cost value of the index can be "Infinity". This implies that respective index cannot deal with specific condition/respective index cannot query the data.
    • Query Engine then picks the index which has lowest estimated cost.
    • Note : 
      • Cost value is an estimated worst-case value and hence need not be accurate. 
      • The above said process happens whenever a query is executed.
  • Example: 
    • If a query is written to get all pages with specific "jcr:title" value, query engine need not traverse the entire repository for jcr:title, instead looks for selected index(based on cost) and then fetch/filter results from that indexed content.
    • oak:index definition for jcr:title of page is available OOB in /oak:index/cqPageLucene/indexRules/cq:Page/properties/jcrTitle
Need for creating custom index:
  • When the query engine has to traverse the entire repository/more than allowed nodes(per configuration, its100000 - Apache Jackrabbit Query Engine Settings - In memory read limit) for what we have queried for, it will result in slow query and eventually throw UnsupportedOperationException to stop further processing. (Observing the logs, we would notice a warn level message suggesting to create an index as shown below)
*DEBUG* [0:0:0:0:0:0:0:1 [1588022611960] GET /libs/cq/search/content/querydebug.html HTTP/1.1] org.apache.jackrabbit.oak.query.QueryImpl no proper index was found for filter Filter(query=select [jcr:path], [jcr:score], * from [nt:unstructured] as a where isdescendantnode(a, '/') /* xpath: /jcr:root//element(*, nt:unstructured) */, path=//*)
*WARN* [0:0:0:0:0:0:0:1 [1588022611960] GET /libs/cq/search/content/querydebug.html HTTP/1.1] org.apache.jackrabbit.oak.query.QueryImpl Traversal query (query without index): select [jcr:path], [jcr:score], * from [nt:unstructured] as a where isdescendantnode(a, '/') /* xpath: /jcr:root//element(*, nt:unstructured) */; consider creating an index

Screenshot of OSGI config for reference:
query engine settings

 Play around in your local instance (to visualize the flow mentioned above)
  • Create a new logger entry in sling log for below highlighted APIs
  • Navigate to http://localhost:4502/libs/cq/search/content/querydebug.html and frame a sample query predicates and execute. (Can also try to intentionally induce a traversal query in local and observe the logs)
    path=/content/we-retail
    type=cq:Page
    1_property=@jcr:content/jcr:title
    1_property.value=English
    p.limit=-1
  • Observe the logs either in log file or directly in /system/console/slinglog (Have highlighted the points mentioned in the flow)

Comments

Post a Comment

Popular posts from this blog

Embedding Third party dependency/OSGi bundle in AEM application hosted in AEMasCS

Image
This post illustrate the ways of embedding third party dependency in AEM maven project. Steps are mentioned considering the project structure to be followed for AEMasCS while the same can be followed for classic versions too.  Before we begin, I would like to explain the scenario of including a dependency in our AEM maven project.  When we would like to make use of an API that is not already available in our project, we look for that in maven-central repository and specify the dependency in main pom.xml and desired module's pom.xml file. (Lets say, AWS S3 Java SDK V2 dependency) Upon updating the project in the IDE, we would see our class files (making use of those APIs) are compiled/build successfully without any errors.  Now, in the context of AEM/OSGi environment, when the same package and hence the bundle is deployed to AEM instance, dependencies that we added new might not be resolved/available in the OSGi as a bundle. Something like the below This indicates that the added dep
Read more

OSGI Factory Configuration implementation

OSGI Factory configuration is used to have many instances of an OSGI config (represented/identified via an identifier) for the single OSGI service/PID. Most often used OOB Factory configurations for example: Logger Factory/writer configuration (for log statements) Service user Mapper service Amendment (for using service resource resolver in projects) Few sample real time use case for the need of OSGI factory config: For multi tenant projects, if we have generic service used across tenants with properties/attributes specific to tenant. For single tenant, generic service targeting all locales with properties/attributes specific to locale. (where generic service -> holds logic in the lines of calling third party service for retrieving/updating information.) Implementation: ( In terms of OSGI R6 annotations ): Create Object Class Definition(OCD) defining desired properties for the OSGI service. - OSGIFactoryConfigOCD.java OSGI Service/Factory service to retrieve
Read more

Embed Third party dependency using bnd-maven-plugin

Image
This post explains the use of bnd-maven-plugin to embed third party dependency  Two main instructions of bnd-maven-plugin that serves the purpose are detailed below.  -conditionalpackage  pick packages from respective library and copy them in the bundle. Any dependencies of those copied packages will also be copied if they match the selectors. This means we need to explicitly list all the Java package names of desired dependency as part of this instruction (including its transitive dependencies) More about conditionalpackage instruction from official docs -  https://bnd.bndtools.org/instructions/conditionalpackage.html -includeresource  For including resource inline -includeresource: @artifactId-version.jar!/* For embedding as a jar with an entry in bundle class path -includeresource: artifactId-version.jar;lib:=true For several other possible combinations for including the resource, please check - https://bnd.bndtools.org/instructions/includeresource.html With complete understandin
Read more