Things-Search Case Sensitivity Option

[dtmb91]

Currently, in the Ditto API for Things-Search there is no option for case sensitivity.

It is explicitly mentioned here: "The search is case sensitive. In case you don't know how exactly the spelling of value of the namespace, name, attribute, feature etc. is, use the like notation instead of eq for filtering." https://www.eclipse.org/ditto/http-api-doc.html#/Things-Search/get_search_things

My proposal is that the "like" filter accepts a third parameter dictating if case will be ignored or not.

[thjaeckle]

Hi @dtmb91 That's a good idea supporting case insensitive searches. I suggest using a new "function" name for that, e.g. likeIgnoringCase(<key>,"valUE*")

A PullRequest for that would be very welcome, as we currently don't have a priority on adding that. Please let us know if you need guidance where to make additions to the code for supporting that.

[thjaeckle]

Here some hints regarding how to implement:

• https://github.com/eclipse/ditto/blob/master/rql/query/src/main/java/org/eclipse/ditto/rql/query/criteria/LikePredicateImpl.java is the implementation of like - I would copy from that
• look at where the constructor of LikePredicateImpl is used - that is CriteriaFactoryImpl in the CriteriaFactory.like(Object) implementation
  • add another interface method to CriteriaFactory for the new like with case insensitivity
• look at where CriteriaFactory.like(Object) is used, that is in ParameterPredicateVisitor
  • add something new to ParameterPredicateVisitor similar to the existing one for "like":

  SINGLE_COMPARISON_NODE_MAPPING.put(SingleComparisonNode.Type.LIKE, CriteriaFactory::like);
  

• look where SingleComparisonNode.Type.LIKE enum value is used
  • that is the Scala based RqlPredicateParser.like: https://github.com/eclipse/ditto/blob/master/rql/parser/src/main/scala/org/eclipse/ditto/rql/parser/internal/RqlPredicateParser.scala#L105
  • add something new for that as well
  • as well as for its usage, etc.
• enhance PredicateVisitor and its implementations with the added "like ignoring case sensitivty": https://github.com/eclipse/ditto/blob/master/rql/query/src/main/java/org/eclipse/ditto/rql/query/criteria/visitors/PredicateVisitor.java
  • e.g. CreateBsonPredicateVisitor will map that to the MongoDB query which is finally done to search into the DB: https://github.com/eclipse/ditto/blob/master/thingsearch/service/src/main/java/org/eclipse/ditto/thingsearch/service/persistence/read/criteria/visitors/CreateBsonPredicateVisitor.java#L94

  return fieldName -> Filters.regex(fieldName, valueWithoutLeadingOrTrailingWildcard, "");
  

  • MongoDB's used Filters.regex should have an option to search case insensitive

[Abhijeetmishr]

@thjaeckle can I pick up this if this is not assigned to anyone esle ?

[thjaeckle]

@Abhijeetmishr sure, please go ahead. Noone is working on this as far as I know. The old hints regarding implementation should still be valid.

[thjaeckle]

Another interpretation of RQL can be found here: https://connect.cloudblue.com/community/developers/api/rql/

The original RQL spec document did not include "like", so Ditto added it. Same as the linked resource. They use ilike for case insensitive search. Which is also common for SQL. So I would suggest to introduce "ilike".

[Abhijeetmishr]

@thjaeckle please assign this to me I will try to do it as earliest as possible.

[thjaeckle]

@thjaeckle please assign this to me I will try to do it as earliest as possible.

Done

[Abhijeetmishr]

@thjaeckle My approach:-

1. Create a method in interface CriteriaFactory with name ilike.
2. implement ilike method in CriteriaFactoryImpl.
3. add ilike functionality similar to like in all the places.
4. create helper function similar to LikeHelper and change this line ->

String escapedString = Pattern.compile(Pattern.quote(valueString), Pattern.CASE_INSENSITIVE).toString(); will this work ?

Regards, Abhijeet

[thjaeckle]

@Abhijeetmishr my comment should still be valid: https://github.com/eclipse-ditto/ditto/issues/1087#issuecomment-860780016

What you proposed is probably a part of the solution, but applying it to the MongoDB query was missing from your suggestion.

Adjusting the in Scala written RQL parser is also something to do.

[Abhijeetmishr]

Hey @thjaeckle I have created one PR please review it #1570. And I wanted to understand whole project structure and high level flow.

Regards, Abhijeet