datawave icon indicating copy to clipboard operation
datawave copied to clipboard

Published 20 hours ago • verified publisher icon NationalSecurityAgency

IngestTypePruningVisitor can produce invalid Jexl Syntax

Open apmoriarty opened this issue 6 months ago • 1 comments

When the IngestTypePruning visitor is run after the RangeStream it can produce a query string with invalid Jexl

One example is the token ( && (A == '1')).

The root cause of this is unknown, but some possibilities include:

  • The query tree pre-RangeStream has invalid node parentage
  • The query tree post-RangeStream has invalid node parentage
  • The IngestTypePruningVisitor breaks node parentage during it's operation
  • Some unhandled edge case in the IngestTypePruningVisitor

apmoriarty avatar May 30 '25 16:05 apmoriarty

Do you have some examples of queries that produce the invalid Jexl?

SethSmucker avatar Jun 10 '25 16:06 SethSmucker

FYI, I found that the UniqueExpressionTermsVisitor can also produce a non-flattened tree in the end. The query that exposed this was query in the ExecutableVisitorExpansionTest.testGeowaveExpansion in the process of working on a different ticket. The answer was to simply add a flatten call after the visitor was executed.

ivakegg avatar Jul 02 '25 12:07 ivakegg

Can this be closed? Seems this was addressed in https://github.com/NationalSecurityAgency/datawave/pull/3016 and that PR has since been merged.

foster33 avatar Aug 01 '25 16:08 foster33

Yes

apmoriarty avatar Aug 01 '25 16:08 apmoriarty