[FEATURE] Create a PPL compatibility verification framework

[dai-chen]

Is your feature request related to a problem?

Yes. When implementing the PPL language on top of various query engines (e.g., OpenSearch, Spark, Trino), it’s crucial to assess whether the behavior conforms to the PPL specification. Currently, there is no standardized framework to validate PPL query compatibility across engines. This makes it hard to detect semantic mismatches in areas like type coercion, null handling, expression evaluation, or operator behavior.

Without such a framework, each backend must create its own ad hoc tests, leading to duplicated effort, inconsistent coverage, and gaps in spec compliance.

What solution would you like?

Introduce a PPL compatibility test framework that includes:

• A reference test dataset shared across engines
• A suite of PPL queries covering core syntax and semantics (e.g., source, eval, where, stats, sort, etc.)
• Expected result files for each query
• A lightweight test runner that executes the queries against a given backend and compares the actual output to the expected results

This framework would help backend implementers verify conformance to the PPL language spec and avoid regressions when introducing new features.

What alternatives have you considered?

• Each backend sticks with its current backend-specific test suites as below

Do you have any additional context?

References

• PartiQL test suite: https://github.com/partiql/partiql-tests
• ZetaSQL compliance test suite: https://github.com/google/zetasql

Current PPL IT

There are currently several PPL-related test suites across the opensearch-sql and opensearch-spark repositories. However, these are tightly coupled to specific engines and not designed as a general-purpose compatibility framework.

• OpenSearch
  • OS PPL Calcite IT: https://github.com/opensearch-project/sql/tree/main/integ-test/src/test/java/org/opensearch/sql/calcite
  • OS PPL V2 IT: https://github.com/opensearch-project/sql/tree/main/integ-test/src/test/java/org/opensearch/sql/ppl
• Spark
  • Spark PPL IT: https://github.com/opensearch-project/opensearch-spark/tree/main/integ-test/src/integration/scala/org/opensearch/flint/spark/ppl
  • Spark PPL e2e IT: https://github.com/opensearch-project/opensearch-spark/blob/main/e2e-test/src/test/scala/org/opensearch/spark/e2e/EndToEndITSuite.scala