[SPARK-48343][SQL] Introduction of SQL Scripting interpreter

[davidm-db]

What changes were proposed in this pull request?

Previous PR introduced parser changes for SQL Scripting. This PR is a follow-up to introduce the interpreter for SQL Scripting language and proposes the following changes:

• SqlScriptingExecutionNode - introduces execution nodes for SQL scripting, used during interpretation phase:
  • SingleStatementExec - executable node for SingleStatement logical node; wraps logical plan of the single statement.
  • CompoundNestedStatementIteratorExec - implements base recursive iterator logic for all nesting statements.
  • CompoundBodyExec - concrete implementation of CompoundNestedStatementIteratorExec for CompoundBody logical node.
• SqlScriptingInterpreter - introduces the interpreter for SQL scripts. Product of interpretation is the iterator over the statements that should be executed.

Follow-up PRs will introduce further statements, support for exceptions thrown from parser/interpreter, exception handling in SQL, etc. More details can be found in Jira item for this task and its parent (where the design doc is uploaded as well).

Why are the changes needed?

The intent is to add support for SQL scripting (and stored procedures down the line). It gives users the ability to develop complex logic and ETL entirely in SQL.

Until now, users had to write verbose SQL statements or combine SQL + Python to efficiently write the logic. This is an effort to breach that gap and enable complex logic to be written entirely in SQL.

Does this PR introduce any user-facing change?

No. This PR is second in series of PRs that will introduce changes to sql() API to add support for SQL scripting, but for now, the API remains unchanged. In the future, the API will remain the same as well, but it will have new possibility to execute SQL scripts.

How was this patch tested?

There are tests for newly introduced parser changes:

• SqlScriptingExecutionNodeSuite - unit tests for execution nodes.
• SqlScriptingInterpreterSuite - tests for interpreter (with parser integration).

Was this patch authored or co-authored using generative AI tooling?

No.