maxRows option for executeStatement is not taken into account

[quadristan]

Version

I am using the 1.8.4 version

Description

We want to limit the amount of RAM usage. passing a default maxRows option has no effect, the whole query is returned

How to reproduce

    const operation = await this.session.executeStatement(
      `WITH numbers AS (
    SELECT num
    FROM (SELECT EXPLODE(SEQUENCE(1, 100)) AS num) AS seq
)
SELECT num
FROM numbers
ORDER BY RAND()`,
      { maxRows: 10 },
    );
    const fetched = await operation.fetchAll();
    console.log(fetched.length);

You will notice here that fetched has a size of 100, not 10.

Workaround

The workaround is to use fetchChunk instead:

const chunk = await operation.fetchChunk({
        maxRows: limit - result.length,
      });

Please deprecate or fix maxRows options for the executeStatement method of the session object

[kravets-levko]

Hi @quadristan! Your question is a duplicate of databricks/databricks-sql-nodejs#86 and databricks/databricks-sql-nodejs#155 - please read it. In short - maxRows in executeStatement is related to direct results feature and provides a hint to server about how much data it can return immediately