bq_table_function : Added functionality to parse table functionsin BigQuery

[moh-lch]

Brief summary of the change made

Are there any other side effects of this change that we should be aware of?

Pull Request checklist

• [ ] Please confirm you have completed any of the necessary steps below.

• Included test cases to demonstrate any code changes, which may be one or more of the following:

  • .yml rule test cases in test/fixtures/rules/std_rule_cases.
  • .sql/.yml parser test cases in test/fixtures/dialects (note YML files can be auto generated with tox -e generate-fixture-yml).
  • Full autofix test cases in test/fixtures/linter/autofix.
  • Other.

• Added appropriate documentation for the change.

• Created GitHub issues for any relevant followup/future enhancements if appropriate.

[greg-finley]

Great start! Could you add some tests like you see in this PR?

Ref: https://github.com/sqlfluff/sqlfluff/wiki/Contributing-Dialect-Changes#testing-your-changes

[github-actions[bot]]

Coverage Results ✅

Name    Stmts   Miss  Cover   Missing
-------------------------------------
TOTAL   17016      0   100%

222 files skipped due to complete coverage.

[coveralls]

coverage: 100.0%. remained the same when pulling a285505022873215a8d5ba59462765086b52646b on moh-lch:bq_table_function into 4668f1de8cdbf47d6f4f76b0bcc13617ad670268 on sqlfluff:main.

[moh-lch]

Great start! Could you add some tests like you see in this PR?

Ref: https://github.com/sqlfluff/sqlfluff/wiki/Contributing-Dialect-Changes#testing-your-changes

@greg-finley I did try and: 1 - Create a .sql file for simple table function. 2 - Run the BQ dialect test by running : python test/generate_parse_fixture_yml.py -d bigquery

• I get this output always : _"Match Pattern Received: filter=* dialect=bigquery new-only=False Found 109 file(s) to generate 'bigquery' complete. 1 fails. ⚠️
  • 'create_or_replace_table_function.sql' FAILED TO GENERATE ALL CASES"_

3 - I traced the issue to the file "conftest.py" line 121 in the function "parse_example_file" getting the tree value based on the token. *** Token Passed :** (<WordSegment: ([L: 1, P: 1]) 'CREATE'>, <WhitespaceSegment: ([L: 1, P: 7]) ' '>, <WordSegment: ([L: 1, P: 8]) 'OR'>, <WhitespaceSegment: ([L: 1, P: 10]) ' '>, <WordSegment: ([L: 1, P: 11]) 'REPLACE'>, <WhitespaceSegment: ([L: 1, P: 18]) ' '>, <WordSegment: ([L: 1, P: 19]) 'TABLE'>, <WhitespaceSegment: ([L: 1, P: 24]) ' '>, <WordSegment: ([L: 1, P: 25]) 'FUNCTION'>, <WhitespaceSegment: ([L: 1, P: 33]) ' '>, <WordSegment: ([L: 1, P: 34]) 'ds'>, <CodeSegment: ([L: 1, P: 36]) '.'>, <WordSegment: ([L: 1, P: 37]) 'lch'>, <WhitespaceSegment: ([L: 1, P: 40]) ' '>, <WordSegment: ([L: 1, P: 41]) 'AS'>, <WhitespaceSegment: ([L: 1, P: 43]) ' '>, <NewlineSegment: ([L: 1, P: 44]) '\n'>, <WhitespaceSegment: ([L: 2, P: 1]) ' '>, <WordSegment: ([L: 2, P: 5]) 'SELECT'>, <NewlineSegment: ([L: 2, P: 11]) '\n'>, <WhitespaceSegment: ([L: 3, P: 1]) ' '>, <WordSegment: ([L: 3, P: 9]) 'id'>, <WhitespaceSegment: ([L: 3, P: 11]) ' '>, <WordSegment: ([L: 3, P: 12]) 'AS'>, <WhitespaceSegment: ([L: 3, P: 14]) ' '>, <WordSegment: ([L: 3, P: 15]) 'call_id'>, <CodeSegment: ([L: 3, P: 22]) ','>, <NewlineSegment: ([L: 3, P: 23]) '\n'>, <WhitespaceSegment: ([L: 4, P: 1]) ' '>, <WordSegment: ([L: 4, P: 9]) 'creation_date'>, <WhitespaceSegment: ([L: 4, P: 22]) ' '>, <WordSegment: ([L: 4, P: 23]) 'AS'>, <WhitespaceSegment: ([L: 4, P: 25]) ' '>, <WordSegment: ([L: 4, P: 26]) 'created_at'>, <NewlineSegment: ([L: 4, P: 36]) '\n'>, <WhitespaceSegment: ([L: 5, P: 1]) ' '>, <WordSegment: ([L: 5, P: 5]) 'FROM'>, <NewlineSegment: ([L: 5, P: 9]) '\n'>, <WhitespaceSegment: ([L: 6, P: 1]) ' '>, <WordSegment: ([L: 6, P: 9]) 'ds'>, <CodeSegment: ([L: 6, P: 11]) '.'>, <WordSegment: ([L: 6, P: 12]) 'data'>, <NewlineSegment: ([L: 6, P: 16]) '\n'>, <NewlineSegment: ([L: 7, P: 1]) '\n'>, <EndOfFile: ([L: 8, P: 1]) ''>) *** Tree Returned :** {'dedent', 'base' , 'select_clause_element', 'from_clause', 'select_clause', 'alias_expression', 'whitespace', 'from_expression', 'select_statement', 'newline', 'statement', 'column_reference', 'from_expression_element', 'file', 'symbol', 'identifier', 'indent', 'table_reference', 'keyword', 'table_expression', 'end_of_file'}

Conclusion :

• So it fails on the "base" value when the tree value is returned to the function "generate_one_parse_fixture" in the file "generate_parse_fixture_yml.py" based on this if statement :

if "base" in types:
        return example, SQLParseError(f"Unnamed base section when parsing: {sql_path}")

• Part of my confusion is that I cant find the "parser.py" under "sqlfluff\test\core" as its what is generating the tree value based on the token generated by "lexer.py" in "conftest.py" under the function "parse_example_file".

I though Im not missing anything as I tested my changes on my local sqlfluff env and tried linting multiple files and my changes worked so not sure what am not seeing when testing hmmmm maybe you have an idea. Thanks in Advance

[WittierDinosaur]

This is pretty hard to debug remotely. What is the content of the test file you're trying to generate?

[moh-lch]

This is pretty hard to debug remotely. What is the content of the test file you're trying to generate?

@WittierDinosaur sure thing ! you will be able to see it under the commit "bq_table_function : Added file by mistake". I added the .sql test file by mistake and removed it. Let me know if you need something else.

[WittierDinosaur]

I wonder if it's trying to create a table named function. Can you try moving function from unreserved keywords into reserved keywords?

[keraion]

2 - Run the BQ dialect test by running : python test/generate_parse_fixture_yml.py -d bigquery

• I get this output always : _"Match Pattern Received: filter=* dialect=bigquery new-only=False Found 109 file(s) to generate 'bigquery' complete. 1 fails. ⚠️

  • 'create_or_replace_table_function.sql' FAILED TO GENERATE ALL CASES"_

@moh-lch from what I can tell the parsing logic is fine when running against the few examples from the BigQuery docs. I see you tried to run the tests by running python test/generate_parse_fixture_yml.py -d bigquery. Would it be possible to use tox to set up the environment for the tests with tox -e generate-fixture-yml -- --dialect bigquery?