superset
superset copied to clipboard
apache
Starrocks executing a specific query will cause the Query history page to report an error and not load the data
Bug description
Starrocks executing a specific query will cause the Query history page to report an error and not load the data. Error msg: An error occurred while fetching Query historys: Fatal error
https://github.com/user-attachments/assets/dc4bf4da-4725-47ac-9545-ceaac1e429ae
How to reproduce the bug
- Create a data source with mysql, fill in the Starrocks cluster address and account secret.
- Goto Query history page(/sqllab/history/), at this point, you can see the query record normally.
- Goto Sql Lab, select the Starrocks data source you just created.
- Execute the following sql:
select date_add(current_date, -1) as yst_date. - Return to the Query history page, at this point the page reported an error, can not browse the query history.
Screenshots/recordings
superset_app container logs:
2024-08-22 06:53:04,605:ERROR:flask_appbuilder.api:list index out of range
Traceback (most recent call last):
File "/app/superset/sql_parse.py", line 297, in _extract_tables_from_sql
statements = parse(self.stripped(), dialect=self._dialect)
File "/usr/local/lib/python3.10/site-packages/sqlglot/init.py", line 87, in parse
return Dialect.get_or_raise(read or dialect).parse(sql, **opts)
File "/usr/local/lib/python3.10/site-packages/sqlglot/dialects/dialect.py", line 490, in parse
return self.parser(**opts).parse(self.tokenize(sql), sql)
File "/usr/local/lib/python3.10/site-packages/sqlglot/parser.py", line 1153, in parse
return self._parse(
File "/usr/local/lib/python3.10/site-packages/sqlglot/parser.py", line 1219, in _parse
expressions.append(parse_method(self))
File "/usr/local/lib/python3.10/site-packages/sqlglot/parser.py", line 1427, in _parse_statement
expression = self._parse_set_operations(expression) if expression else self._parse_select()
File "/usr/local/lib/python3.10/site-packages/sqlglot/parser.py", line 2486, in parse_select
from = self._parse_from()
File "/usr/local/lib/python3.10/site-packages/sqlglot/parser.py", line 2693, in _parse_from
exp.From, comments=self._prev_comments, this=self._parse_table(joins=joins)
File "/usr/local/lib/python3.10/site-packages/sqlglot/parser.py", line 3067, in _parse_table
subquery = self._parse_select(table=True)
File "/usr/local/lib/python3.10/site-packages/sqlglot/parser.py", line 2501, in _parse_select
self._parse_table()
File "/usr/local/lib/python3.10/site-packages/sqlglot/parser.py", line 3067, in _parse_table
subquery = self._parse_select(table=True)
File "/usr/local/lib/python3.10/site-packages/sqlglot/parser.py", line 2491, in _parse_select
this = self._parse_query_modifiers(this)
File "/usr/local/lib/python3.10/site-packages/sqlglot/parser.py", line 2639, in _parse_query_modifiers
key, expression = parser(self)
File "/usr/local/lib/python3.10/site-packages/sqlglot/parser.py", line 942, in
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/flask_appbuilder/api/init.py", line 110, in wraps
return f(self, *args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/flask_appbuilder/api/init.py", line 182, in wraps
return f(self, *args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/flask_appbuilder/api/init.py", line 1711, in get_list
return self.get_list_headless(**kwargs)
File "/app/superset/queries/api.py", line 340, in get_list_headless
response[flask_appbuilder.const.API_RESULT_RES_KEY] = list_model_schema.dump(lst, many=True)
File "/usr/local/lib/python3.10/site-packages/marshmallow/schema.py", line 557, in dump
result = self._serialize(processed_obj, many=many)
File "/usr/local/lib/python3.10/site-packages/marshmallow/schema.py", line 519, in _serialize
return [
File "/usr/local/lib/python3.10/site-packages/marshmallow/schema.py", line 520, in
Superset version
4.0.2
Python version
3.10
Node version
16
Browser
Firefox
Additional context
No response
Checklist
- [X] I have searched Superset docs and Slack and didn't find a solution to my problem.
- [X] I have searched the GitHub issue tracker and didn't find a similar bug report.
- [X] I have checked Superset's logs for errors and if I found a relevant Python stacktrace, I included it here as text in the "additional context" section.
To resolve the "list index out of range" error in the Query history page when executing a specific query in Starrocks on Apache Superset, you should check the handling of the statements list in the execute_sql_statements function. Specifically, ensure that the list of statements is not empty before attempting to access its elements.
Here is a potential fix:
- Add a check to ensure that
statementsis not empty before proceeding with the execution loop.
def execute_sql_statements(
# pylint: disable=too-many-arguments, too-many-locals, too-many-statements, too-many-branches
query_id: int,
rendered_query: str,
return_results: bool,
store_results: bool,
start_time: Optional[float],
expand_data: bool,
log_params: Optional[dict[str, Any]],
) -> Optional[dict[str, Any]]:
"""Executes the sql query returns the results."""
if store_results and start_time:
# only asynchronous queries
stats_logger.timing("sqllab.query.time_pending", now_as_float() - start_time)
query = get_query(query_id)
payload: dict[str, Any] = {"query_id": query_id}
database = query.database
db_engine_spec = database.db_engine_spec
db_engine_spec.patch()
if database.allow_run_async and not results_backend:
raise SupersetErrorException(
SupersetError(
message=__("Results backend is not configured."),
error_type=SupersetErrorType.RESULTS_BACKEND_NOT_CONFIGURED_ERROR,
level=ErrorLevel.ERROR,
)
)
# Breaking down into multiple statements
parsed_query = ParsedQuery(
rendered_query,
engine=db_engine_spec.engine,
)
if not db_engine_spec.run_multiple_statements_as_one:
statements = parsed_query.get_statements()
logger.info(
"Query %s: Executing %i statement(s)", str(query_id), len(statements)
)
else:
statements = [rendered_query]
logger.info("Query %s: Executing query as a single statement", str(query_id))
# Check if statements list is empty
if not statements:
raise SupersetErrorException(
SupersetError(
message=__("No valid SQL statements found."),
error_type=SupersetErrorType.INVALID_SQL_STATEMENT_ERROR,
level=ErrorLevel.ERROR,
)
)
logger.info("Query %s: Set query to 'running'", str(query_id))
query.status = QueryStatus.RUNNING
query.start_running_time = now_as_float()
db.session.commit()
# Rest of the function...
This check ensures that the statements list is not empty before attempting to execute any SQL statements, which should prevent the "list index out of range" error [1].
To continue talking to Dosu, mention @dosu.
![dosubot[bot] avatar](https://avatars.githubusercontent.com/in/324583?v=4 "Published by a pub.dev verified publisher"){kind=small}
I made a temporary change to superset/sql_parse.py which avoids this issue, but it may cause other problems, so someone still needs to fix the bug!
def _extract_tables_from_sql(self) -> set[Table]:
"""
Extract all table references in a query.
Note: this uses sqlglot, since it's better at catching more edge cases.
"""
try:
statements = parse(self.stripped(), dialect=self._dialect)
except ParseError as ex:
statements = []
except SqlglotError as ex:
...
Hi, does anyone fix it? I got same problem in 4.0.2 version, it happened in Saved Queries page
Maybe @miomiocat has some insight here?
Thanks.
We should use a specialized StarRocks Python client to connect to StarRocks instead of using a MySQL client and this issue does not occur with the StarRocks client.
We will continue to focus on developing and maintaining this StarRocks Python client.
Please refer to this usage link: https://docs.starrocks.io/docs/integrations/BI_integrations/Superset/
Seems that the above answer should suffice, and we can call this resolved. If anyone has further issues here, we can revisit/reopen if needed.