bigquery lineage error makes job hung

[ananbas]

Describe the bug bigquery lineage error makes job hung because of super long sql query

To Reproduce execute a super long SQL query and then trigger a bigquery lineage ingestion, then in the log there is an error during lineage extraction and then nothing happen.

I need to cancel the job manually.

Expected behavior lineage eror but job not hung, still continue to next records.

attached the logs below, in the logs we can see that the SQL query is truncated (not complete). I dont know is this BQ limitation in the audit logs or it is in the sql lineage extractor.

exec-urn_li_dataHubExecutionRequest_f20bd39a-dccc-448b-a268-fb9bfe58b5c1.log

[treff7es]

@ananbas can you try using bigquery-beta instead of bigquery source? It contains a bunch of bug fixes and hopefully there you won't see this issue.

[github-actions[bot]]

This issue is stale because it has been open for 30 days with no activity. If you believe this is still an issue on the latest DataHub release please leave a comment with the version that you tested it with. If this is a question/discussion please head to https://slack.datahubproject.io. For feature requests please use https://feature-requests.datahubproject.io

[laulpogan]

Hi @ananbas this seems like a troubleshooting issue, rather than a bug. We're happy to provide community support on our Slack channel, but currently reserve git issues for bugs.

If you're still having trouble, please join us at slack.datahubproject.io and we can troubleshoot there. For now, I'm going to close this issue.