[Bug] Runtime Error 'Lexer' object has no attribute '_SQL_REGEX'

[Tonayya]

Is this a new bug in dbt-redshift?

• [X] I believe this is a new bug in dbt-redshift
• [X] I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

dbt Cloud users experiencing an intermittent runtime error in random job runs which stems from the sqlparse library that dbt uses(link). But this appears to only be affecting Redshift connected projects. Issue was also reported on Discourse here.

Some notes from the internal correspondence with Core team:

• sqlparse is a library that we use to combine the SQL from ephemeral models with the non-ephemeral model SQL. It looks like this error would be an even more edge-casey failure than the one that we addressed in the referenced ticket.
• This is likely a variation on the original issue, which is that the sqlparse lexer isn't initialized when we use it to parse the SQL. They set the "default_instance" before it's initialized so another thread checking if the default instance is set in that tiny tiny window will see that it is, but it hasn't been initialized. In order to fix this we'd probably need to write an internal version of their initialization code and pin the code. Or vendor it.
• Looks like dbt-redshift directly uses sqlparse but just for splitting. It's the only adapter that does it though and it looks like it went in as of 1.5 so lines up. It also (on main) no longer does probably with dbt-adapter being pulled out (link).

Expected Behavior

Job runs not failing with the aforementioned error.

Steps To Reproduce

There's no specific step to reproduce this as it happens on random intervals and so far the only consistent thing found in the behaviour has been the dbt-redshift adapter. Error has occurred on dbt version 1.5, 1.6 and 1.7.

Relevant log output

an error:
Runtime Error
  'Lexer' object has no attribute '_SQL_REGEX'

Environment

- OS:
- Python:
- core: Running with dbt=1.6.9
-  adapter: redshift=1.6.7

Additional Context

No response

[mikealfare]

Re-opening until this is deployed.

[mikealfare]

User confirmed this is resolved.

[Tonayya]

Hiya @mikealfare , user just got back to us saying this has occurred again today and asked if this was backported to 1.6 (which I can see in the PR that it was). Should this still be occurring?

[mikealfare]

Hiya @mikealfare , user just got back to us saying this has occurred again today and asked if this was backported to 1.6 (which I can see in the PR that it was). Should this still be occurring?

If it's still occurring in the latest version, than this is a different issue than what we saw in dbt-core. Can you confirm that the user is using dbt-redshift==1.6.7?

[Tonayya]

Hi @mikealfare confirming that they are running with the following:

02:06:34  Running with dbt=1.6.9
02:06:34  Registered adapter: redshift=1.6.7

It's the same error:

02:06:39  Encountered an error:
Runtime Error
  'Lexer' object has no attribute '_SQL_REGEX'

[mikealfare]

Thanks for confirming. We'll need to look into this more then as it appears to be an unknown issue.

[Tonayya]

Hiya @mikealfare hope you're well. Just wanted to check in on this one and see whether we have any idea what's happening? :/ user just reached out to follow-up on an ETA. I have asked them whether this has occurred again since they last reported this on 13th Feb. Thank you! :)

[martynydbt]

@Tonayya is there currently a work around? Sorry, just joined in here and getting up to speed.

[Tonayya]

@martynydbt thanks for getting back to me. So basically re-running the jobs usually does the trick but it is causing an issue intermittently and disrupting their flow. It has been impacting a few of our customers for quite some time now.

[Tonayya]

Hi @martynydbt hope you're well. Just checking in to see if we know anything further about this? Was there any info needed from this side at all?

[Tonayya]

Hi all, just wondering if any updates on this? :/

[stisti]

We also see this sometimes. Here's the log from today morning run in Jenkins:

[2024-03-15T07:04:03.810Z] + docker run --rm -u 1000 -v /tmp/workspace/_lake-etls-dbt-fast_cycle_master:/etl -w /etl -v /tmp/tmp.Si51YcFO73:/profiles -e DBT_PROFILES_DIR=/profiles -e DBT_TARGET_PATH -e DBT_LOG_PATH -e USER -e AWS_DEFAULT_REGION -e AWS_METADATA_SERVICE_NUM_ATTEMPTS xxxxx.dkr.ecr.eu-west-1.amazonaws.com/ds/dbt:1.7.8 source freshness --target prd --select +tag:fast_cycle
[2024-03-15T07:04:06.447Z] 07:04:06  Running with dbt=1.7.8
[2024-03-15T07:04:06.721Z] 07:04:06  Registered adapter: redshift=1.7.3
[2024-03-15T07:04:06.721Z] 07:04:06  Unable to do partial parsing because saved manifest not found. Starting full parse.
[2024-03-15T07:04:11.035Z] 07:04:10  Found 64 models, 11 seeds, 111 tests, 22 sources, 0 exposures, 0 metrics, 836 macros, 0 groups, 0 semantic models
[2024-03-15T07:04:11.035Z] 07:04:10  
[2024-03-15T07:04:14.441Z] 07:04:13  Encountered an error:
[2024-03-15T07:04:14.441Z] Runtime Error
[2024-03-15T07:04:14.441Z]   'Lexer' object has no attribute '_SQL_REGEX'
script returned exit code 2

[jliu0812]

Hello.

I just faced this issue.

[2024-04-12, 14:41:12 CDT] {{subprocess.py:93}} INFO - [0m19:41:10.666850 [error] [MainThread]: Encountered an error:
[2024-04-12, 14:41:12 CDT] {{subprocess.py:93}} INFO - Runtime Error
[2024-04-12, 14:41:12 CDT] {{subprocess.py:93}} INFO -   'Lexer' object has no attribute '_SQL_REGEX'

Version info:

[2024-04-12, 14:44:07 CDT] {{subprocess.py:93}} INFO - 19:44:07  Running with dbt=1.7.11
[2024-04-12, 14:44:08 CDT] {{subprocess.py:93}} INFO - 19:44:08  Registered adapter: redshift=1.7.4
[2024-04-12, 14:44:08 CDT] {{subprocess.py:93}} INFO - 19:44:08  Unable to do partial parsing because saved manifest not found. Starting full parse.
...

[jliu0812]

I reran the same process again, it works fine after that. Seems to be an intermittent issue.