feat(db_engine_specs): big query cost estimation

[zamar-roura]

SUMMARY

We are adding four methods inside superset/db_engine_specs/bigquery.py

For now only Postgres and Presto had cost estimation capabilities. For that they used the cursor object. Bigquery needs to use a dry run to get the query cost estimation. Sadly, you can't use the cursor to execute dry runs (I have tried). And that's why there is a need to override the two functions. estimate_statement_cost and estimate_query_cost. The other two methods get_allow_cost_estimate and query_cost_formatter are just added because the base class doesn't have functionality or gives a False.

Also the QueryEditor.py object doesn't have sqlEditorId anymore in it's variables and the Cost Estimation Reducer is looking for it, thus giving error, that is also changed.

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

BEFORE

After

TESTING INSTRUCTIONS

Add ESTIMATE_QUERY_COST = True inside superset/config.py. Add a valid BigQuery database, activate option Enable query cost estimation inside advanced options. Go to SQL Lab and put a valid query, then press Estimate Cost button.

ADDITIONAL INFORMATION

• [ X] Has associated issue: https://github.com/apache/superset/issues/20832
• [ ] Required feature flags:
• [ ] Changes UI
• [ ] Includes DB Migration (follow approval process in SIP-59)
  • [ ] Migration is atomic, supports rollback & is backwards-compatible
  • [ ] Confirm DB migration upgrade and downgrade tested
  • [ ] Runtime estimates and downtime expectations provided
• [ X] Introduces new feature or API
• [ ] Removes existing feature or API

[zamar-roura]

Thanks for running the tests.

I'll check on the lint errors and try to see why the cypress test is showing the "cant create chart from query error. Maybe the estimate cost button is changing the selector of the other button? Lets see.

[codecov[bot]]

Codecov Report

Merging #21325 (ef5c3b1) into master (9cfbc22) will decrease coverage by 0.03%. The diff coverage is 18.75%.

@@            Coverage Diff             @@
##           master   #21325      +/-   ##
==========================================
- Coverage   67.03%   66.99%   -0.04%     
==========================================
  Files        1859     1859              
  Lines       71043    71090      +47     
  Branches     7776     7776              
==========================================
+ Hits        47622    47630       +8     
- Misses      21397    21436      +39     
  Partials     2024     2024              

Flag	Coverage Δ
hive	52.43% <18.75%> (-0.05%)	:arrow_down:
javascript	53.85% <ø> (ø)
mysql	77.96% <18.75%> (-0.09%)	:arrow_down:
postgres	78.03% <18.75%> (-0.09%)	:arrow_down:
presto	52.33% <18.75%> (-0.05%)	:arrow_down:
python	81.29% <18.75%> (-0.09%)	:arrow_down:
sqlite	76.41% <18.75%> (-0.09%)	:arrow_down:
unit	51.43% <18.75%> (-0.05%)	:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
superset-frontend/src/SqlLab/reducers/sqlLab.js	36.87% <ø> (ø)
.../CRUD/data/database/DatabaseModal/ExtraOptions.tsx	70.37% <ø> (ø)
superset/config.py	91.64% <ø> (-0.03%)	:arrow_down:
superset/db_engine_specs/bigquery.py	69.60% <18.75%> (-12.09%)	:arrow_down:

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

[zamar-roura]

@villebro is it okay like this? The cypress E2E tests sometimes work sometimes don't, but everything else is fixed and working.

[zamar-roura]

Do I need to do something else to merge this?

[fernandofernandezgonzalez]

Wowwww!!! This is a so, so much needed feature!!

[villebro]

@zamar-roura would you be able to rebase the PR to resolve the conflicts?

[zamar-roura]

@villebro Tell me if you need anything else. I'll be

The reducer/SqlLab.js needed some fixes in the ESTIMATE QUERY action as action.query didn't have any sqlEditorId parameter. It always comes as action.query.id not action.query.sqlEditorId.

[villebro]

Thanks @zamar-roura - I'll do a review + test pass tomorrow 👍

[zamar-roura]

Hi @villebro, I'm at your full disposal to improve the merge as much as needed, it's my first merge in superset, if anything is not as it should be tell me and I'll put my full effort into it.

[villebro]

@zamar-roura I haven't forgotten about this, I've just been insanely busy. But I'll do my absolute best to review this today!

[zamar-roura]

@villebro I totally understand, thank you so much for the quick reply!

[villebro]

Restarted CI, the cypress workflow seems to be having issues

[zamar-roura]

Hi @villebro. I don't remember if there was anything else we needed to do. Its a busy end of the year and also full of festivities so if you want we can see what´s needed after all this ends, cheers!

[zamar-roura]

@villebro I'll be checking from time to time to resolve merge conflicts