Replace environment variable with a project flag to gate micorbatch functionality

[MichelleArk]

Resolves #10798

Problem

We had gated the new microbatch feature behind an environment variable in the initial implementation of microbatch (https://github.com/dbt-labs/dbt-core/pull/10594). However, for a better experience, we want people to be setting a project flag (AKA behavior flag).

Solution

Gate microbatch functionality behind a project flag (AKA behavior flag) instead of an environment variable.

Checklist

• [x] I have read the contributing guide and understand what's expected of me.
• [x] I have run this code in development, and it appears to resolve the stated issue.
• [x] This PR includes tests, or tests are not required or relevant for this PR.
• [x] This PR has no interface changes (e.g., macros, CLI, logs, JSON artifacts, config files, adapter interface, etc.) or this PR has already received feedback and approval from Product or DX.
• [x] This PR includes type annotations for new and modified functions.

[github-actions[bot]]

Thank you for your pull request! We could not find a changelog entry for this change. For details on how to document a change, see the contributing guide.

[codecov[bot]]

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 89.07%. Comparing base (30b8a92) to head (0f6b7fb). Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #10799      +/-   ##
==========================================
- Coverage   89.12%   89.07%   -0.06%     
==========================================
  Files         183      183              
  Lines       23592    23626      +34     
==========================================
+ Hits        21027    21044      +17     
- Misses       2565     2582      +17     

Flag	Coverage Δ
integration	86.37% <100.00%> (-0.14%)	:arrow_down:
unit	62.79% <52.94%> (-0.03%)	:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
Unit Tests	62.79% <52.94%> (-0.03%)	:arrow_down:
Integration Tests	86.37% <100.00%> (-0.14%)	:arrow_down:

[QMalcolm]

Related PRs

• https://github.com/dbt-labs/dbt-adapters/pull/323
• https://github.com/dbt-labs/dbt-bigquery/pull/1358

[mirnawong1]

hey @QMalcolm @MichelleArk , the docs team received this github issue.

i was reviewing this dbt-adapters PR and https://github.com/dbt-labs/dbt-core/pull/10799 -- am i correct in thinking that the following needs to be added to the behavior flag doc:

the require_batched_execution_for_custom_microbatch_strategy flag defaults to False, which will raise a deprecation warning. Set it to True to allow custom microbatch strategies (like get_incremental_microbatch_sql) to use batched execution.

[QMalcolm]

hey @QMalcolm @MichelleArk , the docs team received this github issue.

i was reviewing this dbt-adapters PR and #10799 -- am i correct in thinking that the following needs to be added to the behavior flag doc:

the require_batched_execution_for_custom_microbatch_strategy flag defaults to False, which will raise a deprecation warning. Set it to True to allow custom microbatch strategies (like get_incremental_microbatch_sql) to use batched execution.

@mirnawong1 It's actually a little bit different than that. The flag require_batched_execution_for_custom_microbatch_strategy does default to False. However, a user only needs to set require_batched_execution_for_custom_microbatch_strategy to True if they have a custom microbatch macro. If they don't have a custom microbatch macro, everything will work. Additionally, they'll only get a depreciation warning IFF they have a custom microbatch strategy and the flag is False.

[mirnawong1]

hey @QMalcolm oh ok i see! so require_batched_execution_for_custom_microbatch_strategy is for custom microbatch macros only.

so does this also mean we have another flag: require_builtin_microbatch_strategy (pr here), which replaces the DBT_EXPERIMENTAL_MICROBATCH env var? and they should set to true if they had the env_var already enabled?

[QMalcolm]

@mirnawong1 There is no other flag. I recognize that as slightly confusing. For context, the only reason we gated microbatch in the first place was the fear of breaking people's projects who already had a custom microbatch macro. What we've done here is make it so one only needs to set a flag if they have a custom microbatch macro, instead of making everyone set a flag because someone else's project might have a microbatch macro.

For a person setting up a microbatch model for the first time, there is no longer a flag or environment variable they need to set (assuming they don't have a custom microbatch macro)

[mirnawong1]

oh ok got it! so i can update the callout here and replace it with content informing users of the behavior flag should they have a custom microbatch macro only. all other users that are setting up incremental microbatch strategy for the first time w no existing custom macro don't need to do anything apart from their config.

thank you so much for clarifying this!