action-sqlfluff

action-sqlfluff copied to clipboard

v1.9.0: dbt-core v1.9.0

Compare Source

dbt-core 1.9.0 - December 09, 2024

Breaking Changes

• Fix changing the current working directory when using dpt deps, clean and init. (#​8997)

Features

• Parseable JSON and text output in quiet mode for dbt show and dbt compile (#​9840)
• serialize inferred primary key (#​9824)
• Add unit_test: selection method (#​10053)
• Maximally parallelize dbt clone in clone command" (#​7914)
• Add --host flag to dbt docs serve, defaulting to '127.0.0.1' (#​10229)
• Update data_test to accept arbitrary config options (#​10197)
• add pre_model and post_model hook calls to data and unit tests to be able to provide extra config options (#​10198)
• add --empty value to jinja context as flags.EMPTY (#​10317)
• Warning message for snapshot timestamp data types (#​10234)
• Support cumulative_type_params & sub-daily granularities in semantic manifest. (#​10360)
• Add time_granularity to metric spec. (#​10376)
• Support standard schema/database fields for snapshots (#​10301)
• Support ref and source in foreign key constraint expressions, bump dbt-common minimum to 1.6 (#​8062)
• Support new semantic layer time spine configs to enable sub-daily granularity. (#​10475)
• Add order_by and limit fields to saved queries. (#​10531)
• Add support for behavior flags (#​10618)
• Enable --resource-type and --exclude-resource-type CLI flags and environment variables for dbt test (#​10656)
• Allow configuring snapshot column names (#​10185)
• Add custom_granularities to YAML spec for time spines. (#​9265)
• Add basic functionality for creating microbatch incremental models (#​9490, #​10635, #​10637, #​10638, #​10636, #​10662, #​10639)
• Execute microbatch models in batches (#​10700)
• Create 'skip_nodes_if_on_run_start_fails' behavior change flag (#​7387)
• Allow snapshots to be defined in YAML. (#​10246)
• Write microbatch compiled/run targets to separate files, one per batch (#​10714)
• Track incremental_strategy as part of model_run tracking event (#​10761)
• Support required 'begin' config for microbatch models (#​10701)
• Parse-time validation of microbatch configs: require event_time, batch_size, lookback and validate input event_time (#​10709)
• Added the --inline-direct parameter to 'dbt show' (#​10770)
• Enable specification of dbt_valid_to for current records (#​10187)
• Enable retry support for microbatch models (#​10715, #​10729)
• Use unrendered database and schema source properties during state:modified, behind state_modified_compare_more_unrendered_values behavoiur flag (#​9573)
• Ensure microbatch models respect full_refresh model config (#​10785)
• Adds validations for custom_granularities to ensure unique naming. (#​9265)
• Enable use of multi-column unique key in snapshots (#​9992)
• Change gating of microbatch feature to be behind project flag / behavior flag (#​10798)
• Ensure --event-time-start is before --event-time-end (#​10786)
• Ensure microbatch models use same current_time value (#​10819)
• Emit warning when microbatch model has no input with event_time config (#​10926)
• Emit debug logging event whenever artifacts are written (#​10937)
• Support --empty for snapshots (#​10372)
• Add new hard_deletes="new_record" mode for snapshots. (#​10235)
• Allow microbatch batches to run in parallel (#​10853, #​10855)
• Add batch context object to model jinja context (#​11025)
• Ensure pre/post hooks only run on first/last batch respectively for microbatch model batches (#​11094, #​11104)

Fixes

• Remove unused check_new method (#​7586)
• Test case for merge_exclude_columns (#​8267)
• Convert "Skipping model due to fail_fast" message to DEBUG level (#​8774)
• Restore previous behavior for --favor-state: only favor defer_relation if not selected in current command" (#​10107)
• Unit test fixture (csv) returns null for empty value (#​9881)
• Fix json format log and --quiet for ls and jinja print by converting print call to fire events (#​8756)
• Add resource type to saved_query (#​10168)
• Fix: Order-insensitive unit test equality assertion for expected/actual with multiple nulls (#​10167)
• Renaming or removing a contracted model should raise a BreakingChange warning/error (#​10116)
• prefer disabled project nodes to external node (#​10224)
• Fix issues with selectors and inline nodes (#​8943, #​9269)
• Fix snapshot config to work in yaml files (#​4000)
• Improve handling of error when loading schema file list (#​10284)
• Use model alias for the CTE identifier generated during ephemeral materialization (#​5273)
• Implement state:modified for saved queries (#​10294)
• Saved Query node fail during skip (#​10029)
• DOn't warn on unit_test config paths that are properly used (#​10311)
• Fix setting silence of warn_error_options via dbt_project.yaml flags (#​10160)
• Attempt to provide test fixture tables with all values to set types correctly for comparisong with source tables (#​10365)
• Limit data_tests deprecation to root_project (#​9835)
• CLI flags should take precedence over env var flags (#​10304)
• Fix typing for artifact schemas (#​10442)
• Fix over deletion of generated_metrics in partial parsing (#​10450)
• Fix error constructing warn_error_options (#​10452)
• Do not update varchar column definitions if a contract exists (#​10362)
• fix all_constraints access, disabled node parsing of non-uniquely named resources (#​10509)
• respect --quiet and --warn-error-options for flag deprecations (#​10105)
• Propagate measure label when using create_metrics (#​10536)
• Fix state:modified check for exports (#​10138)
• Filter out empty nodes after graph selection to support consistent selection of nodes that depend on upstream public models (#​8987)
• Late render pre- and post-hooks configs in properties / schema YAML files (#​10603)
• Allow the use of env_var function in certain macros in which it was previously unavailable. (#​10609)
• Remove deprecation for tests: to data_tests: change (#​10564)
• Fix --resource-type test for dbt list and dbt build (#​10730)
• Fix unit tests for incremental model with alias (#​10754)
• Allow singular tests to be documented in properties.yml (#​9005)
• Ignore --empty in unit test ref/source rendering (#​10516)
• Ignore rendered jinja in configs for state:modified, behind state_modified_compare_more_unrendered_values behaviour flag (#​9564)
• Improve performance of infer primary key (#​10781)
• Pass test user config to adapter pre_hook by explicitly adding test builder config to node (#​10484)
• Attempt to skip saved query processing when no semantic manifest changes (#​10563)
• Ensure dbt retry of microbatch models doesn't lose prior successful state (#​10800)
• override materialization python models (#​8520)
• Handle edge cases when a specified --event-time-end is equivalent to the batch size truncated batch start time (#​10824)
• Begin tracking execution time of microbatch model batches (#​10825)
• Support disabling unit tests via config. (#​9109, #​10540)
• Allow instances of generic data tests to be documented (#​2578)
• Fix warnings for models referring to a deprecated model (#​10833)
• Change lookback default from 0 to 1 to ensure better data completeness (#​10867)
• Make --event-time-start and --event-time-end mutually required (#​10874)
• Ensure KeyboardInterrupt/SystemExit halts microbatch model execution (#​10862)
• Exclude hook result from results in on-run-end context (#​7387)
• unit tests with versioned refs (#​10880, #​10528, #​10623)
• Implement partial parsing for all-yaml snapshots (#​10903)
• Restore source quoting behaviour when quoting config provided in dbt_project.yml (#​10892)
• Fix bug when referencing deprecated models (#​10915)
• Fix 'model' jinja context variable type to dict (#​10927)
• Take end_time for batches to the ceiling to handle edge case where event_time column is a date (#​10868)
• Handle exceptions in get_execution_status more broadly to better ensure run_results.json gets written (#​10934)
• Fix 'no attribute .config' error when ref-ing a microbatch model from non-Model context (#​10928)
• Ensure inferred primary_key is a List[str] with no null values (#​10983)
• Correct when custom microbatch macro deprecation warning is fired (#​10994)
• Validate manifest has group_map during group_lookup init (#​10988)
• Fix plural of 'partial success' in log message (#​10999)
• Emit batch-level exception with node_info on microbatch batch run failure (#​10840)
• Fix restrict-access to not apply within a package (#​10134)
• Make microbatch models skippable (#​11021)
• Catch DbtRuntimeError for hooks (#​11012)
• Access DBUG flag more consistently with the rest of the codebase in ManifestLoader (#​11068)
• Implement partial parsing for singular data test configs in yaml files (#​10801)

Docs

• Enable display of unit tests (dbt-docs/#​501)
• Unit tests not rendering (dbt-docs/#​506)
• Add support for Saved Query node (dbt-docs/#​486)
• Fix npm security vulnerabilities as of June 2024 (dbt-docs/#​513)

Under the Hood

• Clear error message for Private package in dbt-core (#​10083)
• Enable use of context in serialization (#​10093)
• Make RSS high water mark measurement more accurate on Linux (#​10177)
• Enable record filtering by type. (#​10240)
• Remove IntermediateSnapshotNode (#​10326)
• Additional logging for skipped ephemeral models (#​10389)
• bump black to 24.3.0 (#​10454)
• generate protos with protoc version 5.26.1 (#​10457)
• Move from minimal-snowplow-tracker fork back to snowplow-tracker (#​8409)
• Add group info to RunResultError, RunResultFailure, RunResultWarning log lines (#)
• Improve speed of tree traversal when finding children, increasing build speed for some selectors (#​10434)
• Add test for sources tables with quotes (#​10582)
• Additional type hints for core/dbt/version.py (#​10612)
• Fix typing issues in core/dbt/contracts/sql.py (#​10614)
• Fix type errors in dbt/core/task/clean.py (#​10616)
• Add Snowplow tracking for behavior flag deprecations (#​10552)
• Add test utility patch_microbatch_end_time for adapters testing (#​10713)
• Replace TestSelector with ResourceTypeSelector (#​10718)
• Standardize returning ResourceTypeSelector instances in dbt list and dbt build (#​10739)
• Add group metadata info to LogModelResult and LogTestResult (#​10775)
• Remove support and testing for Python 3.8, which is now EOL. (#​10861)
• Behavior change for mf timespine without yaml configuration (#​10959)
• Behavior change for cumulative metric type param (#​10960)
• Upgrade protobuf (#​10658)

Dependencies

• Remove logbook dependency (#​8027)
• Increase supported version range for dbt-semantic-interfaces. Needed to support custom calendar features. (#​9265)
• Bump minimnum allowed dbt-adapters version to 1.8.0 (#N/A)
• Bump minimum dbt-adapters version to 1.9.0 (#​10996)

Security

• Explicitly bind to localhost in docs serve (#​10209)

Contributors

• @​DevonFulcher (#​10959, #​10960)
• @​McKnight-42 (#​10197, #​10198)
• @​Threynaud (#​11068)
• @​TowardOliver (#​10656)
• @​aliceliu (#​10536, #​10138)
• @​courtneyholcomb (#​10360, #​10376, #​10475, #​9265, #​9265, #​9265)
• @​danlsn (#​10915)
• @​dave-connors-3 (#​9824)
• @​devmessias (#​8520, #​10880, #​10528, #​10623)
• @​jeancochrane (#​5273)
• @​katsugeneration (#​10754)
• @​kevinneville (#​7586)
• @​nakamichiworks (#​10442)
• @​plypaul (#​10531)
• @​rariyama (#​8997)
• @​scottgigante,nevdelap (#​8774)
• @​tsturge (#​9109, #​10540)
• @​ttusing (#​10434)