dbt-data-reliability
dbt-data-reliability copied to clipboard
support query_max_size for insert_rows_method `chunk`
Hey guys,
To fix the issue raised in
https://github.com/elementary-data/dbt-data-reliability/issues/675
I added support for query_max_size to the insert_rows_method chunk
.
This addition is important to prevent errors like The query is too large. The maximum standard SQL query length is 1024.00K characters
(in BigQuery) when use the insert_rows_method chunk
Test
I intended to write a unit test for this change, but I encountered errors running the current test cases even after following the procedure outlined in https://github.com/elementary-data/dbt-data-reliability/blob/master/integration_tests/README.md
I faced conflicting dependencies errors when attempting to install the required packages.
ERROR: Cannot install elementary-data and urllib3==2.0.4 because these package versions have conflicting dependencies.
The conflict is caused by:
The user requested urllib3==2.0.4
dbt-core 1.7.9 depends on urllib3~=1.0
To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict
ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts
After I added elementary-data
and dbt-postgres
to the requirements.txt
, another error occurred.
requirements.txt
pytest
pytest-xdist
pytest-parametrization
pytest-html
filelock
# urllib3==2.0.4
elementary-data==0.14.1
dbt-postgres==1.7.9
error
ERROR elementary.clients.dbt.dbt_runner:dbt_runner.py:175 Encountered an error:
Compilation Error
dbt found 2 package(s) specified in packages.yml, but only 0 package(s) installed in dbt_packages. Run "dbt deps" to install package dependencies.
Therefore, instead of writing a unit test, I tested this PR by running dbt run
in our own environment, and it worked well.