dbt_artifacts icon indicating copy to clipboard operation
dbt_artifacts copied to clipboard
verified publisher icon brooklyn-data

chore: add Redshift support

Open hanslemm opened this issue 1 year ago • 12 comments

Overview

This is an update of @brendan-cook-87 PR for Redshift support.

Just changed the data type to SUPER for ARRAY and JSON.

Also merged the latest updates of the package.

I've tested it using Redshift and works fine.

Anyone wanting to test this can just add the package to your dbt project while this is not merged via:

# packages.yml
packages:
  - git: https://github.com/hanslemm/dbt_artifacts.git
    revision: REDSHIFT_SUPPORT

Update type - breaking / non-breaking

  • [ ] Minor bug fix
  • [ ] Documentation improvements
  • [ ] Quality of Life improvements
  • [x] New features (non-breaking change)
  • [ ] New features (breaking change)
  • [ ] Other (non-breaking change)
  • [ ] Other (breaking change)
  • [ ] Release preparation

What does this solve?

Outstanding questions

What databases have you tested with?

  • [X] Redshift
  • [ ] Snowflake
  • [ ] Google BigQuery
  • [ ] Databricks
  • [ ] Spark
  • [ ] N/A

hanslemm avatar Oct 21 '24 11:10 hanslemm

Hi @llifoawing are you able to review this?

shiv-io avatar Aug 12 '25 20:08 shiv-io

Hi, I faced similar issue https://github.com/brooklyn-data/dbt_artifacts/discussions/524 when using dbt artifacts with redshift. I suspect it can be resolved once the endmacro block is added. Hope to hear back soon. Thank you! @shiv-io @llifoawing

yeexuan26 avatar Sep 16 '25 07:09 yeexuan26

I'm still getting errors:

07:01:49  Failure in model model_executions (models/sources/model_executions.sql)
07:01:49    Database Error in model model_executions (models/sources/model_executions.sql)
  column "adapter_response" is of type character varying but expression is of type super
  compiled code at target/run/dbt_artifacts/models/sources/model_executions.sql


      insert into "personal_jorge_machado_extern"."public"."tests" ("run_started_at", "command_invocation_id", "node_id", "name", "depends_on_nodes", "package_name", "test_path", "tags", "all_results")
    (
        select "run_started_at", "command_invocation_id", "node_id", "name", "depends_on_nodes", "package_name", "test_path", "tags", "all_results"
        from "tests__dbt_tmp065924265201"
    )

jomach avatar Oct 29 '25 07:10 jomach

It seems the error is related to a test run. Over the last year using my package I realized that at least one test out of the standard dbt column tests doesn't work with it (I don't remember now which one, I think it is related with column name test). Can you check which test is the culprit via isolating each one and running the model? I can try to implement a fix afterwards.

hanslemm avatar Oct 29 '25 08:10 hanslemm

So I deleted all tables and runned again. Now it does not fail but the tables are empty :D I also tried another run with dbt build. Also empty tables..

 dbt run --select dbt_artifacts --no-version-check
07:29:27  Running with dbt=1.10.10
07:29:28  Registered adapter: redshift=1.9.5
07:29:28  Found 110 models, 1 snapshot, 6 seeds, 396 data tests, 14 sources, 1043 macros, 19 unit tests
07:29:28  
07:29:28  Concurrency: 6 threads (target='dev')
07:29:28  
07:29:30  1 of 34 START sql incremental model public.exposures ........................... [RUN]
07:29:30  2 of 34 START sql incremental model public.invocations ......................... [RUN]
07:29:30  3 of 34 START sql incremental model public.model_executions .................... [RUN]
07:29:30  4 of 34 START sql incremental model public.models .............................. [RUN]
07:29:30  5 of 34 START sql incremental model public.seed_executions ..................... [RUN]
07:29:30  6 of 34 START sql incremental model public.seeds ............................... [RUN]
07:29:37  6 of 34 OK created sql incremental model public.seeds .......................... [SUCCESS in 7.22s]
07:29:37  7 of 34 START sql incremental model public.snapshot_executions ................. [RUN]
07:29:37  1 of 34 OK created sql incremental model public.exposures ...................... [SUCCESS in 7.47s]
07:29:37  8 of 34 START sql incremental model public.snapshots ........................... [RUN]
07:29:38  4 of 34 OK created sql incremental model public.models ......................... [SUCCESS in 7.67s]
07:29:38  9 of 34 START sql incremental model public.sources ............................. [RUN]
07:29:38  5 of 34 OK created sql incremental model public.seed_executions ................ [SUCCESS in 7.92s]
07:29:38  3 of 34 OK created sql incremental model public.model_executions ............... [SUCCESS in 7.93s]
07:29:38  2 of 34 OK created sql incremental model public.invocations .................... [SUCCESS in 7.93s]
07:29:38  10 of 34 START sql incremental model public.test_executions .................... [RUN]
07:29:38  11 of 34 START sql incremental model public.tests .............................. [RUN]
07:29:38  12 of 34 START sql view model public.stg_dbt__seeds ............................ [RUN]
07:29:39  12 of 34 OK created sql view model public.stg_dbt__seeds ....................... [SUCCESS in 1.64s]
07:29:39  13 of 34 START sql view model public.stg_dbt__exposures ........................ [RUN]
07:29:41  13 of 34 OK created sql view model public.stg_dbt__exposures ................... [SUCCESS in 1.45s]
07:29:41  14 of 34 START sql view model public.stg_dbt__models ........................... [RUN]
07:29:41  7 of 34 OK created sql incremental model public.snapshot_executions ............ [SUCCESS in 3.99s]
07:29:41  15 of 34 START sql view model public.stg_dbt__seed_executions .................. [RUN]
07:29:43  14 of 34 OK created sql view model public.stg_dbt__models ...................... [SUCCESS in 2.00s]
07:29:43  15 of 34 OK created sql view model public.stg_dbt__seed_executions ............. [SUCCESS in 1.80s]
07:29:43  16 of 34 START sql view model public.stg_dbt__model_executions ................. [RUN]
07:29:43  17 of 34 START sql view model public.stg_dbt__invocations ...................... [RUN]
07:29:45  17 of 34 OK created sql view model public.stg_dbt__invocations ................. [SUCCESS in 2.46s]
07:29:45  18 of 34 START sql view model public.dim_dbt__seeds ............................ [RUN]
07:29:45  16 of 34 OK created sql view model public.stg_dbt__model_executions ............ [SUCCESS in 2.47s]
07:29:45  19 of 34 START sql view model public.dim_dbt__exposures ........................ [RUN]
07:29:46  9 of 34 OK created sql incremental model public.sources ........................ [SUCCESS in 7.95s]
07:29:46  20 of 34 START sql view model public.stg_dbt__snapshot_executions .............. [RUN]
07:29:46  8 of 34 OK created sql incremental model public.snapshots ...................... [SUCCESS in 8.44s]
07:29:46  21 of 34 START sql view model public.dim_dbt__models ........................... [RUN]
07:29:46  11 of 34 OK created sql incremental model public.tests ......................... [SUCCESS in 8.14s]
07:29:46  22 of 34 START sql view model public.fct_dbt__seed_executions .................. [RUN]
07:29:46  10 of 34 OK created sql incremental model public.test_executions ............... [SUCCESS in 8.54s]
07:29:46  23 of 34 START sql view model public.fct_dbt__invocations ...................... [RUN]
07:29:48  19 of 34 OK created sql view model public.dim_dbt__exposures ................... [SUCCESS in 2.35s]
07:29:48  24 of 34 START sql view model public.dim_dbt__current_models ................... [RUN]
07:29:48  21 of 34 OK created sql view model public.dim_dbt__models ...................... [SUCCESS in 2.05s]
07:29:48  18 of 34 OK created sql view model public.dim_dbt__seeds ....................... [SUCCESS in 2.45s]
07:29:48  20 of 34 OK created sql view model public.stg_dbt__snapshot_executions ......... [SUCCESS in 2.33s]
07:29:48  27 of 34 START sql view model public.stg_dbt__snapshots ........................ [RUN]
07:29:48  26 of 34 START sql view model public.stg_dbt__sources .......................... [RUN]
07:29:48  25 of 34 START sql view model public.fct_dbt__model_executions ................. [RUN]
07:29:49  22 of 34 OK created sql view model public.fct_dbt__seed_executions ............. [SUCCESS in 2.67s]
07:29:49  23 of 34 OK created sql view model public.fct_dbt__invocations ................. [SUCCESS in 2.27s]
07:29:49  28 of 34 START sql view model public.stg_dbt__tests ............................ [RUN]
07:29:49  29 of 34 START sql view model public.stg_dbt__test_executions .................. [RUN]
07:29:51  25 of 34 OK created sql view model public.fct_dbt__model_executions ............ [SUCCESS in 2.90s]
07:29:51  27 of 34 OK created sql view model public.stg_dbt__snapshots ................... [SUCCESS in 2.90s]
07:29:51  26 of 34 OK created sql view model public.stg_dbt__sources ..................... [SUCCESS in 2.90s]
07:29:51  30 of 34 START sql view model public.fct_dbt__snapshot_executions .............. [RUN]
07:29:51  24 of 34 OK created sql view model public.dim_dbt__current_models .............. [SUCCESS in 3.00s]
07:29:51  31 of 34 START sql view model public.dim_dbt__snapshots ........................ [RUN]
07:29:51  32 of 34 START sql view model public.dim_dbt__sources .......................... [RUN]
07:29:52  29 of 34 OK created sql view model public.stg_dbt__test_executions ............. [SUCCESS in 3.51s]
07:29:52  33 of 34 START sql view model public.fct_dbt__test_executions .................. [RUN]
07:29:52  28 of 34 OK created sql view model public.stg_dbt__tests ....................... [SUCCESS in 3.51s]
07:29:52  34 of 34 START sql view model public.dim_dbt__tests ............................ [RUN]
07:29:53  31 of 34 OK created sql view model public.dim_dbt__snapshots ................... [SUCCESS in 2.37s]
07:29:53  32 of 34 OK created sql view model public.dim_dbt__sources ..................... [SUCCESS in 2.37s]
07:29:53  30 of 34 OK created sql view model public.fct_dbt__snapshot_executions ......... [SUCCESS in 2.37s]
07:29:54  33 of 34 OK created sql view model public.fct_dbt__test_executions ............. [SUCCESS in 1.80s]
07:29:54  34 of 34 OK created sql view model public.dim_dbt__tests ....................... [SUCCESS in 1.80s]
07:29:54  
07:29:54  Finished running 11 incremental models, 23 view models in 0 hours 0 minutes and 25.99 seconds (25.99s).
07:29:55  
07:29:55  Completed successfully
07:29:55  
07:29:55  Done. PASS=34 WARN=0 ERROR=0 SKIP=0 NO-OP=0 TOTAL=34

jomach avatar Oct 31 '25 07:10 jomach

ok, After adding the upload hook everything works fine.

jomach avatar Oct 31 '25 09:10 jomach

Is it planned to get this merged ? We wanted to use this in production...

jomach avatar Nov 06 '25 09:11 jomach

Over a year and none of the main branch maintainers left a comment @jomach, so I don't think we will have it in the official package sadly.

hanslemm avatar Nov 06 '25 16:11 hanslemm

Can we ping someone ? The builds are failing

jomach avatar Nov 06 '25 16:11 jomach

maybe @jared-rimmer can take a look?

hanslemm avatar Nov 06 '25 16:11 hanslemm

Over a year and none of the main branch maintainers left a comment @jomach, so I don't think we will have it in the official package sadly.

Yeah. They've never been responsive at all lol. I gave up with the same issues of completely unrelated builds failing...

brendan-cook-87 avatar Nov 06 '25 20:11 brendan-cook-87

I'm switching to elementary

jomach avatar Nov 06 '25 21:11 jomach