sagerx icon indicating copy to clipboard operation
sagerx copied to clipboard

Rxnorm dbt update and intermediate tables

Open lprzychodzien opened this issue 2 months ago • 1 comments

Resolves #270

Explanation

Changes rxnorm staging and intermediate queries to using dbt's jinja table references.

Most important is that it sets intermediate models to materialize as tables (vs views).

Rationale

dbt mart models are made up of complex logic that should be captured in the intermediate models. The issues already that we ran into (issue #270 ) was that performing these aggregations for each query took 10+ minutes. Therefore, bringing this aggregation into a separated intermediate model to be materialized as a table is a good solution. The materialization of these intermediate tables can take awhile but will significantly speed up queries.

Additional work can be done to optimize the queries that build out intermediate models in the future.

Tests

  1. What testing did you do? dbt run --full-refresh
  2. Attach testing logs inside a summary block:
testing logs

lprzychodzien avatar Apr 28 '24 20:04 lprzychodzien