deepdive
deepdive copied to clipboard
Schema support in DD8
We've always wanted to write intermediate DD output to different database schemas, so that we can run multiple extractors in the same database. Some advantages:
- never loose any intermediate DD output
- access shared inputs (in different schema), without first loading into the database when running the extractor
It turns out that using different schemas is very simple in DD8 and doesn't require any code change. Here are the necessary steps:
- Add a file
input/init.sh
with the following content:
deepdive sql "CREATE SCHEMA myschema"
- Now set the search path in an environment variable:
export PGOPTIONS="-c search_path=myschema,public"
- Now run any of the
deepdive ..
commands, eg.deepdive run
. Voila. The output is written into the new schema.
Thanks for the clear steps! I'm planning to introduce a higher level operation that integrates this PG schema support with the rest of the process timestamps etc. This will be super helpful for that direction.