deepdive icon indicating copy to clipboard operation
deepdive copied to clipboard

Schema support in DD8

Open raphaelhoffmann opened this issue 9 years ago • 1 comments

We've always wanted to write intermediate DD output to different database schemas, so that we can run multiple extractors in the same database. Some advantages:

  • never loose any intermediate DD output
  • access shared inputs (in different schema), without first loading into the database when running the extractor

It turns out that using different schemas is very simple in DD8 and doesn't require any code change. Here are the necessary steps:

  1. Add a file input/init.sh with the following content:
deepdive sql "CREATE SCHEMA myschema"
  1. Now set the search path in an environment variable:
export PGOPTIONS="-c search_path=myschema,public"
  1. Now run any of the deepdive .. commands, eg. deepdive run. Voila. The output is written into the new schema.

raphaelhoffmann avatar Feb 23 '16 23:02 raphaelhoffmann

Thanks for the clear steps! I'm planning to introduce a higher level operation that integrates this PG schema support with the rest of the process timestamps etc. This will be super helpful for that direction.

netj avatar Feb 24 '16 00:02 netj