Benson Tucker

Add explicit variable name scope for output layer

1

Fixes https://github.com/tomlepaine/fast-wavenet/issues/5 Also addressed style issues that my linter (pyflakes) complained about

Support schema path templates in database source relations

1

# Summary Extract for database targets doesn't support the power config rendering that's available for static sources. In `extract`, the output target directory comes from `relation.data_directory`, whereas for static sources...

Getting-started guides?

4

# Summary I'm trying to set up a fresh project and wonder if there are any templates for the 'sibling' repo. (I have the fortunate position of vaguely remembering how...

Docker setup fails with permissions error

7

# Summary After a fresh install, attempted to run arthur: ``` ./bin/run_arthur.sh ++ pwd + docker run --rm --interactive --tty {volumes omitted} --env DATA_WAREHOUSE_CONFIG=/opt/data-warehouse/warehouse_config --env ARTHUR_DEFAULT_PREFIX=bhtucker arthur-redshift-etl:latest + cd /opt/src/arthur-redshift-etl...

TableNames and Schemas share 'position' abstraction too loosely

1

# Summary We have a position abstraction in logic but not in our objects. # Details Managed schemas can be promoted from one 'position' to another (ie, staging -> standard,...

Concurrent load is too vulnerable to slow extracts

1

If extracts stop for an hour, a concurrent load will exit with a `ETLRuntimeError`, regardless of whether the relations it is waiting on extracts for are required.

Extract parallelism might exceed capacity and fail

1

Extracting with Sqoop, code must be generated and compiled each time an `extract_table` attempt is made. With a thread per database source calling Sqoop, you may not have enough memory...

In a concurrent load, extracts may be observed to fail or may stall for long enough that the load process 'times out' waiting for them. When this happens, the last-loaded...