bigmetadata
bigmetadata copied to clipboard
Redo/Refactor/Fix metadata logic
Related #294
During the related task I've found some problems with the metadata tasks:
- Some tasks like
ColumnTask
is not idempotent and does operations that are not recommended by Luigi like:- Have logic in the
complete
method - Call and get the
output
objects in therun
method - Call
complete
fromoutput
method - They're not idempotent because they don't have a clear output
- Have logic in the
- Also I've found that the ORM model is more complex than needed so we probably have to check if we can simplify it
- We have more tasks without an idempotent
output
so we have to fix that too
Meeting summary and conclusions
Yesterday we had a meeting to discuss what we should do with this ticket and if we block it or continue working on it.
The conclusion was:
- Block the ticket and stop spending time on it because we have to fix first some other issues like the metadata one (that is covered in this ticket)
- Create and be aware of this ticket but don't start working on it immediately because the problem is not a blocking one.
As a specific conclusion of "be aware of this ticket", we agreed on considering creating a simpler task class hierarchy when we tackle the "declarative ETL" task in the future.