bigmetadata icon indicating copy to clipboard operation
bigmetadata copied to clipboard

Redo/Refactor/Fix metadata logic

Open ethervoid opened this issue 7 years ago • 2 comments

Related #294

During the related task I've found some problems with the metadata tasks:

  • Some tasks like ColumnTask is not idempotent and does operations that are not recommended by Luigi like:
    • Have logic in the complete method
    • Call and get the output objects in the run method
    • Call complete from output method
    • They're not idempotent because they don't have a clear output
  • Also I've found that the ORM model is more complex than needed so we probably have to check if we can simplify it
  • We have more tasks without an idempotent output so we have to fix that too

ethervoid avatar Oct 03 '17 12:10 ethervoid

Meeting summary and conclusions

Yesterday we had a meeting to discuss what we should do with this ticket and if we block it or continue working on it.

The conclusion was:

  • Block the ticket and stop spending time on it because we have to fix first some other issues like the metadata one (that is covered in this ticket)
  • Create and be aware of this ticket but don't start working on it immediately because the problem is not a blocking one.

ethervoid avatar Oct 05 '17 08:10 ethervoid

As a specific conclusion of "be aware of this ticket", we agreed on considering creating a simpler task class hierarchy when we tackle the "declarative ETL" task in the future.

juanignaciosl avatar Oct 05 '17 10:10 juanignaciosl