bubbles
bubbles copied to clipboard
Stiivi
Darn
This seemed to be the most complete Python ETL package. Why don't Python nerds need to ETL?
Why don't Python nerds need to ETL?
Maybe they have custom solutions, like me. The most important thing is parallelism, jobs scheduling and execution, a possibility to track issues (inspecting a staging state step-by-step), scheduler state inspection and management, and so on. Abstraction layer for transforms/calculations aren't so important, because they can be done in raw sql. For everyting else, i.e matrix operations, a numpy/scipy packages are just enough. Just my 5cents.
@marcinn what do you use for the more important points that you mentioned? Something like Luigi?
I didn't know about Luigi when I was starting work with ETLs. I created something similar to this, but way limited - without dependency handling, parallelism only within a run() method, just sequence of tasks per job, simple jobs scheduler and runner (2in1, just 1 runner at once).
My ETL was a simple Python script written in one module, at the beginning. It was changing slowly into something bigger, and that's how my custom library and scheduler was created.
Luigi looks more powerful and generic. I think I'll start using it, because maintaining a custom solution isn't cheap and requires a lot of time. Thank you for pointing Luigi here.