Question about Luigi log history

[danielhanbitlee]

Hi Katharine,

I want to thank you for the data-pipeline course on Safari O'Reilly as your tutorials have been helpful.

I have a question about Luigi log history function. I followed what you did in the luigi.cfg file:

[worker]
keep_alive=True
task_limit=10

[scheduler]
retry_count=4
record_task_history=True

[task_history]
db_connection=sqlite:///tasks.db

However, when I go to localhost:8082/history, it only saves the most recent history. Is there a way to save all the histories from the previous attempts? For example, let's say that I run the pipeline 10 times. Is there a way to save all the history from the 10 pipeline executions?

Any help would be greatly appreciated.

Best, Daniel

[kjam]

Hi Daniel,

It seems from the Luigi documentation that in the /history only the last 24 hours are shown. Are there more entries if you connect directly to the tasks.db?

In addition, I found a thread regarding "pruning" of these tasks here: https://groups.google.com/forum/#!topic/luigi-user/XU6Fsl57gGk which might be worth a read.

Hope that helps and thanks for you patience with my delay in responding :smile:

Cheers, katharine