[Feature][Framework] Support downloading logs from Temporal workers

[keon94]

Search before asking

• [X] I had searched in the issues and found no similar feature requirement.

Description

With #2578 it is assumed that all the logs exist on the main devlake service. If we have Temporal running, then we are unable to get logs from the worker instances. We need to be able to fetch those logs as well, and put them all in the same archive.

Use case

Have the download logs API return the logs from all workers that participated in the pipeline execution. Possible solutions:

1. Have the main Devlake service track the workers involved, and upon their completion, query them for their logs. This becomes challenging if we try to build resiliency against a worker dying.

2. Have the workers periodically publish their logs back to the main service (use a message queue on the main service to listen for these events and append to the log files) - I prefer this one, and it's also safe against dying workers.

Related issues

#2578

Are you willing to submit a PR?

• [ ] Yes I am willing to submit a PR!

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.

[github-actions[bot]]

This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.

[github-actions[bot]]

This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future.

[github-actions[bot]]

This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.

[github-actions[bot]]

This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future.