[Bug]: Log Pipeline Does Not Finish Processing on Manual STOP (Missing Log Entries)

[dsanderbi]

Apache Hop version?

2.14

Java version?

17

Operating system

Windows

What happened?

When using Apache Hop's logging framework with a log pipeline that processes log entries from a main workflow, not all log information is preserved if the workflow is manually stopped, especially when the logging pipeline contains steps that take some time to process (e.g., Delay Rows or long-running actions).

This issue was first observed using a Workflow Executor step within a workflow, but it can also be reproduced using a simplified example, see: manual-abort.zip

Reproduction Steps:

• Use the manual-abort.zip
• Start dummy.hwf
• Manually abort after some exectuion time while it is still running.
• Compare output files (target folder)

---

1. The log pipeline does not finish processing all incoming log rows after a manual abort.
2. As a result, part of the logging output is missing, depending on the processing time of the log pipeline.
3. This leads to incomplete logs, which makes debugging and audit trails unreliable.

---

Expected Behavior:

• When a workflow is manually aborted, the log pipeline should be allowed to finish processing all incoming log rows, regardless of any delay or processing time.
• The logging output should be complete and consistent, matching what would be expected if the workflow ended normally.

Issue Priority

Priority: 2

Issue Component

Component: Other

[hansva]

I fear this will be quite an undertaking. The logging pipeline is linked to the extension points, and those get triggered when pressing the stop button. Also, the timers are stopped at that point in time.

[mattcasters]

Great reproduction path. Nice work. So this concerns the option [x] Execute at the end of a pipeline in the Pipeline Log metadata element.

I noticed that if I ONLY specify this option (disabled logging at the start and periodically) that I still get a logging file, but only one, not the delayed file. So this might be a bug with the WorkflowLogging transform or the execution of the logging pipeline somehow.

[mattcasters]

There is some generated confusion in the sample:

• The start/interval/end logging pipelines all run and write to the same file(s).
• The start logging pipeline creates the log-out-DELAY.csv file.
• The end logging pipeline does not go beyond the "Delay row" transform because that transform looks at the stopped state of the pipeline. The stopped state is received from the parent workflow (WorkflowStartLoggingXp set the workflow parent). This is the likely cause of this whole issue.
• Multiple rows are received with details for all the executed actions because the "hwfl-input" transform is asked to do this.

I changed the sample to only log at the end and to also include dates and times in the text file output files.

[mattcasters]

The linked commit fixes the demonstrated issue with the stopped workflows or pipelines.