spring-batch icon indicating copy to clipboard operation
spring-batch copied to clipboard

Performance improvement for step execution retrieval

Open hpoettker opened this issue 3 years ago • 0 comments

This PR is for versions 4.3.x and addresses #3790 and #4135.

Using JobExplorer::getStepExecution for multiple step executions is not ideal. Each invocation will retrieve the corresponding job execution which in turn triggers the retrieval of all step executions of the job execution. If JobExplorer::getStepExecution is used for all step executions, this leads to efforts and memory consumption that scale quadratically in the number of step executions.

The idea of the PR is to only call JobExplorer::getJobExecution and get the step executions from the returned job execution. The PR is intended to be conservative in its changes due to the late stage of the 4.3.x releases. That's why the filtering on the step execution ids are implemented although I'm not 100% sure that they are actually needed in every case.

I tested the change locally with the reproducing project provided here: https://github.com/spring-projects/spring-batch/issues/3790#issuecomment-810761639 With 250 MB of heap, the application no longer crashes and the memory profile looks much better.

For Spring Batch 5, it makes sense to forward port the change, I think. It should also be considered to deprecate JobExplorer::getStepExecution and provide a way to retrieve a StepExecution from a JobExecution by id instead.

hpoettker avatar Oct 05 '22 12:10 hpoettker