spring-projects
How retrieve the Retry/Skipped data for Testing purposes [BATCH-2629]
Manuel Jordan opened BATCH-2629 and commented
For Testing purposes I can do the following in peace (in general)
assertThat(jobExecution.getExitStatus().getExitCode(), is("COMPLETED"));
assertThat(jobExecution.getAllFailureExceptions().size(), is(0));
assertThat(jobExecution.getFailureExceptions().size(), is(0));
or
assertThat(jobExecution.getExitStatus().getExitCode(), is("FAILED"));
assertThat(jobExecution.getAllFailureExceptions().size(), is(1));
assertThat(jobExecution.getAllFailureExceptions().get(0).getClass().getSimpleName(),
is(SkipLimitExceededException.class.getSimpleName()));
assertThat(jobExecution.getAllFailureExceptions().get(0).getMessage(),
is("Skip limit of '5' exceeded"));
assertThat(jobExecution.getFailureExceptions().size(), is(0));
Seems the API does not provide a way to extract detailed data about Retry and Skip information.
For example: for the Steps X, Y, Z (for each one or individually) how many retries were applied and for what row, same appreciation for Skip. Same about (Retry and Skip) but through a global report from the Steps X,Y,Z. Of course if the Job has one Step, the global is the same than the individual
The listeners works fine only for report purposes and their methods return void. Not sure if the current API and Spring Batch db schema support it for testing.
Thank you.
Affects: 3.0.6, 3.0.7
Reference URL: https://stackoverflow.com/questions/44574473/spring-batch-how-retrieve-the-skipped-exceptions-for-testing-purposes
Mahmoud Ben Hassine commented
The listeners works fine only for report purposes and their methods return void. Not sure if the current API and Spring Batch db schema support it for testing.
You can store this information in the execution context through the skip/retry listener and retrieve it afterwards for assertions. Here is an example:
import java.util.Arrays;
import org.springframework.batch.core.Job;
import org.springframework.batch.core.JobExecution;
import org.springframework.batch.core.JobParameters;
import org.springframework.batch.core.Step;
import org.springframework.batch.core.StepExecution;
import org.springframework.batch.core.annotation.BeforeStep;
import org.springframework.batch.core.configuration.annotation.EnableBatchProcessing;
import org.springframework.batch.core.configuration.annotation.JobBuilderFactory;
import org.springframework.batch.core.configuration.annotation.StepBuilderFactory;
import org.springframework.batch.core.launch.JobLauncher;
import org.springframework.batch.core.listener.SkipListenerSupport;
import org.springframework.batch.item.ItemReader;
import org.springframework.batch.item.ItemWriter;
import org.springframework.batch.item.support.ListItemReader;
import org.springframework.context.ApplicationContext;
import org.springframework.context.annotation.AnnotationConfigApplicationContext;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
@Configuration
@EnableBatchProcessing
public class MyJob {
private final JobBuilderFactory jobs;
private final StepBuilderFactory steps;
public MyJob(JobBuilderFactory jobs, StepBuilderFactory steps) {
this.jobs = jobs;
this.steps = steps;
}
@Bean
public ItemReader<Integer> itemReader() {
return new ListItemReader<>(Arrays.asList(1, 2, 3, 4, 5, 6, 7, 8, 9, 10));
}
@Bean
public ItemWriter<Integer> itemWriter() {
return items -> {
if (items.contains(3) || items.contains(8)) {
throw new IllegalArgumentException();
}
for (Integer item : items) {
System.out.println("item = " + item);
}
};
}
@Bean
public Step step() {
return steps.get("step")
.<Integer, Integer>chunk(5)
.reader(itemReader())
.writer(itemWriter())
.listener(new MySkipListener())
.faultTolerant()
.skipLimit(3)
.skip(IllegalArgumentException.class)
.build();
}
@Bean
public Job job() {
return jobs.get("job")
.start(step())
.build();
}
public static void main(String[] args) throws Exception {
ApplicationContext context = new AnnotationConfigApplicationContext(MyJob.class);
JobLauncher jobLauncher = context.getBean(JobLauncher.class);
Job job = context.getBean(Job.class);
JobExecution jobExecution = jobLauncher.run(job, new JobParameters());
int skipCount = jobExecution.getStepExecutions().iterator().next().getExecutionContext().getInt("skipCount");
System.out.println("skipCount = " + skipCount);
}
public class MySkipListener extends SkipListenerSupport<Integer, Integer> {
private StepExecution stepExecution;
private int skipCount = 0;
@BeforeStep
public void setStepExecution(StepExecution stepExecution) {
this.stepExecution = stepExecution;
}
@Override
public void onSkipInWrite(Integer item, Throwable t) {
stepExecution.getExecutionContext().put("skipCount", ++skipCount);
System.out.println("skipped item = " + item);
}
}
This sample prints:
item = 1
item = 2
item = 4
skipped item = 3
item = 5
item = 6
item = 7
item = 9
skipped item = 8
item = 10
skipCount = 2
The sample shows how to store the skipped items count in the execution context and retrieve it after the job execution. The sample store the item count but you can do the same and store the items themselves.
So for me it is already possible to achieve what you are looking for and there is nothing to do for this JIRA ticket.
@Manuel Jordan Do you agree?
Manuel Jordan commented
Hello
Thanks for the reply and for the sample code, I appreciate that.
Because your solution has output seems correct, but I can see it only works around about skipped items. Nothing about Retries yet. I will do a research later.
Not sure if the SB API by itself was enhanced to support the original request/features. Remember I created this post in June 2017.
Thank you.
Mahmoud Ben Hassine commented
The same example would work with a retry listener and allows you to retrieve retry/skip data as requested. I don't understand what feature you are asking for. Can you please provide an example of what exactly you are expecting Spring Batch to provide?
Manuel Jordan commented
Sure.
For testing purposes: I need do assertions, for multiple @Test scenarios, for retry and skip. It for a Step and Job.
Consider the same feature to be applied in runtime through logging, with or without JMX. In this case how report.
I think same as we got stepExecution.getAllFailureExceptions(), we should have stepExecution.getAllSkippedExceptions() in a built-in way, that would be very useful, especially when using AlwaysSkipItemSkipPolicy/ExceptionClassifierSkipPolicy or the defaults but not reaching the skipLimit
