QueryBatcher JobReport wrong failure number

[maffelbaffel]

I am querying documents using a QueryBatcher and applying a transformation using a ApplyTransformListener. After all batches completed, I want to know if any of the batches failed. The JobReport seems to be the way to go for this. My problem is that the JobReport always reports that every batch was successful even if there were failures. For testing purpose batch size is 1 so that each document gets processed in a single batch.

final ApplyTransformListener transformListener = new ApplyTransformListener()
    .withApplyResult(ApplyTransformListener.ApplyResult.REPLACE)
    .withTransform(new ServerTransform(transformName))
    .onSuccess(batch -> {
        if (log.isTraceEnabled()) {
            for (String item : batch.getItems()) {
                log.trace("Batch #{}: item {} successfully executed.", batch.getForestBatchNumber(), item);
            }
        }
        log.debug("Batch #{}: finished executed.", batch.getForestBatchNumber());
    })
    .onFailure((batch, throwable) -> {
        log.error("Batch #{}: failed.", batch.getForestBatchNumber(), throwable);
    })
    .onSkipped(batch -> Arrays.stream(batch.getItems())
        .forEach(it -> log.warn("Skipped processing document {}.", it))
    );

final QueryBatcher batcher = dmm.newQueryBatcher(queryDef)
    .withBatchSize(batchSize)
    .withConsistentSnapshot()
    .onUrisReady(transformListener);

try {
    final JobTicket jobTicket = dmm.startJob(batcher);
    batcher.awaitCompletion();
    final JobReport jobReport = dmm.getJobReport(jobTicket);

    if (jobReport.getFailureBatchesCount() > 0) {
        // expected to be at least 1
        throw new MagicException(String.format("%d batches failed to executed.", jobReport.getFailureBatchesCount()));
    }
    dmm.stopJob(jobTicket);
    log.debug("Successfully executed {} batches.", jobReport.getSuccessBatchesCount());
} catch (final Exception ex) {
    System.out.println(ex);
}

These are the produced logs:

11:04:10.950 [pool-2-thread-2] TRACE - Batch #1: item test/Cat.xml successfully executed.
11:04:10.950 [pool-2-thread-2] DEBUG - Batch #1: finished executed.
11:04:10.952 [pool-2-thread-1] TRACE - Batch #1: item test/Cat3.xml successfully executed.
11:04:10.952 [pool-2-thread-1] DEBUG - Batch #1: finished executed.
11:04:10.971 [pool-2-thread-3] ERROR - Batch #2: failed.
com.marklogic.client.FailedRequestException: Local message: failed to apply resource at internal/apply-transform: Internal Server Error. Server Message: error (err:FOER0000): . See the MarkLogic server error log for further detail.
    at com.marklogic.client.impl.OkHttpServices.checkStatus(OkHttpServices.java:4395) ~[marklogic-client-api-4.2.0.jar:?]
    at com.marklogic.client.impl.OkHttpServices.postResource(OkHttpServices.java:3377) ~[marklogic-client-api-4.2.0.jar:?]
    at com.marklogic.client.impl.OkHttpServices.postResource(OkHttpServices.java:3323) ~[marklogic-client-api-4.2.0.jar:?]
    at com.marklogic.client.impl.OkHttpServices.postResource(OkHttpServices.java:3314) ~[marklogic-client-api-4.2.0.jar:?]
    at com.marklogic.client.datamovement.ApplyTransformListener.processEvent(ApplyTransformListener.java:144) [marklogic-client-api-4.2.0.jar:?]
    at com.marklogic.client.datamovement.impl.QueryBatcherImpl$QueryTask.run(QueryBatcherImpl.java:674) [marklogic-client-api-4.2.0.jar:?]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_222]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_222]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_222]
11:04:10.974 [main] DEBUG - Successfully executed 3/3 batches.

As you can see there is indeed an error thrown and logged in my onFailure listener.

The transformation is quite simple and only for testing purposes. It throws an error if some value is not equal to one:

xquery version "1.0-ml";
module namespace transform = "http://marklogic.com/rest-api/transform/magic-test/cat.xml";
declare function transform($context as map:map, $params as map:map, $content as document-node()) as document-node(){
    if (xs:integer($content/cats/age) eq 1) then
        document {
           <cats>
               {$content/cats/uri}
               {$content/cats/age}
               <name>Tiger</name>
           </cats>
        }
     else fn:error()
};

This is how my data looks:

<cats>
    <uri>test/Cat</uri>
    <name>cat</name>
    <age>1</age>
</cats>
<cats>
    <uri>test/Cat2</uri>
    <name>cat two</name>
    <age>2</age>
</cats>
<cats>
    <uri>test/Cat3</uri>
    <name>cat three</name>
    <age>1</age>
</cats>

So we can address your issue, please include the following:

Version of MarkLogic Java Client API

4.2.0

Version of MarkLogic Server

9.0-10

Java version

Java 8

OS and version

Linux

StackOverflow Reference

[ehennum]

@anu3990 , please take a look at reproducing this issue.

[ehennum]

As @georgeajit pointed out to me, the JobReport reflects the success and failure of the uri batch retrieval and not of subsequent processing of the uri batch such as by ApplyTransform.

It makes sense to me that the success or failure of the batch processing is also useful to reflect in the JobReport, so I've recategorized this issue as an RFE for prioritization.

In the meantime, the best approach would be to count failure and success on the uri batch processing in the listener for ApplyTransform.

[rjrudin]

Not sure if this is still relevant, but from my analysis of JobReport and what it captures - I believe it's working as designed, and unfortunately that design only accounts for the retrieval of batches of URIs when using QueryBatcher - it explicitly ignores what happens with those batches. Which is why a concrete listener like ApplyTransformListener provides its own success/failure listeners, which should definitely be used when using ApplyTransformListener.

I can see justification for this approach when considering a scenario where a user has attached 10 different listeners. And for any given batch, some of those listeners may succeed on some URIs and some may fail on other URIs. Whether or not that means the batch "succeeded" or "failed" is entirely up to the user. JobReport stays out of that and focuses solely on how many batches it was able to retrieve.

It certainly seems in hindsight that JobReport is far more useful when writing documents, as that's a one step process - i.e. either a batch of documents succeeds or fails to be written to the database. Whereas querying is a two step process - first try to retrieve a batch of URIs, and then try to process that batch of URIs with N batch listeners.

So in summary - I don't see JobReport ever changing to account for the processing of batches; it's really up to the individual batch listeners to determine what constitutes success or failure, particularly when multiple listeners exist.

The one action worth taking here is that the Javadocs are misleading for JobReport as they use the term "processed" instead of "retrieved". We'll get that updated for the 6.2.0 release.