Incomplete transfer of DAM without error

[soujiro32167]

Hi

I am trying to transfer a DAM path with 4737 assets. However, this is the result I get from Grabbit every time:

{
        "endTime": "2017-01-16T22:48:46+0000",
        "exitStatus": {
            "exitCode": "COMPLETED",
            "exitDescription": "",
            "running": false
        },
        "jcrNodesWritten": 398,
        "jobExecutionId": 4067630146893983297,
        "path": "/content/dam/estore",
        "startTime": "2017-01-16T22:47:11+0000",
        "timeTaken": 95151,
        "transactionID": 3971212595291231010
    }

Looking at the logs, there are no errors. Running in DEBUG mode reveals that the process always (5 tries) ends just before the original rendition of a ~400MB .zip file:

/content/dam/estore/popup/Autodesk_Realtime_Renderer_Tutorial_en.zip/jcr:content/renditions/cq5dam.web.1280.1280.jpeg, /jcr:content]
17.01.2017 19:19:57.002 *INFO* [clientJobLauncherTaskExecutor-7] com.twcable.grabbit.client.batch.steps.jcrnodes.JcrNodesWriter StopWatch 'Refreshing session: session-6804754': running time (millis) = 0
17.01.2017 19:19:57.009 *INFO* [clientJobLauncherTaskExecutor-7] com.twcable.grabbit.client.batch.steps.jcrnodes.LoggingStepExecutionListener JcrNodes Step Complete. Current Path : /content/dam/estore . Total nodes written : 398

Is there a size limit to transferred nodes? Or perhaps some sort of connection timeout that would cause the transfer to stop without error?

[jbornemann]

Hey @soujiro32167,

Could you provide an example configuration based off the configuration you used to produce this? What AEM version are you running?

Are you running a delta sync? If so - if the asset was successfully synched previously, and not updated on the sending server - it would not be sent across, regardless of it's state on the receiving server - for e.g, if it was since deleted.

Can you try to sync just the 400MB asset, ensuring delta sync is off?

Either a connection timeout, or hitting technical size limitations I would expect to result in a job failure.

[soujiro32167]

Thanks for the quick response! I'll try to migrate the file alone and send over the config

[soujiro32167]

# Client Type: author

   # Information for connecting to the source content
   serverUsername : ''
   serverPassword : ''
   serverScheme : http
   serverHost : ''
   serverPort : 4502

   deltaContent : true # default for all the paths

   # A reference to the standard set of workflow configuration ids that
   # we want to turn off when working with DAM assets.
   damWorkflows: &ourDamWorkflows
     - /etc/workflow/launcher/config/update_asset_mod
     - /etc/workflow/launcher/config/update_asset_create
     - /etc/workflow/launcher/config/dam_xmp_nested_writeback
     - /etc/workflow/launcher/config/dam_xmp_writeback


   # Each of the paths to include in the copy
   pathConfigurations :
    -
      path : /content/dam/estore
      workflowConfigIds : *ourDamWorkflows
      deltaContent : false
      deleteBeforeWrite : true

[mzgubin]

We have seen something similar as well. We used a batch size of 100 to copy dam images and they get copied over but there are some misses. I wonder if the batch size is too big for the allotted heap for the JVM of AEM. There should be some errors thrown somewhere to indicate truncation or something similar. We have 4GB for heap allocated to both source and target JVMs.

[vhonchar]

We also have the same issue. Only 4475 assets from 4491 were copied.

Config is almost the same:

{
    "serverUsername" : "",
    "serverPassword" : "",
    "serverScheme" : "http",
    "serverHost" : "",
    "serverPort" : "4503",
    "batchSize" : 150,
    "deltaContent" : false,
    "pathConfigurations" :  [
        {
            "path" : "/content/dam/site",
            "workflowConfigIds" :
                [
                    "/etc/workflow/launcher/config/update_asset_mod",
                    "/etc/workflow/launcher/config/update_asset_create",
                    "/etc/workflow/launcher/config/dam_xmp_nested_writeback",
                    "/etc/workflow/launcher/config/dam_xmp_writeback"
                ]
        }
    
    ]
}

Status of job is successful. In receiver logs I found:

27.08.2018 17:07:13.651 *INFO* [clientJobLauncherTaskExecutor-4] com.twcable.grabbit.client.batch.steps.jcrnodes.LoggingStepExecutionListener JcrNodes Step Complete. Current Path : /content/dam/site . Total nodes written : 8890

However error.log of sender contains error at the same time:

27.08.2018 17:07:13.181 *ERROR* [172.40.1.21 [1535382311345] GET /grabbit/content HTTP/1.1] org.springframework.batch.core.step.AbstractStep Encountered an error executing step serverJcrNodes in job serverJob
java.lang.OutOfMemoryError: Java heap space
	at java.util.Arrays.copyOf(Arrays.java:3332)
	at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124)
	at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:448)
	at java.lang.StringBuilder.append(StringBuilder.java:136)
	at java.lang.StringBuilder.append(StringBuilder.java:76)
	at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:484)
	at java.lang.StringBuilder.append(StringBuilder.java:166)
	at java.lang.StringBuilder.append(StringBuilder.java:76)
	at com.google.protobuf.TextFormat$TextGenerator.write(TextFormat.java:567)
	at com.google.protobuf.TextFormat$TextGenerator.print(TextFormat.java:556)
	at com.google.protobuf.TextFormat$Printer.printFieldValue(TextFormat.java:425)
	at com.google.protobuf.TextFormat$Printer.printSingleField(TextFormat.java:353)
	at com.google.protobuf.TextFormat$Printer.printField(TextFormat.java:312)
	at com.google.protobuf.TextFormat$Printer.print(TextFormat.java:299)
	at com.google.protobuf.TextFormat$Printer.printFieldValue(TextFormat.java:434)
	at com.google.protobuf.TextFormat$Printer.printSingleField(TextFormat.java:353)
	at com.google.protobuf.TextFormat$Printer.printField(TextFormat.java:309)
	at com.google.protobuf.TextFormat$Printer.print(TextFormat.java:299)
	at com.google.protobuf.TextFormat$Printer.printFieldValue(TextFormat.java:434)
	at com.google.protobuf.TextFormat$Printer.printSingleField(TextFormat.java:353)
	at com.google.protobuf.TextFormat$Printer.printField(TextFormat.java:309)
	at com.google.protobuf.TextFormat$Printer.print(TextFormat.java:299)
	at com.google.protobuf.TextFormat$Printer.printFieldValue(TextFormat.java:434)
	at com.google.protobuf.TextFormat$Printer.printSingleField(TextFormat.java:353)
	at com.google.protobuf.TextFormat$Printer.printField(TextFormat.java:309)
	at com.google.protobuf.TextFormat$Printer.print(TextFormat.java:299)
	at com.google.protobuf.TextFormat$Printer.access$400(TextFormat.java:273)
	at com.google.protobuf.TextFormat.print(TextFormat.java:76)
	at com.google.protobuf.TextFormat.printToString(TextFormat.java:143)
	at com.google.protobuf.AbstractMessage.toString(AbstractMessage.java:79)
	at org.codehaus.groovy.runtime.InvokerHelper.format(InvokerHelper.java:630)
	at org.codehaus.groovy.runtime.InvokerHelper.format(InvokerHelper.java:577)

[mzgubin]

This is the one that may have been addressed in issue #2

[sagarsane]

Thanks @mzgubin .. @vhonchar can you please retest this issue with the latest release? Hopefully your issue is now resolved.

[vhonchar]

Fix helped. Thank you guys.