apache jenkins: Failed to save the JUnit test result

[dweiss]

Description

Just noticed this in run reports. If anybody wishes to dig what this may be - feel free to.

FATAL: Failed to save the JUnit test result
java.io.IOException: java.lang.RuntimeException: Failed to serialize hudson.tasks.junit.TestResult#suites for class hudson.tasks.junit.TestResult
        at hudson.XmlFile.write(XmlFile.java:223)
        at PluginClassLoader for junit//hudson.tasks.junit.TestResultAction.setResult(TestResultAction.java:135)
        at PluginClassLoader for junit//hudson.tasks.junit.TestResultAction.<init>(TestResultAction.java:96)
        at PluginClassLoader for junit//hudson.tasks.junit.JUnitResultArchiver.parseAndSummarize(JUnitResultArchiver.java:301)
        at PluginClassLoader for junit//hudson.tasks.junit.JUnitResultArchiver.perform(JUnitResultArchiver.java:194)
        at jenkins.tasks.SimpleBuildStep.perform(SimpleBuildStep.java:123)
        at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:80)
        at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
        at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:818)
        at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:767)
        at hudson.model.Build$BuildExecution.post2(Build.java:179)
        at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:711)
        at hudson.model.Run.execute(Run.java:1856)
        at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:44)
        at hudson.model.ResourceController.execute(ResourceController.java:101)
        at hudson.model.Executor.run(Executor.java:446)
Caused by: java.lang.RuntimeException: Failed to serialize hudson.tasks.junit.TestResult#suites for class hudson.tasks.junit.TestResult
        at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:274)
        at hudson.util.RobustReflectionConverter$2.visit(RobustReflectionConverter.java:241)
        at com.thoughtworks.xstream.converters.reflection.PureJavaReflectionProvider.visitSerializableFields(PureJavaReflectionProvider.java:174)
        at hudson.util.RobustReflectionConverter.doMarshal(RobustReflectionConverter.java:226)
        at hudson.util.RobustReflectionConverter.marshal(RobustReflectionConverter.java:163)
        at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:68)
        at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:59)
        at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:44)
        at com.thoughtworks.xstream.core.TreeMarshaller.start(TreeMarshaller.java:83)
        at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.marshal(AbstractTreeMarshallingStrategy.java:37)
        at com.thoughtworks.xstream.XStream.marshal(XStream.java:1307)
        at com.thoughtworks.xstream.XStream.marshal(XStream.java:1296)
        at com.thoughtworks.xstream.XStream.toXML(XStream.java:1269)
        at hudson.XmlFile.write(XmlFile.java:216)
        ... 15 more
Caused by: java.lang.RuntimeException: Failed to serialize hudson.tasks.junit.SuiteResult#stderr for class hudson.tasks.junit.SuiteResult
        at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:274)
        at hudson.util.RobustReflectionConverter$2.visit(RobustReflectionConverter.java:241)
        at com.thoughtworks.xstream.converters.reflection.PureJavaReflectionProvider.visitSerializableFields(PureJavaReflectionProvider.java:174)
        at hudson.util.RobustReflectionConverter.doMarshal(RobustReflectionConverter.java:226)
        at hudson.util.RobustReflectionConverter.marshal(RobustReflectionConverter.java:163)
        at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:68)
        at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:59)
        at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:44)
        at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:87)
        at com.thoughtworks.xstream.converters.collections.AbstractCollectionConverter.writeBareItem(AbstractCollectionConverter.java:94)
        at com.thoughtworks.xstream.converters.collections.AbstractCollectionConverter.writeItem(AbstractCollectionConverter.java:66)
        at com.thoughtworks.xstream.converters.collections.AbstractCollectionConverter.writeCompleteItem(AbstractCollectionConverter.java:81)
        at com.thoughtworks.xstream.converters.collections.CollectionConverter.marshal(CollectionConverter.java:75)
        at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:68)
        at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:59)
        at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:83)
        at hudson.util.RobustReflectionConverter.marshallField(RobustReflectionConverter.java:283)
        at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:270)
        ... 28 more
Caused by: [com.thoughtworks.xstream.io](http://com.thoughtworks.xstream.io/).StreamException: Invalid character 0xd83a in XML stream
        at hudson.util.PrettyPrintWriter.lambda$writeText$0(PrettyPrintWriter.java:253)
        at java.base/java.lang.StringUTF16$CodePointsSpliterator.advance(StringUTF16.java:1319)
        at java.base/java.lang.StringUTF16$CodePointsSpliterator.forEachRemaining(StringUTF16.java:1291)
        at java.base/java.util.stream.IntPipeline$Head.forEach(IntPipeline.java:617)
        at hudson.util.PrettyPrintWriter.writeText(PrettyPrintWriter.java:214)
        at hudson.util.PrettyPrintWriter.writeText(PrettyPrintWriter.java:210)
        at hudson.util.PrettyPrintWriter.setValue(PrettyPrintWriter.java:192)
        at [com.thoughtworks.xstream.io](http://com.thoughtworks.xstream.io/).WriterWrapper.setValue(WriterWrapper.java:45)
        at com.thoughtworks.xstream.converters.SingleValueConverterWrapper.marshal(SingleValueConverterWrapper.java:45)
        at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:50)
        at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:59)
        at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:83)
        at hudson.util.RobustReflectionConverter.marshallField(RobustReflectionConverter.java:283)
        at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:270)
        ... 45 more

Version and environment details

No response

[uschindler]

Does this happen on all builds? To me it looks like it serializes some stdout/stderr and one of the chars is not on the XML whitelist.

Unfortunately it does not say which file is broken. Best is to grep through workspace for the hex entity above.

[dweiss]

I've seen it a few times. It's an odd error. Doesn't show up in build scans, for example, only in emails. The workspace was wiped out when I checked the one above. Just something to keep an eye on.

[uschindler]

ASF Jenkins or Policeman? Could be Jenkins bug.

[dweiss]

ASF. https://lists.apache.org/[email protected]:lte=1M:%22failed%20to%20save%22

[rmuir]

Caused by: com.thoughtworks.xstream.io.StreamException: Invalid character 0xd83a in XML stream

probably caused by an unpaired surrogate in the test output?

[dweiss]

Cannot be reproduced and happens only occasionally though. Strange.

[rmuir]

We try to prevent these elsewhere in the tests but i think it's difficult to prevent in eg assertion strings or exception strings.

Maybe a worst case option is to reduce the amount of data sent as strings to the junit plugin, eg via configuration. I've done this with Jenkins in other programming languages just to reduce ram usage for Jenkins controller node.

[rmuir]

Here's the bug: https://github.com/jenkinsci/junit-plugin/blob/5173dddf0feb54dca1ac7fbe6b62d1b464c68019/src/main/java/hudson/tasks/junit/CaseResult.java#L394-L395

the junit plugin truncates long output by default, and "long" is relative and not that long. I stare at these outputs quite often in jenkins at work.

the problem is that it may truncate in the middle of a surrogate pair because it uses String.subSequence() to do it. It should at least do a Character.isHighSurrogate() check on the last character of the truncated string.

[rmuir]

The issue is some work to fix there. If the file is < 1MB, the entire File is read into a java.lang.String, and the linked code above is called which truncates via subSequence()

Otherwise for big files, whole file is not read into memory but instead "truncated reads" are done with shenanigans in a different repository:

• head() using java.io.File https://github.com/jenkinsci/jenkins/blob/master/core/src/main/java/hudson/util/TextFile.java#L116
• fastTail() using java.io.RandomAccessFile https://github.com/jenkinsci/jenkins/blob/master/core/src/main/java/hudson/util/TextFile.java#L131-L161

[rmuir]

https://github.com/jenkinsci/junit-plugin/pull/701

[rmuir]

In the meantime, one potential workaround would be to simply clean up tests that print too much.

But we'd have to really really reduce TestRuleLimitSysouts default limit to dodge their truncation logic: https://github.com/jenkinsci/junit-plugin/blob/5173dddf0feb54dca1ac7fbe6b62d1b464c68019/src/main/java/hudson/tasks/junit/CaseResult.java#L438-L439

Currently by default, we allow suites to spam up to 8KB.

Note that even improving just some of the tests that spam a lot, might at least make the failures rarer, so it would be progress.

[timja]

fix is available now: https://github.com/jenkinsci/junit-plugin/releases/tag/1334.vd3b_b_2094e438

[uschindler]

Policeman Jenkins gets it installed, ASF Jenkins needs to be pinged:

[dweiss]

Thank you!