junit-plugin icon indicating copy to clipboard operation
junit-plugin copied to clipboard

[JENKINS-71139] Recover gracefully from corrupt `junitResult.xml`

Open jglick opened this issue 1 year ago • 2 comments

Amending #521. On our servers, we were seeing tons of errors on every startup (or even thereafter, due to lazy-loading) from junitResult.xml files containing the invalid null sequence, as you are warned. So I wrote a script and ran it using https://github.com/jenkinsci/log-cli-plugin/pull/131

find $JENKINS_HOME/jobs -type f -name junitResult.xml -exec fgrep -q '�' {} \; -print -delete

to clean them up. But now our logs are instead full of

java.nio.file.NoSuchFileException: /var/jenkins_home/jobs/…/builds/1/junitResult.xml
	at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
	at java.base/sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:219)
	at java.base/java.nio.file.Files.newByteChannel(Files.java:371)
	at java.base/java.nio.file.Files.newByteChannel(Files.java:422)
	at java.base/java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:420)
	at java.base/java.nio.file.Files.newInputStream(Files.java:156)
	at hudson.XmlFile.read(XmlFile.java:164)
	at hudson.tasks.junit.TestResultAction.load(TestResultAction.java:233)
	at hudson.tasks.junit.TestResultAction.getResult(TestResultAction.java:157)
	at hudson.tasks.junit.TestResultAction.getFailedTests(TestResultAction.java:212)
	at …

which is just as irritating!

Tried to write a test reproducing the situation and confirming the fix, but to my confusion, in this test environment TestResultAction.load succeeds with no error (the SuiteResult.stdout is simply null). And of course with all the faulty result files having been deleted, it is now trickier to go back and determine what is different in the functional test. Do you know of something I am forgetting here @timja?

jglick avatar Jul 31 '23 19:07 jglick

Manual cleanup FTR

for (p in Jenkins.instance.getAllItems(Job)) {
  for (b in p.builds) {
    def a = b.getAction(hudson.tasks.junit.TestResultAction);
    if (a != null && !new File(b.rootDir, 'junitResult.xml').file) {
      println(b);
      b.removeActions(hudson.tasks.junit.TestResultAction);
      b.save();
    }
  }
}

or

for (p in Jenkins.instance.getAllItems(jenkins.model.ParameterizedJobMixIn.ParameterizedJob)) {
  for (b in p._getRuns().getLoadedBuilds().values()) {
    def a = b.getAction(hudson.tasks.junit.TestResultAction);
    if (a != null && !new File(b.rootDir, 'junitResult.xml').file) {
      println(b);
      b.removeActions(hudson.tasks.junit.TestResultAction);
      b.save();
    }
  }
}

jglick avatar Jul 31 '23 20:07 jglick

Do you know of something I am forgetting here @timja?

Unsure

timja avatar Jul 31 '23 20:07 timja

Probably only of historical interest now.

jglick avatar Jul 26 '24 11:07 jglick