Gaffer icon indicating copy to clipboard operation
Gaffer copied to clipboard

Tests involving Hadoop/Accumulo do not work on Windows

Open GCHQDeveloper314 opened this issue 1 year ago • 0 comments

While looking at #2591 it was discovered that tests involving Hadoop/Accumulo do not work on Windows.

For Hadoop "Native IO is mandatory on Windows and without it you will not be able to get your installation working" (from Hadoop Wiki). This involves installing Hadoop libraries, setting the HADOOP_HOME environment variable and compiling and installing Native IO for windows. The Hadoop Native IO libraries for Windows are not distributed by Apache and must either be built from source or downloaded from a 3rd party.

Due to the above situation with Hadoop, all modules which have a test dependency on Hadoop fail to run tests on Windows. This includes at least hdfs-library,accumulo-store,accumulo-rest(plus most other rest modules), flink-library and all spark modules.

In addition there is what could be a separate problem with flink and JUnit where a temp directory cannot be deleted.

[ERROR] uk.gov.gchq.gaffer.flink.integration.operation.handler.AddElementsFromKafkaHandlerIT.shouldAddElementsWithByteArrayConsumer  Time elapsed: 8.093 s  <<< ERROR!
java.io.IOException: Failed to delete temp directory C:\Users\User\AppData\Local\Temp\junit5479111707429912277. The following paths could not be deleted (see suppressed exceptions for details): , kafkaLogDir, kafkaLogDir\51c41179-8027-4641-b8ff-7d191bafdcb5-0, kafkaLogDir\51c41179-8027-4641-b8ff-7d191bafdcb5-0\00000000000000000000.timeindex
        at org.junit.jupiter.engine.extension.TempDirectory$CloseablePath.createIOExceptionWithAttachedFailures(TempDirectory.java:280)
        at org.junit.jupiter.engine.extension.TempDirectory$CloseablePath.close(TempDirectory.java:188)
        at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
        at org.junit.jupiter.engine.execution.ExtensionValuesStore.lambda$closeAllStoredCloseableValues$3(ExtensionValuesStore.java:68)
        at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
        at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
        ...
        Suppressed: java.nio.file.DirectoryNotEmptyException: C:\Users\User\AppData\Local\Temp\junit5479111707429912277
                at sun.nio.fs.WindowsFileSystemProvider.implDelete(WindowsFileSystemProvider.java:266)
                at sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103)
                at java.nio.file.Files.delete(Files.java:1126)
                at org.junit.jupiter.engine.extension.TempDirectory$CloseablePath$1.deleteAndContinue(TempDirectory.java:228)
                at org.junit.jupiter.engine.extension.TempDirectory$CloseablePath$1.postVisitDirectory(TempDirectory.java:223)
                at org.junit.jupiter.engine.extension.TempDirectory$CloseablePath$1.postVisitDirectory(TempDirectory.java:199)
                at java.nio.file.Files.walkFileTree(Files.java:2688)
                at java.nio.file.Files.walkFileTree(Files.java:2742)
                at org.junit.jupiter.engine.extension.TempDirectory$CloseablePath.deleteAllFilesAndDirectories(TempDirectory.java:199)
                at org.junit.jupiter.engine.extension.TempDirectory$CloseablePath.close(TempDirectory.java:186)
                ... 63 more
        Suppressed: java.nio.file.FileSystemException: C:\Users\User\AppData\Local\Temp\junit5479111707429912277\kafkaLogDir\51c41179-8027-4641-b8ff-7d191bafdcb5-0\00000000000000000000.timeindex: The process cannot access the file because it is being used by another process.
                at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:86)
                at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
                at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:102)
                at sun.nio.fs.WindowsFileSystemProvider.implDelete(WindowsFileSystemProvider.java:269)
                at sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103)
                at java.nio.file.Files.delete(Files.java:1126)
                at org.junit.jupiter.engine.extension.TempDirectory$CloseablePath$1.deleteAndContinue(TempDirectory.java:228)
                at org.junit.jupiter.engine.extension.TempDirectory$CloseablePath$1.visitFile(TempDirectory.java:218)
                at org.junit.jupiter.engine.extension.TempDirectory$CloseablePath$1.visitFile(TempDirectory.java:199)
                at java.nio.file.Files.walkFileTree(Files.java:2670)
                at java.nio.file.Files.walkFileTree(Files.java:2742)
                at org.junit.jupiter.engine.extension.TempDirectory$CloseablePath.deleteAllFilesAndDirectories(TempDirectory.java:199)
                at org.junit.jupiter.engine.extension.TempDirectory$CloseablePath.close(TempDirectory.java:186)
                ... 63 more

There's also a potential path problem with path path processing in common-rest which may just be linked to Hadoop:

[ERROR] Failures:
[ERROR]   DefaultGraphFactoryTest.shouldThrowRuntimeExceptionIfGraphLibraryClassDoesNotExist:55
Expecting message to be:
  "Unable to deserialise graph config"
but was:
  "Illegal char <:> at index 2: /C:/Users/User/Documents/Gaffer/rest-api/common-rest/target/test-classes/store.properties"
Throwable that failed the check:
java.nio.file.InvalidPathException: Illegal char <:> at index 2: /C:/Users/User/Documents/Gaffer/rest-api/common-rest/target/test-classes/store.properties
        at sun.nio.fs.WindowsPathParser.normalize(WindowsPathParser.java:182)
        at sun.nio.fs.WindowsPathParser.parse(WindowsPathParser.java:153)
        at sun.nio.fs.WindowsPathParser.parse(WindowsPathParser.java:77)
        at sun.nio.fs.WindowsPath.parse(WindowsPath.java:94)
        at sun.nio.fs.WindowsFileSystem.getPath(WindowsFileSystem.java:255)
        at java.nio.file.Paths.get(Paths.java:84)
        at uk.gov.gchq.gaffer.store.StoreProperties.loadStoreProperties(StoreProperties.java:135)
        at uk.gov.gchq.gaffer.rest.factory.DefaultGraphFactory.createGraphBuilder(DefaultGraphFactory.java:97)
        at uk.gov.gchq.gaffer.rest.factory.GraphFactory.createGraph(GraphFactory.java:55)
        at uk.gov.gchq.gaffer.rest.factory.DefaultGraphFactory.getGraph(DefaultGraphFactory.java:71)
        at org.assertj.core.api.ThrowableAssert.catchThrowable(ThrowableAssert.java:63)
        at org.assertj.core.api.ThrowableTypeAssert.isThrownBy(ThrowableTypeAssert.java:59)
        at uk.gov.gchq.gaffer.rest.factory.DefaultGraphFactoryTest.shouldThrowRuntimeExceptionIfGraphLibraryClassDoesNotExist(DefaultGraphFactoryTest.java:54)

GCHQDeveloper314 avatar Jul 22 '22 10:07 GCHQDeveloper314