Gaffer
Gaffer copied to clipboard
Tests involving Hadoop/Accumulo do not work on Windows
While looking at #2591 it was discovered that tests involving Hadoop/Accumulo do not work on Windows.
For Hadoop "Native IO is mandatory on Windows and without it you will not be able to get your installation working" (from Hadoop Wiki). This involves installing Hadoop libraries, setting the HADOOP_HOME
environment variable and compiling and installing Native IO for windows. The Hadoop Native IO libraries for Windows are not distributed by Apache and must either be built from source or downloaded from a 3rd party.
Due to the above situation with Hadoop, all modules which have a test dependency on Hadoop fail to run tests on Windows. This includes at least hdfs-library
,accumulo-store
,accumulo-rest
(plus most other rest modules), flink-library
and all spark
modules.
In addition there is what could be a separate problem with flink and JUnit where a temp directory cannot be deleted.
[ERROR] uk.gov.gchq.gaffer.flink.integration.operation.handler.AddElementsFromKafkaHandlerIT.shouldAddElementsWithByteArrayConsumer Time elapsed: 8.093 s <<< ERROR!
java.io.IOException: Failed to delete temp directory C:\Users\User\AppData\Local\Temp\junit5479111707429912277. The following paths could not be deleted (see suppressed exceptions for details): , kafkaLogDir, kafkaLogDir\51c41179-8027-4641-b8ff-7d191bafdcb5-0, kafkaLogDir\51c41179-8027-4641-b8ff-7d191bafdcb5-0\00000000000000000000.timeindex
at org.junit.jupiter.engine.extension.TempDirectory$CloseablePath.createIOExceptionWithAttachedFailures(TempDirectory.java:280)
at org.junit.jupiter.engine.extension.TempDirectory$CloseablePath.close(TempDirectory.java:188)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.jupiter.engine.execution.ExtensionValuesStore.lambda$closeAllStoredCloseableValues$3(ExtensionValuesStore.java:68)
at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
...
Suppressed: java.nio.file.DirectoryNotEmptyException: C:\Users\User\AppData\Local\Temp\junit5479111707429912277
at sun.nio.fs.WindowsFileSystemProvider.implDelete(WindowsFileSystemProvider.java:266)
at sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103)
at java.nio.file.Files.delete(Files.java:1126)
at org.junit.jupiter.engine.extension.TempDirectory$CloseablePath$1.deleteAndContinue(TempDirectory.java:228)
at org.junit.jupiter.engine.extension.TempDirectory$CloseablePath$1.postVisitDirectory(TempDirectory.java:223)
at org.junit.jupiter.engine.extension.TempDirectory$CloseablePath$1.postVisitDirectory(TempDirectory.java:199)
at java.nio.file.Files.walkFileTree(Files.java:2688)
at java.nio.file.Files.walkFileTree(Files.java:2742)
at org.junit.jupiter.engine.extension.TempDirectory$CloseablePath.deleteAllFilesAndDirectories(TempDirectory.java:199)
at org.junit.jupiter.engine.extension.TempDirectory$CloseablePath.close(TempDirectory.java:186)
... 63 more
Suppressed: java.nio.file.FileSystemException: C:\Users\User\AppData\Local\Temp\junit5479111707429912277\kafkaLogDir\51c41179-8027-4641-b8ff-7d191bafdcb5-0\00000000000000000000.timeindex: The process cannot access the file because it is being used by another process.
at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:86)
at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:102)
at sun.nio.fs.WindowsFileSystemProvider.implDelete(WindowsFileSystemProvider.java:269)
at sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103)
at java.nio.file.Files.delete(Files.java:1126)
at org.junit.jupiter.engine.extension.TempDirectory$CloseablePath$1.deleteAndContinue(TempDirectory.java:228)
at org.junit.jupiter.engine.extension.TempDirectory$CloseablePath$1.visitFile(TempDirectory.java:218)
at org.junit.jupiter.engine.extension.TempDirectory$CloseablePath$1.visitFile(TempDirectory.java:199)
at java.nio.file.Files.walkFileTree(Files.java:2670)
at java.nio.file.Files.walkFileTree(Files.java:2742)
at org.junit.jupiter.engine.extension.TempDirectory$CloseablePath.deleteAllFilesAndDirectories(TempDirectory.java:199)
at org.junit.jupiter.engine.extension.TempDirectory$CloseablePath.close(TempDirectory.java:186)
... 63 more
There's also a potential path problem with path path processing in common-rest
which may just be linked to Hadoop:
[ERROR] Failures:
[ERROR] DefaultGraphFactoryTest.shouldThrowRuntimeExceptionIfGraphLibraryClassDoesNotExist:55
Expecting message to be:
"Unable to deserialise graph config"
but was:
"Illegal char <:> at index 2: /C:/Users/User/Documents/Gaffer/rest-api/common-rest/target/test-classes/store.properties"
Throwable that failed the check:
java.nio.file.InvalidPathException: Illegal char <:> at index 2: /C:/Users/User/Documents/Gaffer/rest-api/common-rest/target/test-classes/store.properties
at sun.nio.fs.WindowsPathParser.normalize(WindowsPathParser.java:182)
at sun.nio.fs.WindowsPathParser.parse(WindowsPathParser.java:153)
at sun.nio.fs.WindowsPathParser.parse(WindowsPathParser.java:77)
at sun.nio.fs.WindowsPath.parse(WindowsPath.java:94)
at sun.nio.fs.WindowsFileSystem.getPath(WindowsFileSystem.java:255)
at java.nio.file.Paths.get(Paths.java:84)
at uk.gov.gchq.gaffer.store.StoreProperties.loadStoreProperties(StoreProperties.java:135)
at uk.gov.gchq.gaffer.rest.factory.DefaultGraphFactory.createGraphBuilder(DefaultGraphFactory.java:97)
at uk.gov.gchq.gaffer.rest.factory.GraphFactory.createGraph(GraphFactory.java:55)
at uk.gov.gchq.gaffer.rest.factory.DefaultGraphFactory.getGraph(DefaultGraphFactory.java:71)
at org.assertj.core.api.ThrowableAssert.catchThrowable(ThrowableAssert.java:63)
at org.assertj.core.api.ThrowableTypeAssert.isThrownBy(ThrowableTypeAssert.java:59)
at uk.gov.gchq.gaffer.rest.factory.DefaultGraphFactoryTest.shouldThrowRuntimeExceptionIfGraphLibraryClassDoesNotExist(DefaultGraphFactoryTest.java:54)