Gaffer
Gaffer copied to clipboard
Published
20 hours ago •
gchq
gchq
Investigate intermittent RoadTrafficRestApiITs test failure
When run with maven, the RoadTrafficRestApiITs>RoadTrafficTestQueries.checkRoadJunctionsInSouthWestHeavilyUsedByBusesIn2000 test fails intermittently.
The error messages seem to indicate that the store is not returning correct number of expected elements:
[ERROR] Failures:
[ERROR] RoadTrafficRestApiITs>RoadTrafficTestQueries.checkRoadJunctionsInSouthWestHeavilyUsedByBusesIn2000:306
expected: <[Junction,Bus Count, M32:2,1411, M5:LA Boundary,1067, M4:LA Boundary,1958]>
but was: <[Junction,Bus Count, M5:LA Boundary,1060, M32:2,1411, M5:LA Boundary,1067, M4:LA Boundary,1958]>
This error only seems to happen when running using maven.
This could indicate something isn't right with the MiniAccumuloCluster but more investigation is needed.
Here's a dump of the logs from a failed attempt. The cause of the Accumulo Exception might be something to look into:
Dec 04, 2020 3:59:47 PM uk.gov.gchq.gaffer.traffic.listeners.DataLoader loadData
INFO: Loading data
client.impl.TabletServerBatchWriter ERROR - Server side error on ubuntu:36533: org.apache.thrift.TApplicationException: Internal error processing closeUpdate
client.impl.TabletServerBatchWriter ERROR - Failed to send tablet server ubuntu:36533 its batch : Error on server ubuntu:36533
org.apache.accumulo.core.client.impl.AccumuloServerException: Error on server ubuntu:36533
at org.apache.accumulo.core.client.impl.TabletServerBatchWriter$MutationWriter.sendMutationsToTabletServer(TabletServerBatchWriter.java:1038)
at org.apache.accumulo.core.client.impl.TabletServerBatchWriter$MutationWriter.access$2100(TabletServerBatchWriter.java:675)
at org.apache.accumulo.core.client.impl.TabletServerBatchWriter$MutationWriter$SendTask.send(TabletServerBatchWriter.java:896)
at org.apache.accumulo.core.client.impl.TabletServerBatchWriter$MutationWriter$SendTask.run(TabletServerBatchWriter.java:852)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.thrift.TApplicationException: Internal error processing closeUpdate
at org.apache.thrift.TApplicationException.read(TApplicationException.java:111)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79)
at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.recv_closeUpdate(TabletClientService.java:459)
at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.closeUpdate(TabletClientService.java:445)
at org.apache.accumulo.core.client.impl.TabletServerBatchWriter$MutationWriter.sendMutationsToTabletServer(TabletServerBatchWriter.java:997)
... 9 more
gaffer.accumulostore.AccumuloStore WARN - Accumulo batch writer failed to close
org.apache.accumulo.core.client.MutationsRejectedException: # constraint violations : 0 security codes: {} # server errors 1 # exceptions 1
at org.apache.accumulo.core.client.impl.TabletServerBatchWriter.checkForFailures(TabletServerBatchWriter.java:593)
at org.apache.accumulo.core.client.impl.TabletServerBatchWriter.close(TabletServerBatchWriter.java:373)
at org.apache.accumulo.core.client.impl.BatchWriterImpl.close(BatchWriterImpl.java:54)
at uk.gov.gchq.gaffer.accumulostore.AccumuloStore.insertGraphElements(AccumuloStore.java:479)
at uk.gov.gchq.gaffer.accumulostore.AccumuloStore.addElements(AccumuloStore.java:424)
at uk.gov.gchq.gaffer.accumulostore.operation.handler.AddElementsHandler.addElements(AddElementsHandler.java:47)
at uk.gov.gchq.gaffer.accumulostore.operation.handler.AddElementsHandler.doOperation(AddElementsHandler.java:34)
at uk.gov.gchq.gaffer.accumulostore.operation.handler.AddElementsHandler.doOperation(AddElementsHandler.java:29)
at uk.gov.gchq.gaffer.store.Store.handleOperation(Store.java:941)
at uk.gov.gchq.gaffer.store.operation.handler.OperationChainHandler.doOperation(OperationChainHandler.java:47)
at uk.gov.gchq.gaffer.store.operation.handler.OperationChainHandler.doOperation(OperationChainHandler.java:35)
at uk.gov.gchq.gaffer.store.Store.handleOperation(Store.java:941)
at uk.gov.gchq.gaffer.store.Store.execute(Store.java:386)
at uk.gov.gchq.gaffer.store.Store.execute(Store.java:380)
at uk.gov.gchq.gaffer.graph.Graph._execute(Graph.java:334)
at uk.gov.gchq.gaffer.graph.Graph.execute(Graph.java:184)
at uk.gov.gchq.gaffer.graph.Graph.execute(Graph.java:157)
at uk.gov.gchq.gaffer.traffic.generator.RoadTrafficDataLoader.load(RoadTrafficDataLoader.java:88)
at uk.gov.gchq.gaffer.traffic.generator.RoadTrafficDataLoader.load(RoadTrafficDataLoader.java:61)
at uk.gov.gchq.gaffer.traffic.listeners.DataLoader.loadData(DataLoader.java:58)
at uk.gov.gchq.gaffer.traffic.listeners.DataLoader.contextInitialized(DataLoader.java:43)
at uk.gov.gchq.gaffer.traffic.RoadTrafficRestApiITs.prepareRestApi(RoadTrafficRestApiITs.java:72)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:686)
at org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
at org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
at org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:149)
at org.junit.jupiter.engine.extension.TimeoutExtension.interceptLifecycleMethod(TimeoutExtension.java:126)
at org.junit.jupiter.engine.extension.TimeoutExtension.interceptBeforeAllMethod(TimeoutExtension.java:68)
at org.junit.jupiter.engine.execution.ExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(ExecutableInvoker.java:115)
at org.junit.jupiter.engine.execution.ExecutableInvoker.lambda$invoke$0(ExecutableInvoker.java:105)
at org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
at org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
at org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
at org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
at org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:104)
at org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:98)
at org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.lambda$invokeBeforeAllMethods$8(ClassBasedTestDescriptor.java:375)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.invokeBeforeAllMethods(ClassBasedTestDescriptor.java:373)
at org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.before(ClassBasedTestDescriptor.java:193)
at org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.before(ClassBasedTestDescriptor.java:78)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:132)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:125)
at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:135)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:123)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:122)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:80)
at java.util.ArrayList.forEach(ArrayList.java:1259)
at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:38)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:139)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:125)
at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:135)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:123)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:122)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:80)
at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit(SameThreadHierarchicalTestExecutorService.java:32)
at org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor.execute(HierarchicalTestExecutor.java:57)
at org.junit.platform.engine.support.hierarchical.HierarchicalTestEngine.execute(HierarchicalTestEngine.java:51)
at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:248)
at org.junit.platform.launcher.core.DefaultLauncher.lambda$execute$5(DefaultLauncher.java:211)
at org.junit.platform.launcher.core.DefaultLauncher.withInterceptedStreams(DefaultLauncher.java:226)
at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:199)
at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:132)
at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.execute(JUnitPlatformProvider.java:188)
at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invokeAllTests(JUnitPlatformProvider.java:154)
at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invoke(JUnitPlatformProvider.java:128)
at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:428)
at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:162)
at org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:562)
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:548)
Caused by: org.apache.accumulo.core.client.impl.AccumuloServerException: Error on server ubuntu:36533
at org.apache.accumulo.core.client.impl.TabletServerBatchWriter$MutationWriter.sendMutationsToTabletServer(TabletServerBatchWriter.java:1038)
at org.apache.accumulo.core.client.impl.TabletServerBatchWriter$MutationWriter.access$2100(TabletServerBatchWriter.java:675)
at org.apache.accumulo.core.client.impl.TabletServerBatchWriter$MutationWriter$SendTask.send(TabletServerBatchWriter.java:896)
at org.apache.accumulo.core.client.impl.TabletServerBatchWriter$MutationWriter$SendTask.run(TabletServerBatchWriter.java:852)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.thrift.TApplicationException: Internal error processing closeUpdate
at org.apache.thrift.TApplicationException.read(TApplicationException.java:111)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79)
at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.recv_closeUpdate(TabletClientService.java:459)
at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.closeUpdate(TabletClientService.java:445)
at org.apache.accumulo.core.client.impl.TabletServerBatchWriter$MutationWriter.sendMutationsToTabletServer(TabletServerBatchWriter.java:997)
... 9 more
Dec 04, 2020 4:00:05 PM uk.gov.gchq.gaffer.traffic.listeners.DataLoader loadData
INFO: Sample data has been loaded
Dec 04, 2020 4:00:08 PM org.glassfish.grizzly.http.server.NetworkListener shutdownNow
INFO: Stopped listener bound to [localhost:8080]
[ERROR] Tests run: 4, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 38.144 s <<< FAILURE! - in uk.gov.gchq.gaffer.traffic.RoadTrafficRestApiITs
[ERROR] uk.gov.gchq.gaffer.traffic.RoadTrafficRestApiITs.checkM4Junction17To16RoadUse Time elapsed: 0.533 s <<< FAILURE!
org.opentest4j.AssertionFailedError: expected: <true> but was: <false>
[ERROR] uk.gov.gchq.gaffer.traffic.RoadTrafficRestApiITs.checkRoadJunctionsInSouthWestHeavilyUsedByBusesIn2000 Time elapsed: 1.275 s <<< FAILURE!
org.opentest4j.AssertionFailedError: expected: <[Junction,Bus Count, M32:2,1411, M5:LA Boundary,1067, M4:LA Boundary,1958]> but was: <[Junction,Bus Count, M5:LA Boundary,1067, M4:LA Boundary,1958]>
I have never seen this error. @GCHQDev404 @t92549 - Have you seen this at all? If not I think this should be closed as no longer an issue.
