DeserializeException

[dkurillo]

From time to time I get the following exception during checkin process of one model:

8113966 [pool-1-thread-1] ERROR org.bimserver.longaction.LongAction - [UNKNOWN_DESERIALIZER_ERROR] [firebox.ifc] Error on line 2370:  (null) #6329= IFCMATERIALLAYER(#6330,15.,$,'Layer',$,$,$);
org.bimserver.shared.exceptions.UserException: [firebox.ifc] Error on line 2370:  (null) #6329= IFCMATERIALLAYER(#6330,15.,$,'Layer',$,$,$);
	at org.bimserver.database.actions.StreamingCheckinDatabaseAction.execute(StreamingCheckinDatabaseAction.java:412)
	at org.bimserver.database.actions.StreamingCheckinDatabaseAction.execute(StreamingCheckinDatabaseAction.java:84)
	at org.bimserver.database.DatabaseSession.executeAndCommitAction(DatabaseSession.java:775)
	at org.bimserver.database.DatabaseSession.executeAndCommitAction(DatabaseSession.java:752)
	at org.bimserver.longaction.LongStreamingCheckinAction.execute(LongStreamingCheckinAction.java:73)
	at org.bimserver.longaction.LongActionManager$1.run(LongActionManager.java:55)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
	at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.base/java.lang.Thread.run(Unknown Source)
Caused by: org.bimserver.plugins.deserializers.DeserializeException: Error on line 2370:  (null) #6329= IFCMATERIALLAYER(#6330,15.,$,'Layer',$,$,$);
	at org.bimserver.ifc.step.deserializer.IfcStepStreamingDeserializer.read(IfcStepStreamingDeserializer.java:205)
	at org.bimserver.ifc.step.deserializer.IfcStepStreamingDeserializer.read(IfcStepStreamingDeserializer.java:175)
	at org.bimserver.database.actions.StreamingCheckinDatabaseAction.execute(StreamingCheckinDatabaseAction.java:196)
	... 10 common frames omitted
Caused by: java.nio.BufferOverflowException: null
	at java.base/java.nio.Buffer.nextPutIndex(Unknown Source)
	at java.base/java.nio.HeapByteBuffer.putLong(Unknown Source)
	at org.bimserver.shared.ByteBufferVirtualObject.<init>(ByteBufferVirtualObject.java:62)
	at org.bimserver.ifc.step.deserializer.IfcStepStreamingDeserializer.newVirtualObject(IfcStepStreamingDeserializer.java:304)
	at org.bimserver.ifc.step.deserializer.IfcStepStreamingDeserializer.processRecord(IfcStepStreamingDeserializer.java:333)
	at org.bimserver.ifc.step.deserializer.IfcStepStreamingDeserializer.processLine(IfcStepStreamingDeserializer.java:252)
	at org.bimserver.ifc.step.deserializer.IfcStepStreamingDeserializer.read(IfcStepStreamingDeserializer.java:193)
	... 12 common frames omitted

The strange thing is that all IFC viewers correctly read the model. And BIMServer itself after some activities such as uploading other models correctly uploads this model. But sometimes it crushes with this error. Other instances of BIMServer checkin this model also without errors. But at one time something happens and it throws error.

[hlg]

This seemingly nondeterministic behavior is due to the way how memory is allocated for processing a line with the streaming deserializer. There is MetricCollector which keeps track of the total number of lines and bytes reported. Then in IfcStepStreamingDeserializer it will register the lines processed by any serializer instance and use the collected metrics to estimate required buffer size. Apparently this is heuristic and dependent on what happened before, but if you check in the same files in the same order it should always either succeed or fail in the same place.

In your failure case, there seems to be a rather short line compared to the memory needed and thus the estimation is too low and you see the overflow. To avoid this, we could change the metrics, but if we keep the estimation simple like that and just adjust the factor, this means to slightly increase memory allocations and thus memory usage overall.

[dkurillo]

Taking into account your explanation I see two ways to resolve this:

1. Just increase default factor from 1.1D to, let's say, 1.2 and see if exception persists. If yes, then increase more...
2. Catch BuffereOverflowException and restart the line processing with larger buffer. This way seems to be obvious, why it is not implemented?