routing Exception contracting europe-latest.osm.pbf

I'm compiling europe-latest.osm.pbf to a routerdb. Machine has 144GB memory. Using standard Car.lua and processRestrictions=true.

Without contraction routerDb.LoadOsmData() and routerDb.Serialize() finishes successfully. With contraction the following exception occurs after about 1h running in routerDb.AddContracted(vec.Fastest(), processRestrictions); . Smaller pbf-files are fine. Is Europe too much? Or is it suppose to compile? There is int32 in the exception, should this maybe be changed to int64?

Array dimensions exceeded supported range.
   at System.Collections.Generic.Dictionary`2.Resize(Int32 newSize, Boolean forceNewHashCodes)
   at System.Collections.Generic.Dictionary`2.Insert(TKey key, TValue value, Boolean add)
   at Itinero.Algorithms.Contracted.Dual.FastHierarchyBuilder`1.NotifyContracted(UInt32 vertex)
   at Itinero.Algorithms.Contracted.Dual.FastHierarchyBuilder.Contract()
   at Itinero.Algorithms.Contracted.Dual.FastHierarchyBuilder`1.DoRun(CancellationToken cancellationToken)
   at Itinero.Algorithms.AlgorithmBase.Run(CancellationToken cancellationToken)
   at Itinero.RouterDbExtensions.AddContracted(RouterDb db, Profile profile, Boolean forceEdgeBased, CancellationToken cancellationToken)
   at Itinero.RouterDbExtensions.AddContracted(RouterDb db, Profile profile, Boolean forceEdgeBased)
   at QvxItinero.QmrTools.<>c__DisplayClass0_0.<CompileQmr>b__0() in C:\Users\Tom\QuoVadis\Qvx1\QvxItinero\Qmr\CompileQmr.cs:line 77
   at System.Threading.Tasks.Task.Execute()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
   at QvxItinero.QmrTools.<CompileQmr>d__0.MoveNext() in C:\Users\Tom\QuoVadis\Qvx1\QvxItinero\Qmr\CompileQmr.cs:line 35

Mar 22 '19 16:03 TomQv

Same error with asia-latest.osm, which is 7.2GB. africa-latest.osm.pbf (3GB), germany-latest.osm.pbf (3.1GB) and south-america-latest.osm.pbf (1.5GB) was ok.

Is there a limit in size?

Mar 25 '19 08:03 TomQv

A standard .NET dictionary/array is limited in size. Replacing this by HugeDictionary could fix the issue:

https://github.com/itinero/routing/blob/develop/src/Itinero/Algorithms/Contracted/Dual/FastHierarchyBuilder.cs#L19 https://github.com/itinero/routing/blob/develop/src/Itinero/Algorithms/Contracted/Dual/FastHierarchyBuilder.cs#L20

https://github.com/itinero/routing/blob/develop/src/Itinero/Algorithms/Collections/HugeDictionary.cs

Will try to do this asap. It's strange however that I never ran into this issue myself.

Mar 25 '19 08:03 xivk

Did you contract europe already successfully before? I recently upgraded to .net 4.7.2, maybe there is a problem?

Mar 25 '19 09:03 TomQv

FYI, I'm testing with europe currently. The fix done does seem to solve the issue you were having but I ran into another issue:

Something is going wrong during compression of the graph. I hope it's not too big and overflows uint range.

Mar 26 '19 09:03 xivk

same here... I was almost getting excited, because it was running the whole night without error, but just now it also ended with this exception..

Mar 26 '19 10:03 TomQv

The latest commit from Mar 28, did it fix this issue?

Mar 30 '19 08:03 TomQv

yes, europe was now successful :) It took 26h, but cpu-load didn't exceed 8% on a 16-core cpu. Seems, that only one core is working?

Mar 31 '19 13:03 TomQv

routing routing copied to clipboard

Exception contracting europe-latest.osm.pbf

routing
routing copied to clipboard