core
core copied to clipboard
Published
20 hours ago •
OsmSharp
OsmSharp
Improve OSM PBF read performance.
- OsmSharp Version 6.2.0
Opening osm pbf files and counting the elements in a decently large osm.pbf file is very slow compared to other languages osm libraries/tools.
For example, running the example snippet in the readme (without the console prints) on a ~9GB osm.pbf file takes over 10 minutes. Running the osmium tags-filter tool with a complex filter command (which reads the file multiple times) over the same file on the same machine takes less than 3 minutes. Something funky is going on in the osm pbf stream reading logic or the readme provided example.
using OsmSharp;
using OsmSharp.Streams;
class Program
{
static void Main(string[] args)
{
// 1,255,681,519 entries
using(var fileStream = new FileInfo(@".\north-america-latest.osm.pbf").OpenRead())
{
var source = new PBFOsmStreamSource(fileStream);
int count = 0;
foreach (var element in source)
count++;
Console.WriteLine($"Total: {count}");
// Runtime is > 10 minutes on modern hardware!
}
}
}
System and dotnet version (although I ran into the same thing on my ubuntu18 machine).
(Click to expand)
PS D:\repositories\sandbox> dotnet --info .NET Core SDK (reflecting any global.json): Version: 3.1.101 Commit: b377529961Runtime Environment: OS Name: Windows OS Version: 10.0.18362 OS Platform: Windows RID: win10-x64 Base Path: C:\Program Files\dotnet\sdk\3.1.101
dottrace output of all "OsmSharp" functions. Time values are in milliseconds. Total time was 853211 ms (~14 minutes).
- Total Time: ~14 minutes
- OsmSharp.Streams.PBFOsmStreamSource.MoveNext: ~14 minutes
- OsmSharp.Streams.PBFOsmStreamSource.MoveToNextPrimitive: ~ 9 minutes
- OsmSharp.IO.PBF.PBFReader.MoveNext: ~7 minutes
- I believe this is the biggest issue.
- OsmSharp.IO.PBF.PBFReader.MoveNext: ~7 minutes
- OsmSharp.IO.PBF.Encoder.DecodeNode: ~2.5 minutes
- OsmSharp.IO.PBF.Encoder.DecodeWay: ~2 minutes
- OsmSharp.Streams.PBFOsmStreamSource.MoveToNextPrimitive: ~ 9 minutes
(Click to expand)
<Report>
<Function Id="0x00200001" FQN="OsmSandbox.Program.Main" TotalTime="853211" OwnTime="3654" Samples="142205" Instances="1" />
<Function Id="0x00400285" FQN="OsmSharp.IO.PBF.DenseInfo..ctor" TotalTime="90" OwnTime="24" Samples="15" Instances="1" />
<Function Id="0x004002A8" FQN="OsmSharp.IO.PBF.DenseNodes..ctor" TotalTime="12" OwnTime="6.0" Samples="2" Instances="1" />
<Function Id="0x00400223" FQN="OsmSharp.IO.PBF.Encoder.Decode" TotalTime="85647" OwnTime="33119" Samples="14275" Instances="2">
<Instance TotalTime="85629" OwnTime="33119" Samples="14272" />
<Instance TotalTime="18" OwnTime="0" Samples="3" />
</Function>
<Function Id="0x00400232" FQN="OsmSharp.IO.PBF.Encoder.DecodeLatLon" TotalTime="3935" OwnTime="3935" Samples="656" Instances="1" />
<Function Id="0x00400226" FQN="OsmSharp.IO.PBF.Encoder.DecodeNode" TotalTime="140975" OwnTime="33719" Samples="23500" Instances="2">
<Instance TotalTime="140933" OwnTime="33713" Samples="23493" />
<Instance TotalTime="42" OwnTime="6.0" Samples="7" />
</Function>
<Function Id="0x0040022E" FQN="OsmSharp.IO.PBF.Encoder.DecodeRelation" TotalTime="2010" OwnTime="288" Samples="335" Instances="1" />
<Function Id="0x0040022B" FQN="OsmSharp.IO.PBF.Encoder.DecodeWay" TotalTime="108122" OwnTime="11030" Samples="18022" Instances="1" />
<Function Id="0x004002D5" FQN="OsmSharp.IO.PBF.InputStream.get_CanRead" TotalTime="6.0" OwnTime="6.0" Samples="1" Instances="1" />
<Function Id="0x004002CD" FQN="OsmSharp.IO.PBF.InputStream.Read" TotalTime="179183" OwnTime="294" Samples="29863" Instances="24">
<Instance TotalTime="3501" OwnTime="24" Samples="584" />
<Instance TotalTime="6.0" OwnTime="0" Samples="1" />
<Instance TotalTime="2094" OwnTime="0" Samples="349" />
<Instance TotalTime="5880" OwnTime="6.0" Samples="980" />
<Instance TotalTime="15246" OwnTime="12" Samples="2541" />
<Instance TotalTime="647" OwnTime="0" Samples="108" />
<Instance TotalTime="864" OwnTime="0" Samples="144" />
<Instance TotalTime="10074" OwnTime="42" Samples="1679" />
<Instance TotalTime="12" OwnTime="0" Samples="2" />
<Instance TotalTime="2988" OwnTime="6.0" Samples="498" />
<Instance TotalTime="7374" OwnTime="6.0" Samples="1229" />
<Instance TotalTime="96281" OwnTime="132" Samples="16046" />
<Instance TotalTime="798" OwnTime="6.0" Samples="133" />
<Instance TotalTime="5605" OwnTime="0" Samples="934" />
<Instance TotalTime="720" OwnTime="0" Samples="120" />
<Instance TotalTime="4889" OwnTime="18" Samples="815" />
<Instance TotalTime="14539" OwnTime="24" Samples="2423" />
<Instance TotalTime="7585" OwnTime="18" Samples="1264" />
<Instance TotalTime="18" OwnTime="0" Samples="3" />
<Instance TotalTime="6.0" OwnTime="0" Samples="1" />
<Instance TotalTime="12" OwnTime="0" Samples="2" />
<Instance TotalTime="6.0" OwnTime="0" Samples="1" />
<Instance TotalTime="12" OwnTime="0" Samples="2" />
<Instance TotalTime="24" OwnTime="0" Samples="4" />
</Function>
<Function Id="0x004002DB" FQN="OsmSharp.IO.PBF.LimitedStream..ctor" TotalTime="12" OwnTime="12" Samples="2" Instances="1" />
<Function Id="0x004002DC" FQN="OsmSharp.IO.PBF.LimitedStream.ReadNextBlock" TotalTime="3567" OwnTime="6.0" Samples="595" Instances="3">
<Instance TotalTime="3477" OwnTime="6.0" Samples="580" />
<Instance TotalTime="6.0" OwnTime="0" Samples="1" />
<Instance TotalTime="84" OwnTime="0" Samples="14" />
</Function>
<Function Id="0x0040029C" FQN="OsmSharp.IO.PBF.Node..ctor" TotalTime="27601" OwnTime="13188" Samples="4600" Instances="2">
<Instance TotalTime="27589" OwnTime="13188" Samples="4598" />
<Instance TotalTime="12" OwnTime="0" Samples="2" />
</Function>
<Function Id="0x004002C5" FQN="OsmSharp.IO.PBF.PBFExtensions.FromUnixTime" TotalTime="2514" OwnTime="1543" Samples="419" Instances="3">
<Instance TotalTime="2250" OwnTime="1380" Samples="375" />
<Instance TotalTime="258" OwnTime="156" Samples="43" />
<Instance TotalTime="6.0" OwnTime="6.0" Samples="1" />
</Function>
<Function Id="0x004002C8" FQN="OsmSharp.IO.PBF.PBFReader..ctor" TotalTime="169" OwnTime="0" Samples="28" Instances="1" />
<Function Id="0x004002CA" FQN="OsmSharp.IO.PBF.PBFReader.MoveNext" TotalTime="432090" OwnTime="132" Samples="72014" Instances="2">
<Instance TotalTime="431874" OwnTime="132" Samples="71978" />
<Instance TotalTime="216" OwnTime="0" Samples="36" />
</Function>
<Function Id="0x0040026E" FQN="OsmSharp.IO.PBF.PrimitiveGroup..ctor" TotalTime="30" OwnTime="30" Samples="5" Instances="1" />
<Function Id="0x004002B9" FQN="OsmSharp.IO.PBF.Relation..ctor" TotalTime="180" OwnTime="60" Samples="30" Instances="1" />
<Function Id="0x004002B0" FQN="OsmSharp.IO.PBF.Way..ctor" TotalTime="9613" OwnTime="2255" Samples="1602" Instances="1" />
<Function Id="0x004002DA" FQN="OsmSharp.IO.PBF.ZLibStreamWrapper.ReadNextBlock" TotalTime="252" OwnTime="180" Samples="42" Instances="13">
<Instance TotalTime="12" OwnTime="12" Samples="2" />
<Instance TotalTime="12" OwnTime="12" Samples="2" />
<Instance TotalTime="6.0" OwnTime="6.0" Samples="1" />
<Instance TotalTime="12" OwnTime="12" Samples="2" />
<Instance TotalTime="72" OwnTime="72" Samples="12" />
<Instance TotalTime="12" OwnTime="12" Samples="2" />
<Instance TotalTime="18" OwnTime="0" Samples="3" />
<Instance TotalTime="36" OwnTime="36" Samples="6" />
<Instance TotalTime="18" OwnTime="18" Samples="3" />
<Instance TotalTime="12" OwnTime="0" Samples="2" />
<Instance TotalTime="6.0" OwnTime="0" Samples="1" />
<Instance TotalTime="24" OwnTime="0" Samples="4" />
<Instance TotalTime="12" OwnTime="0" Samples="2" />
</Function>
<Function Id="0x004001FA" FQN="OsmSharp.IO.Zip.Checksum.Adler32.Update" TotalTime="18198" OwnTime="18198" Samples="3033" Instances="16">
<Instance TotalTime="198" OwnTime="198" Samples="33" />
<Instance TotalTime="593" OwnTime="593" Samples="99" />
<Instance TotalTime="120" OwnTime="120" Samples="20" />
<Instance TotalTime="1861" OwnTime="1861" Samples="310" />
<Instance TotalTime="84" OwnTime="84" Samples="14" />
<Instance TotalTime="78" OwnTime="78" Samples="13" />
<Instance TotalTime="1380" OwnTime="1380" Samples="230" />
<Instance TotalTime="84" OwnTime="84" Samples="14" />
<Instance TotalTime="7741" OwnTime="7741" Samples="1290" />
<Instance TotalTime="780" OwnTime="780" Samples="130" />
<Instance TotalTime="246" OwnTime="246" Samples="41" />
<Instance TotalTime="2435" OwnTime="2435" Samples="406" />
<Instance TotalTime="1853" OwnTime="1853" Samples="309" />
<Instance TotalTime="96" OwnTime="96" Samples="16" />
<Instance TotalTime="630" OwnTime="630" Samples="105" />
<Instance TotalTime="18" OwnTime="18" Samples="3" />
</Function>
<Function Id="0x00400171" FQN="OsmSharp.IO.Zip.DeflaterHuffman.BitReverse" TotalTime="720" OwnTime="720" Samples="120" Instances="21">
<Instance TotalTime="6.0" OwnTime="6.0" Samples="1" />
<Instance TotalTime="6.0" OwnTime="6.0" Samples="1" />
<Instance TotalTime="12" OwnTime="12" Samples="2" />
<Instance TotalTime="168" OwnTime="168" Samples="28" />
<Instance TotalTime="18" OwnTime="18" Samples="3" />
<Instance TotalTime="6.0" OwnTime="6.0" Samples="1" />
<Instance TotalTime="6.0" OwnTime="6.0" Samples="1" />
<Instance TotalTime="5.9" OwnTime="5.9" Samples="1" />
<Instance TotalTime="24" OwnTime="24" Samples="4" />
<Instance TotalTime="360" OwnTime="360" Samples="60" />
<Instance TotalTime="18" OwnTime="18" Samples="3" />
<Instance TotalTime="6.0" OwnTime="6.0" Samples="1" />
<Instance TotalTime="6.0" OwnTime="6.0" Samples="1" />
<Instance TotalTime="6.0" OwnTime="6.0" Samples="1" />
<Instance TotalTime="6.1" OwnTime="6.1" Samples="1" />
<Instance TotalTime="36" OwnTime="36" Samples="6" />
<Instance TotalTime="6.0" OwnTime="6.0" Samples="1" />
<Instance TotalTime="6.0" OwnTime="6.0" Samples="1" />
<Instance TotalTime="6.0" OwnTime="6.0" Samples="1" />
<Instance TotalTime="6.0" OwnTime="6.0" Samples="1" />
<Instance TotalTime="6.0" OwnTime="6.0" Samples="1" />
</Function>
<Function Id="0x00400176" FQN="OsmSharp.IO.Zip.Inflater..ctor" TotalTime="1344" OwnTime="270" Samples="224" Instances="1" />
<Function Id="0x0040017C" FQN="OsmSharp.IO.Zip.Inflater.Decode" TotalTime="6605" OwnTime="192" Samples="1101" Instances="18">
<Instance TotalTime="30" OwnTime="5.9" Samples="5" />
<Instance TotalTime="1962" OwnTime="36" Samples="327" />
<Instance TotalTime="96" OwnTime="0" Samples="16" />
<Instance TotalTime="12" OwnTime="0" Samples="2" />
<Instance TotalTime="234" OwnTime="24" Samples="39" />
<Instance TotalTime="30" OwnTime="0" Samples="5" />
<Instance TotalTime="180" OwnTime="0" Samples="30" />
<Instance TotalTime="3714" OwnTime="126" Samples="619" />
<Instance TotalTime="96" OwnTime="0" Samples="16" />
<Instance TotalTime="18" OwnTime="0" Samples="3" />
<Instance TotalTime="12" OwnTime="0" Samples="2" />
<Instance TotalTime="42" OwnTime="0" Samples="7" />
<Instance TotalTime="138" OwnTime="0" Samples="23" />
<Instance TotalTime="12" OwnTime="0" Samples="2" />
<Instance TotalTime="6.0" OwnTime="0" Samples="1" />
<Instance TotalTime="6.0" OwnTime="0" Samples="1" />
<Instance TotalTime="12" OwnTime="0" Samples="2" />
<Instance TotalTime="6.0" OwnTime="0" Samples="1" />
</Function>
<Function Id="0x00400178" FQN="OsmSharp.IO.Zip.Inflater.DecodeHeader" TotalTime="18" OwnTime="12" Samples="3" Instances="1" />
<Function Id="0x0040017A" FQN="OsmSharp.IO.Zip.Inflater.DecodeHuffman" TotalTime="147766" OwnTime="29196" Samples="24626" Instances="22">
<Instance TotalTime="1842" OwnTime="504" Samples="307" />
<Instance TotalTime="12943" OwnTime="3018" Samples="2157" />
<Instance TotalTime="5101" OwnTime="1428" Samples="850" />
<Instance TotalTime="569" OwnTime="126" Samples="95" />
<Instance TotalTime="744" OwnTime="204" Samples="124" />
<Instance TotalTime="7674" OwnTime="1902" Samples="1279" />
<Instance TotalTime="12" OwnTime="12" Samples="2" />
<Instance TotalTime="2670" OwnTime="774" Samples="445" />
<Instance TotalTime="6247" OwnTime="1717" Samples="1041" />
<Instance TotalTime="4009" OwnTime="1020" Samples="668" />
<Instance TotalTime="83326" OwnTime="14003" Samples="13887" />
<Instance TotalTime="696" OwnTime="204" Samples="116" />
<Instance TotalTime="6697" OwnTime="1789" Samples="1116" />
<Instance TotalTime="2796" OwnTime="306" Samples="466" />
<Instance TotalTime="11798" OwnTime="2058" Samples="1966" />
<Instance TotalTime="600" OwnTime="126" Samples="100" />
<Instance TotalTime="6.0" OwnTime="0" Samples="1" />
<Instance TotalTime="6.0" OwnTime="6.0" Samples="1" />
<Instance TotalTime="6.0" OwnTime="0" Samples="1" />
<Instance TotalTime="6.0" OwnTime="0" Samples="1" />
<Instance TotalTime="12" OwnTime="0" Samples="2" />
<Instance TotalTime="6.0" OwnTime="0" Samples="1" />
</Function>
<Function Id="0x00400182" FQN="OsmSharp.IO.Zip.Inflater.Inflate" TotalTime="174380" OwnTime="258" Samples="29062" Instances="22">
<Instance TotalTime="2082" OwnTime="0" Samples="347" />
<Instance TotalTime="15060" OwnTime="6.0" Samples="2510" />
<Instance TotalTime="5874" OwnTime="12" Samples="979" />
<Instance TotalTime="647" OwnTime="0" Samples="108" />
<Instance TotalTime="846" OwnTime="0" Samples="141" />
<Instance TotalTime="9984" OwnTime="36" Samples="1664" />
<Instance TotalTime="12" OwnTime="0" Samples="2" />
<Instance TotalTime="2976" OwnTime="0" Samples="496" />
<Instance TotalTime="7302" OwnTime="6.0" Samples="1217" />
<Instance TotalTime="5569" OwnTime="12" Samples="928" />
<Instance TotalTime="95609" OwnTime="174" Samples="15934" />
<Instance TotalTime="786" OwnTime="0" Samples="131" />
<Instance TotalTime="7543" OwnTime="6.0" Samples="1257" />
<Instance TotalTime="4841" OwnTime="0" Samples="807" />
<Instance TotalTime="14455" OwnTime="0" Samples="2409" />
<Instance TotalTime="720" OwnTime="0" Samples="120" />
<Instance TotalTime="18" OwnTime="6.0" Samples="3" />
<Instance TotalTime="6.0" OwnTime="0" Samples="1" />
<Instance TotalTime="6.0" OwnTime="0" Samples="1" />
<Instance TotalTime="6.0" OwnTime="0" Samples="1" />
<Instance TotalTime="12" OwnTime="0" Samples="2" />
<Instance TotalTime="24" OwnTime="0" Samples="4" />
</Function>
<Function Id="0x00400180" FQN="OsmSharp.IO.Zip.Inflater.SetInput" TotalTime="18" OwnTime="0" Samples="3" Instances="3">
<Instance TotalTime="6.0" OwnTime="0" Samples="1" />
<Instance TotalTime="6.0" OwnTime="0" Samples="1" />
<Instance TotalTime="6.0" OwnTime="0" Samples="1" />
</Function>
<Function Id="0x0040018D" FQN="OsmSharp.IO.Zip.InflaterDynHeader.BuildDistTree" TotalTime="984" OwnTime="12" Samples="164" Instances="9">
<Instance TotalTime="6.0" OwnTime="0" Samples="1" />
<Instance TotalTime="6.0" OwnTime="0" Samples="1" />
<Instance TotalTime="258" OwnTime="6.0" Samples="43" />
<Instance TotalTime="48" OwnTime="0" Samples="8" />
<Instance TotalTime="18" OwnTime="0" Samples="3" />
<Instance TotalTime="582" OwnTime="6.0" Samples="97" />
<Instance TotalTime="36" OwnTime="0" Samples="6" />
<Instance TotalTime="12" OwnTime="0" Samples="2" />
<Instance TotalTime="18" OwnTime="0" Samples="3" />
</Function>
<Function Id="0x0040018C" FQN="OsmSharp.IO.Zip.InflaterDynHeader.BuildLitLenTree" TotalTime="2760" OwnTime="84" Samples="460" Instances="13">
<Instance TotalTime="12" OwnTime="0" Samples="2" />
<Instance TotalTime="60" OwnTime="6.0" Samples="10" />
<Instance TotalTime="834" OwnTime="24" Samples="139" />
<Instance TotalTime="78" OwnTime="12" Samples="13" />
<Instance TotalTime="6.0" OwnTime="0" Samples="1" />
<Instance TotalTime="42" OwnTime="0" Samples="7" />
<Instance TotalTime="1530" OwnTime="42" Samples="255" />
<Instance TotalTime="90" OwnTime="0" Samples="15" />
<Instance TotalTime="18" OwnTime="0" Samples="3" />
<Instance TotalTime="6.0" OwnTime="0" Samples="1" />
<Instance TotalTime="6.0" OwnTime="0" Samples="1" />
<Instance TotalTime="6.0" OwnTime="0" Samples="1" />
<Instance TotalTime="72" OwnTime="0" Samples="12" />
</Function>
<Function Id="0x0040018B" FQN="OsmSharp.IO.Zip.InflaterDynHeader.Decode" TotalTime="2609" OwnTime="702" Samples="435" Instances="14">
<Instance TotalTime="6.0" OwnTime="0" Samples="1" />
<Instance TotalTime="24" OwnTime="6.0" Samples="4" />
<Instance TotalTime="827" OwnTime="246" Samples="138" />
<Instance TotalTime="84" OwnTime="18" Samples="14" />
<Instance TotalTime="6.0" OwnTime="0" Samples="1" />
<Instance TotalTime="36" OwnTime="12" Samples="6" />
<Instance TotalTime="1470" OwnTime="390" Samples="245" />
<Instance TotalTime="48" OwnTime="6.0" Samples="8" />
<Instance TotalTime="12" OwnTime="6.0" Samples="2" />
<Instance TotalTime="24" OwnTime="0" Samples="4" />
<Instance TotalTime="6.0" OwnTime="6.0" Samples="1" />
<Instance TotalTime="12" OwnTime="0" Samples="2" />
<Instance TotalTime="48" OwnTime="12" Samples="8" />
<Instance TotalTime="6.0" OwnTime="0" Samples="1" />
</Function>
<Function Id="0x00400190" FQN="OsmSharp.IO.Zip.InflaterHuffmanTree..cctor" TotalTime="6.0" OwnTime="0" Samples="1" Instances="1" />
<Function Id="0x00400191" FQN="OsmSharp.IO.Zip.InflaterHuffmanTree..ctor" TotalTime="6.0" OwnTime="0" Samples="1" Instances="1" />
<Function Id="0x00400192" FQN="OsmSharp.IO.Zip.InflaterHuffmanTree.BuildTree" TotalTime="4289" OwnTime="2256" Samples="715" Instances="31">
<Instance TotalTime="6.0" OwnTime="0" Samples="1" />
<Instance TotalTime="12" OwnTime="6.0" Samples="2" />
<Instance TotalTime="246" OwnTime="156" Samples="41" />
<Instance TotalTime="732" OwnTime="396" Samples="122" />
<Instance TotalTime="6.0" OwnTime="0" Samples="1" />
<Instance TotalTime="54" OwnTime="36" Samples="9" />
<Instance TotalTime="12" OwnTime="12" Samples="2" />
<Instance TotalTime="282" OwnTime="96" Samples="47" />
<Instance TotalTime="6.0" OwnTime="6.0" Samples="1" />
<Instance TotalTime="48" OwnTime="18" Samples="8" />
<Instance TotalTime="66" OwnTime="30" Samples="11" />
<Instance TotalTime="36" OwnTime="18" Samples="6" />
<Instance TotalTime="432" OwnTime="240" Samples="72" />
<Instance TotalTime="12" OwnTime="6.0" Samples="2" />
<Instance TotalTime="42" OwnTime="24" Samples="7" />
<Instance TotalTime="18" OwnTime="12" Samples="3" />
<Instance TotalTime="36" OwnTime="18" Samples="6" />
<Instance TotalTime="12" OwnTime="12" Samples="2" />
<Instance TotalTime="18" OwnTime="12" Samples="3" />
<Instance TotalTime="552" OwnTime="210" Samples="92" />
<Instance TotalTime="1422" OwnTime="792" Samples="237" />
<Instance TotalTime="90" OwnTime="72" Samples="15" />
<Instance TotalTime="6.0" OwnTime="0" Samples="1" />
<Instance TotalTime="72" OwnTime="54" Samples="12" />
<Instance TotalTime="18" OwnTime="6.0" Samples="3" />
<Instance TotalTime="12" OwnTime="6.0" Samples="2" />
<Instance TotalTime="6.0" OwnTime="6.0" Samples="1" />
<Instance TotalTime="12" OwnTime="0" Samples="2" />
<Instance TotalTime="6.0" OwnTime="0" Samples="1" />
<Instance TotalTime="12" OwnTime="12" Samples="2" />
<Instance TotalTime="6.0" OwnTime="0" Samples="1" />
</Function>
<Function Id="0x00400193" FQN="OsmSharp.IO.Zip.InflaterHuffmanTree.GetSymbol" TotalTime="76304" OwnTime="36616" Samples="12717" Instances="32">
<Instance TotalTime="683" OwnTime="336" Samples="114" />
<Instance TotalTime="1951" OwnTime="943" Samples="325" />
<Instance TotalTime="6.0" OwnTime="0" Samples="1" />
<Instance TotalTime="4321" OwnTime="1968" Samples="720" />
<Instance TotalTime="228" OwnTime="84" Samples="38" />
<Instance TotalTime="6.0" OwnTime="0" Samples="1" />
<Instance TotalTime="210" OwnTime="66" Samples="35" />
<Instance TotalTime="342" OwnTime="168" Samples="57" />
<Instance TotalTime="3389" OwnTime="1727" Samples="565" />
<Instance TotalTime="6.0" OwnTime="0" Samples="1" />
<Instance TotalTime="1068" OwnTime="474" Samples="178" />
<Instance TotalTime="2736" OwnTime="1248" Samples="456" />
<Instance TotalTime="30" OwnTime="24" Samples="5" />
<Instance TotalTime="306" OwnTime="108" Samples="51" />
<Instance TotalTime="50059" OwnTime="24183" Samples="8343" />
<Instance TotalTime="1692" OwnTime="792" Samples="282" />
<Instance TotalTime="3007" OwnTime="1465" Samples="501" />
<Instance TotalTime="4080" OwnTime="1926" Samples="680" />
<Instance TotalTime="6.0" OwnTime="0" Samples="1" />
<Instance TotalTime="18" OwnTime="6.0" Samples="3" />
<Instance TotalTime="1308" OwnTime="660" Samples="218" />
<Instance TotalTime="252" OwnTime="96" Samples="42" />
<Instance TotalTime="510" OwnTime="282" Samples="85" />
<Instance TotalTime="12" OwnTime="12" Samples="2" />
<Instance TotalTime="12" OwnTime="6.0" Samples="2" />
<Instance TotalTime="24" OwnTime="12" Samples="4" />
<Instance TotalTime="12" OwnTime="6.0" Samples="2" />
<Instance TotalTime="6.0" OwnTime="6.0" Samples="1" />
<Instance TotalTime="6.0" OwnTime="6.0" Samples="1" />
<Instance TotalTime="6.0" OwnTime="6.0" Samples="1" />
<Instance TotalTime="6.0" OwnTime="0" Samples="1" />
<Instance TotalTime="6.0" OwnTime="6.0" Samples="1" />
</Function>
<Function Id="0x004001BB" FQN="OsmSharp.IO.Zip.Streams.InflaterInputBuffer..ctor" TotalTime="396" OwnTime="54" Samples="66" Instances="1" />
<Function Id="0x004001C3" FQN="OsmSharp.IO.Zip.Streams.InflaterInputBuffer.Fill" TotalTime="540" OwnTime="66" Samples="90" Instances="12">
<Instance TotalTime="6.0" OwnTime="0" Samples="1" />
<Instance TotalTime="48" OwnTime="12" Samples="8" />
<Instance TotalTime="18" OwnTime="6.0" Samples="3" />
<Instance TotalTime="12" OwnTime="0" Samples="2" />
<Instance TotalTime="12" OwnTime="0" Samples="2" />
<Instance TotalTime="6.0" OwnTime="0" Samples="1" />
<Instance TotalTime="342" OwnTime="36" Samples="57" />
<Instance TotalTime="42" OwnTime="0" Samples="7" />
<Instance TotalTime="6.0" OwnTime="0" Samples="1" />
<Instance TotalTime="18" OwnTime="6.0" Samples="3" />
<Instance TotalTime="6.0" OwnTime="6.0" Samples="1" />
<Instance TotalTime="24" OwnTime="0" Samples="4" />
</Function>
<Function Id="0x004001C2" FQN="OsmSharp.IO.Zip.Streams.InflaterInputBuffer.SetInflaterInput" TotalTime="24" OwnTime="6.0" Samples="4" Instances="3">
<Instance TotalTime="6.0" OwnTime="0" Samples="1" />
<Instance TotalTime="12" OwnTime="6.0" Samples="2" />
<Instance TotalTime="6.0" OwnTime="0" Samples="1" />
</Function>
<Function Id="0x004001CD" FQN="OsmSharp.IO.Zip.Streams.InflaterInputStream..ctor" TotalTime="450" OwnTime="48" Samples="75" Instances="1" />
<Function Id="0x004001D2" FQN="OsmSharp.IO.Zip.Streams.InflaterInputStream.Fill" TotalTime="570" OwnTime="30" Samples="95" Instances="12">
<Instance TotalTime="6.0" OwnTime="0" Samples="1" />
<Instance TotalTime="48" OwnTime="0" Samples="8" />
<Instance TotalTime="18" OwnTime="0" Samples="3" />
<Instance TotalTime="12" OwnTime="0" Samples="2" />
<Instance TotalTime="18" OwnTime="6.0" Samples="3" />
<Instance TotalTime="6.0" OwnTime="0" Samples="1" />
<Instance TotalTime="366" OwnTime="24" Samples="61" />
<Instance TotalTime="42" OwnTime="0" Samples="7" />
<Instance TotalTime="6.0" OwnTime="0" Samples="1" />
<Instance TotalTime="18" OwnTime="0" Samples="3" />
<Instance TotalTime="6.0" OwnTime="0" Samples="1" />
<Instance TotalTime="24" OwnTime="0" Samples="4" />
</Function>
<Function Id="0x004001DF" FQN="OsmSharp.IO.Zip.Streams.InflaterInputStream.Read" TotalTime="175142" OwnTime="168" Samples="29189" Instances="22">
<Instance TotalTime="2094" OwnTime="6.0" Samples="349" />
<Instance TotalTime="15138" OwnTime="24" Samples="2523" />
<Instance TotalTime="5874" OwnTime="0" Samples="979" />
<Instance TotalTime="647" OwnTime="0" Samples="108" />
<Instance TotalTime="858" OwnTime="0" Samples="143" />
<Instance TotalTime="10020" OwnTime="18" Samples="1670" />
<Instance TotalTime="12" OwnTime="0" Samples="2" />
<Instance TotalTime="2982" OwnTime="0" Samples="497" />
<Instance TotalTime="7356" OwnTime="12" Samples="1226" />
<Instance TotalTime="5593" OwnTime="0" Samples="932" />
<Instance TotalTime="96077" OwnTime="90" Samples="16012" />
<Instance TotalTime="792" OwnTime="0" Samples="132" />
<Instance TotalTime="7567" OwnTime="0" Samples="1261" />
<Instance TotalTime="4853" OwnTime="6.0" Samples="809" />
<Instance TotalTime="14479" OwnTime="6.0" Samples="2413" />
<Instance TotalTime="720" OwnTime="0" Samples="120" />
<Instance TotalTime="18" OwnTime="0" Samples="3" />
<Instance TotalTime="6.0" OwnTime="0" Samples="1" />
<Instance TotalTime="6.0" OwnTime="0" Samples="1" />
<Instance TotalTime="12" OwnTime="6.0" Samples="2" />
<Instance TotalTime="12" OwnTime="0" Samples="2" />
<Instance TotalTime="24" OwnTime="0" Samples="4" />
</Function>
<Function Id="0x004001E7" FQN="OsmSharp.IO.Zip.Streams.OutputWindow.CopyOutput" TotalTime="1572" OwnTime="1572" Samples="262" Instances="14">
<Instance TotalTime="12" OwnTime="12" Samples="2" />
<Instance TotalTime="72" OwnTime="72" Samples="12" />
<Instance TotalTime="12" OwnTime="12" Samples="2" />
<Instance TotalTime="180" OwnTime="180" Samples="30" />
<Instance TotalTime="6.0" OwnTime="6.0" Samples="1" />
<Instance TotalTime="72" OwnTime="72" Samples="12" />
<Instance TotalTime="5.9" OwnTime="5.9" Samples="1" />
<Instance TotalTime="654" OwnTime="654" Samples="109" />
<Instance TotalTime="90" OwnTime="90" Samples="15" />
<Instance TotalTime="30" OwnTime="30" Samples="5" />
<Instance TotalTime="180" OwnTime="180" Samples="30" />
<Instance TotalTime="180" OwnTime="180" Samples="30" />
<Instance TotalTime="6.0" OwnTime="6.0" Samples="1" />
<Instance TotalTime="72" OwnTime="72" Samples="12" />
</Function>
<Function Id="0x004001E2" FQN="OsmSharp.IO.Zip.Streams.OutputWindow.Repeat" TotalTime="25430" OwnTime="25406" Samples="4238" Instances="16">
<Instance TotalTime="468" OwnTime="468" Samples="78" />
<Instance TotalTime="1248" OwnTime="1248" Samples="208" />
<Instance TotalTime="4806" OwnTime="4806" Samples="801" />
<Instance TotalTime="1459" OwnTime="1453" Samples="243" />
<Instance TotalTime="138" OwnTime="138" Samples="23" />
<Instance TotalTime="174" OwnTime="174" Samples="29" />
<Instance TotalTime="931" OwnTime="925" Samples="155" />
<Instance TotalTime="84" OwnTime="84" Samples="14" />
<Instance TotalTime="7272" OwnTime="7266" Samples="1212" />
<Instance TotalTime="1182" OwnTime="1176" Samples="197" />
<Instance TotalTime="588" OwnTime="588" Samples="98" />
<Instance TotalTime="4819" OwnTime="4819" Samples="803" />
<Instance TotalTime="852" OwnTime="852" Samples="142" />
<Instance TotalTime="156" OwnTime="156" Samples="26" />
<Instance TotalTime="1248" OwnTime="1248" Samples="208" />
<Instance TotalTime="6.0" OwnTime="6.0" Samples="1" />
</Function>
<Function Id="0x004001E1" FQN="OsmSharp.IO.Zip.Streams.OutputWindow.SlowRepeat" TotalTime="174" OwnTime="174" Samples="29" Instances="5">
<Instance TotalTime="6.0" OwnTime="6.0" Samples="1" />
<Instance TotalTime="42" OwnTime="42" Samples="7" />
<Instance TotalTime="96" OwnTime="96" Samples="16" />
<Instance TotalTime="24" OwnTime="24" Samples="4" />
<Instance TotalTime="6.0" OwnTime="6.0" Samples="1" />
</Function>
<Function Id="0x004001E0" FQN="OsmSharp.IO.Zip.Streams.OutputWindow.Write" TotalTime="14537" OwnTime="14537" Samples="2422" Instances="15">
<Instance TotalTime="108" OwnTime="108" Samples="18" />
<Instance TotalTime="198" OwnTime="198" Samples="33" />
<Instance TotalTime="402" OwnTime="402" Samples="67" />
<Instance TotalTime="606" OwnTime="606" Samples="101" />
<Instance TotalTime="48" OwnTime="48" Samples="8" />
<Instance TotalTime="42" OwnTime="42" Samples="7" />
<Instance TotalTime="222" OwnTime="222" Samples="37" />
<Instance TotalTime="78" OwnTime="78" Samples="13" />
<Instance TotalTime="10893" OwnTime="10893" Samples="1815" />
<Instance TotalTime="414" OwnTime="414" Samples="69" />
<Instance TotalTime="174" OwnTime="174" Samples="29" />
<Instance TotalTime="570" OwnTime="570" Samples="95" />
<Instance TotalTime="288" OwnTime="288" Samples="48" />
<Instance TotalTime="42" OwnTime="42" Samples="7" />
<Instance TotalTime="450" OwnTime="450" Samples="75" />
</Function>
<Function Id="0x004001EB" FQN="OsmSharp.IO.Zip.Streams.StreamManipulator.DropBits" TotalTime="7980" OwnTime="7980" Samples="1330" Instances="21">
<Instance TotalTime="6.0" OwnTime="6.0" Samples="1" />
<Instance TotalTime="66" OwnTime="66" Samples="11" />
<Instance TotalTime="126" OwnTime="126" Samples="21" />
<Instance TotalTime="420" OwnTime="420" Samples="70" />
<Instance TotalTime="342" OwnTime="342" Samples="57" />
<Instance TotalTime="42" OwnTime="42" Samples="7" />
<Instance TotalTime="42" OwnTime="42" Samples="7" />
<Instance TotalTime="48" OwnTime="48" Samples="8" />
<Instance TotalTime="5310" OwnTime="5310" Samples="885" />
<Instance TotalTime="5.9" OwnTime="5.9" Samples="1" />
<Instance TotalTime="318" OwnTime="318" Samples="53" />
<Instance TotalTime="114" OwnTime="114" Samples="19" />
<Instance TotalTime="6.0" OwnTime="6.0" Samples="1" />
<Instance TotalTime="168" OwnTime="168" Samples="28" />
<Instance TotalTime="60" OwnTime="60" Samples="10" />
<Instance TotalTime="30" OwnTime="30" Samples="5" />
<Instance TotalTime="138" OwnTime="138" Samples="23" />
<Instance TotalTime="384" OwnTime="384" Samples="64" />
<Instance TotalTime="342" OwnTime="342" Samples="57" />
<Instance TotalTime="6.0" OwnTime="6.0" Samples="1" />
<Instance TotalTime="6.0" OwnTime="6.0" Samples="1" />
</Function>
<Function Id="0x004001EA" FQN="OsmSharp.IO.Zip.Streams.StreamManipulator.PeekBits" TotalTime="34732" OwnTime="34732" Samples="5788" Instances="42">
<Instance TotalTime="78" OwnTime="78" Samples="13" />
<Instance TotalTime="6.0" OwnTime="6.0" Samples="1" />
<Instance TotalTime="396" OwnTime="396" Samples="66" />
<Instance TotalTime="282" OwnTime="282" Samples="47" />
<Instance TotalTime="276" OwnTime="276" Samples="46" />
<Instance TotalTime="12" OwnTime="12" Samples="2" />
<Instance TotalTime="294" OwnTime="294" Samples="49" />
<Instance TotalTime="12" OwnTime="12" Samples="2" />
<Instance TotalTime="18" OwnTime="18" Samples="3" />
<Instance TotalTime="882" OwnTime="882" Samples="147" />
<Instance TotalTime="1932" OwnTime="1932" Samples="322" />
<Instance TotalTime="6.0" OwnTime="6.0" Samples="1" />
<Instance TotalTime="144" OwnTime="144" Samples="24" />
<Instance TotalTime="192" OwnTime="192" Samples="32" />
<Instance TotalTime="102" OwnTime="102" Samples="17" />
<Instance TotalTime="24" OwnTime="24" Samples="4" />
<Instance TotalTime="6.0" OwnTime="6.0" Samples="1" />
<Instance TotalTime="1056" OwnTime="1056" Samples="176" />
<Instance TotalTime="174" OwnTime="174" Samples="29" />
<Instance TotalTime="66" OwnTime="66" Samples="11" />
<Instance TotalTime="1314" OwnTime="1314" Samples="219" />
<Instance TotalTime="102" OwnTime="102" Samples="17" />
<Instance TotalTime="480" OwnTime="480" Samples="80" />
<Instance TotalTime="1171" OwnTime="1171" Samples="195" />
<Instance TotalTime="36" OwnTime="36" Samples="6" />
<Instance TotalTime="174" OwnTime="174" Samples="29" />
<Instance TotalTime="20561" OwnTime="20561" Samples="3426" />
<Instance TotalTime="150" OwnTime="150" Samples="25" />
<Instance TotalTime="18" OwnTime="18" Samples="3" />
<Instance TotalTime="198" OwnTime="198" Samples="33" />
<Instance TotalTime="6.0" OwnTime="6.0" Samples="1" />
<Instance TotalTime="732" OwnTime="732" Samples="122" />
<Instance TotalTime="24" OwnTime="24" Samples="4" />
<Instance TotalTime="12" OwnTime="12" Samples="2" />
<Instance TotalTime="510" OwnTime="510" Samples="85" />
<Instance TotalTime="1200" OwnTime="1200" Samples="200" />
<Instance TotalTime="168" OwnTime="168" Samples="28" />
<Instance TotalTime="6.0" OwnTime="6.0" Samples="1" />
<Instance TotalTime="126" OwnTime="126" Samples="21" />
<Instance TotalTime="1770" OwnTime="1770" Samples="295" />
<Instance TotalTime="6.0" OwnTime="6.0" Samples="1" />
<Instance TotalTime="12" OwnTime="12" Samples="2" />
</Function>
<Function Id="0x004001F3" FQN="OsmSharp.IO.Zip.Streams.StreamManipulator.SetInput" TotalTime="18" OwnTime="18" Samples="3" Instances="3">
<Instance TotalTime="6.0" OwnTime="6.0" Samples="1" />
<Instance TotalTime="6.0" OwnTime="6.0" Samples="1" />
<Instance TotalTime="6.0" OwnTime="6.0" Samples="1" />
</Function>
<Function Id="0x00400088" FQN="OsmSharp.Streams.OsmStreamSource.get_Current" TotalTime="804" OwnTime="804" Samples="134" Instances="1" />
<Function Id="0x00400086" FQN="OsmSharp.Streams.OsmStreamSource.GetEnumerator" TotalTime="6.0" OwnTime="6.0" Samples="1" Instances="1" />
<Function Id="0x0040007B" FQN="OsmSharp.Streams.OsmStreamSource.MoveNext" TotalTime="1375" OwnTime="912" Samples="229" Instances="1" />
<Function Id="0x004000B3" FQN="OsmSharp.Streams.PBFOsmStreamSource.Current" TotalTime="402" OwnTime="402" Samples="67" Instances="1" />
<Function Id="0x004000BA" FQN="OsmSharp.Streams.PBFOsmStreamSource.DeQueuePrimitive" TotalTime="8410" OwnTime="1861" Samples="1402" Instances="2">
<Instance TotalTime="8404" OwnTime="1861" Samples="1401" />
<Instance TotalTime="6.0" OwnTime="0" Samples="1" />
</Function>
<Function Id="0x004000B2" FQN="OsmSharp.Streams.PBFOsmStreamSource.MoveNext" TotalTime="847309" OwnTime="26392" Samples="141225" Instances="2">
<Instance TotalTime="846846" OwnTime="26392" Samples="141148" />
<Instance TotalTime="463" OwnTime="0" Samples="77" />
</Function>
<Function Id="0x004000B7" FQN="OsmSharp.Streams.PBFOsmStreamSource.MoveToNextPrimitive" TotalTime="538067" OwnTime="11859" Samples="89678" Instances="2">
<Instance TotalTime="537821" OwnTime="11853" Samples="89637" />
<Instance TotalTime="246" OwnTime="6.0" Samples="41" />
</Function>
<Function Id="0x004000BB" FQN="OsmSharp.Streams.PBFOsmStreamSource.ProcessNode" TotalTime="9131" OwnTime="2057" Samples="1522" Instances="1" />
<Function Id="0x004000BD" FQN="OsmSharp.Streams.PBFOsmStreamSource.ProcessRelation" TotalTime="6.0" OwnTime="0" Samples="1" Instances="1" />
<Function Id="0x004000BC" FQN="OsmSharp.Streams.PBFOsmStreamSource.ProcessWay" TotalTime="600" OwnTime="138" Samples="100" Instances="1" />
<Function Id="0x0040004C" FQN="OsmSharp.Tags.TagsCollection.AddOrReplace" TotalTime="11656" OwnTime="726" Samples="1943" Instances="3">
<Instance TotalTime="3773" OwnTime="84" Samples="629" />
<Instance TotalTime="7782" OwnTime="642" Samples="1297" />
<Instance TotalTime="102" OwnTime="0" Samples="17" />
</Function>
<Function Id="0x0040004F" FQN="OsmSharp.Tags.TagsCollection.TryGetValue" TotalTime="342" OwnTime="342" Samples="57" Instances="3">
<Instance TotalTime="90" OwnTime="90" Samples="15" />
<Instance TotalTime="240" OwnTime="240" Samples="40" />
<Instance TotalTime="12" OwnTime="12" Samples="2" />
</Function>
<Function Id="0x00400053" FQN="OsmSharp.Tags.TagsCollectionBase.Add" TotalTime="21838" OwnTime="1392" Samples="3640" Instances="3">
<Instance TotalTime="5159" OwnTime="234" Samples="860" />
<Instance TotalTime="16475" OwnTime="1146" Samples="2746" />
<Instance TotalTime="204" OwnTime="12" Samples="34" />
</Function>
</Report>
Steps to reproduce dotTrace output:
(Click to expand)
- Download dotTrace command line tools (free for command line, have to pay for fancy ide integration stuff) https://www.jetbrains.com/profiler/download/#section=commandline
.\ConsoleProfiler.exe start ..\Program.exe
.\Reporter.exe *.dtp --pattern=patterns.xml --save-to=result.xml
<Patterns> <Pattern>Main</Pattern> <Pattern>OsmSharp</Pattern> </Patterns>
I ran the profiling tool again, this time also looking at protobuf and the entire system namespace. OsmSharpDotTraceResults.zip
Thanks for testing this out. Performance hasn't been a major focus for OsmSharp for a while but with all the new features now available to use in .NET core there is a lot that can be improved.
My resources are limited to spend time on OsmSharp and for now I'm happy performance-wise but I have a list in my head of things we can do:
- Change the streaming model so that it doesn't have to create a new object for each OSM object it encounters.
- Spanify all IO and compression/decompression stuff.
- See if protobuf-net has new spanified features we can use.
- Use async everywhere.
- Figure out how to do this using netstandard2.1 while still supporting netstandard2.0.
@xivk Do you have ideas on how to implement those? Especially regarding the streaming model. I plan on using OsmSharp in a big project.
I was thinking about instead of using Current() to return an OsmGeo object we use properties on the OsmStreamSource class exposing the data (ID, changeset, user, nodes, members) OR (and this is risky) we mute the object we return in Current() forcing consumers to clone it if they want to keep it around.
Another option is to use an internal property that exposes an internal mutable object.
It would also be a good idea to refactor the nodes and members array into something we can reused. Now this is an array in both cases and we have to rebuild the array when the # of nodes or members change. It's also probably a good idea to define lightweight objects without meta-data, a node being just a lat/lon when that's all that is needed. Also the tags collections could use an update and a better approach with regards to performance.
Just to say, there is a lot to be done and considered here, I can help you get started, but I believe we should start with some experimentation before making design decisions just to confirm we are fixing the right things.
Multicore support? I hadn't considered that yet but it could be an option to read pbf data into a cache and use that to enumerate the OSM objects.
Some more info on this, I looked into this, the main effort is in reading the PBF blocks (obviously).
From what I can see about how protobuf-net works, our only option to improve this is to write our own PBF reader/writer specifically for OSM data, customized and optimized.
I guess this requires a lot of effort. I am not that familiar with protobuf-net. So the "easiest" part is reduce allocations and other operations slowing down the process?
The OSM PBF format is a bit strange, it seems to use two steps, one with raw data encoded in a protobuf message that then again is a protobuf message.
I think the biggest is with the decoding for the Blob message, it allocates a new byte array on every message and then optionally decompresses it into yet another buffer. I think the byte array is just there in the stream. It would need some figuring out to see what protobuf does with a message like that. More info here:
https://wiki.openstreetmap.org/wiki/PBF_Format#Low_level_encoding
The relevant code is here:
https://github.com/OsmSharp/core/blob/develop/src/OsmSharp/IO/PBF/PBFReader.cs#L100
The second thing is the PrimitiveBlock message. I think we can probably decode that data without first dumping it into a list/array/stringtable.
We could also try to get in touch with @mgravell the author of the protobuf-net library, but I guess he has better things to do than help us read PBF OSM as fast as possible! ;-)
Hi! Thought this might be useful to dive into the internals : https://github.com/mapbox/osmpbf-tutorial
it allocates a new byte array on every message and then optionally decompresses it into yet another buffer. I think the byte array is just there in the stream. It would need some figuring out to see what protobuf does with a message like that.
Well the basic idea for a low-allocation reader would be:
- Read 4 bytes that say how big the upcoming
BlobHeaderis - Rent a buffer from a buffer pool that's at least that many bytes
- Read that many bytes into your rented buffer
- Use protobuf-net to interpret those bytes as a
BlobHeader(I think protobuf-net lets you reuse a single instance ofBlobHeaderand just overwrite its contents every time). - Return your rented buffer to the pool
- Rent another buffer from the buffer pool that's big enough for the size of the
Blobas listed in theBlobHeader. - Read that many bytes into your rented buffer
- Use a lower-level API from protobuf-net to walk through the fields in order to identify the section of your buffer that holds the payload.
- If the payload is compressed, then you should also know the size that it will be after decompressing it, from another field.
- If the data is compressed:
- Rent another buffer that's big enough to hold the uncompressed payload
- Decompress your data to the rented buffer, this is your new payload
- Return your old rented buffer since you don't need it anymore.
- Use protobuf-net to interpret your payload bytes as either a
PrimitiveBlockorHeaderBlock, depending on theBlobHeader's "type". - Return your payload's rented buffer to the pool.
- Do whatever processing you want on the
PrimitiveBlockorHeaderBlock, then go back to step 1.
The actual tricky part is getting a PrimitiveBlock not to allocate completely new object instances for its nested members every time you read a PrimitiveBlock. See https://stackoverflow.com/a/11966947/1083771 for a reasonable way to do this.
Another weird part is that we usually don't see many examples of compressing or decompressing from one buffer to another without allocating extra garbage each time we do so. Assuming you're using Ionic.Zlib, it's basically (warning: I haven't compiled or tested this exact version of this snippet, so it might be slightly broken):
static void InflateAssumingExactSize(this ZlibCodec inflater, ArraySegment<byte> compressed, ArraySegment<byte> decompressed)
{
inflater.InitializeInflate();
inflater.InputBuffer = compressed.Array;
inflater.NextIn = compressed.Offset;
inflater.AvailableBytesIn = compressed.Count;
inflater.OutputBuffer = decompressed.Array;
inflater.NextOut = decompressed.Offset;
inflater.AvailableBytesOut = decompressed.Count;
inflater.Inflate(FlushType.Finish);
}
// you only need one instance of this:
ZlibCodec inflater = new ZlibCodec(CompressionMode.Decompress);
// for each data block, usage is something like:
ArraySegment<byte> compressed = /* you should have this already. */;
int rawDataSize = /* you should have this already. */;
byte[] rentedDecompressedBuffer = ArrayPool<byte>.Shared.Rent(rawDataSize);
try
{
ArraySegment<byte> decompressed = new ArraySegment<byte>(rentedDecompressedBuffer, 0, rawDataSize);
inflater.InflateAssumingExactSize(compressed, decompressed);
DoStuffWith(decompressed);
}
finally
{
ArrayPool<byte>.Shared.Return(rentedDecompressedBuffer);
}
There's also something I came across when getting complete relations (and thus reading multiple times the same PBF) : It's hard to know where the Ways and Relations start. It would be great to get from the blob bytes the type of data coming (Nodes, Ways, Relations) without decompressing and deserializing the whole blob.
When getting relations in my first pass (not specific to osmsharp) I have to read and decompress every blob to know what the PrimitiveBlock is holding.
I will add https://github.com/dotnet/BenchmarkDotNet here just in case. One idea could be to first add this benchmarking library .NET repo also use and then benchmark the code to be refactored. This way there are quantfiable results and gradually increasing performance coverage.
