osm2pgsql-dev
Huge performance loss when using --extra-atributes
When importing data with osm2pgsql and the --extra-atributes switch is used the performance of reading the PBF file drops below a 10th of what it was than without using the extra-attributes switch.
Just for comparison:
- Before: Node 2994 k/s, Way 52,54 k/s, Relations 710/s
- After: Node: 122 k/s, Way 22,54 k/s, Realtions /s
I also tested the new C++ version from PL https://github.com/openstreetmap/osm2pgsql/pull/187 but having the same issue.
System is a Xeon-1231v3, 16GB Ram, SSD running Elementary OS (aka Ubuntu 14.10)
First guess - it's creating objects in the rendering tables for every single object, as they now have the extra attributes tags.
What's the full osm2pgsql command line you're using?
osm2pgsql --number-processes 8 -c -d $DATABASE -U $USER -p de_osm -C 12000 -S $BASE/config/osm2pgsql/commuter_simulation.style -x germany-latest.pbf
Style file: https://github.com/boerngen-schmidt/commuter-simulation/blob/master/config/osm2pgsql/commuter_simulation.style
With the PL #187 the import crashes, since it runs out of ram and swap.
With the C Version import seems to finish (still running). But it is painfully slow. Also I noticed on a smaller import (bremen-latest.pdf ~12MB) that analysing of the tables is much slower.
Any reason you can't use --hstore-match-only, or do you want every single object in the OSM file to be in the rendering tables, even if it has no real tags?
Well I had --hstore-match-only before and removed it due to not having any hstore column in my style anymore, so I thought removing anything that has to do with hstore will deactivate it.
So I added --hstore and --hstore-match-only to the above command, but performance is still bad. Importing the osm for Bremen (Germany) takes 17s without --extra-attributes and with 45s. (C-Version). With the C++-Version it takes 12s without and 43s with --extra-attributes
All I want to achieve is to have the osm_version tag in the DB to make an entry unique
Well I had --hstore-match-only before and removed it due to not having any hstore column in my style anymore, so I thought removing anything that has to do with hstore will deactivate it.
Oh, I misread the initial command.
Hmm. I'm surprised, do you get the same reduced performance in slim mode?
Slimmode (C++)
-
With -x
Processing: Node(1042k 173.8k/s) Way(196k 49.02k/s) Relation(4090 2045.00/s) parse time: 12s Node stats: total(1042778), max(3113420717) in 6s Way stats: total(196066), max(306438655) in 4s Relation stats: total(4099), max(4084730) in 2s
-
Without -x
Processing: Node(1042k 260.7k/s) Way(196k 65.36k/s) Relation(4090 2045.00/s) parse time: 9s Node stats: total(1042778), max(3113420717) in 4s Way stats: total(196066), max(306438655) in 3s Relation stats: total(4099), max(4084730) in 2s
Slim mode is slow anyways, so the difference is not much, but still just half
The times you posted are probably not statistically different - run to run variations will overwhelm any actual performance differences.
I know that these numbers are not statistically significant, but slim mode is slow anyways. I could do a 100 imports and see what happens then, but the point still remains, that using --extra-attributes has a huge performance impact.
And yet I still fail to see where our discussion is going. Yes the numbers I posted are just examples, but they still show that something in the code slows down the import process by a very big margin. I tried to read the code, but it is not very well commented and my C or C++ skills are not that good. So what I could understand from the code was:
- -x sets a flag in osmdata_t in osm2pgslq.c
- parse-pdf.c checks if this flag is set and adds user_id, version, timestamp (still todo) and changeset to the tags of the current processed object (node, way, relation)
What happens after this is a little foggy to me, so thats the part where one of the programmers of the software has to step in :-)
From your comments I understand, that Objects are only added if certain criteria are true. These are:
- Object has a tag that is in .style file
- other criteria
So if I now add osm_version to my style file all objects will fullfil the criteria to be added to the DB. A wild guess is, that the extra_attributes, once added to the tag list of the Object are considered for the criteria, but they should not be, since they are only extra attributes.
So if I now add osm_version to my style file all objects will fullfil the criteria to be added to the DB. A wild guess is, that the extra_attributes, once added to the tag list of the Object are considered for the criteria, but they should not be, since they are only extra attributes.
This is probably correct - except that nothing in the style is extra. Adding columns but not adding objects based on those columns isn't supported, but see #125.
Okay then how about first using the tags to consider if the Object should be added and after that add the extra_attributes as tags?
https://github.com/openstreetmap/osm2pgsql/blob/master/parse-pbf.c#L253 adds the extra_attributes to the tags https://github.com/openstreetmap/osm2pgsql/blob/master/parse-pbf.c#L257 adds all the tag tags to tags https://github.com/openstreetmap/osm2pgsql/blob/master/parse-pbf.c#L266 checks if we want the node
So placing L253 after the want node check would only add the extra attributes if the node is wanted
Update: So I tried placing the placing the check later and removing the tags from the style, but still performance is down. So it seems to be somewhere else in the code.
Okay I done further testing and narrowed down the performance impact to the processOsmDataDenseNodes function
Disableing the if statement at https://github.com/openstreetmap/osm2pgsql/blob/master/parse-pbf.c#L321 bumps up the import of nodes to around 650k/s
The Performance impact seems to start at https://github.com/openstreetmap/osm2pgsql/blob/master/parse-pbf.c#L318 where it starts to drop down to ~900k/s
As far as I could dig down the problem was, that it has something to do with the addItem function in https://github.com/openstreetmap/osm2pgsql/blob/master/keyvals.c#L223, because disableing all calls to addIntItem and addItem in the processOsmDataDenseNodes boosted the importspeed to around 2900k/s again.
Hope this helps you guy tracking down the problem
Googling brought up the following result http://www.codeproject.com/Articles/340797/Number-crunching-Why-you-should-never-ever-EVER-us which tells me that the LinkedList addItem is using is bad. Also I found something about memory pools, that allocate memory in advance to minimize the malloc calls performance impact.
The speed decrease probably isn't from slower osm2pgsql code, but from having to insert more items in the DB. See #125.
The only immediate workaround that comes to mind is to use -hstore-column osm_* instead of adding the columns to your style file.
If the slowness deep inside the PBF parsing as opposed to adding too many DB objects, it's doubtful anyone will get it to soon, though I would kind of like to rewrite the PBF parser post-C++ (#187) to make use of the C++ PBF library.
-O null might help identify the cause if you need to
This issue is very old, PBF parsing has moved to libosmium in the meantime, all code references mentioned in https://github.com/openstreetmap/osm2pgsql/issues/189#issuecomment-58274096 no longer exist.
Is this still relevant? Closing?
osmdata_t::{node,way,relation}_add only processes an object, if it has at least one tag. In case of nodes, more than 97% of all nodes don't have any tags and would be discarded right away. However, when providing the "Extra parameters" option, every single object needs to be processed.
Depending on the output, this much larger amount of data to be processed could make a difference: for the Flex backend, turning on extra attributes for a completely empty style file reduces throughput by about 50%.
I wonder if this should be reflected in docs/flex.md. At this time, the page says:
osm2pgsql.process_node(object): Called for each node.
There's no mentioning of a tag filter, or the role of the extra-attributes parameter. Its role is described as turning on/off meta data. That's unrelated to the question of the object having tags or not.