Copy multiple object values -> RuntimeError: Node callback keeps reference to OSM object. This is not allowed.

[muran123]

Hi Sarah and Jochen

First of all thank you very much for the library you've build. I'm pretty new to OSM (and in general to geographic information). My goal it to extract certain entities e.g. libraries worldwide with the information about name, latitude, longitude, address etc. and later do it for other entities.

After trying https://overpass-turbo.eu/ and running into timeouts (I guess the query as too big) I thought I use the OSM extracts from http://download.geofabrik.de/ and process it with pyosmium locally. I'm running the code on Ubuntu 20.04, python 3.8.10 and use PyCharm 2022.2.

To get started I wanted to reuse the example for hotels to avoid the RuntimeError: Node callback keeps reference to OSM object. This is not allowed. described there https://github.com/osmcode/pyosmium/blob/master/doc/intro.rst Building upon the proposed solution

class HotelHandler(osmium.SimpleHandler):
    def __init__(self):
        super(HotelHandler, self).__init__()
        self.hotels = []

    def node(self, o):
        if o.tags.get('tourism') == 'hotel' and 'name' in o.tags:
            self.hotels.append(o.tags['name'])


h = HotelHandler()
h.apply_file(some_file)

print(sorted(h.hotels))

I adopted it to also capture the latitude and longitude (changes in the append() using a dict with multiple fields and also added parameters to apply_file() to get the coordinates):

class HotelHandler(osmium.SimpleHandler):
    def __init__(self):
        super(HotelHandler, self).__init__()
        self.hotels = []

    def node(self, o):
        if o.tags.get('tourism') == 'hotel' and 'name' in o.tags:
            self.hotels.append({'name': o.tags['name'], 'latitude': float(o.location.lat),
                                'longitude': float(o.location.lon)})

h = HotelHandler()
h.apply_file(some_file, locations=True, idx='flex_mem')

When I run this code with http://download.geofabrik.de/europe/germany-latest.osm.pbf I can process the first 2x hotels (Waldhotel Klaholz, Hotel-Restaurant Waldhotel zum Bergsee) and afterwards I get the exception RuntimeError: Node callback keeps reference to OSM object. This is not allowed. I'm rather new to python and deepcopy, but what I understood is that I should not append the object itself like explained in the intro.rst self.hotels.append(o) # THIS IS WRONG! itself, but the object values, which I tried. May I ask you to look into this and indicate, what is going wrong (because I would like to extend it as well to use even more tags like address and check if these other tags are available) and run it not only on nodes, but also ways)?

Thank you very much

[lonvia]

There is nothing wrong with the code and it works fine when I test it with a smaller Liechtenstein excerpt.

I don't know what exactly is going on here. Problem is that I'd need a reproducable case for these 'reference keeping' issues. But things seem to just creep up now and then on very different systems.

[waltherg]

Not sure this is helpful but running PyOsmium handlers in a Jupyter notebook I've noticed that when I start with a clean kernel (kernel restart) and run an apply_file I won't get this error. Once my cell has run and failed for some unrelated reason, I fix the issue (changing the handler class), then re-running this updated handler class with apply_file I'll pretty much always get RuntimeError: Node callback keeps reference to OSM object. This is not allowed.. Re-starting the notebook kernel and apply_file-running the updated handler class always works without the above error.

Actually, the handler doesn't need to fail first. Just re-running a handler on the same .osm.pbf file without restarting the kernel seems to produce this behaviour.

So my hunch is there's something stateful in apply_file or another component deeper down - maybe some file pointer or somesuch?

[lonvia]

I suspect that Jupyter is doing something evil in its exception handlers. It would indeed help to have a reproducible example there. Unfortunately, I did not manage to reproduce what you describe, @waltherg. If you are able to share a simple Jupyter notebook with steps to reproduce triggering the error, then that would be helpful.

[lonvia]

The new code without the reference check is now available as a pre-release. Install with pip install --pre osmium. Please test this and report if there are any regressions.