Subscribing to parsing progress events

[nyurik]

is it possible to track apply_file progress? The simplest way might be to add a progress(self, percentage, total) member function to the osmium.SimpleHandler, and if present, it would get called. Thanks!

[lonvia]

This is possible for a subset of uses for apply() but not exactly trivial. libosmium's internal apply() would need to be replaced with a custom processing function that loops over the buffers, calling a custom callback after processing the data in each buffer.

[wiktorn]

You can always track progress on your own. Here is osm_file_stats.py extended with tqdm:

"""
Simple example that counts the number of objects in an osm file.

Shows how to write a handler for the different types of objects.
"""
import osmium as o
import sys

import tqdm


class FileStatsHandler(o.SimpleHandler):
    def __init__(self, reporthook):
        super(FileStatsHandler, self).__init__()
        self.nodes = 0
        self.ways = 0
        self.rels = 0
        self.reporthook = reporthook

    def node(self, n):
        self.nodes += 1
        self.reporthook(1)

    def way(self, w):
        self.ways += 1
        self.reporthook(1)

    def relation(self, r):
        self.rels += 1
        self.reporthook(1)


if __name__ == '__main__':
    if len(sys.argv) != 2:
        print("Usage: python osm_file_stats.py <osmfile>")
       sys.exit(-1)
    with tqdm.tqdm() as pb:
        h = FileStatsHandler(pb.update)
        h.apply_file(sys.argv[1])

    print("Nodes: %d" % h.nodes)
    print("Ways: %d" % h.ways)
    print("Relations: %d" % h.rels)

However it gives ~50% slowdown (for my test file - from 210 seconds to 410 seconds with tqdm).

I'm not sure if osmium is aware how far in the file it has progressed.