OSMnx generated Chinese road network program crashes, RAM is full, RAM is 180GB

[zy1992]

Contributing guidelines

• [x] I understand the contributing guidelines

Documentation

• [x] My problem is not addressed by the documentation or examples

Existing issues

• [x] My problem does not appear in an existing issue

What operating system and Python version are you using?

ubuntu 22.04 python 3.8

What OSMnx version are you using?

v 1.9.4

Environment packages and versions

~~~~

How did you install OSMnx?

Other (describe below)

Problem description

I hope OSMNX can convert the road network in the Chinese region into a network dataset

Complete minimal reproducible example

#!/usr/bin/env python
# coding:utf-8

# import osm2gmns as og

# net = og.getNetFromFile(r'C:\code\graphhopper\tw.osm')
# # 对网络中的复杂交叉口进行合并处理，以简化网络结构, 参数 auto_identify=True 表示自动识别复杂交叉口并进行合并
# og.consolidateComplexIntersections(net, auto_identify=True)

# # 构建多分辨率网络，用于支持不同精度级别的交通分析和模拟, 这将根据原始网络生成不同分辨率的网络版本，以便在不同精度要求下使用
# og.buildMultiResolutionNets(net)
# # 使用 outputNetToCSV 函数将网络导出为 CSV 文件
# og.outputNetToCSV(net)
# og.show(net)
# print('finish')
import osmnx as ox
import gc
# 分割路网->OSM  OSM->shp
ox.settings.use_cache = True
ox.settings.cache_folder = "./osmnx_cache"
ox.settings.all_oneway = True
ox.settings.log_console = True
ox.settings.useful_tags_way = ['fclass', 'code', 'osm_id', 'bridge', 'tunnel', 'oneway', 'ref', 'name', 'layer',
                               'maxspeed']
gc.collect()
print('----------'+ox.__version__+'----------')
G = ox.graph_from_xml('../data/gis_osm_roads_free_hainan.osm', bidirectional=False, simplify=True, retain_all=False)
"""
    Create a graph from data in a .osm formatted XML file.
    Parameters 
    ----------
    filepath : string or pathlib.Path
        path to file containing OSM XML data
    bidirectional : bool
        如果为True,则为单行道创建双向边
    simplify : bool
        如果为True,则使用`simplefy_graph`函数简化图拓扑
    retain_all : bool
        如果为True,则返回整个图,即使它没有连接。
        否则，只保留最大的弱连接组件。

    Returns
    -------
    G : networkx.MultiDiGraph
    """
# 构建节点和边集合
print('loading data success!')
gdf_nodes, gdf_edges = ox.graph_to_gdfs(G, nodes=True, edges=True, node_geometry=True, fill_edge_geometry=True)
# output shp
gdf_edges.to_file("../data/result/edges.shp", dirver="ESRI Shapefile", encoding='utf-8')
gdf_nodes.to_file("../data/result/nodes.shp", dirver="ESRI Shapefile", encoding='utf-8')
# output gpkg
# gdf_edges.to_file("./data/result/edges.gpkg", driver="GPKG")
# gdf_nodes.to_file("./data/result/nodes.gpkg", driver="GPKG")
print("Ok")

[gboeing]

Thanks for using OSMnx! If you have a usage question, please ask on StackOverflow as we reserve this issue tracker for proposing features and reporting bugs.

Your code example isn't reproducible, since it relies on external data files which we don't have. Thus we cannot reproduce, or in turn troubleshoot, what you have reported. And we have no way to guess what is in this data file or how large it is or many many nodes/edges it contains or at what level of detail you're modeling the road network.

However, the memory needs of constructing very large NetworkX graphs are well-documented. Additionally, there has been much discussion and advice regarding large (e.g., country-size) graphs in OSMnx (e.g., here, here, here, here, and here).

A couple other quick thoughts. You are using a Python version that reached end-of-life a year ago, and OSMnx v1. Note that OSMnx v2 introduced many memory efficiency improvements. Nonetheless, you face the fundamental trade-off discussed elsewhere:

There are inherent trade-offs when you want to model a large study area such as an entire region or an entire country: 1) model precision vs 2) area size vs 3) memory/speed. You need to trade off one of these three.

For the first, you can model a coarser-grained network, such as only major roads in the region/country, rather than millions of fine-grained residential streets and paths. For the second, you can study a smaller area. For the third, you can provision a machine with lots of memory and then let the script run for a while to complete the process. What you trade off will be up to your own needs for this analysis.

Finally, the OSMnx project is always open to enhancements. If you'd like to propose an specific enhancement to further improve memory efficiency, please open a new issue providing implementation details and memory/runtime benchmarks for your proposed solution.