Too much RAM usage on drc_rosetta.py

[hakan-demirli]

Description

https://github.com/The-OpenROAD-Project/OpenLane/blob/master/scripts/drc_rosetta.py Every one of these functions creates the whole database in RAM and write it back after it is finished. The problem is if the design is huge (more than 100k cells) the RAM consumption increase substantially. Especially tr_to_klayout at its peak use more than 60 GBs of RAM.

Proposal

Rather than saving the conversion object as a variable write each line directly to the file.

[donn]

tr_to_klayout is especially problematic since the entire XML object has to be constructed first before writing, which I know, sucks for RAM usage. Unfortunately, the KLayout DRC output may very well be the most important one- KLayout is the easiest of the tools to use.

The rest though are fairly easy. Thank you for reporting- if you have any ideas on how to improve the KLayout, please, by all means.

[hakan-demirli]

I see that a dictionary is used as a backend to store the processed data. A disk based dictionary can solve the problem. It may hurt the performance for the machines with 64GB+ RAM but it should be unnoticeable or slightly slower for small/medium designs and PCs with less RAM since the SWAP RAM is also on the disk.

An easy solution would be to use standard shelve library. It can be used as if it is a dictionary.

import shelve
my_dict = shelve.open('my_dict.db', writeback=True)
my_dict['key1'] = 'value1'
my_dict['key2'] = 'value2'
print(my_dict['key1'])  # Output: 'value1'

At the end of the tr_to_klayout the shelve database will be ready on the disk. Then we can call a function shelve_to_xml which reads shelve database in chunks converts them to xml and appends to the xml file. shelve_to_xml is a really simple function that can finish the task in less than a minute since there is no data dependency or random access. It is just a simple read modify write operation on a file which is less than a GB.

[donn]

To be precise, we are not in control of the backend being used. That's Python's XML library.

We'd have to move a library such as lxml for example, which IS doable. We'll investigate