Cannot deepcopy a pypowsybl network

[BDonnot]

• Do you want to request a feature or report a bug? This is both a "feature request" and an "unexpected behaviour python side" (that can be called "a bug")

• What is the current behavior? Once a network is loaded, for example with:

import pypowsybl as pp
net = pp.network.create_ieee14()

It is not possible to "pickle" it, preventing its use in multi processing framework, preventing also the copying of such grid etc.

import copy
copy.deepcopy(net)

And it failes with:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.8/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/usr/lib/python3.8/copy.py", line 270, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/lib/python3.8/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python3.8/copy.py", line 230, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python3.8/copy.py", line 161, in deepcopy
    rv = reductor(4)
TypeError: cannot pickle 'PyCapsule' object

• If the current behavior is a bug, please provide the steps to reproduce and if possible a minimal demo of the problem As mentionned above, you can:

import pypowsybl as pp
import copy
net = pp.network.create_ieee14()
copy.deepcopy(net)

• What is the expected behavior? It is expected that making a deepcopy of an object works. Sometimes it's not possible, but it's a great addition that such possibility is implemented.

• What is the motivation / use case for changing the behavior? Pickle is often use in case of multiprocessing, it is also interesting to save python object using this format and the vast majority of python framework expect manipulated objects to be pickleable.

It is relatively easy to set this up with pybind11 (https://pybind11.readthedocs.io/en/stable/advanced/classes.html#pickling-support, https://pybind11.readthedocs.io/en/stable/advanced/classes.html#deepcopy-support) or here https://github.com/BDonnot/lightsim2grid/blob/90cf65bfa26e610a2b836844d19bd651bea1b8ad/src/main.cpp#L158 for a concrete example.

I don't know however if having this feature pybind11 side is enough.

• Please tell us about your environment:
  • PowSyBl Version: ...
  • OS Version: ...

Current versions are:

Powsybl versions:
+-----------------------------+-----------------------+------------+------------------------------------------+-------------------------------+
| Repository name             | Maven project version | Git branch | Git version                              | Build timestamp               |
+-----------------------------+-----------------------+------------+------------------------------------------+-------------------------------+
| powsybl-open-loadflow       | 0.11.0                | UNKNOWN    | eb1f53612a03c0468a6156de89aacc4012674443 | 2021-05-27T09:50:07.767+02:00 |
| powsybl-single-line-diagram | 2.2.0                 | UNKNOWN    | b97b2d54108bad50fd2a9c1050bc0423f2499e86 | 2021-05-26T22:08:52.780+02:00 |
| powsybl-core                | 4.2.0                 | UNKNOWN    | 929bbaf93ff7bfb6481d04824c21e81b8800764f | 2021-05-25T11:14:38.408+02:00 |
| powsybl-rte-core            | 3.2.0                 | UNKNOWN    | 76c88c56886fd55b82bf0e76c8a2efeda55a75ce | 2021-05-27T11:29:00.743+02:00 |
+-----------------------------+-----------------------+------------+------------------------------------------+-------------------------------+

[geofjamg]

Indeed, it would makes sens to support this feature, but here is quite a lot of work in our case as we have to support pickle serialization to automatically support deep cloning. As far as I understand pickle something specific to Python and our internal data model is in Java. But it should be feasible.

[BDonnot]

If an object can be pickled, it can be (deep) copied.

But there are other ways to support deepcopy (with the implementation of the __copy__ and __deepcopy__ methods for example).

This might be easier.

Though, the support for pickling is often a good idea, for example to use the framework alongside python multiprocessing module for example.

This is not a problem at the moment, but I also think this would make sense to have this feature :-)

[sylvlecl]

I think it should work and be easy to implement by implementing pickling via XIIDM serialization.

Actually, so far, there is no other way to deep copy a network, even in the java lib.

[sylvlecl]

Implemented in #157

[vinault]

Hi !

I'm working on the pypowsybl grid2op backend and realized that the deepcopy behavior is probably not the one expected. See the mentionned issue above. Basically, the dump_to_string method (for the getstate method used by deepcopy) writes the xml with the load's order "messed up". See attached the file generated by the code below. ieee14.xiidm.txt

import pypowsybl.network as pn
network = pn.create_ieee14()
xml = network.dump_to_string()
with open("ieee14.xiidm", "w") as f:
    f.writelines(xml)

The load_network_from_string method then probably builds the get_loads DataFrame with the index sorted as in the file (I'm guessing here).

For the grid2op backend purpose, I temporarily "overloaded" the pypowsybl Network here to keep the loads original order in the DataFrame returned by get_loads

[AnkurArohi]

I just came across the same use case, what we implemented as a solution was to use a inmemory database and to simply store the network converterd to pickled bytes format (then zipped) in the databank with the index as case_Date. In this way it is possible by using an extra boolean setting in load function to store the network in database for reference.

There is another point that I would like to mention, I think pypowsybl also supports working varints, maybe you can stor ethe initial working variant as a reference.

[nicow-elia]

Though it works right now, a deepcopy operation seems to re-order some things (look at net2 busbars 5-9):

[geofjamg]

Though it works right now, a deepcopy operation seems to re-order some things (look at net2 busbars 5-9):

Fixed by https://github.com/powsybl/pypowsybl/pull/750