htm.core icon indicating copy to clipboard operation
htm.core copied to clipboard

Python pickle fails for Regions and Network

Open fcr opened this issue 5 years ago • 6 comments

I would like to multiprocess several parallel networks, in particular the location framework, but this fails on pickle errors for networks and regions. I added pickle tests to the Python bindings unit tests for region and network to check if the problem is me, or is the bindings. It seems it is the bindings unless I am missing something. I have attached the modified unit tests for your convenience.

test.zip

fcr avatar Oct 28 '19 12:10 fcr

Good catch.

Serialization of the network is performed by calling network.saveToFile(filename) or save(stream). However, from the Python side a pickle of a network should result in save( ) being called. The stream would need to be captured and copied into the pickle stream. All of this would need to be performed by the bindings. PyBindRegion has code to perform the pickle in the other direction (C++ serializing the python region), but not for a pickle starting on the python side.

For the region, we should have this as well... After a region has been deserialized it would need to be reconnected to the network before it could be used.

@fcr would you like to take on this project?

dkeeney avatar Oct 28 '19 13:10 dkeeney

@dkeeney if I was comfortable enough with C++ I would have been happy to. But it is way too far out of my comfort zone so I must decline.

fcr avatar Oct 28 '19 13:10 fcr

ok, let me see what I can do.

dkeeney avatar Oct 28 '19 13:10 dkeeney

Thanks. It would be much appreciated.

fcr avatar Oct 28 '19 13:10 fcr

BTW this is part of my efforts to get around the performance bottleneck of the ColumnPooler.

fcr avatar Oct 28 '19 13:10 fcr

I am able to reproduce your error:

======================================================================
ERROR: testAL246aCorticalColumnPickle (L46aNetwork_test.TestSimpleSPTMNetwork)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "J:\Projects\AI\htmA\py\tests\advanced\regions\L46aNetwork_test.py", line 117, in testAL246aCorticalColumnPickle
    network2 = pickle.loads(pickledColumn)
RuntimeError: CHECK FAILED: "srcOutput.get() != o" addLink -- link from region L4 output winnerCells to region L6a input anchorGrowthCandidates already exists

I don't know the cause yet but I will setup another PR to address this.

dkeeney avatar Nov 06 '19 15:11 dkeeney