pyarrow._plasma.PlasmaObjectExists: object already exists in the plasma store

[nsbruce]

In one terminal I start the plasma store

plasma_store -m 1000000000 -s /tmp/plasma

In another terminal I have

i=0
while True:
     brain['this']=int(datetime.datetime.now().timestamp())
     time.sleep(0.5)
     print(i)
     i += 1

in a third terminal I have

 i=0
while True:
     print(f"{i}: {brain['this']}")
     brain['that'] = brain['this']+1
    time.sleep(0.25)
     i+=1

And after a varying number of iterations I get one of two errors. Either:

1. the first loop throws pyarrow._plasma.PlasmaObjectExists: object already exists in the plasma store.

The full error is

PlasmaObjectExists                        Traceback (most recent call last)
<ipython-input-17-5973602b2a84> in <module>
      1 i=0
      2 while True:
----> 3     brain['this']=int(datetime.datetime.now().timestamp())
      4     time.sleep(0.5)
      5     print(i)

~/path/to/my/venv/lib/python3.9/site-packages/brain_plasma/brain.py in __setitem__(self, name, item)
     41     ##########################################################################################
     42     def __setitem__(self, name, item):
---> 43         self.learn(name, item)
     44 
     45     def __getitem__(self, name):

~/path/to/my/venv/lib/python3.9/site-packages/brain_plasma/brain.py in learn(self, name, thing, description)
    119             # REPLACE THE METADATA OBJECT
    120             self.client.delete([metadata_id])
--> 121             self.client.put(metadata, metadata_id)
    122 
    123             # (3)

~/path/to/my/venv/lib/python3.9/site-packages/brain_plasma/brain_client.py in put(self, *args, **kwargs)
     18 
     19     def put(self, *args, **kwargs):
---> 20         return self.client.put(*args, **kwargs)
     21 
     22     def get(self, *args, **kwargs):

~/path/to/my/venv/lib/python3.9/site-packages/pyarrow/_plasma.pyx in pyarrow._plasma.PlasmaClient.put()

~/path/to/my/venv/lib/python3.9/site-packages/pyarrow/_plasma.pyx in pyarrow._plasma.PlasmaClient.create()

~/path/to/my/venv/lib/python3.9/site-packages/pyarrow/_plasma.pyx in pyarrow._plasma.plasma_check_status()

PlasmaObjectExists: object already exists in the plasma store

In [18]: i=0
    ...: while True:
    ...:     brain['this']=int(datetime.datetime.now().timestamp())
    ...:     time.sleep(0.5)
    ...:     print(i)
    ...:     i += 1
    ...: 
---------------------------------------------------------------------------
PlasmaObjectExists                        Traceback (most recent call last)
<ipython-input-18-5973602b2a84> in <module>
      1 i=0
      2 while True:
----> 3     brain['this']=int(datetime.datetime.now().timestamp())
      4     time.sleep(0.5)
      5     print(i)

~/path/to/my/venv/lib/python3.9/site-packages/brain_plasma/brain.py in __setitem__(self, name, item)
     41     ##########################################################################################
     42     def __setitem__(self, name, item):
---> 43         self.learn(name, item)
     44 
     45     def __getitem__(self, name):

~/path/to/my/venv/lib/python3.9/site-packages/brain_plasma/brain.py in learn(self, name, thing, description)
    119             # REPLACE THE METADATA OBJECT
    120             self.client.delete([metadata_id])
--> 121             self.client.put(metadata, metadata_id)
    122 
    123             # (3)

~/path/to/my/venv/lib/python3.9/site-packages/brain_plasma/brain_client.py in put(self, *args, **kwargs)
     18 
     19     def put(self, *args, **kwargs):
---> 20         return self.client.put(*args, **kwargs)
     21 
     22     def get(self, *args, **kwargs):

~/path/to/my/venv/lib/python3.9/site-packages/pyarrow/_plasma.pyx in pyarrow._plasma.PlasmaClient.put()

~/path/to/my/venv/lib/python3.9/site-packages/pyarrow/_plasma.pyx in pyarrow._plasma.PlasmaClient.create()

~/path/to/my/venv/lib/python3.9/site-packages/pyarrow/_plasma.pyx in pyarrow._plasma.plasma_check_status()

PlasmaObjectExists: object already exists in the plasma store

or else

2. the second loop throws KeyError: 'Name this does not exist.'

The full error is

<ipython-input-22-2dfa8d4cfc6f> in <module>
      9 
     10     print(f"{i}: {brain['this']}")
---> 11     brain['that'] = brain['this']+1
     12     time.sleep(0.25)
     13     i+=1

~/path/to/my/venv/lib/python3.9/site-packages/brain_plasma/brain.py in __getitem__(self, name)
     44 
     45     def __getitem__(self, name):
---> 46         return self.recall(name)
     47 
     48     def __delitem__(self, name):

~/path/to/my/venv/lib/python3.9/site-packages/brain_plasma/brain.py in recall(self, name)
    153         """
    154         if not self.exists(name):
--> 155             raise KeyError(f"Name {name} does not exist.")
    156 
    157         metadata_id = self._name_to_namespace_hash(name)

KeyError: 'Name this does not exist.'

This is on macOS 11.2.3 (20D91) with python version 3.9.1.

[nsbruce]

I expect the second error occurs when the loop is reading 'this' while brain-plasma has just deleted 'this' in order to update it's value. I didn't realize that plasma's put() doesn't let you update values and so you have to delete/re-put. If that's the case this may not be solve-able at high refresh rates.

[nsbruce]

The first error may be caused by the same thing if brain-plasma calls are executed on separate threads. It could be putting the next iteration before the prior one has been properly deleted/re inserted.

[nsbruce]

Looks like the same error as you recorded a while back @russellromney

https://community.plotly.com/t/show-and-tell-brain-plasma-fast-sharing-for-large-objects-between-callbacks/23194/4?u=nsbruce

[russellromney]

@nsbruce excellent error documentation, looks like you've nailed the second issue - the call fails bc the client is mid-delete. Generally the deletes are so fast as to be safe; in this case they are not. A way to solve this would be to have three maps: a map of names to hashes and a map of hashes to Plasma keys (hashed with some arbitrary blue/green string) - then when replacing, simply

• check the current "color" for this name
• make a new plasma entry with the new color's name hash
• switch the name->hash mapping to point to the new color
• delete the old "color" from the name hash map

so fetching would be eventually consistent, but in a way that prevents both (1) and (2) failures

I may draft up a version of this over the next bit of time.

[russellromney]

I expect the second error occurs when the loop is reading 'this' while brain-plasma has just deleted 'this' in order to update it's value. I didn't realize that plasma's put() doesn't let you update values and so you have to delete/re-put. If that's the case this may not be solve-able at high refresh rates.

This issue is a huge problem at fast refresh rates...updates between clients cannot be strictly consistent. A single-threaded process that supports updates (Redis) is a better solution for this.