Bug: custom serializers aren't fully supported (unless they serialize info)

[kodonnell]

E.g. if I write custom serializers for my objects, this fails when the info is serialized (using the same serializer). See e.g. here. Solution would be to serialize info in a separate/standard way. Unless I've missed something ....

[peter-wangxu]

Any more information? I did not get you point here?

[kodonnell]

OK, MRE:

from persistqueue import Queue

class Point:

    def __init__(self, x, y):
        self.x = x
        self.y = y

    def __repr__(self):
        return f"{self.x}|{self.y}"

    @staticmethod
    def dump(value, f):
        print("dumping:", value)
        f.write(f"{value.x}|{value.y}\n".encode('ascii'))

    @staticmethod
    def load(f):
        x, y = f.readline().strip().split('|')
        return Point(int(x), int(y))


if __name__ == '__main__':

    q = Queue("./queue", serializer=Point)
    p = Point(1, 2)
    print("p:", p)
    print("putting p")
    q.put(p)
    print("getting p")
    p = q.get()
    print("p:", p)
    q.task_done()

Outputs:

p: 1|2
putting p
dumping: 1|2
dumping: {'chunksize': 100, 'size': 1, 'tail': [0, 0, 0], 'head': [0, 1, 4]}
Traceback (most recent call last):
  File "pq.py", line 29, in <module>
    q.put(p)
  File "<redacted>\site-packages\persistqueue\queue.py", line 161, in put
    self._put(item)
  File "<redacted>\site-packages\persistqueue\queue.py", line 182, in _put
    self._saveinfo()
  File "<redacted>\site-packages\persistqueue\queue.py", line 273, in _saveinfo
    self.serializer.dump(self.info, tmpfo)
  File "pq.py", line 15, in dump
    f.write(f"{value.x}|{value.y}\n".encode('ascii'))
AttributeError: 'dict' object has no attribute 'x'

As you can see, my serializer works correctly for adding/removing my Point objects to the queue (and presumably from, though the code doesn't get that far). But this fails because the queue tries to serialize info with the same serializer, which obviously fails.

Is that clearer now?

[peter-wangxu]

Got it.

You may need to fall back the pickle if the load/dump if the value is not Point

Alternatively, you might give a try of the sqlite queue?

Thanks Peter