Pyrebase4 icon indicating copy to clipboard operation
Pyrebase4 copied to clipboard

stream() is very slow with large messages

Open bringert opened this issue 3 years ago • 2 comments

In current HEAD, with Python 3.6.5, I found that stream() is extremely slow. It takes many seconds on a Raspberry Pi 3 to parse a few tens of kilobytes, and it gets progressively slower.

It looks like pyre_sseclient._event_complete() searches the message from the start for every character read, which makes detecting the end of a message quadratic in the length of the message.

bringert avatar Dec 12 '21 15:12 bringert

https://github.com/btubbs/sseclient/blob/master/sseclient.py doesn't seem to have this quadratic behavior, so one solution would be to change to that.

bringert avatar Dec 12 '21 16:12 bringert

Changing it so that the search for end_of_field just searches the end of the buffer speeds it up a lot, but it's still a lot slower than https://github.com/btubbs/sseclient/blob/master/sseclient.py.

It looks like it's because pyre_sseclient reads one character at a time, whereas sseclient.py uses a default chunk size of 1024.

bringert avatar Dec 12 '21 19:12 bringert