october icon indicating copy to clipboard operation
october copied to clipboard

Implement more efficient way of submitting highlights

Open marcus-crane opened this issue 2 years ago • 4 comments

At the moment, everything is just fired off to Readwise.

This is fine but as your Kobo highlights grow, the upload process will naturally become slower.

I was thinking that you could easily tell if a book hasn't changed (and finished books don't) by simply saving hashes for books and seeing if they've changed. We kind of need to retain what content has actually been submitted (not just number of bookmarks) since highlight positions and text annotations can change after the fact.

If they have changed then we ideally just figure out what items are new so perhaps we just hash each bookmark.

It all falls down if someone either is doing a first upload with a squillion highlights (these should be chunked to avoid rate limiting) and/or they use a new computer with a previous upload DB.

It's really cheap to make database calls but expensive (relatively speaking) to fire JSON across the web.

marcus-crane avatar Feb 07 '22 08:02 marcus-crane