rope icon indicating copy to clipboard operation
rope copied to clipboard

Remove usage of pickle from `objectdb` and `history`

Open lieryan opened this issue 3 years ago • 0 comments

pickle is insecure and it's not very efficient for larger, more complex datasets.

This will be a multiple steps plan towards moving into sqlite-backed storage:

  1. [x] Remove rarely used features
    1. [x] Deprecate objectdb and history compression: This was never the default and disk space nowadays makes this unlikely to be an issue nowadays, maybe we can re-introduce compression when sqlite migration is complete (#606)
    2. [x] Deprecate importing of old format (objectdb.pickle and history.pickle): rope hasn't been generating this for decades (#607)
  2. [x] Make ScopeInfo JSON-serializable: there are some data here that are python class instances, but they really doesn't need to be, we can just __getstate__()/__setstate__() them instead when de/serializing. #611
  3. [ ] Implement sqlite database schema versioning #565
  4. [ ] These files contains a bunch of nested dictionaries, these can be translated to sqlite tables in a straightforward manner, we'll likely will still store JSON documents inside the sqlite tables instead of normalizing everything
  5. [ ] Make sure that the tables have appropriate indexes
  6. [ ] Remove write support for legacy objectdb and history files
  7. [ ] Remove read support for legacy objectdb and history files

lieryan avatar Dec 16 '22 04:12 lieryan