importmagic

importmagic copied to clipboard

Creating a symbol index of a well filled sys.path takes a long time. Caching would help, but raises questions.

• granularity: cache whole indices (serialization with metadata about index search path and indexed modules) or symbols of individual modules
• format: JSON is (somewhat) human readable, pickle is (probably, untested!) faster but unsafe (does it matter?)
• invalidation: go by size/mtime or content hash
• location: let the caller decide, or use a global location like ~/.cache/importmagic

The indexing is one of the biggest outstanding issues. My long term plan was to modify the storage so that it would support incremental updates. Probably by separating each package into its own file so that it could be modified without having to rewrite the entire index.

I did some small tests with pickle, and it's faster than Json but no faster than Ujson for reads (still faster for writes).

I like the idea of use global location for cache, like:

• ~/.cache/importmagic on Unix
• ~/Library/Caches/importmagic on OS X
• %APPDATA%\importmagic on Windows

I also agree with what @alecthomas said about split the cache in different files but maybe not by package, we should test what is better.

Btw, I am just busy, still here.