pyjson_tricks
pyjson_tricks copied to clipboard
Make `ignore_comments` not default (next major release)
Stripping comments from files that are multiple megabytes takes an obscene amount of time, even if there are no comments.
I don't think I can make it much faster without writing a whole json parser (in something other than Python, or it can never compete).
So I'm thinking of something like:
- Make
ignore_comments=Falsethe default - Ignore comments only for small files, unless
ignore_commentsis used to override - Only parse comments if the file starts with a comment, unless
ignore_commentsis used to override - Only strip comments on uncompressed files, unless
ignore_commentsis used to override
As a way to address
- Performance
- Excessive deprecation messages
I think it might be good to always try to
- Parse without stripping comments if ignore_comments is None
- Then only if it fails (or ignore_comments is True) re-try with stripping comments
This way
- The deprecation warning is only shown if strip_comments was None and there were actual comments (and only the first time)
- The performance without comments is better, at the cost of performance of implicitly stripping comments
Released v3.16.1 with this optimization,