pyjson_tricks icon indicating copy to clipboard operation
pyjson_tricks copied to clipboard

Make `ignore_comments` not default (next major release)

Open mverleg opened this issue 5 years ago • 2 comments

Stripping comments from files that are multiple megabytes takes an obscene amount of time, even if there are no comments.

I don't think I can make it much faster without writing a whole json parser (in something other than Python, or it can never compete).

So I'm thinking of something like:

  • Make ignore_comments=False the default
  • Ignore comments only for small files, unless ignore_comments is used to override
  • Only parse comments if the file starts with a comment, unless ignore_comments is used to override
  • Only strip comments on uncompressed files, unless ignore_comments is used to override

mverleg avatar Apr 05 '20 20:04 mverleg

As a way to address

  • Performance
  • Excessive deprecation messages

I think it might be good to always try to

  • Parse without stripping comments if ignore_comments is None
  • Then only if it fails (or ignore_comments is True) re-try with stripping comments

This way

  • The deprecation warning is only shown if strip_comments was None and there were actual comments (and only the first time)
  • The performance without comments is better, at the cost of performance of implicitly stripping comments

mverleg avatar Nov 03 '22 19:11 mverleg

Released v3.16.1 with this optimization,

mverleg avatar Nov 04 '22 16:11 mverleg