Incompatible with latest version of chardet

[prettybits]

Since commit a98992c5818129949b11912cbf11b5b4972430c0 polyfile uses a narrower version specifier for the version of chardet expected. In the meantime chardet 5.1.0 has been released (and packaged on Arch Linux), which leads to a DistributionNotFound error since pkg_resources is explicitly imported at runtime:

Traceback (most recent call last):
  File "/home/prettybits/.local/lib/python3.10/site-packages/pkg_resources/__init__.py", line 581, in _build_master
    ws.require(__requires__)
  File "/home/prettybits/.local/lib/python3.10/site-packages/pkg_resources/__init__.py", line 909, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/home/prettybits/.local/lib/python3.10/site-packages/pkg_resources/__init__.py", line 800, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (chardet 5.1.0 (/usr/lib/python3.10/site-packages), Requirement.parse('chardet~=5.0.0'), {'polyfile'})

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/bin/polyfile", line 33, in <module>
    sys.exit(load_entry_point('polyfile==0.5.0', 'console_scripts', 'polyfile')())
  File "/usr/bin/polyfile", line 25, in importlib_load_entry_point
    return next(matches).load()
  File "/usr/lib/python3.10/importlib/metadata/__init__.py", line 171, in load
    module = import_module(match.group('module'))
  File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 992, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/usr/lib/python3.10/site-packages/polyfile/__init__.py", line 1, in <module>
    from . import nes, pdf, jpeg, zipmatcher, nitf, kaitaimatcher, languagematcher, polyfile
  File "/usr/lib/python3.10/site-packages/polyfile/nes.py", line 6, in <module>
    from .polyfile import register_parser, InvalidMatch, Submatch
  File "/usr/lib/python3.10/site-packages/polyfile/polyfile.py", line 8, in <module>
    import pkg_resources
  File "/home/prettybits/.local/lib/python3.10/site-packages/pkg_resources/__init__.py", line 3260, in <module>
    def _initialize_master_working_set():
  File "/home/prettybits/.local/lib/python3.10/site-packages/pkg_resources/__init__.py", line 3234, in _call_aside
    f(*args, **kwargs)
  File "/home/prettybits/.local/lib/python3.10/site-packages/pkg_resources/__init__.py", line 3272, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/home/prettybits/.local/lib/python3.10/site-packages/pkg_resources/__init__.py", line 583, in _build_master
    return cls._build_from_requirements(__requires__)
  File "/home/prettybits/.local/lib/python3.10/site-packages/pkg_resources/__init__.py", line 596, in _build_from_requirements
    dists = ws.resolve(reqs, Environment())
  File "/home/prettybits/.local/lib/python3.10/site-packages/pkg_resources/__init__.py", line 795, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'chardet~=5.0.0' distribution was not found and is required by polyfile

This was observed on Arch Linux with the AUR package for polyfile at version 0.5.

Can the version specifier be safely loosened to chardet>=5.0.0 again?

[danieldjewell]

On a related note, the current install_requires dependencies are (as of the time of posting this comment):

https://github.com/trailofbits/polyfile/blob/438628fea2d32ee97b9f23a7aef7ffa3fdc80a0a/setup.py#L26-L39

With the disclaimer that I fully understand (1) why versioning dependencies can be important (2) "virutalenvs" exist for a reason (3) I might be wishing for something that isn't possible and (4) there are a multitude of different viewpoints on what is "right" in terms of dependency versioning, etc.

If at all possible, it would be nice if there was a way to remove restrictions on specific versions for dependencies. I am fully aware that it's easier said than done. Certainly a > or >= is much easier to integrate than a == or ~=.

This particular issue is probably more impactful on rolling distributions like Kali or Arch as compared to Ubuntu or Debian (at least, Debian other than sid)...

That said, if there's any way to remove "hard" dependencies - that'd be great. Alternatively, using a submodule and/or integrating specific functionality from a specific version of a different module/package might be an option too. Though I recognize and detest the idea of not maximizing code-reuse (i.e. having, essentially, multiple versions of a package installed because the code from one package is incorporated as a sub-part of another.)