asciimatics Reducing number of dependencies

We are using asciimatics with an embedded device and it requires lots of dependencies. The main offender is Pillow which also requires 3-6 other dependencies and some of them do not compile very well for anything other than basic i386/amd64 platforms. I also noticed that sub dependencies are some sort of imaging libraries for jpeg and png and I can hardly imagine they have anything to do with TUI library. So, would it be possible to remove dependency on Pillow and revisit other dependencies as well.

Sep 18 '17 22:09 dtoubelis

The Pillow dependency is for image to text conversion, as per this (https://asciinema.org/a/19920) demo.

If you avoid the ImageFile and ColourImageFile renderers, it should be safe to run without Pillow. Of course, you then can't import any renderers without fixing the source to avoid the import...

I'm open to suggestions for how to package this more neatly, but don't want to remove the feature entirely as it is required for the animatics aspect of the package (which is where it all started).

Sep 18 '17 23:09 peterbrittain

Was looking at this last night. What I'd really like to have is an "opt out" option on the installation so that most people will just get the whole solution. In fact, what I'd like is this: https://github.com/pypa/setuptools/issues/163 - which isn't implemented yet. Sigh.

Sep 20 '17 11:09 peterbrittain

That would work. We were chosen asciimatics as more modern alternative to urwid and we just expected it to use ncurses and everything else to be optional. So, if it is possible to turn off the optional features it would be awesome.

Sep 20 '17 13:09 dtoubelis

Hmmm... Opt outs will be hard and need more time than I can invest into setuptools...

After more thinking, maybe I can get what I need with a "default specifier" setting that gets used if you don't specify any specifier for extras on the install. This way, I could define a default which picks all the extras, but allows you to reduce that set by selecting a different (reduced) set of options if you so desire.

Oct 03 '17 01:10 peterbrittain

I think this in an excellent idea. Realistically, if we can have some sensible defaults for majority of uses and be able to overwrite them if needed that would be the perfect solution.

Oct 03 '17 02:10 dtoubelis

Looks like this is going to take a while to get agreed. Do you have a suitable workaround while that happens? If not, I think that a simple one would be to:

clone this repo
remove asciimatics/renderers.py
remove the Pillow dependency from setup.py
build your own distribution
use that as a dependency for your TUI application.

I've had a quick test and this can run the contact_list demo, but obviously cannot run any of the others that use a Renderer.

Oct 11 '17 10:10 peterbrittain

That sounds like it may be a reasonable workaround. We build our packages with yocto, so it should be rather straightforward to create a patch and apply it to the base release at build time. Let me try that.

Oct 11 '17 13:10 dtoubelis

I ended up stripping asciimatics of dependencies as well (pyfiglet, Pillow, and future), but now I'm stuck on an older version. Definitely would like to see Pillow as optional.

Oct 18 '17 16:10 brycepg

Given that I'm just aiming at supporting pip installations, maybe the option in https://github.com/pypa/setuptools/issues/1139 will work. The theory would be having everything installed if no extras are specified and skip Pillow and pyfiglet for a tui extra.

I could just then add some import protection in renderers to handle the missing dependencies a little more gracefully.

Oct 21 '17 18:10 peterbrittain

OK, so I've just tried using the pip only solution I found above... I think the code in https://github.com/peterbrittain/asciimatics/tree/slim_tui now allows you to pip install asciimatics[tui] to avoid figlet and Pillow dependencies.

It also prevents you from trying to use any renderers until you have a full installation using the normal pip install.

Does this do the trick for your environments?

Nov 02 '17 23:11 peterbrittain

Yes, that sounds like a neat idea. Let me try it in our environment. However, one of the problems we were facing was that our developers could not use pip in our yocto builds and they were using python setup.py install instead but I will check where we stand on this.

Nov 08 '17 14:11 dtoubelis

Any news?

Nov 18 '17 09:11 peterbrittain

Thanks for working on this Peter.

With latest version of setuptools I'm getting an exception when i run the slim_tui branch with python setup.py install, pip install '.[tui]', or pip install . However when I use setuptools==20.4 I do not get an exception.

It looks like this issue is supposed to be fixed?: https://github.com/pypa/setuptools/issues/523

Here's the traceback from setuptools 37.0

(pm5-dev) bguinta  [1]➜  asciimatics git:(slim_tui) ✗ python setup.py install         
running install
running bdist_egg
running egg_info
writing asciimatics.egg-info/PKG-INFO
writing dependency_links to asciimatics.egg-info/dependency_links.txt
writing requirements to asciimatics.egg-info/requires.txt
writing top-level names to asciimatics.egg-info/top_level.txt
reading manifest template 'MANIFEST.in'
writing manifest file 'asciimatics.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_py
copying asciimatics/version.py -> build/lib/asciimatics
creating build/bdist.linux-x86_64/egg
creating build/bdist.linux-x86_64/egg/asciimatics
copying build/lib/asciimatics/__init__.py -> build/bdist.linux-x86_64/egg/asciimatics
copying build/lib/asciimatics/effects.py -> build/bdist.linux-x86_64/egg/asciimatics
copying build/lib/asciimatics/event.py -> build/bdist.linux-x86_64/egg/asciimatics
copying build/lib/asciimatics/particles.py -> build/bdist.linux-x86_64/egg/asciimatics
copying build/lib/asciimatics/paths.py -> build/bdist.linux-x86_64/egg/asciimatics
copying build/lib/asciimatics/scene.py -> build/bdist.linux-x86_64/egg/asciimatics
copying build/lib/asciimatics/sprites.py -> build/bdist.linux-x86_64/egg/asciimatics
copying build/lib/asciimatics/utilities.py -> build/bdist.linux-x86_64/egg/asciimatics
copying build/lib/asciimatics/exceptions.py -> build/bdist.linux-x86_64/egg/asciimatics
copying build/lib/asciimatics/renderers.py -> build/bdist.linux-x86_64/egg/asciimatics
copying build/lib/asciimatics/screen.py -> build/bdist.linux-x86_64/egg/asciimatics
copying build/lib/asciimatics/widgets.py -> build/bdist.linux-x86_64/egg/asciimatics
copying build/lib/asciimatics/version.py -> build/bdist.linux-x86_64/egg/asciimatics
byte-compiling build/bdist.linux-x86_64/egg/asciimatics/__init__.py to __init__.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/asciimatics/effects.py to effects.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/asciimatics/event.py to event.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/asciimatics/particles.py to particles.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/asciimatics/paths.py to paths.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/asciimatics/scene.py to scene.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/asciimatics/sprites.py to sprites.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/asciimatics/utilities.py to utilities.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/asciimatics/exceptions.py to exceptions.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/asciimatics/renderers.py to renderers.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/asciimatics/screen.py to screen.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/asciimatics/widgets.py to widgets.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/asciimatics/version.py to version.cpython-36.pyc
creating build/bdist.linux-x86_64/egg/EGG-INFO
copying asciimatics.egg-info/PKG-INFO -> build/bdist.linux-x86_64/egg/EGG-INFO
copying asciimatics.egg-info/SOURCES.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying asciimatics.egg-info/dependency_links.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying asciimatics.egg-info/requires.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying asciimatics.egg-info/top_level.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
zip_safe flag not set; analyzing archive contents...
creating 'dist/asciimatics-1.8.1.dev56+g8aea4e5-py3.6.egg' and adding 'build/bdist.linux-x86_64/egg' to it
removing 'build/bdist.linux-x86_64/egg' (and everything under it)
Processing asciimatics-1.8.1.dev56+g8aea4e5-py3.6.egg
Removing /home/bguinta/miniconda3/envs/pm5-dev/lib/python3.6/site-packages/asciimatics-1.8.1.dev56+g8aea4e5-py3.6.egg
Copying asciimatics-1.8.1.dev56+g8aea4e5-py3.6.egg to /home/bguinta/miniconda3/envs/pm5-dev/lib/python3.6/site-packages
asciimatics 1.8.1.dev56+g8aea4e5 is already the active version in easy-install.pth

Installed /home/bguinta/miniconda3/envs/pm5-dev/lib/python3.6/site-packages/asciimatics-1.8.1.dev56+g8aea4e5-py3.6.egg
Processing dependencies for asciimatics==1.8.1.dev56+g8aea4e5
Traceback (most recent call last):
  File "/home/bguinta/miniconda3/envs/pm5-dev/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2650, in _dep_map
    return self.__dep_map
  File "/home/bguinta/miniconda3/envs/pm5-dev/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2725, in __getattr__
    raise AttributeError(attr)
AttributeError: _Distribution__dep_map

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "setup.py", line 78, in <module>
    test_suite='nose.collector',
  File "/home/bguinta/miniconda3/envs/pm5-dev/lib/python3.6/site-packages/setuptools/__init__.py", line 129, in setup
    return distutils.core.setup(**attrs)
  File "/home/bguinta/miniconda3/envs/pm5-dev/lib/python3.6/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/home/bguinta/miniconda3/envs/pm5-dev/lib/python3.6/distutils/dist.py", line 955, in run_commands
    self.run_command(cmd)
  File "/home/bguinta/miniconda3/envs/pm5-dev/lib/python3.6/distutils/dist.py", line 974, in run_command
    cmd_obj.run()
  File "/home/bguinta/miniconda3/envs/pm5-dev/lib/python3.6/site-packages/setuptools/command/install.py", line 67, in run
    self.do_egg_install()
  File "/home/bguinta/miniconda3/envs/pm5-dev/lib/python3.6/site-packages/setuptools/command/install.py", line 117, in do_egg_install
    cmd.run()
  File "/home/bguinta/miniconda3/envs/pm5-dev/lib/python3.6/site-packages/setuptools/command/easy_install.py", line 411, in run
    self.easy_install(spec, not self.no_deps)
  File "/home/bguinta/miniconda3/envs/pm5-dev/lib/python3.6/site-packages/setuptools/command/easy_install.py", line 653, in easy_install
    return self.install_item(None, spec, tmpdir, deps, True)
  File "/home/bguinta/miniconda3/envs/pm5-dev/lib/python3.6/site-packages/setuptools/command/easy_install.py", line 700, in install_item
    self.process_distribution(spec, dist, deps)
  File "/home/bguinta/miniconda3/envs/pm5-dev/lib/python3.6/site-packages/setuptools/command/easy_install.py", line 745, in process_distribution
    [requirement], self.local_index, self.easy_install
  File "/home/bguinta/miniconda3/envs/pm5-dev/lib/python3.6/site-packages/pkg_resources/__init__.py", line 878, in resolve
    new_requirements = dist.requires(req.extras)[::-1]
  File "/home/bguinta/miniconda3/envs/pm5-dev/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2669, in requires
    dm = self._dep_map
  File "/home/bguinta/miniconda3/envs/pm5-dev/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2658, in _dep_map
    if invalid_marker(marker):
  File "/home/bguinta/miniconda3/envs/pm5-dev/lib/python3.6/site-packages/pkg_resources/__init__.py", line 1441, in invalid_marker
    evaluate_marker(text)
  File "/home/bguinta/miniconda3/envs/pm5-dev/lib/python3.6/site-packages/pkg_resources/__init__.py", line 1459, in evaluate_marker
    return marker.evaluate()
  File "/home/bguinta/miniconda3/envs/pm5-dev/lib/python3.6/site-packages/pkg_resources/_vendor/packaging/markers.py", line 301, in evaluate
    return _evaluate_markers(self._markers, current_environment)
  File "/home/bguinta/miniconda3/envs/pm5-dev/lib/python3.6/site-packages/pkg_resources/_vendor/packaging/markers.py", line 226, in _evaluate_markers
    lhs_value = _get_env(environment, lhs.value)
  File "/home/bguinta/miniconda3/envs/pm5-dev/lib/python3.6/site-packages/pkg_resources/_vendor/packaging/markers.py", line 208, in _get_env
    "{0!r} does not exist in evaluation environment.".format(name)
pkg_resources.extern.packaging.markers.UndefinedEnvironmentName: 'extra' does not exist in evaluation environment.

Nov 22 '17 18:11 brycepg

OK - I've reproduced your issue exactly. I agree that it is setuptools issue 523, but have found a later open issue that is tracking this. Have followed up there and am hoping that we'll get a response at some point.

Nov 23 '17 10:11 peterbrittain

I've been looking at this again. I now think it was a regression introduced by release 36.2 of setuptools. Plenty of other versions before that, including 36.1, work fine. I'll see if I can find the cause...

Dec 17 '17 09:12 peterbrittain

Getting there. Have tried to repro the bug in setuptools unit tests and think I'm there. Trying to get an official patch in https://github.com/pypa/setuptools/issues/1139

Dec 20 '17 00:12 peterbrittain

Sweet. I'm making a conda package for asciimatics (since I use the conda toolchain, which also requires me to make a pyfiglet conda package). When you get this branch into a release, I'll update the package to make pyfiglet optional.

Dec 21 '17 18:12 brycepg

Some kind person has submitted the PR for me. See https://github.com/pypa/setuptools/pull/1503 for progress.

Oct 02 '18 11:10 peterbrittain

Hmmm... Looks like that PR has been dropped. Maybe the suggestion of specifying a blank extras definition will do the trick here?

Oct 11 '19 06:10 peterbrittain

Voting in favor of the slim_tui branch :+1:

Nov 16 '19 15:11 pawamoy

I would dearly love to merge that change, but setuptools doesn't work with that use of extras. The best I can do is make the default install of asciimatics not include Pillow and pyfiglet, then define an all option that users must specify when using the non-tui features.

I've been reluctant to make that default change up to now as this will break existing package definitions. I suppose I could move to a new major version and handle it that way.

Nov 16 '19 18:11 peterbrittain

Oh I see, I thought the change in the slim_tui branch was functional. That's too bad! Then I'm totally OK with waiting for a working solution that will not break existing code (or for a new major version) :slightly_smiling_face:

Nov 16 '19 18:11 pawamoy

Ok - let's make that the plan of record. Will convert the install when I bump the major version.

Nov 17 '19 13:11 peterbrittain

asciimatics asciimatics copied to clipboard

Reducing number of dependencies

asciimatics
asciimatics copied to clipboard