dependencies

[bruceravel]

Updated just now. Upon building, I saw this:

#==============================================================#
#=== Warning: Some recommended Python Packages are missing:
     pyFAI:  needed for xrd modules
     xrayutilities:  needed for xrd modules
     CifFile:  needed for xrd modules
     fabio:  needed for xrd modules
     epics:  needed for using the EPICS control system

Some functionality will not work until these are installed.
See INSTALL for further information.
#==============================================================#

OK. Time to pip!

CifFile is not a package that is available via pip. The package that provides CifFile seems to be called PyCifRW, which installed readily.

pyFAI seems to expect something called pyOpenCl. At least it complained with WARNING:pyFAI.opencl:Unable to import pyOpenCl. Please install it from: http://pypi.python.org/pypi/pyopencl. This lead me down a bit of a rabbit hole, which pip was unable to resolve on its own.

pyOpenCl in turn needs something called mako. Also something called pycparser. Also something called cffi. Also I had to install OpenCL using my system's package manager.

I am willing to believe that the python installation on my computer is somehow borked and if I had just started clean, than all this would have worked fine. That said, it seems like a lot to expect a newbie to wade through all of that. At the very least, the INSTALL file and the document should be made clear about all these installation requirements.

[newville]

Hi Bruce,

Sorry for delay

On Thu, Apr 6, 2017 at 10:12 AM, Bruce Ravel [email protected] wrote:

Updated just now. Upon building, I saw this:

#==============================================================# #=== Warning: Some recommended Python Packages are missing: pyFAI: needed for xrd modules xrayutilities: needed for xrd modules CifFile: needed for xrd modules fabio: needed for xrd modules epics: needed for using the EPICS control system

Some functionality will not work until these are installed. See INSTALL for further information. #==============================================================#

OK. Time to pip!

CifFile is not a package that is available via pip. The package that provides CifFile seems to be called PyCifRW, which installed readily.

pyFAI seems to expect something called pyOpenCl. At least it complained with WARNING:pyFAI.opencl:Unable to import pyOpenCl. Please install it from: http://pypi.python.org/pypi/pyopencl. This lead me down a bit of a rabbit hole, which pip was unable to resolve on its own.

pyOpenCl in turn needs something called mako. Also something called pycparser. Also something called cffi. Also I had to install OpenCL using my system's package manager.

I am willing to believe that the python installation on my computer is somehow borked and if I had just started clean, than all this would have worked fine. That said, it seems like a lot to expect a newbie to wade through all of that. At the very least, the INSTALL file and the document should be made clear about all these installation requirements.

Yeah, sorry for all that and thanks. We're trying to get to the point where a single Anaconda channel can install all these dependencies too, but we don't have all of these packages worked out. PyFAI looks to be the main trouble spot, but should be solvable.

I agree that we should have instructions for non-Anaconda installs too. We'll keep working on this.

[jn287]

I tried to install larch 0.9.33 on Scientific Linux 7 (Red Hat Enterprise Linux clone) and I had dependency problems similar to those of Bruce. I was able to get to this point

johan@localhost:~/kk/xraylarch$ python setup.py build
Checking dependencies....
running build
running build_py

#==============================================================#
#=== Warning: Some recommended Python Packages are missing:
     pyFAI:  needed for xrd modules
     CifFile:  needed for xrd modules

Some functionality will not work until these are installed.
See INSTALL for further information.
#==============================================================#

If I install pyfai and pycifrw via pip this is what I get:

johan@localhost:~/kk/xraylarch$ python setup.py build
Checking dependencies....
WARNING:pyFAI.opencl:Unable to import pyOpenCl. Please install it from: http://pypi.python.org/pypi/pyopencl
Traceback (most recent call last):
  File "setup.py", line 48, in <module>
    x = __import__(mod)
  File "/usr/lib64/python2.7/site-packages/CifFile/__init__.py", line 2, in <module>
    from .CifFile import CifFile,CifDic,CifError,CifBlock,ReadCif,ValidCifFile,ValidCifError,Validate
  File "/usr/lib64/python2.7/site-packages/CifFile/CifFile.py", line 2849
    exec(func_def, globals()) #(re)defines element_convert in global namespace
SyntaxError: unqualified exec is not allowed in function 'convert_list_values' it contains a nested function with free variables
johan@localhost:~/kk/xraylarch$ 

I could not solve these issues so I uninstalled pyfai and pycifrw and continued with the installation, which worked fine. I'm not doing any xrd stuff so I guess I'm fine for the moment.

[newville]

@jn287 Thanks for the report Johan. I think we need to figure out how to deal with these dependencies better. Using Anaconda even on linux might be the simplest approach, though I know that this also has a few oddities....

[jn287]

I tried a bit more and I managed to get CifFile working by installing an older version, maybe there has been some regression in the upstream project pip install pycifrw==4.2.1 I still get the warning for pyFAI, but from what I can read pyopencl should be an optional dependency for GPU acceleration, anyway apart from this warning I can build and install larch and it seems to be working fine.

johan@localhost:~/kk/xraylarch$ larch
WARNING:pyFAI.opencl:Unable to import pyOpenCl. Please install it from: http://pypi.python.org/pypi/pyopencl
==============================================================================
Larch 0.9.33 (2017-Apr-3) M. Newville, M. Koker, B. Ravel, and others
Python 2.7.5 (default, Nov  3 2016, 22:05:29) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-11)],
numpy 1.12.1, scipy 0.19.0, matplotlib 2.0.2, wxpython 2.8.12.0
==============================================================================

larch> 

Although it has its problems I think managing the dependencies with pip works ok if you have a rough idea of what you need to install, I think the lack of documentation is the biggest problem in the short term. I could contribute some hints on how to make an installation from source, at least on linux, if you think that would be helpful.

[bruceravel]

I completely understand that conda is the preferred installation route AND I followed the recent conversation about installation routes. That said, when doing a python setup.py build, this gets written to the screen:

If you need some of these capabilities, you can install them with
    `pip install <Package Name>` or `conda install <Package Name>`

Since I am sitting this morning at a machine with pip but without conda, I figured i would give pip a go. Immediately, I only need the boring, ol' XAS functionality. But I tried to install all the other fancy things, as well.

Mostly no complaints, but there were a small number of friction points which could probably be addressed rather readily.

1. Installing pyFAI required cython. Fine, but that's kind of a significant requirement that should probably be specified.

2. Once pyFAI was installed, python setup.py build failed with an error about object has no attribute FabioImage. I simply unistalled pyFAI.

3. pyFAI needs pyOpenCl. For some reason, pip was unable to resolve that dependency. I ended up letting apt install that for me.

4. pip does not seem to know what tomopy is.

As I said, all of these issues are things that don't affect me. And I understand that conda is the more sanctified installation route. Just trying to be helpful by documenting and sharing my experience with a route that you seem to still intend to support.

[bruceravel]

I guess part of the point of the previous post is that if you want to direct people to use conda, don't explicitly say that using pip is OK. That suggests that pip is fully supported. I, for one, am happy to interact with larch in the way that makes your life easiest. FSM knows that keeping track of installation problems is the bane of my own software project...!

[newville]

@bruceravel It should be that pyFAI, tomopy, etc are not required, but only needed for some XRD and tomography functionality. The functionality in larch that do require those optional packages should be listed when doing python setup.py install and the routines that use these packages should fail gracefully when running larch without those packages installed. I for sure run many larch installations with pyfai or tomopy.

Those package may be installable with pip or with conda. I guess we should emphasize that "may".
As it turns out, pyFAI and tomopy are difficult to support on all combinations of OS and python version. We are not going to solve this soon, and are going have some functionality in larch that may not work on every OS/Python combination. Still, we do want to keep for those routines for those systems that have it.

That's independent of whether one installs larch itself with pip. Using the binary installers or conda packages is definitely recommended: the install docs to not actually mention using pip install for larch. As it turns out, using pip install larch partially works: it may not get all the dependencies (though I think that is now fixed). The python code is installed, but the binaries (say for Feff, etc) are only for one OS (currently macOS). This ought to be solvable, perhaps using "wheels" (yippee! another python install format!), but it will take some work and testing. Just to be clear, unlike CPAN (I think), pip install XXX does not do: wget/curl ..../XXX.tar.gz tar xvzf XXX.tar.gz cd XXX python setup.py install it definitely gives different results. Pip kind of sucks in that it mostly works, except when it doesn't.

The binary installers and the conda packages also enable (and forces) a dedicated environment that is separate from the system-installed python. I think we want to discourage using system-installed python. Many linuxes will have numpy, scipy, etc packages as part of their system packaging and we want to not be constrained by those choices or break other code by overwriting packages outside the system package manager.

So, having pip install xraylarch might be nice, but would take some effort, I'm reluctant to force people to use Anaconda, but it does seem like the desire for pip install aims at a use-case that I think we should not be encouraging. Which leads me to the conclusion that it is "would be nice to have, not the highest priority". Does that seem reasonable to you?

Sorry this was so long.

[maurov]

@bruceravel we have made huge progress on Larch packaging since... 2018! I think it makes no sense to keep this issue open. If not, do not hesitate opening a fresh one.