dials Debian package

[picca]

Hello,

I try to build an official package of dials for Debian. So I need to use all system libraries. I would like your advice in order to build it the right way.

I already face a few issues.

here the target for Debian which build dials

override_dh_auto_build:
	# install scons for the build
	ln -s /usr/lib/scons modules/

	python3 installer/bootstrap.py build

• I need to link the source code of the system scons into the dials sources.
• I need to rename the boosy_python library which are called

/usr/lib/alpha-linux-gnu/libboost_python.a
/usr/lib/alpha-linux-gnu/libboost_python.so
/usr/lib/alpha-linux-gnu/libboost_python27.a
/usr/lib/alpha-linux-gnu/libboost_python27.so
/usr/lib/alpha-linux-gnu/libboost_python3-py37.a
/usr/lib/alpha-linux-gnu/libboost_python3-py37.so
/usr/lib/alpha-linux-gnu/libboost_python3.a
/usr/lib/alpha-linux-gnu/libboost_python3.so
/usr/lib/alpha-linux-gnu/libboost_python37.a
/usr/lib/alpha-linux-gnu/libboost_python37.so
/usr/lib/alpha-linux-gnu/libboost_python38.a
/usr/lib/alpha-linux-gnu/libboost_python38.so

we append the pythonversion to the library name.

So what is the best solution to integrate this in the dials build system.

• I skip the last step call precommitbx by not running this

modules/dials/precommitbx/installer.py

• Once I have a build directory, which is the command in order to install the files ate the right place. On linux system under /bin, /lib, /share etc... . It seems to me that the build directory contain .o which does not need to be installed at the end.

thanks for your help.

Frederic

[picca]

Hello,

I managed to make it build and run tests on Debian, for now only for python3.8

https://salsa.debian.org/science-team/dials/-/jobs/649387/raw

Can you tel me if there is someting strange in the test suite. This is the first time that I see it :).

thanks for considering

[dagewa]

Hi Frederic,

Your output contains some test results for cctbx, but not for DIALS or dxtbx, as far as I can see. Running pytest at the root of the source tree for these should give test output, though testing is limited when regression data are not available.

Cheers -- David

On Mon, 6 Apr 2020 at 07:07, picca [email protected] wrote:

Hello,

I managed to make it build and run tests on Debian, for now only for python3.8

https://salsa.debian.org/science-team/dials/-/jobs/649387/raw

Can you tel me if there is someting strange in the test suite. This is the first time that I see it :).

thanks for considering

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/dials/dials/issues/1218#issuecomment-609584054, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACXYEAZJURS74SXDFLWX6MTRLFWSDANCNFSM4L2YPDLA .

[picca]

Hello,

How big are the regression data ? and how can I get them ?

thanks for your help.

Frederic

[picca]

found it, sorry for the noise

cd ../modules
curl http://dials.diamond.ac.uk/developers/dials_regression.tgz > dials_regression.tgz
tar -xzvf dials_regression.tgz
libtbx.configure dials_regression

[picca]

Except that the package is empty ???

curl http://dials.diamond.ac.uk/developers/dials_regression.tgz > dials_regression.tgz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   243  100   243    0     0    313      0 --:--:-- --:--:-- --:--:--   313

[picca]

Indeed

Forbidden

You don't have permission to access /developers/ on this server.

Is it possible to provide these data publicly ?

[Anthchirp]

Unfortunately dials_regression is not a public repository, which is why we are moving away from using it, and I don't advise using it at all. If you make sure the dials-data python package is installed in the environment then you can just run pytest --regression in the dials source directory to download the data required for most of the tests.

[picca]

ok, I will work with this dials-data package.

other questions

1. about the .so files generated during the build. I will build dials for all the supported version of python available in Debian. So I would like to understand who is who :).

   it seems that there is .so files which are python extensions. So I need to build these extension with the right python version. the other are just normal .so which are not python extension.

   Can you give me the list of the real python extensions.

2. setup.py there is also pyton code to install what is the right way to build these python modules. run python setup.py everywhere I find a setup.py ?

thanks

[Anthchirp]

What I would have thought to be the biggest stumbling block of the entire approach is that to build/bundle/run dials you first need to build/bundle/run cctbx and then dxtbx. Did you achieve that already?

If you have built a dxtbx bundle then you should check that dxtbx tests pass, for this the trick is again to ensure dials-data is installed and then run pytest --regression.

Because DIALS relies on the entire cctbx build system -which doesn't know anything about modern python packaging- there is no setup.py. The loose equivalent is libtbx.refresh, which dxtbx and DIALS use to set entry points and libtbx_config which we use to sort of define package dependencies. Running tests on dxtbx will check whether those entry points are set correctly. It doesn't make any sense to even try to get DIALS working unless dxtbx passes.

Finally, DIALS does not officially support Python 3.7/3.8 yet (mainly because cctbx doesn't, https://github.com/cctbx/cctbx_project/pull/409), but I would not expect that to be an issue for DIALS itself.

[dagewa]

Someone else may correct me here, but running pytest --regression might download the public regression data automatically (about 5.3 GB). There is other regression data not available publicly, but we have been trying to move away from that. I don't know what the current status is.

[dagewa]

Sorry, I missed the messages above, my page had not refreshed

[picca]

What I would have thought to be the biggest stumbling block of the entire approach is that to build/bundle/run dials you first need to build/bundle/run cctbx and then dxtbx. Did you achieve that already?

for now I checkout the dials repository. importe all the modules as explained in the documentation. then I started my package from this starting point. I just discover that a source package is available with all this.

If you have built a dxtbx bundle then you should check that dxtbx tests pass, for this the trick is again to ensure dials-data is installed and then run pytest --regression.

I think that all this is available in the link of my build. Can you confirme this ?

Because DIALS relies on the entire cctbx build system -which doesn't know anything about modern python packaging- there is no setup.py. The loose equivalent is libtbx.refresh, which dxtbx and DIALS use to set entry points and libtbx_config which we > use to sort of define package dependencies. Running tests on dxtbx will check whether those entry points are set correctly. It doesn't make any sense to even try to get DIALS working unless dxtbx passes.

Ok, so If I want to install all the binary, I need to use libtbx.refresh in order to use "sort of'' prefix=/usr.

In the build there is more than 700 files under build/bin, so I am not sure thaht I understand everythings for now.

inally, DIALS does not officially support Python 3.7/3.8 yet (mainly because cctbx doesn't, cctbx/cctbx_project#409https://github.com/cctbx/cctbx_project/pull/409), but I would not expect that to be an issue for DIALS itself.

If I need to packahe separatly cctbx and dials, I need to know if dials is a pure python code or if it use the cctbx include files ?

Cheers

[Anthchirp]

dials contains C++ code and depends on header files in (at least) cctbx_project and dxtbx, dxtbx contains C++ code and depends on header files in (at least) cctbx_project. cctbx_project has its own dependencies. This is part of the reason why we are nowhere near a stage where we can provide an easily installable bundle. There is some work towards providing cctbx as conda-forge packages, which may be useful, https://github.com/cctbx/cctbx_project/issues/451.

[picca]

ok thanks.

• How can I change the build dir ?

for now I use this

pythonXY installer/bootstrap.py build \
	--config-flags="--build=debug_optimized" \
	--config-flags="--use_environment_flags" \
	--config-flags="--enable_openmp_if_possible=1" \
	--config-flags="--enable_cxx11"

I want to use build27 build37 build38 for each python version: (I know this does not work for 37 and 38) butI prepare the futur :).

• I want to move all the wrapper into /usr/bin.

Is it possible ?

• I do not want to build the module ccp4io

because I hve already the depende cies on the system.

how can I do this ?

• last :), how can I modify these wrapper in order to find the right scripts installed a python modules under

/usr/lib/python3/dist-packages/ for python3 or the other relavant for python2 ?

thanks for your help

Frederic

[picca]

Still working on the system library,

I can find this in ccp4io_adaptbx/SConscript

c_files = []
c_files.extend(["%s/%s" % (ccp4_src, bn ) for bn in """\
library_err.c
library_file.c
library_utils.c
ccp4_array.c
ccp4_parser.c
ccp4_unitcell.c
cvecmat.c
cmtzlib.c
""".splitlines()])
open(op.join(build_ccp4io_adaptbx, "csymlib.c"), "w").write(
  open(op.join(path_lib_src, "csymlib.c")).read()
    .replace(
      "static int reported_syminfo = 0",
      "static int reported_syminfo = 1"))
source.append(op.join("#ccp4io_adaptbx", "csymlib.c"))

It seems that csymlib.c is patched and then a copy is added to the adaptbx sources. but I can not find a reference to the reported_syminfo variable in the code. so it seems useless.

[picca]

All the fortran part of ccp4 is also patched by replacing printf and fprintf into ccp4io_printf and ccp4io_fprintf. So the fortran part is no the official cpp4 fortran library.

need_f_c = (
     libtbx.env.has_module("solve_resolve")
  or libtbx.env.find_in_repositories(relative_path="mosflm_fable"))
if (need_f_c or os.name != "nt"):
  source.append(op.join("#ccp4io_adaptbx", "fortran_call_stubs.c"))
  for file_name in """\
ccp4_diskio_f.c
ccp4_general.c
ccp4_general_f.c
ccp4_parser_f.c
ccp4_program.c
ccp4_unitcell_f.c
cmaplib_f.c
cmtzlib_f.c
csymlib_f.c
library_f.c
""".splitlines():
    open(op.join(build_ccp4io_adaptbx, file_name), "w").write(
      replace_printf(file_name=file_name))
    source.append(op.join("#ccp4io_adaptbx", file_name))
  source.append(op.join("#ccp4io_adaptbx", "printf_wrappers.c"))

[dagewa]

I think issues regarding the ccp4io_adaptbx build are best directed to the cctbx_project. We (DIALS) don't maintain this package here.

[picca]

Ok for ccp4io_adaptbx, but for the other question regarding the build system ?

[picca]

While workink on dials packaging I found the clipper library but when I build your project is seems that your version dos not build the cctbx_project clipper part.

is it normal ?

Relocatable paths anchored at: /builds/science-team/dials/debian/output/dials-2.2.0+modules/build
Top-down list of all modules involved:
  iota            "/builds/science-team/dials/debian/output/dials-2.2.0+modules/modules/cctbx_project/iota"
  xia2            "/builds/science-team/dials/debian/output/dials-2.2.0+modules/modules/xia2"
  dials           "/builds/science-team/dials/debian/output/dials-2.2.0+modules/modules/dials"
  xfel            "/builds/science-team/dials/debian/output/dials-2.2.0+modules/modules/cctbx_project/xfel"
  prime           "/builds/science-team/dials/debian/output/dials-2.2.0+modules/modules/cctbx_project/prime"
  simtbx          "/builds/science-team/dials/debian/output/dials-2.2.0+modules/modules/cctbx_project/simtbx"
  cma_es          "/builds/science-team/dials/debian/output/dials-2.2.0+modules/modules/cctbx_project/cma_es"
  crys3d          "/builds/science-team/dials/debian/output/dials-2.2.0+modules/modules/cctbx_project/crys3d"
  rstbx           "/builds/science-team/dials/debian/output/dials-2.2.0+modules/modules/cctbx_project/rstbx"
  spotfinder      "/builds/science-team/dials/debian/output/dials-2.2.0+modules/modules/cctbx_project/spotfinder"
  annlib+adaptbx  "/builds/science-team/dials/debian/output/dials-2.2.0+modules/modules/annlib"
                  "/builds/science-team/dials/debian/output/dials-2.2.0+modules/modules/annlib_adaptbx"
  wxtbx           "/builds/science-team/dials/debian/output/dials-2.2.0+modules/modules/cctbx_project/wxtbx"
  gltbx           "/builds/science-team/dials/debian/output/dials-2.2.0+modules/modules/cctbx_project/gltbx"
  mmtbx           "/builds/science-team/dials/debian/output/dials-2.2.0+modules/modules/cctbx_project/mmtbx"
  dxtbx           "/builds/science-team/dials/debian/output/dials-2.2.0+modules/modules/dxtbx"
  iotbx           "/builds/science-team/dials/debian/output/dials-2.2.0+modules/modules/cctbx_project/iotbx"
  ccp4io+adaptbx  "/builds/science-team/dials/debian/output/dials-2.2.0+modules/modules/ccp4io"
                  "/builds/science-team/dials/debian/output/dials-2.2.0+modules/modules/ccp4io_adaptbx"
  smtbx           "/builds/science-team/dials/debian/output/dials-2.2.0+modules/modules/cctbx_project/smtbx"
  ucif            "/builds/science-team/dials/debian/output/dials-2.2.0+modules/modules/cctbx_project/ucif"
  cbflib+adaptbx  "/builds/science-team/dials/debian/output/dials-2.2.0+modules/modules/cbflib"
                  "/builds/science-team/dials/debian/output/dials-2.2.0+modules/modules/cctbx_project/cbflib_adaptbx"
  cctbx           "/builds/science-team/dials/debian/output/dials-2.2.0+modules/modules/cctbx_project/cctbx"
  scitbx          "/builds/science-team/dials/debian/output/dials-2.2.0+modules/modules/cctbx_project/scitbx"
  fable           "/builds/science-team/dials/debian/output/dials-2.2.0+modules/modules/cctbx_project/fable"
  omptbx          "/builds/science-team/dials/debian/output/dials-2.2.0+modules/modules/cctbx_project/omptbx"
  boost+adaptbx   "/builds/science-team/dials/debian/output/dials-2.2.0+modules/modules/cctbx_project/boost"
                  "/builds/science-team/dials/debian/output/dials-2.2.0+modules/modules/cctbx_project/boost_adaptbx"
  tbxx            "/builds/science-team/dials/debian/output/dials-2.2.0+modules/modules/cctbx_project/tbxx"
  chiltbx         "/builds/science-team/dials/debian/output/dials-2.2.0+modules/modules/cctbx_project/chiltbx"
  libtbx          "/builds/science-team/dials/debian/output/dials-2.2.0+modules/modules/cctbx_project/libtbx"

[Anthchirp]

I don't think dials uses clipper anywhere. I don't believe it's part of our default environment either, but I could be wrong. Again, this id probably something that should be figured out upstream (ie. with cctbx)

[picca]

thanks, It is just that I am working on dials for now from the dials-v2-2-1-source.tar.xz so I did not tryed to build an independant cctbx. There is no release of cctbx ;)

[picca]

Hello, So I progress, now I built all the extesions and the library, now I need to install the python code. So I am wondering how you are using the setup.py of dials and xia2 ?

I tryed them but I end up with no dials namespace for all the submodiules

something like

buid/lib/array_familly instread of build/lib/dials/array_familly

I had to tune the setup.py like this in order to build a proper module in the debian name space.

packages = ['dials'] + ["dials." + p for p in find_packages('.')]
package_dir={"dials": "../dials"},

with this it is possible to remove the

    data_files=[
        ("dials", ["conftest.py", "__init__.py", "libtbx_refresh.py", "run_tests.py"])
    ],

which seems to be a workaround for the namespace problem.

just for info the run_test.py is not available in the current release 2.2.1 sources.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. The label will be removed automatically if any activity occurs. Thank you for your contributions.

[graeme-winter]

We now have a build process which allows (i) cmake builds and (ii) conda / mamba install dials - would this work for your use case?

[picca]

Hello,

for now I need to run cmake by hand like this

export PYBUILD_NAME=dials export PYBUILD_SYSTEM=distutils export PYBUILD_AFTER_CONFIGURE=cmake -DPython_EXECUTABLE=/usr/bin/{interpreter} -S . -B {build_dir}/lib export PYBUILD_AFTER_BUILD=make -j $(shell nproc) -C {build_dir}/lib && cp {build_dir}/lib/lib/*.so {build_dir} && rm -rf {build_dir}/lib export PYBUILD_DISABLE=test

Is it still necessary ?