pyFAI icon indicating copy to clipboard operation
pyFAI copied to clipboard

Importing `worker` takes a lot of time ...

Open kif opened this issue 1 year ago • 3 comments

Condition: ewoks worker running on slurm cluster; reported by Edgar. Several seconds are reported ...

kif avatar Mar 21 '24 07:03 kif

python3 -X importtime -c "import pyFAI.worker" gives a hint ... the timings are in µs.

kif avatar Mar 21 '24 08:03 kif

On lintaillefer, this takes: 2.1s out of which:

 193.521 ms	        fabio
 423.257 ms	      pyFAI.utils
 435.484 ms	    pyFAI
1466.502 ms	              silx.opencl.common
1466.716 ms	            silx.opencl
1466.735 ms	          silx.opencl.common
1467.515 ms	        pyFAI.opencl
1570.422 ms	      pyFAI.load_integrators
1574.027 ms	    pyFAI.azimuthalIntegrator
2109.442 ms	  pyFAI.worker

Apparently, it is the initialization of OpenCL (which probes all devices) which takes most of the time.

kif avatar Mar 21 '24 08:03 kif

The selection of the OpenCL device can be tuned with OCL_ICD_VENDORS. In this example I point to an ICD for a FPGA board that I don't have:

  54.466 ms	      pyFAI.io.image
  59.140 ms	                numpy.core
 101.420 ms	              numpy
 102.072 ms	    pyFAI.average
 123.890 ms	            fabio.fabioimage
 124.134 ms	          fabio.fabioformats
 138.252 ms	          pkg_resources
 138.401 ms	        pyFAI.resources
 186.043 ms	        fabio
 401.359 ms	      pyFAI.utils
 414.103 ms	    pyFAI
 558.349 ms	  pyFAI.worker

kif avatar Mar 21 '24 08:03 kif