How can I configure HPAT in a cluster enviroment?

[ogreyesp]

Hello,

I'm really interested in using HPAT in my project. I have a HPC cluster with 12 worker nodes. Where can I find an example to configure HPAT in order to use all my nodes and cores?

Best regards

Oscar

[ehsantn]

@ogreyesp HPAT doesn't need any especial configuration; it works on regular MPI clusters. If you can use mpiexec -n 12 ./program for MPI binaries, then you can also use mpiexec -n 12 python hpat_program.py.

[ogreyesp]

Hi @ehsantn ,

Thanks for your response. I have another isue. I have followed the instructions to install HPAT and everything was installed ok. The error that I get is when executing the example of pi estimation. This is the complete error stack trace after executing "mpiexec -n 8 python pi.py". Can you help me?, Thanks.

Traceback (most recent call last): File "/home/ogreyes/anaconda3/envs/HPAT/lib/python3.6/site-packages/numba/targets/options.py", line 17, in from_dict ctor = self.OPTIONS[k] KeyError: 'user_pipeline_funcs'

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "pi.py", line 14, in calc_pi(2 * 10**8) File "/home/ogreyes/anaconda3/envs/HPAT/lib/python3.6/site-packages/numba/dispatcher.py", line 360, in _compile_for_args raise e File "/home/ogreyes/anaconda3/envs/HPAT/lib/python3.6/site-packages/numba/dispatcher.py", line 311, in _compile_for_args return self.compile(tuple(argtypes)) File "/home/ogreyes/anaconda3/envs/HPAT/lib/python3.6/site-packages/numba/dispatcher.py", line 618, in compile cres = self._compiler.compile(args, return_type) File "/home/ogreyes/anaconda3/envs/HPAT/lib/python3.6/site-packages/numba/dispatcher.py", line 74, in compile self.targetdescr.options.parse_as_flags(flags, self.targetoptions) File "/home/ogreyes/anaconda3/envs/HPAT/lib/python3.6/site-packages/numba/targets/options.py", line 27, in parse_as_flags opt.from_dict(options) File "/home/ogreyes/anaconda3/envs/HPAT/lib/python3.6/site-packages/numba/targets/options.py", line 20, in from_dict raise KeyError(fmt % (self.class, k)) KeyError: "<class 'numba.targets.cpu.CPUTargetOptions'> does not support option: 'user_pipeline_funcs'" Traceback (most recent call last): File "/home/ogreyes/anaconda3/envs/HPAT/lib/python3.6/site-packages/numba/targets/options.py", line 17, in from_dict ctor = self.OPTIONS[k] KeyError: 'user_pipeline_funcs'

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "pi.py", line 14, in calc_pi(2 * 10**8) File "/home/ogreyes/anaconda3/envs/HPAT/lib/python3.6/site-packages/numba/dispatcher.py", line 360, in _compile_for_args raise e File "/home/ogreyes/anaconda3/envs/HPAT/lib/python3.6/site-packages/numba/dispatcher.py", line 311, in _compile_for_args return self.compile(tuple(argtypes)) File "/home/ogreyes/anaconda3/envs/HPAT/lib/python3.6/site-packages/numba/dispatcher.py", line 618, in compile cres = self._compiler.compile(args, return_type) File "/home/ogreyes/anaconda3/envs/HPAT/lib/python3.6/site-packages/numba/dispatcher.py", line 74, in compile self.targetdescr.options.parse_as_flags(flags, self.targetoptions) File "/home/ogreyes/anaconda3/envs/HPAT/lib/python3.6/site-packages/numba/targets/options.py", line 27, in parse_as_flags opt.from_dict(options) File "/home/ogreyes/anaconda3/envs/HPAT/lib/python3.6/site-packages/numba/targets/options.py", line 20, in from_dict raise KeyError(fmt % (self.class, k)) KeyError: "<class 'numba.targets.cpu.CPUTargetOptions'> does not support option: 'user_pipeline_funcs'" Traceback (most recent call last): File "/home/ogreyes/anaconda3/envs/HPAT/lib/python3.6/site-packages/numba/targets/options.py", line 17, in from_dict ctor = self.OPTIONS[k] KeyError: 'user_pipeline_funcs'

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "pi.py", line 14, in calc_pi(2 * 10**8) File "/home/ogreyes/anaconda3/envs/HPAT/lib/python3.6/site-packages/numba/dispatcher.py", line 360, in _compile_for_args raise e File "/home/ogreyes/anaconda3/envs/HPAT/lib/python3.6/site-packages/numba/dispatcher.py", line 311, in _compile_for_args return self.compile(tuple(argtypes)) File "/home/ogreyes/anaconda3/envs/HPAT/lib/python3.6/site-packages/numba/dispatcher.py", line 618, in compile cres = self._compiler.compile(args, return_type) File "/home/ogreyes/anaconda3/envs/HPAT/lib/python3.6/site-packages/numba/dispatcher.py", line 74, in compile self.targetdescr.options.parse_as_flags(flags, self.targetoptions) File "/home/ogreyes/anaconda3/envs/HPAT/lib/python3.6/site-packages/numba/targets/options.py", line 27, in parse_as_flags opt.from_dict(options) File "/home/ogreyes/anaconda3/envs/HPAT/lib/python3.6/site-packages/numba/targets/options.py", line 20, in from_dict raise KeyError(fmt % (self.class, k)) KeyError: "<class 'numba.targets.cpu.CPUTargetOptions'> does not support option: 'user_pipeline_funcs'" Traceback (most recent call last): File "/home/ogreyes/anaconda3/envs/HPAT/lib/python3.6/site-packages/numba/targets/options.py", line 17, in from_dict ctor = self.OPTIONS[k] KeyError: 'user_pipeline_funcs'

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "pi.py", line 14, in calc_pi(2 * 10**8) File "/home/ogreyes/anaconda3/envs/HPAT/lib/python3.6/site-packages/numba/dispatcher.py", line 360, in _compile_for_args raise e File "/home/ogreyes/anaconda3/envs/HPAT/lib/python3.6/site-packages/numba/dispatcher.py", line 311, in _compile_for_args return self.compile(tuple(argtypes)) File "/home/ogreyes/anaconda3/envs/HPAT/lib/python3.6/site-packages/numba/dispatcher.py", line 618, in compile cres = self._compiler.compile(args, return_type) File "/home/ogreyes/anaconda3/envs/HPAT/lib/python3.6/site-packages/numba/dispatcher.py", line 74, in compile self.targetdescr.options.parse_as_flags(flags, self.targetoptions) File "/home/ogreyes/anaconda3/envs/HPAT/lib/python3.6/site-packages/numba/targets/options.py", line 27, in parse_as_flags opt.from_dict(options) File "/home/ogreyes/anaconda3/envs/HPAT/lib/python3.6/site-packages/numba/targets/options.py", line 20, in from_dict raise KeyError(fmt % (self.class, k)) KeyError: "<class 'numba.targets.cpu.CPUTargetOptions'> does not support option: 'user_pipeline_funcs'" Traceback (most recent call last): File "/home/ogreyes/anaconda3/envs/HPAT/lib/python3.6/site-packages/numba/targets/options.py", line 17, in from_dict ctor = self.OPTIONS[k] KeyError: 'user_pipeline_funcs'

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "pi.py", line 14, in calc_pi(2 * 10**8) File "/home/ogreyes/anaconda3/envs/HPAT/lib/python3.6/site-packages/numba/dispatcher.py", line 360, in _compile_for_args raise e File "/home/ogreyes/anaconda3/envs/HPAT/lib/python3.6/site-packages/numba/dispatcher.py", line 311, in _compile_for_args return self.compile(tuple(argtypes)) File "/home/ogreyes/anaconda3/envs/HPAT/lib/python3.6/site-packages/numba/dispatcher.py", line 618, in compile cres = self._compiler.compile(args, return_type) File "/home/ogreyes/anaconda3/envs/HPAT/lib/python3.6/site-packages/numba/dispatcher.py", line 74, in compile self.targetdescr.options.parse_as_flags(flags, self.targetoptions) File "/home/ogreyes/anaconda3/envs/HPAT/lib/python3.6/site-packages/numba/targets/options.py", line 27, in parse_as_flags opt.from_dict(options) File "/home/ogreyes/anaconda3/envs/HPAT/lib/python3.6/site-packages/numba/targets/options.py", line 20, in from_dict raise KeyError(fmt % (self.class, k)) KeyError: "<class 'numba.targets.cpu.CPUTargetOptions'> does not support option: 'user_pipeline_funcs'" Traceback (most recent call last): File "/home/ogreyes/anaconda3/envs/HPAT/lib/python3.6/site-packages/numba/targets/options.py", line 17, in from_dict ctor = self.OPTIONS[k] KeyError: 'user_pipeline_funcs'

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "pi.py", line 14, in calc_pi(2 * 10**8) File "/home/ogreyes/anaconda3/envs/HPAT/lib/python3.6/site-packages/numba/dispatcher.py", line 360, in _compile_for_args raise e File "/home/ogreyes/anaconda3/envs/HPAT/lib/python3.6/site-packages/numba/dispatcher.py", line 311, in _compile_for_args return self.compile(tuple(argtypes)) File "/home/ogreyes/anaconda3/envs/HPAT/lib/python3.6/site-packages/numba/dispatcher.py", line 618, in compile cres = self._compiler.compile(args, return_type) File "/home/ogreyes/anaconda3/envs/HPAT/lib/python3.6/site-packages/numba/dispatcher.py", line 74, in compile self.targetdescr.options.parse_as_flags(flags, self.targetoptions) File "/home/ogreyes/anaconda3/envs/HPAT/lib/python3.6/site-packages/numba/targets/options.py", line 27, in parse_as_flags opt.from_dict(options) File "/home/ogreyes/anaconda3/envs/HPAT/lib/python3.6/site-packages/numba/targets/options.py", line 20, in from_dict raise KeyError(fmt % (self.class, k)) KeyError: "<class 'numba.targets.cpu.CPUTargetOptions'> does not support option: 'user_pipeline_funcs'" Traceback (most recent call last): File "/home/ogreyes/anaconda3/envs/HPAT/lib/python3.6/site-packages/numba/targets/options.py", line 17, in from_dict ctor = self.OPTIONS[k] KeyError: 'user_pipeline_funcs'

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "pi.py", line 14, in calc_pi(2 * 10**8) File "/home/ogreyes/anaconda3/envs/HPAT/lib/python3.6/site-packages/numba/dispatcher.py", line 360, in _compile_for_args raise e File "/home/ogreyes/anaconda3/envs/HPAT/lib/python3.6/site-packages/numba/dispatcher.py", line 311, in _compile_for_args return self.compile(tuple(argtypes)) File "/home/ogreyes/anaconda3/envs/HPAT/lib/python3.6/site-packages/numba/dispatcher.py", line 618, in compile cres = self._compiler.compile(args, return_type) File "/home/ogreyes/anaconda3/envs/HPAT/lib/python3.6/site-packages/numba/dispatcher.py", line 74, in compile self.targetdescr.options.parse_as_flags(flags, self.targetoptions) File "/home/ogreyes/anaconda3/envs/HPAT/lib/python3.6/site-packages/numba/targets/options.py", line 27, in parse_as_flags opt.from_dict(options) File "/home/ogreyes/anaconda3/envs/HPAT/lib/python3.6/site-packages/numba/targets/options.py", line 20, in from_dict raise KeyError(fmt % (self.class, k)) KeyError: "<class 'numba.targets.cpu.CPUTargetOptions'> does not support option: 'user_pipeline_funcs'" Traceback (most recent call last): File "/home/ogreyes/anaconda3/envs/HPAT/lib/python3.6/site-packages/numba/targets/options.py", line 17, in from_dict ctor = self.OPTIONS[k] KeyError: 'user_pipeline_funcs'

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "pi.py", line 14, in calc_pi(2 * 10**8) File "/home/ogreyes/anaconda3/envs/HPAT/lib/python3.6/site-packages/numba/dispatcher.py", line 360, in _compile_for_args raise e File "/home/ogreyes/anaconda3/envs/HPAT/lib/python3.6/site-packages/numba/dispatcher.py", line 311, in _compile_for_args return self.compile(tuple(argtypes)) File "/home/ogreyes/anaconda3/envs/HPAT/lib/python3.6/site-packages/numba/dispatcher.py", line 618, in compile cres = self._compiler.compile(args, return_type) File "/home/ogreyes/anaconda3/envs/HPAT/lib/python3.6/site-packages/numba/dispatcher.py", line 74, in compile self.targetdescr.options.parse_as_flags(flags, self.targetoptions) File "/home/ogreyes/anaconda3/envs/HPAT/lib/python3.6/site-packages/numba/targets/options.py", line 27, in parse_as_flags opt.from_dict(options) File "/home/ogreyes/anaconda3/envs/HPAT/lib/python3.6/site-packages/numba/targets/options.py", line 20, in from_dict raise KeyError(fmt % (self.class, k)) KeyError: "<class 'numba.targets.cpu.CPUTargetOptions'> does not support option: 'user_pipeline_funcs'"

[ehsantn]

Looks like your HPAT package might be old, since it seems to expect another Numba version. Could you install from scratch? You could also try installing from source.