UltraNest icon indicating copy to clipboard operation
UltraNest copied to clipboard

Issue when running on SLURM cluster

Open rfranceschi opened this issue 1 year ago • 4 comments

  • UltraNest version: >= 3.3.3
  • Python version: 3.8.15
  • Operating System: SUSE Linux Enterprise Server 12 SP5

Description

Running UltraNest on a remote SLURM cluster on multiple nodes using Intel MPI 2019.9.

What I Did

Previously my code has been running without issues. There have been no changes in the fitting part of my code, but now it fails with the following message:

File ".../fitters.py", line 437, in fit
    for i, result in enumerate(self.sampler.run_iter(**self.run_kwargs)):
File ".../lib/python3.8/site-packages/ultranest/integrator.py", line 2579, in run_iter
    dlogz_min_num_live_points, (Llo_KL, Lhi_KL), (Llo_ess, Lhi_ess) = self._find_strategy(
  File ".../lib/python3.8/site-packages/ultranest/integrator.py", line 1545, in _find_strategy
    widthratio = 1 - np.exp(logweights[1:,0] - logweights[:-1,0])

This is how the sampler is initialized, nothing special happening here:

self.sampler = ultranest.ReactiveNestedSampler(
            [str(param) for param in self.parameters],
            lnprob,
            self.transform,
            log_dir=self.log_dir,
            resume=self.resume,
            storage_backend=self.storage_backend,
            **self.fitter_kwargs
        )

This is not happening on my local machine and I suspect this may be due to a change in the cluster. Could this be the case, or could this be an issue within UltraNest or my code?

Thank you in advance!

rfranceschi avatar Nov 24 '22 16:11 rfranceschi