ValueError: cannot convert float NaN to integer

[IamSreeman]

I am running the following cell in Jupyter notebook

# Export the results to GetDist
from getdist.mcsamples import loadMCSamples
# Notice loadMCSamples requires a *full path*
import os

gd_sample = loadMCSamples(os.path.abspath(info_from_yaml["output"]), settings={'ignore_rows':0.3})
# Analyze and plot
print("H0: Mean=",gd_sample.mean("H0"),", Sigma=",gd_sample.std("H0"))
print("omega_b: Mean=",gd_sample.mean("omega_b"),", Sigma=",gd_sample.std("omega_b"))
print("omega_cdm: Mean=",gd_sample.mean("omega_cdm"),", Sigma=",gd_sample.std("omega_cdm"))
print("Omega_m: Mean=",gd_sample.mean("Omega_m"),", Sigma=",gd_sample.std("Omega_m"))
print("Omega_Lambda: Mean=",gd_sample.mean("Omega_Lambda"),", Sigma=",gd_sample.std("Omega_Lambda"))
%matplotlib inline
import getdist.plots as gdplt

gdplot = gdplt.get_subplot_plotter(width_inch= 15)
gdplot.triangle_plot(gd_sample, ["H0", "omega_b", "omega_cdm", "Omega_m"], filled=True)

I am getting the following error even though the output files have normal real numbers

WARNING:root:outlier fraction 0.13043478260869565 
H0: Mean= 7.269795215476191e+01 , Sigma= 0.6005556020278313
omega_b: Mean= 0.03431773125 , Sigma= 0.0009386554812050743
omega_cdm: Mean= 0.13674215666666667 , Sigma= 0.003244985511507143
Omega_m: Mean= 0.32498746928571426 , Sigma= 0.010351214635157764
Omega_Lambda: Mean= 0.6749439983333334 , Sigma= 0.010352008717849429
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Input In [22], in <cell line: 18>()
     15 import getdist.plots as gdplt
     17 gdplot = gdplt.get_subplot_plotter(width_inch= 15)
---> 18 gdplot.triangle_plot(gd_sample, ["H0", "omega_b", "omega_cdm", "Omega_m"], filled=True)

File ~/anaconda3/lib/python3.9/site-packages/getdist/plots.py:2476, in GetDistPlotter.triangle_plot(self, roots, params, legend_labels, plot_3d_with_param, filled, shaded, contour_args, contour_colors, contour_ls, contour_lws, line_args, label_order, legend_ncol, legend_loc, title_limit, upper_roots, upper_kwargs, upper_label_right, diag1d_kwargs, markers, marker_args, param_limits, **kwargs)
   2472     self.plot_3d(roots, pair + [col_param], color_bar=False, line_offset=1, add_legend_proxy=False,
   2473                  do_xlabel=i2 == plot_col - 1, do_ylabel=i == 0, contour_args=contour_args,
   2474                  no_label_no_numbers=self.settings.no_triangle_axis_labels, ax=ax, **kwargs)
   2475 else:
-> 2476     self.plot_2d(roots, param_pair=pair, do_xlabel=i2 == plot_col - 1, do_ylabel=i == 0,
   2477                  no_label_no_numbers=self.settings.no_triangle_axis_labels, shaded=shaded,
   2478                  add_legend_proxy=i == 0 and i2 == 1, contour_args=contour_args, ax=ax, **kwargs)
   2479 if marker is not None:
   2480     self.add_x_marker(marker, ax=ax, **marker_args)

File ~/anaconda3/lib/python3.9/site-packages/getdist/plots.py:1686, in GetDistPlotter.plot_2d(self, roots, param1, param2, param_pair, shaded, add_legend_proxy, line_offset, proxy_root_exclude, ax, **kwargs)
   1684 contour_args = self._make_contour_args(len(roots), **kwargs)
   1685 for i, root in enumerate(roots):
-> 1686     res = self.add_2d_contours(root, param_pair[0], param_pair[1], line_offset + i, of=len(roots), ax=ax,
   1687                                add_legend_proxy=add_legend_proxy and root not in proxy_root_exclude,
   1688                                **contour_args[i])
   1689     xbounds, ybounds = self._update_limits(res, xbounds, ybounds)
   1690 if xbounds is None:

File ~/anaconda3/lib/python3.9/site-packages/getdist/plots.py:1057, in GetDistPlotter.add_2d_contours(self, root, param1, param2, plotno, of, cols, contour_levels, add_legend_proxy, param_pair, density, alpha, ax, **kwargs)
   1055     density = root.marginalizedMixture(params=[param1, param2]).density2D()
   1056 else:
-> 1057     density = self.sample_analyser.get_density_grid(root, param1, param2,
   1058                                                     conts=self.settings.num_plot_contours,
   1059                                                     likes=self.settings.shade_meanlikes)
   1060 if density is None:
   1061     if add_legend_proxy:

File ~/anaconda3/lib/python3.9/site-packages/getdist/plots.py:613, in MCSampleAnalysis.get_density_grid(self, root, param1, param2, conts, likes)
    611 if not density:
    612     samples = self.samples_for_root(root)
--> 613     density = samples.get2DDensityGridData(param1.name, param2.name, num_plot_contours=conts, meanlikes=likes)
    614     if density is None:
    615         return None

File ~/anaconda3/lib/python3.9/site-packages/getdist/mcsamples.py:1719, in MCSamples.get2DDensityGridData(self, j, j2, num_plot_contours, get_density, meanlikes, **kwargs)
   1716 if smooth_scale < 2:
   1717     logging.warning('fine_bins_2D not large enough for optimal density: %s, %s', parx.name, pary.name)
-> 1719 winw = int(round(2.5 * smooth_scale))
   1721 Cinv = np.linalg.inv(np.array([[ry ** 2, rx * ry * corr], [rx * ry * corr, rx ** 2]]))
   1722 ix1, ix2 = np.mgrid[-winw:winw + 1, -winw:winw + 1]

ValueError: cannot convert float NaN to integer

[cmbant]

You can use seterr to raise and an exception where the NaN is generated to see where it's coming from (presumably the bandwidth estimation, but does not normally give this error) https://numpy.org/doc/stable/reference/generated/numpy.seterr.html

[ShazAlvi]

Hi. I just ran into a problem that gives the same error but perhaps they come from different sources. The chains which had been working on the previous version of GetDist are no longer working. This is the error that I am getting:

 File "/home/shazalvi/anaconda3/envs/BasePython/lib/python3.9/site-packages/getdist/mcsamples.py", line 2159, in getMargeStats
    self._setDensitiesandMarge1D()
  File "/home/shazalvi/anaconda3/envs/BasePython/lib/python3.9/site-packages/getdist/mcsamples.py", line 2253, in _setDensitiesandMarge1D
    self.get1DDensityGridData(j, paramConfid=paramConfid, meanlikes=meanlikes)
  File "/home/shazalvi/anaconda3/envs/BasePython/lib/python3.9/site-packages/getdist/mcsamples.py", line 1453, in get1DDensityGridData
    bandwidth = self.getAutoBandwidth1D(bins, par, j, mult_bias_correction_order,
  File "/home/shazalvi/anaconda3/envs/BasePython/lib/python3.9/site-packages/getdist/mcsamples.py", line 1166, in getAutoBandwidth1D
    N_eff = self._get1DNeff(par, param)
  File "/home/shazalvi/anaconda3/envs/BasePython/lib/python3.9/site-packages/getdist/mcsamples.py", line 1144, in _get1DNeff
    par.N_eff_kde = self.getEffectiveSamplesGaussianKDE(param, scale=par.sigma_range)
  File "/home/shazalvi/anaconda3/envs/BasePython/lib/python3.9/site-packages/getdist/chains.py", line 508, in getEffectiveSamplesGaussianKDE
    maxoff = int(self.getCorrelationLength(d, weight_units=False) * 1.5) + 4
ValueError: cannot convert float NaN to integer

Has anyone been able to fix this problem?

[cmbant]

Can you try my suggestion above about using "seterr" to raise a more useful message where the NaN is actually produced?

Otherwise, some chains to reproduce might be useful.

[ShazAlvi]

Hi! I wanted to ask where I should put the seterr command. I am using a python file to produce plots. I can also provide the chains.

[cmbant]

You can probably call it as soon as you've loaded numpy

[ShazAlvi]

I inserted "np.seterr(all='raise')" right after I call numpy.

I get the following error, which I think is the same error that I got before.

Traceback (most recent call last):
  File "/media/shazalvi/Seagate Expansion Drive/SHAHBAZ/Unife_GoogleDrive/Fisica_Ferrara_PhD/Results_And_ValDocs/Parameter_Constraint/MontePython/DCDM/NewCLASS/GetDist_Plotter_r.py", line 25, in <module>
    Marge_S1 = samples1.getMargeStats()
  File "/home/shazalvi/anaconda3/envs/BasePython/lib/python3.9/site-packages/getdist/mcsamples.py", line 2159, in getMargeStats
    self._setDensitiesandMarge1D()
  File "/home/shazalvi/anaconda3/envs/BasePython/lib/python3.9/site-packages/getdist/mcsamples.py", line 2253, in _setDensitiesandMarge1D
    self.get1DDensityGridData(j, paramConfid=paramConfid, meanlikes=meanlikes)
  File "/home/shazalvi/anaconda3/envs/BasePython/lib/python3.9/site-packages/getdist/mcsamples.py", line 1453, in get1DDensityGridData
    bandwidth = self.getAutoBandwidth1D(bins, par, j, mult_bias_correction_order,
  File "/home/shazalvi/anaconda3/envs/BasePython/lib/python3.9/site-packages/getdist/mcsamples.py", line 1166, in getAutoBandwidth1D
    N_eff = self._get1DNeff(par, param)
  File "/home/shazalvi/anaconda3/envs/BasePython/lib/python3.9/site-packages/getdist/mcsamples.py", line 1144, in _get1DNeff
    par.N_eff_kde = self.getEffectiveSamplesGaussianKDE(param, scale=par.sigma_range)
  File "/home/shazalvi/anaconda3/envs/BasePython/lib/python3.9/site-packages/getdist/chains.py", line 508, in getEffectiveSamplesGaussianKDE
    maxoff = int(self.getCorrelationLength(d, weight_units=False) * 1.5) + 4
ValueError: cannot convert float NaN to integer

[cmbant]

Thanks weird, not sure how you still get NaNs. Can you attach zip of chains and code to reproduce?

[ShazAlvi]

Sure. I am attaching them here. Chains.zip

[cmbant]

Thanks - also code to reproduce? Chain metadata files? (if from Cobaya/cosmomc)

[ShazAlvi]

Ahh, right. Here are the files. Chains.zip

And here is the code:

mult_order = 1.0
ignore_r = 0.1
print('rLDDM_PD_HighTTTEEE_LowTTEE_Lens_BAOSDSSDR16_MGS6dF_BICEP2')
samples1 = loadMCSamples('With_HF/rLDDM_PD_HighTTTEEE_LowTTEE_Lens_BAOSDSSDR16_MGS6dF_BICEP2/2022-07-28_800000_', settings={'ignore_rows':ignore_r, 'boundary_correction_order':0,
                                 'mult_bias_correction_order':mult_order})
g = plots.get_subplot_plotter(width_inch=8)
g.settings.figure_legend_frame = False
g.settings.legend_fontsize = 18
g.settings.axes_labelsize = 22
g.settings.axes_fontsize = 16
g.fig_width_inch = 20

g.triangle_plot([samples1], ['omega_ini_dcdm', 'Gamma_18', 'sigma8', 'H0', 'r'], legend_labels=Labels, filled_compare=True, 
                legend_loc = 'upper right', param_limits={'Gamma_18': [0.0,0.25]}, contour_colors=[('#F4D03F', '#EB984E'), ('#F5B7B1', '#EC7063'), ('#D7BDE2', '#AF7AC5')])

g.export('Parameter_Plots/' + filename + '.png')
g.export('Parameter_Plots/' + filename + '.pdf')
plt.show()

[cmbant]

This doesn't look like a working script - "labels" is undefined, and /2022-07-28_800000_.paramnames does not have "r" as a parameter ?

[ShazAlvi]

Sorry for the confusion. I took the code out of a bigger file that has other chains as well and I missed the part which defines labels. Also, the file paramnames was also changed to have nice symbols in the plot and I mistakenly put the wrong file in the zip. The following should be okay.

Chains.zip

Code:

mult_order = 1.0
ignore_r = 0.1
samples1 = loadMCSamples('With_HF/rLDDM_PD_HighTTTEEE_LowTTEE_Lens_BAOSDSSDR16_MGS6dF_BICEP2/2022-07-28_800000_', settings={'ignore_rows':ignore_r, 'boundary_correction_order':0,
                                 'mult_bias_correction_order':mult_order})
Labels = [r'Planck T&P + lensing + BAO, $\Lambda\mathrm{DDM} + r$']
g = plots.get_subplot_plotter(width_inch=8)
g.settings.figure_legend_frame = False
g.settings.legend_fontsize = 18
g.settings.axes_labelsize = 22
g.settings.axes_fontsize = 16
g.fig_width_inch = 20

g.triangle_plot([samples1], ['omega_ini_dcdm', 'Gamma_18', 'sigma8', 'H0', 'r'], legend_labels=Labels, filled_compare=True, 
                legend_loc = 'upper right', param_limits={'Gamma_18': [0.0,0.25]}, contour_colors=[('#F4D03F', '#EB984E'), ('#F5B7B1', '#EC7063'), ('#D7BDE2', '#AF7AC5')])
plt.show()

[cmbant]

Thanks, though this now runs OK for me without raising an error. It may be something do with a numpy/scipy update (since this part of getdist has not changed recently, so some other module version change could be the issue). My config is Win64 with

[IamSreeman]

@cmbant can this issue be related to whether we use ifort or gfortan?

[cmbant]

No, getdist is pure python

---

From: Kasi Reddy Sreeman Reddy @.> Sent: Tuesday, August 2, 2022 7:49:17 PM To: cmbant/getdist @.> Cc: Antony Lewis @.>; Mention @.> Subject: Re: [cmbant/getdist] ValueError: cannot convert float NaN to integer (Issue #83)

@cmbanthttps://github.com/cmbant can this issue be related to whether we use ifort or gfortan?

— Reply to this email directly, view it on GitHubhttps://github.com/cmbant/getdist/issues/83#issuecomment-1203090991, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AA6GQIZCQ2E3GIMWZ7FIC7TVXFUS3ANCNFSM54NHGW4A. You are receiving this because you were mentioned.Message ID: @.***>

[IamSreeman]

No, getdist is pure python … ________________________________ From: Kasi Reddy Sreeman Reddy @.> Sent: Tuesday, August 2, 2022 7:49:17 PM To: cmbant/getdist @.> Cc: Antony Lewis @.>; Mention @.> Subject: Re: [cmbant/getdist] ValueError: cannot convert float NaN to integer (Issue #83) @cmbanthttps://github.com/cmbant can this issue be related to whether we use ifort or gfortan? — Reply to this email directly, view it on GitHub<#83 (comment)>, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AA6GQIZCQ2E3GIMWZ7FIC7TVXFUS3ANCNFSM54NHGW4A. You are receiving this because you were mentioned.Message ID: @.***>

Ok, thanks.

[cmbant]

I also can't reproduce in a vanilla Ubuntu setup. What modules/system gives the problem? Perhaps you could trace through the code with e.g. PyCharm to see where the NaN is appearing.

[ShazAlvi]

Hi,

Thanks for looking into this error. I check GetDist versions in two different python environments in Conda. Here is the configuration: In the following configuration, there is no error. In the following configuration I get the errors that I mentioned before:

I have not explored further the problem at the moment.