QMCPACK
Nexus: Custom is_int() function in qmcpack_input.py Script Will Make Scientific Notation "Int" Valued for Python 3.12.2
Describe the bug I am piloting using QMCPack on my university's HPC by running the "QMCPack Summer School 2025" tutorials. While executing the lih_workflow.py in "session3_statistics_and_workflows/02_lih_hf_vmc" on my HPC, in the I get the following error when the script tries to read in the *.xml in the vmc step:
"ValueError: could not convert string '3.0139241961e+00' to int64 at row 1, column 1."
I have identified that the issue comes from the "qmcpack_input.py" script (directory: qmcpack-4.1.0/nexus/lib) [Line 1610]. When reading the structure xml document, the "read" function calls to another custom function called "is_int" to check if the value is an int before parsing it as an int type. The float test comes afterwards.
To Reproduce
The custom "is_int" uses Python's built in int() function to see if the passed value turns into a int or not. If the variable becomes an int, the test passes. For my system's Python 3.12.2 version, a Python variable assigned a value like "3.0139241961e+00" will automatically get read as a float, which can get turned into an integer with int(). (See custom Python test output attached where I checked this.) Since the tests passes, the "read" function in qmcpack_input.py assumes the value read in from the structure value is an int64 type, and subsequently fails when trying to read in the float value.
Expected behavior I assume that some Python int() versions will give scientific notation a value error so the float condition gets accepted next (i.e., the one used in the summer school virtual box).
However, in case this is an issue for later Python versions, could an updated read-in filter that can handle scientific notation get created? Also, does anyone know of any short-term fixes in case I run into this for other tests?
System:
- system name: Prime (located at NDSU's CCAST)
- modules loaded: 1) DefaultModules 9) mkl/2023.2.0 2) openpbs/openpbs/22.05.11 10) openblas/dynamic/0.3.18
- gcc/11.2.0 11) lapack/gcc/64/3.10.0 4) tbb/2021.10.0 12) boost/1.88.0
- compiler-rt/2023.0.0 13) fftw/3.3.10 6) oclfpga/2023.2.0 14) cmake/3.30.3
- compiler/2023.0.0 15) hdf5/1.12.1-intel 8) mpi/2021.10.0 16) libxml2/2.9.10-intel-ab5p ---Loaded and execute in a Python Virtual environment which includes all needed modules; Python 3.12.2; note that int() is a default Python function
- other systems where this is reproducible [none; virtual box does not do this]
Additional context I am the first person testing QMCPack on the the Prime HPC system.
Thanks for reporting and apologies for the issue. We'll take a look.
@Kcorbyerd Given your #5625 , this is very timely. Any thoughts?
I had a response typed up that would work in theory but actually failed to capture the larger error at play here.
What happened here is that the position array contains 5 int and one float, but the check that we have for integers in that only checks the first value because it assumes the array has a homogeneous type. In this case, the first 3 numbers are int, the 4th is float, and the remaining 2 are int again.
There are two possible fixes you can try, first and perhaps the most simple is to rewrite your XML file to include trailing periods, like so:
<?xml version="1.0"?>
<qmcsystem>
<particleset name="ion0" size="2">
<group name="Li">
<parameter name="charge">1</parameter>
<parameter name="valence">1</parameter>
<parameter name="atomicnumber">3</parameter>
</group>
<group name="H">
<parameter name="charge">1</parameter>
<parameter name="valence">1</parameter>
<parameter name="atomicnumber">1</parameter>
</group>
<attrib name="position" datatype="posArray">
0. 0. 0.
3.0139241961e+00 0. 0.
</attrib>
<attrib name="ionid" datatype="stringArray">
Li H
</attrib>
</particleset>
<particleset name="e" random="yes" randomsrc="ion0">
<group name="u" size="1">
<parameter name="charge">-1</parameter>
</group>
<group name="d" size="1">
<parameter name="charge">-1</parameter>
</group>
</particleset>
</qmcsystem>
Alternatively, I have a current bandaid patch that should work to directly circumvent the issue in the XML file you provided.
You can replace line 1603 in qmcpack_input.py with the following code:
try:
val = loadtxt(StringIO(xml.text),int)
except ValueError:
val = loadtxt(StringIO(xml.text),float)
This is a very messy fix, but will circumvent the error you're finding currently. There may be errors elsewhere that weren't caught due to the program stopping after this one, but I don't know where those will be.
I will look into a more robust fix while I work on Issue #5625 but until then this may have to suffice.
One additional comment @hgriffin-ux.
The test you provided didn't show the error right away because of a bit of nuance in Python's int() functionality.
For the test you provided, we see
>>> int(3.0139241961e+00)
3
>>> int(3.01)
3
>>> int(3)
3
with no errors raised, however if we were to run
>>> int("3.0139241961e+00")
Traceback (most recent call last):
File "<python-input-0>", line 1, in <module>
int("3.0139241961e+00")
~~~^^^^^^^^^^^^^^^^^^^^
ValueError: invalid literal for int() with base 10: '3.0139241961e+00'
we get the observed ValueError.
This is because there are two different int() functions in Python. One occurs when you call int(float) and simply truncates the fractional part of the float while calling int(str) tries to parse the string in base-10 format by default.
Thank @Kcorbyerd for explaining what the issue is in the script, and for the temporary solution. Updating the structure *.xml position coordinates from 0 to 0. worked. Also, thank you for explaining more on how the int() is intended to work with a string input.
@hgriffin-ux and/or @Kcorbyerd, please can you post here a sequence of one-line cases that show the issue, along with the current output from the is_int, etc, functions?
We should fix the is_int function directly. The logic is still correct even if the function is buggy. I expect it won't be hard.
Original is_int()
def is_int(var):
try:
int(var)
return True
except ValueError:
return False
#end try
#end def is_int
>>> is_int( 1 )
True
>>> is_int( 1.1 )
False
>>> is_int( 1e0 )
False
>>> is_int( 1.e0 )
False
>>> is_int( 1.1e0 )
False
>>> is_int( 1e+00 )
False
>>> is_int( 1.1e+00 )
False
>>> is_int( 1e-00 )
False
>>> is_int( 1e-01 )
False
>>> is_int( 1.01e-00 )
False
>>> is_int( 1.0e-00 )
False
>>> is_int( "1" )
True
>>> is_int( "1.1" )
False
>>> is_int( "1e0" )
False
>>> is_int( "1.e0" )
False
>>> is_int( "1.1e0" )
False
>>> is_int( "1e+00" )
False
>>> is_int( "1.1e+00" )
False
>>> is_int( "1e-00" )
False
>>> is_int( "1e-01" )
False
>>> is_int(b"1.01e-00")
False
>>> is_int(b"1.0e-00" )
False
>>> is_int( "abcdefg" )
False
NumPy < 1.23
>>> ex_str = "0 1\n2 3.1"
>>> is_int(ex_str[0])
True
>>> np.loadtxt(StringIO(c), int)
ValueError: invalid literal for int() with base 10: '3.1'
Complete failure to parse here, and in qmcpack_input.py this happens in both try... except: cases since the first part of ex_str is an integer, but it is not homogeneous, and so the entire section of code will fail since the second error is not properly caught and handled.
NumPy >= 1.23
>>> ex_str = "0 1\n2 3.1"
>>> is_int(ex_str[0])
True
>>> val = np.loadtxt(StringIO(c), int)
DeprecationWarning: loadtxt(): Parsing an integer via a float is deprecated. To avoid this warning, you can:
* make sure the original data is stored as integers.
* use the `converters=` keyword argument. If you only use
NumPy 1.23 or later, `converters=float` will normally work.
* Use `np.loadtxt(...).astype(np.int64)` parsing the file as
floating point and then convert it. (On all NumPy versions.)
(Deprecated NumPy 1.23)
val = np.loadtxt(StringIO(ex_str), int)
>>> print(val)
[[0 1]
[2 3]]
Here the code will work, but will yield both a deprecation warning and also will force all floats to be integers, so if there are non-integer numbers with non-zero digits after the decimal that will be lost.
Possibly fix is_int()
I am not sure it is feasible to fix is_int() and have it retain its current functionality. Part of the issue is with the logic in lines 1600-1621 in qmcpack_input.py that only checks for the identity of the first item in the xml.text string, so if the data is not homogeneous then it will fail on the int conversion. If the is_int() function were to be updated to try and check if the number is mathematically an integer instead of computationally (i.e. 1.0 is an int in the mathematics sense, but is a float in the computational sense), then the behavior would change from those cases being changed to floats to being changed to ints.
The fix in PR #5628 retains the same behavior, but does not fail with inhomogeneous inputs.
Original is_float()
def is_float(var):
try:
float(var)
return True
except ValueError:
return False
#end try
#end def is_float
>>> is_float( 1 )
True
>>> is_float( 1.1 )
True
>>> is_float( 1e0 )
True
>>> is_float( 1.e0 )
True
>>> is_float( 1.1e0 )
True
>>> is_float( 1e+00 )
True
>>> is_float( 1.1e+00 )
True
>>> is_float( 1e-00 )
True
>>> is_float( 1e-01 )
True
>>> is_float( 1.01e-00 )
True
>>> is_float( 1.0e-00 )
True
>>> is_float( "1" )
True
>>> is_float( "1.1" )
True
>>> is_float( "1e0" )
True
>>> is_float( "1.e0" )
True
>>> is_float( "1.1e0" )
True
>>> is_float( "1e+00" )
True
>>> is_float( "1.1e+00" )
True
>>> is_float( "1e-00" )
True
>>> is_float( "1e-01" )
True
>>> is_float(b"1.01e-00")
True
>>> is_float(b"1.0e-00" )
True
>>> is_float( "abcdefg" )
False
This shows that swapping is_int() and is_float() will essentially make is_int() obsolete since is_float() will catch all of the numbers that is_int() would normally catch.
Following extensive offline conversation, @Kcorbyerd and I have developed a fix.
This issue highlights problems with other int dectection functions in other places in Nexus that should be prioritized for fixes.