spcl
Variable corruption when a dictionary is used in subfunction
Describe the bug While running the following code:
import numpy as np
import dace
def sub_fn(x) -> dace.float32[3]:
STATIC_DATA = {
1: 1.06,
2: 0.98,
3: 99.7,
}
stuff = []
for i in x:
stuff.append(STATIC_DATA[i])
stuff = np.array(stuff)
print(x)
return stuff
@dace.program
def run(x):
stuff: dace.float32[3] = sub_fn(x)
return stuff
if __name__ == "__main__":
ret = run([1, 2, 3])
print(ret)
the output is the following:
[4607452634777659638 4607002274814922588 4636716180731382989]
[ 1.06 0.98 99.7 ]
While the final return value is correct, the argument x has been corrupted after being read.
Expected behavior
I'm new to dace, so I might be doing something wrong here. However, I'd expect x to not be corrupted as such, since it doesn't make any sense at the Python level.
One thing that I find odd is that dace won't run this code if it's all in the same function because of the dictionary (TypeError: unhashable type: 'collections.OrderedDict'). I imagine that dictionaries can't be used with dace and that subfunctions are just currently not checked? What is the best way to handle complex programs that require this type of branching?
Desktop (please complete the following information):
- OS: Linux Mint 22.1
- Python: 3.12.3
- dace: 1.0.2
Thanks for opening the issue! One thing that might be happening is that the printout inside the function is not seeing the correct internal type correctly (looks like it sees a list of integers instead of floats). However, internally DaCe sees it as an array of floats so the return value is correct. We will take a look and see who is not doing the conversion properly.
The second part of the issue (if everything is in the same function) is some early exception being raised that makes the error cryptic. Static dictionaries should work, but dynamically updated dictionaries and lists (such as the one you append to) are not supported and should raise an (informative) error. The error should look nicer than unhashable type (normally it should show up with a file and line number too), so thank you for reporting! We will make sure the correct error is raised.
If you want to ensure the static dictionary works, you can also move it outside the function entirely (to the global scope) and it will be seen as a constant.
Thanks for the precisions, here are a couple more observations.
I thought that the issue might just be in the printing, but I got a KeyError exception later in my full code when trying to use x as key a second time. Here is also a minimal example that seems to indicate the actual modification of x:
import numpy as np
import dace
STATIC_DATA = {
1: 1.06,
2: 0.98,
3: 99.7,
}
def sub_fn(x) -> dace.float32[3]:
print("Sum2: ", sum(x))
stuff = []
for i in x:
stuff.append(STATIC_DATA[i])
stuff = np.array(stuff)
print("Sum3: ", sum(x))
print(x)
return stuff
@dace.program
def run(x):
print("Sum1: ", sum(x))
stuff: dace.float32[3] = sub_fn(x)
return stuff
if __name__ == "__main__":
ret = run([1, 2, 3])
print(ret)
This code produces the following:
/home/raphael/.../lib/python3.12/site-packages/dace/frontend/python/newast.py:4605: UserWarning: Performance warning: Automatically creating callback to Python interpreter from method "print". If you would like to know why parsing failed, please place a @dace.program decorator on the function. If a DaCe function cannot be provided (for example, due to recursion), register a replacement through "dace.frontend.common.op_repository".
warnings.warn('Performance warning: Automatically creating '
Exception Function returns 0 values but 1 provided
encountered in File "/home/raphael/tmp/dace_bug/go.py", line 12, column 4 raised while parsing DaCe program:
in File "/home/raphael/tmp/dace_bug/go.py", line 12
stuff = []
/home/raphael/.../lib/python3.12/site-packages/dace/frontend/python/newast.py:4605: UserWarning: Performance warning: Automatically creating callback to Python interpreter from method "sub_fn". If you would like to know why parsing failed, please place a @dace.program decorator on the function. If a DaCe function cannot be provided (for example, due to recursion), register a replacement through "dace.frontend.common.op_repository".
warnings.warn('Performance warning: Automatically creating '
WARNING: Casting list argument "x" to ndarray
Sum1: 6
Sum2: 6
/home/raphael/tmp/dace_bug/go.py:16: RuntimeWarning: overflow encountered in scalar add
print("Sum3: ", sum(x))
Sum3: -4595572983385586401
[4607452634777659638 4607002274814922588 4636716180731382989]
[ 1.06 0.98 99.7 ]
And indeed the last sum would be the sum of the printed x:
>>> sum([4607452634777659638, 4607002274814922588, 4636716180731382989]) - 2**64
-4595572983385586401
Also, to be sure I understand the proper way to do things, this operation with the dynamically constructed array should be done outside the DaCe program, right? Or is there a proper way to do it inside? A list comprehension was my initial approach, but it seems to be identical down the hood from the point of view of dace.
Thanks a lot!
Here's another odd case:
import numpy as np
import dace
def sub_fn(x) -> dace.float32[3]:
stuff = np.zeros((3,)) # This unsets x for some reason
print("Sum2: ", sum(x))
return x
@dace.program
def run(x):
print("Sum1: ", sum(x))
stuff: dace.float32[3] = sub_fn(x)
return stuff
if __name__ == "__main__":
ret = run([1, 2, 3])
print(ret)
This code outputs:
/home/raphael/.../lib/python3.12/site-packages/dace/frontend/python/newast.py:4605: UserWarning: Performance warning: Automatically creating callback to Python interpreter from method "print". If you would like to know why parsing failed, please place a @dace.program decorator on the function. If a DaCe function cannot be provided (for example, due to recursion), register a replacement through "dace.frontend.common.op_repository".
warnings.warn('Performance warning: Automatically creating '
Exception Return values of a data-centric function must always have the same type and shape
encountered in File "/home/raphael/tmp/dace_bug/go.py", line 7, column 4 raised while parsing DaCe program:
in File "/home/raphael/tmp/dace_bug/go.py", line 7
return x
/home/raphael/.../lib/python3.12/site-packages/dace/frontend/python/newast.py:4605: UserWarning: Performance warning: Automatically creating callback to Python interpreter from method "sub_fn". If you would like to know why parsing failed, please place a @dace.program decorator on the function. If a DaCe function cannot be provided (for example, due to recursion), register a replacement through "dace.frontend.common.op_repository".
warnings.warn('Performance warning: Automatically creating '
WARNING: Casting list argument "x" to ndarray
Sum1: 6
Sum2: 0
[0. 0. 0.]
meaning that x has been overwritten. Just calling np.zeros((3,)) without assigning the value to a new variable does the same thing.
However, this code works as expected, despite the same warning messages:
import numpy as np
import dace
def sub_fn(x) -> dace.float32[3]:
stuff = np.zeros((3,), dtype=np.float32) # The datatype is explicitly defined here
print("Sum2: ", sum(x))
return x
@dace.program
def run(x):
print("Sum1: ", sum(x))
stuff: dace.float32[3] = sub_fn(x)
return stuff
if __name__ == "__main__":
ret = run([1, 2, 3])
print(ret)
It outputs the following:
/home/raphael/.../lib/python3.12/site-packages/dace/frontend/python/newast.py:4605: UserWarning: Performance warning: Automatically creating callback to Python interpreter from method "print". If you would like to know why parsing failed, please place a @dace.program decorator on the function. If a DaCe function cannot be provided (for example, due to recursion), register a replacement through "dace.frontend.common.op_repository".
warnings.warn('Performance warning: Automatically creating '
Exception Return values of a data-centric function must always have the same type and shape
encountered in File "/home/raphael/tmp/dace_bug/go.py", line 8, column 4 raised while parsing DaCe program:
in File "/home/raphael/tmp/dace_bug/go.py", line 8
return x
/home/raphael/.../lib/python3.12/site-packages/dace/frontend/python/newast.py:4605: UserWarning: Performance warning: Automatically creating callback to Python interpreter from method "sub_fn". If you would like to know why parsing failed, please place a @dace.program decorator on the function. If a DaCe function cannot be provided (for example, due to recursion), register a replacement through "dace.frontend.common.op_repository".
warnings.warn('Performance warning: Automatically creating '
WARNING: Casting list argument "x" to ndarray
Sum1: 6
Sum2: 6
[1. 2. 3.]
Moreover, the output signature also seems to trigger this behavior:
import numpy as np
import dace
def sub_fn(x) -> dace.float32[3]: # Bug with signature, no bug without
np.zeros((3,))
return x
@dace.program
def run(x):
return sub_fn(x)
if __name__ == "__main__":
ret = run([1, 2, 3])
print(ret)
A mismatch between the output signature and the datatype used seems to be linked to the bug:
import numpy as np
import dace
def sub_fn(x) -> dace.float32[3]:
np.zeros((3,), dtype=np.float64) # This unsets x, but only if the datatype is `np.float64` or `np.int64`, not `np.float32`, `np.float16`, `np.int32` or `np.int16`.
return x
@dace.program
def run(x):
return sub_fn(x)
if __name__ == "__main__":
ret = run([1, 2, 3])
print(ret)