AMDMIGraphX icon indicating copy to clipboard operation
AMDMIGraphX copied to clipboard

UInt64 Overflow with Higher Batch Sizes

Open eddieliao opened this issue 6 months ago • 0 comments

There appears to be an occasional issue in which we try to allocate a buffer to the gpu that seems to be an overflow of an UInt64. @kahmed10 has reportedly gotten the following error when testing higher batch sizes:

what():  /long_pathname_so_that_rpms_can_package_the_debug_info/src/extlibs/AMDMIGraphX/src/targets/gpu/hip.cpp:109: allocate_gpu: Memory not available to allocate buffer: 18446744040423555073

I have gotten a similar issue when using the std::accumulate function to sum up the size of parameters. I was able to resolve this by casting the final parameter 0 to a unsigned long long:

size_t shape_size = std::accumulate(static_param_shapes.begin(),
                                    static_param_shapes.end(),
                                    static_cast<unsigned long long>(0),
                                    [](auto s, auto x) { return s + x.second.bytes(); });

Need to investigate whether this is related to the original error.

eddieliao avatar Jul 26 '24 16:07 eddieliao