ColossalAI
ColossalAI copied to clipboard
[autoparallel] add data_ptr
What's new?
Since all data in OperationData
are meta tensors, their tensor.data_ptr()=0
. I replace the original tensor.data_ptr()
with a customized function generated by uuid4()
.
Why data_ptr?
Generally speaking, for fine-grained memory calculations, I need to specify which tensors are the same tensors which occupy the same piece of memory on the devices.
Test
This is still a draft?