tvm
tvm copied to clipboard
[Virtual Machine] Implementation of 'set_output_zero_copy'
This is a draft of implementation of 'set_output_zero_copy' method on VirtualMachine side.
Brief description of approach.
- There is python API function 'set_output' which save external outputs in VM outputs_ field (map) for specified func name. It looks like 'set_input' method.
- During 'invoke' outputs_ are saved in register_file. For this the register indices of output tensors are found from code_ field. I observed in tests for different models that AllocTensor and AllocADT ops are used for result tensors. Let's consider these two cases: result index is destination for AllocTensor op or AllocADT op. At the first case instead of construction new NDArray the outside output tensor is used. At the second one the fields of AllocADT are analyzed and register indices are extracted. During tests I observed that ReshapeTensor operation is rarely used as final one (SqueezeNet-v1.0 and DUC). Mechanism for replacement by external output tensors was also implemented for this op.
Notes: 1. I'm not sure that it works for many frames. Practically it looks like we need code_ not frame and any number of frames does not change ops stack (code_). Other thing I observed that code_ does not depend on func_name, may be it should, it is not threadsafe just now. 2. It was implemented for CPU, I plan to check GPU specifics. 3. It seems that tensor(s) is(are) allocated over storage with prepared memory. It means that skipped AllocTensor and AllocADT can keep memory in RAM, it is not good thing but it generates questions about scenarios and VM flexibility. 4. Possibly CI tests are needed to cover some base scenarios
Hello @altanh and @mbs-octoml! Could you see the draft?
Hello @tkonolige! Could you review this PR instead of Mark?
Hello @tkonolige! Sorry, I was on vacation and the development was stopped for a moment. Additional questions: Where should documentation be? In description for python and native methods or in some tutorials?
I'd put the documentation in the description for the methods. I think that's where people will look if using them.