tvm icon indicating copy to clipboard operation
tvm copied to clipboard

[Bug][TVMC] Map is not supported by RPC

Open lhutton1 opened this issue 3 years ago • 2 comments
trafficstars

Hi, I'm attempting to run a model via RPC and TVMC but I'm running into an issue:

ValueError: Cannot pass argument 1, type Map is not supported by RPC

The TVMC command I'm using is similar to:

tvmc run --repeat=1 --print-top=5 --rpc-tracker=mytracker.com mymodel.tar

I seem to be able to get the model to run again when I revert #9889 which introduced get_input_info in the graph executor. Is this a case of RPC missing an implementation for Map, or should we do something in TVMC to handle get_input_info differently?

Full backtrace:

tvm.error.RPCError: Traceback (most recent call last):
  8: TVMFuncCall
        at ../src/runtime/c_runtime_api.cc:477
  7: tvm::runtime::PackedFuncObj::CallPacked(tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*) const
        at ../include/tvm/runtime/packed_func.h:1217
  6: tvm::runtime::RPCWrappedFunc::operator()(tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*) const
        at ../src/runtime/rpc/rpc_module.cc:127
  5: tvm::runtime::RPCClientSession::CallFunc(void*, TVMValue const*, int const*, int, std::function<void (tvm::runtime::TVMArgs)> const&)
        at ../src/runtime/rpc/rpc_endpoint.cc:1009
  4: tvm::runtime::RPCEndpoint::CallFunc(void*, TVMValue const*, int const*, int, std::function<void (tvm::runtime::TVMArgs)>)
        at ../src/runtime/rpc/rpc_endpoint.cc:800
  3: tvm::runtime::RPCEndpoint::HandleUntilReturnEvent(bool, std::function<void (tvm::runtime::TVMArgs)>)
        at ../src/runtime/rpc/rpc_endpoint.cc:644
  2: tvm::runtime::RPCEndpoint::EventHandler::HandleNextEvent(bool, bool, std::function<void (tvm::runtime::TVMArgs)>)
        at ../src/runtime/rpc/rpc_endpoint.cc:135
  1: tvm::runtime::RPCEndpoint::EventHandler::HandleProcessPacket(std::function<void (tvm::runtime::TVMArgs)>)
        at ../src/runtime/rpc/rpc_endpoint.cc:315
  0: tvm::runtime::RPCEndpoint::EventHandler::HandleReturn(tvm::runtime::RPCCode, std::function<void (tvm::runtime::TVMArgs)>)
        at ../src/runtime/rpc/rpc_endpoint.cc:376
  [bt] (8) libtvm_runtime.so(tvm::runtime::RPCEndpoint::EventHandler::HandleNextEvent(bool, bool, std::function<void (tvm::runtime::TVMArgs)>)+0x464) [0xffffacf33c54]
  [bt] (7) libtvm_runtime.so(tvm::runtime::RPCSession::AsyncCallFunc(void*, TVMValue const*, int const*, int, std::function<void (tvm::runtime::RPCCode, tvm::runtime::TVMArgs)>)+0x48) [0xffffacf45dd0]
  [bt] (6) libtvm_runtime.so(tvm::runtime::LocalSession::CallFunc(void*, TVMValue const*, int const*, int, std::function<void (tvm::runtime::TVMArgs)> const&)+0x84) [0xffffacf38fac]
  [bt] (5) libtvm_runtime.so(tvm::runtime::LocalSession::EncodeReturn(tvm::runtime::TVMRetValue, std::function<void (tvm::runtime::TVMArgs)> const&)+0x74) [0xffffacf38e1c]
  [bt] (4) libtvm_runtime.so(+0x125ac4) [0xffffacf45ac4]
  [bt] (3) libtvm_runtime.so(std::_Function_handler<void (tvm::runtime::RPCCode, tvm::runtime::TVMArgs), tvm::runtime::RPCEndpoint::EventHandler::HandleNormalCallFunc()::{lambda(tvm::runtime::RPCCode, tvm::runtime::TVMArgs)#1}>::_M_invoke(std::_Any_data const&, tvm::runtime::RPCCode&&, tvm::runtime::TVMArgs&&)+0x4c) [0xffffacf31aa4]
  [bt] (2) libtvm_runtime.so(tvm::runtime::RPCEndpoint::EventHandler::ValidateArguments(TVMValue const*, int const*, int)+0x1ec) [0xffffacf311f4]
  [bt] (1) tvm_rpc(tvm::runtime::detail::LogFatal::Entry::Finalize()+0x6c) [0x40fef4]
  [bt] (0) libtvm_runtime.so(tvm::runtime::Backtrace[abi:cxx11]()+0x1c) [0xffffaceb6d84]
  File "../src/runtime/rpc/rpc_endpoint.cc", line 376
RPCError: Error caught from RPC call:
ValueError: Cannot pass argument 1, type Map is not supported by RPC

cc @Mousius @gromero @leandron @leeexyz @masahi

lhutton1 avatar Jun 14 '22 15:06 lhutton1

Submitted #11757 to work around the issue for now.

lhutton1 avatar Jun 21 '22 08:06 lhutton1

This is the intended behavior of RPC for now to restrict the object that we can support (so we can support minimum cases like uTVM). Would be a good starting pt to document the related behavior

tqchen avatar Jul 26 '22 19:07 tqchen