mlc-llm [Bug] Check failed: (it != type_key2index_.end()) is false: Cannot find type ObjectPath. Did you forget to register the node by TVM_REGISTER_NODE

🐛 Bug

python build.py --hf-path=databricks/dolly-v2-12b --target=cuda
Traceback (most recent call last):
  File "/mnt/f/mlc-llm/build.py", line 8, in <module>
    import tvm
  File "/home/zhipeng/.local/lib/python3.10/site-packages/tvm/__init__.py", line 33, in <module>
    from .runtime.object import Object
  File "/home/zhipeng/.local/lib/python3.10/site-packages/tvm/runtime/__init__.py", line 22, in <module>
    from .object_path import ObjectPath, ObjectPathPair
  File "/home/zhipeng/.local/lib/python3.10/site-packages/tvm/runtime/object_path.py", line 44, in <module>
    class ObjectPath(Object):
  File "/home/zhipeng/.local/lib/python3.10/site-packages/tvm/_ffi/registry.py", line 69, in register
    check_call(_LIB.TVMObjectTypeKey2Index(c_str(object_name), ctypes.byref(tidx)))
  File "/home/zhipeng/.local/lib/python3.10/site-packages/tvm/_ffi/base.py", line 348, in check_call
    raise get_last_ffi_error()
tvm._ffi.base.TVMError: Traceback (most recent call last):
  [bt] (4) /usr/local/lib/libtvm.so(TVMObjectTypeKey2Index+0x73) [0x7f8f27948ac3]
  [bt] (3) /usr/local/lib/libtvm.so(tvm::runtime::Object::TypeKey2Index(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xdb) [0x7f8f27948a3b]
  [bt] (2) /usr/local/lib/libtvm.so(+0xd9596) [0x7f8f27948596]
  [bt] (1) /usr/local/lib/libtvm.so(tvm::runtime::detail::LogFatal::Entry::Finalize()+0x3b) [0x7f8f278ffeab]
  [bt] (0) /usr/local/lib/libtvm.so(tvm::runtime::Backtrace[abi:cxx11]()+0x2c) [0x7f8f2792f6bc]
  File "/mnt/f/mlc-llm/3rdparty/tvm/src/runtime/object.cc", line 165
TVMError:
---------------------------------------------------------------
An error occurred during the execution of TVM.
For more information, please see: https://tvm.apache.org/docs/errors.html
---------------------------------------------------------------
  Check failed: (it != type_key2index_.end()) is false: Cannot find type ObjectPath. Did you forget to register the node by TVM_REGISTER_NODE_TYPE ?

To Reproduce

Steps to reproduce the behavior:

python build.py --hf-path=databricks/dolly-v2-12b --target=cuda
python -c "import tvm.runtime.object_path"

Expected behavior

Environment

Platform (e.g. WebGPU/Vulkan/IOS/Android/CUDA): cuda
Operating system (e.g. Ubuntu/Windows/MacOS/...): ubuntu
Device (e.g. iPhone 12 Pro, PC+RTX 3090, ...) 3070ti
How you installed MLC-LLM (conda, source): source
How you installed TVM-Unity (pip, source): source
Python version (e.g. 3.10): 3.10
GPU driver version (if applicable): 252 or 531.41
CUDA/cuDNN version (if applicable): 11.7 or 11.8 or 12.1
TVM Unity Hash Tag (python -c "import tvm; print('\n'.join(f'{k}: {v}' for k, v in tvm.support.libinfo().items()))", applicable if you compile models): Crash same error
Any other relevant information:

# mlc-ai-nightly
mlc-ai-nightly==0.12.dev1088+g6fd55bcfe

# tvm source code
https://github.com/mlc-ai/relax.git
commit: 6fd55bcfecc7abcc707339d7a8ba493f0048b613
branch: mlc

Additional context

Jun 10 '23 13:06 Ox0400

https://www.tiktok.com/@mansour_alzayid?_t=8d3EtnrkJLW&_r=1

Jun 10 '23 13:06 atea911

I try update tvm/_ffi/registry.py source code like this

 62     def register(cls):
 63         """internal register function"""
 64         if hasattr(cls, "_type_index"):
 65             tindex = cls._type_index
 66         else:
 67             tidx = ctypes.c_uint()
 68             if not _RUNTIME_ONLY:
 69                 try:
 70                     check_call(_LIB.TVMObjectTypeKey2Index(c_str(object_name), ctypes.byref(tidx)))
 71                     print ('register success', cls, object_name)
 72                 except:
 73                     print ('register failed', cls, object_name)
 74             else:
 75                 # directly skip unknown objects during runtime.
 76                 ret = _LIB.TVMObjectTypeKey2Index(c_str(object_name), ctypes.byref(tidx))
 77                 if ret != 0:
 78                     return cls
 79             tindex = tidx.value
 80         _register_object(tindex, cls)
 81         return cls

Test import, Most of them failed to register

python -c "import tvm.runtime.object_path"
register failed <class 'tvm.runtime.object_path.ObjectPath'> ObjectPath
register failed <class 'tvm.runtime.object_path.RootPath'> RootPath
register failed <class 'tvm.runtime.object_path.AttributeAccessPath'> AttributeAccessPath
register failed <class 'tvm.runtime.object_path.UnknownAttributeAccessPath'> UnknownAttributeAccessPath
register failed <class 'tvm.runtime.object_path.ArrayIndexPath'> ArrayIndexPath
register failed <class 'tvm.runtime.object_path.MissingArrayElementPath'> MissingArrayElementPath
register failed <class 'tvm.runtime.object_path.MapValuePath'> MapValuePath
register failed <class 'tvm.runtime.object_path.MissingMapEntryPath'> MissingMapEntryPath
register failed <class 'tvm.runtime.object_path.ObjectPathPair'> ObjectPathPair
register failed <class 'tvm.runtime.script_printer.PrinterConfig'> node.PrinterConfig
register success <class 'tvm.runtime.ndarray.NDArray'> runtime.NDArray
register success <class 'tvm.runtime.profiling.Report'> runtime.profiling.Report
register success <class 'tvm.runtime.profiling.Count'> runtime.profiling.Count
register success <class 'tvm.runtime.profiling.Duration'> runtime.profiling.Duration
register success <class 'tvm.runtime.profiling.Percent'> runtime.profiling.Percent
register success <class 'tvm.runtime.profiling.Ratio'> runtime.profiling.Ratio
register success <class 'tvm.runtime.profiling.MetricCollector'> runtime.profiling.MetricCollector
register success <class 'tvm.runtime.profiling.DeviceWrapper'> runtime.profiling.DeviceWrapper
register success <class 'tvm.runtime.container.ADT'> runtime.ADT
register success <class 'tvm.runtime.container.String'> runtime.String
register success <class 'tvm.runtime.container.ShapeTuple'> runtime.ShapeTuple
register failed <class 'tvm.ir.diagnostics.Diagnostic'> Diagnostic
register failed <class 'tvm.ir.diagnostics.DiagnosticRenderer'> DiagnosticRenderer
register failed <class 'tvm.ir.diagnostics.DiagnosticContext'> DiagnosticContext
register failed <class 'tvm.ir.instrument.PassInstrument'> instrument.PassInstrument
register failed <class 'tvm.ir.instrument.PassTimingInstrument'> instrument.PassInstrument
register failed <class 'tvm.ir.transform.PassInfo'> transform.PassInfo
register failed <class 'tvm.ir.transform.PassContext'> transform.PassContext
register failed <class 'tvm.ir.transform.Pass'> transform.Pass
register failed <class 'tvm.ir.transform.ModulePass'> transform.ModulePass
register failed <class 'tvm.ir.transform.Sequential'> transform.Sequential
register failed <class 'tvm.ir.base.SourceMap'> SourceMap
register failed <class 'tvm.ir.base.SourceName'> SourceName
register failed <class 'tvm.ir.base.Span'> Span
register failed <class 'tvm.ir.base.SequentialSpan'> SequentialSpan
register failed <class 'tvm.ir.base.EnvFunc'> EnvFunc
register failed <class 'tvm.ir.type.PrimType'> PrimType
register failed <class 'tvm.ir.type.PointerType'> PointerType
register failed <class 'tvm.ir.type.TypeVar'> TypeVar
register failed <class 'tvm.ir.type.GlobalTypeVar'> GlobalTypeVar
register failed <class 'tvm.ir.type.TupleType'> TupleType
register failed <class 'tvm.ir.type.TypeConstraint'> TypeConstraint
register failed <class 'tvm.ir.type.FuncType'> FuncType
register failed <class 'tvm.ir.type.IncompleteType'> IncompleteType
register failed <class 'tvm.ir.type.RelayRefType'> relay.RefType
register failed <class 'tvm.ir.expr.GlobalVar'> GlobalVar
register failed <class 'tvm.ir.expr.Range'> Range
register failed <class 'tvm.ir.adt.Constructor'> relay.Constructor
register failed <class 'tvm.ir.adt.TypeData'> relay.TypeData
register failed <class 'tvm.ir.affine_type.TensorAffineType'> TensorAffineType
register failed <class 'tvm.ir.affine_type.TupleAffineType'> TupleAffineType
register failed <class 'tvm.ir.attrs.Attrs'> Attrs
register failed <class 'tvm.ir.attrs.DictAttrs'> DictAttrs
register success <class 'tvm.ir.container.Array'> Array
register success <class 'tvm.ir.container.Map'> Map
register failed <class 'tvm.ir.memory_pools.PoolInfo'> ir.PoolInfo
register failed <class 'tvm.ir.memory_pools.PoolInfoProperties'> ir.PoolInfoProperties
register failed <class 'tvm.ir.memory_pools.ConstantInfo'> ir.ConstantInfo
register failed <class 'tvm.ir.memory_pools.WorkspacePoolInfo'> ir.WorkspacePoolInfo
register failed <class 'tvm.ir.memory_pools.ConstantPoolInfo'> ir.ConstantPoolInfo
register failed <class 'tvm.ir.memory_pools.WorkspaceMemoryPools'> ir.WorkspaceMemoryPools
register failed <class 'tvm.ir.memory_pools.ConstantMemoryPools'> ir.ConstantMemoryPools
register failed <class 'tvm.ir.memory_pools.AllocatedPoolInfo'> ir.AllocatedPoolInfo
register failed <class 'tvm.ir.module.IRModule'> IRModule
register failed <class 'tvm.ir.op.Op'> Op
register failed <class 'tvm.ir.tensor_type.TensorType'> relay.TensorType
register failed <class 'tvm.ir.type_relation.TypeCall'> TypeCall
register failed <class 'tvm.ir.type_relation.TypeRelation'> TypeRelation
register failed <class 'tvm.tir.buffer.Buffer'> tir.Buffer
register failed <class 'tvm.tir.buffer.DataProducer'> tir.DataProducer
register failed <class 'tvm.tir.data_layout.Layout'> tir.Layout
register failed <class 'tvm.tir.data_layout.BijectiveLayout'> tir.BijectiveLayout
register failed <class 'tvm.tir.expr.Var'> tir.Var
register failed <class 'tvm.tir.expr.SizeVar'> tir.SizeVar
register failed <class 'tvm.tir.expr.IterVar'> tir.IterVar
register failed <class 'tvm.tir.expr.CommReducer'> tir.CommReducer
register failed <class 'tvm.tir.expr.Reduce'> tir.Reduce
register failed <class 'tvm.tir.expr.FloatImm'> FloatImm
register failed <class 'tvm.tir.expr.IntImm'> IntImm
register failed <class 'tvm.tir.expr.StringImm'> tir.StringImm
register failed <class 'tvm.tir.expr.Cast'> tir.Cast
register failed <class 'tvm.tir.expr.Add'> tir.Add
register failed <class 'tvm.tir.expr.Sub'> tir.Sub
register failed <class 'tvm.tir.expr.Mul'> tir.Mul
register failed <class 'tvm.tir.expr.Div'> tir.Div
register failed <class 'tvm.tir.expr.Mod'> tir.Mod
register failed <class 'tvm.tir.expr.FloorDiv'> tir.FloorDiv
register failed <class 'tvm.tir.expr.FloorMod'> tir.FloorMod
register failed <class 'tvm.tir.expr.Min'> tir.Min
register failed <class 'tvm.tir.expr.Max'> tir.Max
register failed <class 'tvm.tir.expr.EQ'> tir.EQ
register failed <class 'tvm.tir.expr.NE'> tir.NE
register failed <class 'tvm.tir.expr.LT'> tir.LT
register failed <class 'tvm.tir.expr.LE'> tir.LE
register failed <class 'tvm.tir.expr.GT'> tir.GT
register failed <class 'tvm.tir.expr.GE'> tir.GE
register failed <class 'tvm.tir.expr.And'> tir.And
register failed <class 'tvm.tir.expr.Or'> tir.Or
register failed <class 'tvm.tir.expr.Not'> tir.Not
register failed <class 'tvm.tir.expr.Select'> tir.Select
register failed <class 'tvm.tir.expr.BufferLoad'> tir.BufferLoad
register failed <class 'tvm.tir.expr.ProducerLoad'> tir.ProducerLoad
register failed <class 'tvm.tir.expr.Ramp'> tir.Ramp
register failed <class 'tvm.tir.expr.Broadcast'> tir.Broadcast
register failed <class 'tvm.tir.expr.Shuffle'> tir.Shuffle
register failed <class 'tvm.tir.expr.Call'> tir.Call
register failed <class 'tvm.tir.expr.Let'> tir.Let
register failed <class 'tvm.tir.expr.Any'> tir.Any
register failed <class 'tvm.tir.stmt.LetStmt'> tir.LetStmt
register failed <class 'tvm.tir.stmt.AssertStmt'> tir.AssertStmt
register failed <class 'tvm.tir.stmt.For'> tir.For
register failed <class 'tvm.tir.stmt.While'> tir.While
register failed <class 'tvm.tir.stmt.BufferStore'> tir.BufferStore
register failed <class 'tvm.tir.stmt.BufferRealize'> tir.BufferRealize
register failed <class 'tvm.tir.stmt.ProducerStore'> tir.ProducerStore
register failed <class 'tvm.tir.stmt.Allocate'> tir.Allocate
register failed <class 'tvm.tir.stmt.AllocateConst'> tir.AllocateConst
register failed <class 'tvm.tir.stmt.DeclBuffer'> tir.DeclBuffer
register failed <class 'tvm.tir.stmt.AttrStmt'> tir.AttrStmt
register failed <class 'tvm.tir.stmt.ProducerRealize'> tir.ProducerRealize
register failed <class 'tvm.tir.stmt.SeqStmt'> tir.SeqStmt
register failed <class 'tvm.tir.stmt.IfThenElse'> tir.IfThenElse
register failed <class 'tvm.tir.stmt.Evaluate'> tir.Evaluate
register failed <class 'tvm.tir.stmt.Prefetch'> tir.Prefetch
register failed <class 'tvm.tir.stmt.BufferRegion'> tir.BufferRegion
register failed <class 'tvm.tir.stmt.MatchBufferRegion'> tir.MatchBufferRegion
register failed <class 'tvm.tir.stmt.Block'> tir.Block
register failed <class 'tvm.tir.stmt.BlockRealize'> tir.BlockRealize
register failed <class 'tvm.tir.function.PrimFunc'> tir.PrimFunc
register failed <class 'tvm.tir.function.TensorIntrin'> tir.TensorIntrin
register failed <class 'tvm.tir.function.IndexMap'> tir.IndexMap
register failed <class 'tvm.tir.schedule.block_scope.StmtSRef'> tir.StmtSRef
register failed <class 'tvm.tir.schedule.block_scope.Dependency'> tir.Dependency
register failed <class 'tvm.tir.schedule.block_scope.BlockScope'> tir.BlockScope
register failed <class 'tvm.tir.schedule.instruction.InstructionKind'> tir.InstructionKind
register failed <class 'tvm.tir.schedule.instruction.Instruction'> tir.Instruction
register failed <class 'tvm.tir.schedule.state.ScheduleState'> tir.ScheduleState
register failed <class 'tvm.tir.schedule.trace.Trace'> tir.Trace
register failed <class 'tvm.tir.schedule.schedule.LoopRV'> tir.LoopRV
register failed <class 'tvm.tir.schedule.schedule.BlockRV'> tir.BlockRV
register failed <class 'tvm.tir.schedule.schedule.Schedule'> tir.Schedule
register failed <class 'tvm.tir.schedule.analysis.TensorizeInfo'> tir.schedule.TensorizeInfo
register failed <class 'tvm.tir.schedule.analysis.AutoTensorizeMappingInfo'> tir.schedule.AutoTensorizeMappingInfo
register failed <class 'tvm.tir.transform.function_pass.PrimFuncPass'> tir.PrimFuncPass
register failed <class 'tvm.tir.usmp.utils.BufferInfo'> tir.usmp.BufferInfo
register failed <class 'tvm.tir.usmp.utils.PoolAllocation'> tir.usmp.PoolAllocation
register failed <class 'tvm.target.target.TargetKind'> TargetKind
register failed <class 'tvm.target.target.Target'> Target
register failed <class 'tvm.target.virtual_device.VirtualDevice'> VirtualDevice
register failed <class 'tvm.target.generic_func.GenericFunc'> GenericFunc
register failed <class 'tvm.te.tensor.TensorIntrinCall'> TensorIntrinCall
register failed <class 'tvm.te.tensor.Tensor'> Tensor
register failed <class 'tvm.te.tensor.PlaceholderOp'> PlaceholderOp
register failed <class 'tvm.te.tensor.BaseComputeOp'> BaseComputeOp
register failed <class 'tvm.te.tensor.ComputeOp'> ComputeOp
register failed <class 'tvm.te.tensor.TensorComputeOp'> TensorComputeOp
register failed <class 'tvm.te.tensor.ScanOp'> ScanOp
register failed <class 'tvm.te.tensor.ExternOp'> ExternOp
register failed <class 'tvm.te.tensor.HybridOp'> HybridOp
register failed <class 'tvm.te.schedule.Split'> Split
register failed <class 'tvm.te.schedule.Fuse'> Fuse
register failed <class 'tvm.te.schedule.Singleton'> Singleton
register failed <class 'tvm.te.schedule.Schedule'> Schedule
register failed <class 'tvm.te.schedule.Stage'> Stage
register failed <class 'tvm.te.schedule.SpecializedCondition'> SpecializedCondition
register failed <class 'tvm.te.tensor_intrin.TensorIntrin'> TensorIntrin
register failed <class 'tvm.arith.int_set.IntervalSet'> arith.IntervalSet
register failed <class 'tvm.arith.analyzer.ModularSet'> arith.ModularSet
register failed <class 'tvm.arith.analyzer.ConstIntBound'> arith.ConstIntBound
register failed <class 'tvm.arith.int_solver.IntGroupBounds'> arith.IntGroupBounds
register failed <class 'tvm.arith.int_solver.IntConstraints'> arith.IntConstraints
register failed <class 'tvm.arith.int_solver.IntConstraintsTransform'> arith.IntConstraintsTransform
register failed <class 'tvm.arith.iter_affine_map.IterMark'> arith.IterMark
register failed <class 'tvm.arith.iter_affine_map.IterSplitExpr'> arith.IterSplitExpr
register failed <class 'tvm.arith.iter_affine_map.IterSumExpr'> arith.IterSumExpr

Jun 10 '23 14:06 Ox0400

The issue fixed, do not copy build/tvm/libtvm.so to /usr/local/lib/libtvm.so.

sudo rm /usr/local/lib/libtvm.so
# pip install mlc_ai_nightly_vulkan -f https://mlc.ai/wheels
pip install mlc_ai_nightly_cu121 -f https://mlc.ai/wheels

ln -s /home/zhipeng/.local/lib/python3.10/site-packages/tvm/libtvm.so /usr/local/lib/libtvm.so if need run python3 -m mlc_chat.rest crash the error OSError: /mnt/f/mlc-llm/build/libmlc_llm_module.so: undefined symbol: _ZN3tvm7runtime9BacktraceB5cxx11Ev

Jun 10 '23 14:06 Ox0400

run sudo cp /home/zhipeng/.local/lib/python3.10/site-packages/tvm/*.so /usr/local/lib/ before python build.py --hf-path=databricks/dolly-v2-3b --target=cuda

run sudo cp build/tvm/*.so /usr/local/lib/ before run cli

Jun 10 '23 16:06 Ox0400

https://blog.csdn.net/u012731379/article/details/131153319

Jun 11 '23 07:06 Ox0400

Thanks for doing this and I'm glad you figured it out. Some suggestions:

In your blog post, you mentioned Please only install one mlc-ai-nightly-** , if you installed multiple of them, only one of them would work, and it may cause confusion. btw, we are abandoning mlc-ai-nightly-vulkan and in the future all mlc-ai-nightly wheels (both cpu and cuda) will ship vulkan.

Jun 11 '23 08:06 yzh119

[Bug] Check failed: (it != type_key2index_.end()) is false: Cannot find type ObjectPath. Did you forget to register the node by TVM_REGISTER_NODE_TYPE ?

🐛 Bug

To Reproduce

Expected behavior

Environment

Additional context