ColossalAI icon indicating copy to clipboard operation
ColossalAI copied to clipboard

[BUG]This happened after I ran Colossal after building the environment according to the instructions

Open dong49 opened this issue 2 years ago • 0 comments

🐛 Describe the bug

Traceback (most recent call last): File "/root/anaconda3/bin/colossalai", line 5, in from colossalai.cli import cli File "/root/anaconda3/lib/python3.9/site-packages/colossalai/init.py", line 1, in from .initialize import ( File "/root/anaconda3/lib/python3.9/site-packages/colossalai/initialize.py", line 18, in from colossalai.amp import AMP_TYPE, convert_to_amp File "/root/anaconda3/lib/python3.9/site-packages/colossalai/amp/init.py", line 9, in from .torch_amp import convert_to_torch_amp File "/root/anaconda3/lib/python3.9/site-packages/colossalai/amp/torch_amp/init.py", line 9, in from .torch_amp import TorchAMPLoss, TorchAMPModel, TorchAMPOptimizer File "/root/anaconda3/lib/python3.9/site-packages/colossalai/amp/torch_amp/torch_amp.py", line 10, in from colossalai.nn.optimizer import ColossalaiOptimizer File "/root/anaconda3/lib/python3.9/site-packages/colossalai/nn/init.py", line 1, in from ._ops import * File "/root/anaconda3/lib/python3.9/site-packages/colossalai/nn/_ops/init.py", line 1, in from .addmm import colo_addmm File "/root/anaconda3/lib/python3.9/site-packages/colossalai/nn/_ops/addmm.py", line 5, in from ._utils import GeneralTensor, Number, convert_to_colo_tensor File "/root/anaconda3/lib/python3.9/site-packages/colossalai/nn/_ops/_utils.py", line 8, in from colossalai.nn.layer.utils import divide File "/root/anaconda3/lib/python3.9/site-packages/colossalai/nn/layer/init.py", line 7, in from .moe import * File "/root/anaconda3/lib/python3.9/site-packages/colossalai/nn/layer/moe/init.py", line 1, in from .experts import Experts, FFNExperts, TPExperts File "/root/anaconda3/lib/python3.9/site-packages/colossalai/nn/layer/moe/experts.py", line 8, in from colossalai.zero.init_ctx import no_shard_zero_decrator File "/root/anaconda3/lib/python3.9/site-packages/colossalai/zero/init.py", line 7, in from colossalai.zero.sharded_model.sharded_model_v2 import ShardedModelV2 File "/root/anaconda3/lib/python3.9/site-packages/colossalai/zero/sharded_model/init.py", line 1, in from .sharded_model_v2 import ShardedModelV2 File "/root/anaconda3/lib/python3.9/site-packages/colossalai/zero/sharded_model/sharded_model_v2.py", line 16, in from colossalai.gemini.memory_tracer import MemStatsCollector, StaticMemStatsCollector File "/root/anaconda3/lib/python3.9/site-packages/colossalai/gemini/init.py", line 1, in from .chunk import ChunkManager, TensorInfo, TensorState, search_chunk_configuration File "/root/anaconda3/lib/python3.9/site-packages/colossalai/gemini/chunk/init.py", line 3, in from .search_utils import classify_params_by_dp_degree, search_chunk_configuration File "/root/anaconda3/lib/python3.9/site-packages/colossalai/gemini/chunk/search_utils.py", line 8, in from colossalai.gemini.memory_tracer import MemStats, OrderedParamGenerator File "/root/anaconda3/lib/python3.9/site-packages/colossalai/gemini/memory_tracer/init.py", line 6, in from .static_memstats_collector import StaticMemStatsCollector # isort:skip File "/root/anaconda3/lib/python3.9/site-packages/colossalai/gemini/memory_tracer/static_memstats_collector.py", line 7, in from colossalai.fx.passes.meta_info_prop import MetaInfoProp File "/root/anaconda3/lib/python3.9/site-packages/colossalai/fx/init.py", line 3, in from .passes import MetaInfoProp, metainfo_trace File "/root/anaconda3/lib/python3.9/site-packages/colossalai/fx/passes/init.py", line 2, in from .concrete_info_prop import ConcreteInfoProp File "/root/anaconda3/lib/python3.9/site-packages/colossalai/fx/passes/concrete_info_prop.py", line 10, in from colossalai.fx.profiler import GraphInfo, profile_function, profile_method, profile_module File "/root/anaconda3/lib/python3.9/site-packages/colossalai/fx/profiler/init.py", line 4, in from .opcount import flop_mapping File "/root/anaconda3/lib/python3.9/site-packages/colossalai/fx/profiler/opcount.py", line 273, in aten.upsample_nearest2d_backward.vec: elementwise_flop_counter(0, 1), File "/root/anaconda3/lib/python3.9/site-packages/torch/_ops.py", line 488, in getattr raise AttributeError( AttributeError: The underlying op of 'aten.upsample_nearest2d_backward' has no overload name 'vec'

Environment

cuda kit :12.1 cuda :11.7 cudnn:cudnn-local-repo-ubuntu2004-8.8.0.121_1.0-1_amd64 Python 3.9 pytorch:pytorch.org/whl/cu117

dong49 avatar Mar 19 '23 14:03 dong49