marie-ai
marie-ai copied to clipboard
Published
20 hours ago •
marieai
marieai
CUDA error: initialization error Compile with `TORCH_USE_CUDA_DSA'
Describe the bug
While migrating to torch-2.2.0.dev20231126+cu118 run into the issue. This could be due to the nature that this is a dev release but adding it for tracking purposes.
Describe how you solve it
Use stable version torch 2.1.1
Environment
(marie) greg@xpredator:~/dev/marieai/marie-ai$ python -m detectron2.utils.collect_env
------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
sys.platform linux
Python 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0]
numpy 1.24.1
detectron2 0.6 @/home/greg/dev/3rdparty/detectron2/detectron2
detectron2._C not built correctly: /home/greg/dev/3rdparty/detectron2/detectron2/_C.cpython-310-x86_64-linux-gnu.so: undefined symbol: _ZNR5torch7Library4_defEOSt7variantIJN3c1012OperatorNameENS2_14FunctionSchemaEEEONS_11CppFunctionE
Compiler ($CXX) c++ (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
CUDA compiler Build cuda_11.8.r11.8/compiler.31833905_0
detectron2 arch flags 8.9
DETECTRON2_ENV_MODULE <not set>
PyTorch 2.2.0.dev20231126+cu118 @/home/greg/environment/marie/lib/python3.10/site-packages/torch
PyTorch debug build False
torch._C._GLIBCXX_USE_CXX11_ABI False
GPU available Yes
GPU 0 NVIDIA GeForce RTX 4090 (arch=8.9)
Driver version 545.23.06
CUDA_HOME /usr/local/cuda
Pillow 9.3.0
torchvision 0.17.0.dev20231126+cu118 @/home/greg/environment/marie/lib/python3.10/site-packages/torchvision
torchvision arch flags 3.5, 5.0, 6.0, 7.0, 7.5, 8.0, 8.6
fvcore 0.1.5
iopath 0.1.9
cv2 4.8.1
------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
PyTorch built with:
- GCC 9.3
- C++ Version: 201703
- Intel(R) oneAPI Math Kernel Library Version 2022.2-Product Build 20220804 for Intel(R) 64 architecture applications
- Intel(R) MKL-DNN v3.1.1 (Git Hash 64f6bcbcbab628e96f33a62c3e975f8535a7bde4)
- OpenMP 201511 (a.k.a. OpenMP 4.5)
- LAPACK is enabled (usually provided by MKL)
- NNPACK is enabled
- CPU capability usage: AVX2
- CUDA Runtime 11.8
- NVCC architecture flags: -gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_90,code=sm_90
- CuDNN 8.7
- Magma 2.6.1
- Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.8, CUDNN_VERSION=8.7.0, CXX_COMPILER=/opt/rh/devtoolset-9/root/usr/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=0 -fabi-version=11 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=pedantic -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_DISABLE_GPU_ASSERTS=ON, TORCH_VERSION=2.2.0, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=1, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF,
Screenshots
ERROR main : extract_t/rep-0@1330319
RuntimeError('CUDA error: initialization
error\nCompile with `TORCH_USE_CUDA_DSA` to enable
device-side assertions.\n') during 'WorkerRuntime'
initialization
add "--quiet-error" to suppress the exception
details
Traceback (most recent call last):
File
"/home/greg/dev/marieai/marie-ai/marie/serve/executo…
line 143, in run
runtime = AsyncNewLoopRuntime(
File
"/home/greg/dev/marieai/marie-ai/marie/serve/runtime…
line 93, in __init__
self._loop.run_until_complete(self.async_setup())
File "/usr/lib/python3.10/asyncio/base_events.py",
line 649, in run_until_complete
return future.result()
File
"/home/greg/dev/marieai/marie-ai/marie/serve/runtime…
line 310, in async_setup
self.server = self._get_server()
File
"/home/greg/dev/marieai/marie-ai/marie/serve/runtime…
line 215, in _get_server
return GRPCServer(
File
"/home/greg/dev/marieai/marie-ai/marie/serve/runtime…
line 34, in __init__
super().__init__(**kwargs)
File
"/home/greg/dev/marieai/marie-ai/marie/serve/runtime…
line 70, in __init__
] = (req_handler or self._get_request_handler())
File
"/home/greg/dev/marieai/marie-ai/marie/serve/runtime…
line 95, in _get_request_handler
return self.req_handler_cls(
File
"/home/greg/dev/marieai/marie-ai/marie/serve/runtime…
line 140, in __init__
self._load_executor(
File
"/home/greg/dev/marieai/marie-ai/marie/serve/runtime…
line 379, in _load_executor
self._executor: BaseExecutor =
BaseExecutor.load_config(
File
"/home/greg/dev/marieai/marie-ai/marie/jaml/__init__…
line 792, in load_config
obj = JAML.load(tag_yml, substitute=False,
runtime_args=runtime_args)
File
"/home/greg/dev/marieai/marie-ai/marie/jaml/__init__…
line 174, in load
r = yaml.load(stream,
Loader=get_jina_loader_with_runtime(runtime_args))
File
"/home/greg/dev/marieai/marie-ai/venv/lib/python3.10…
line 81, in load
return loader.get_single_data()
File
"/home/greg/dev/marieai/marie-ai/venv/lib/python3.10…
line 51, in get_single_data
return self.construct_document(node)
File
"/home/greg/dev/marieai/marie-ai/venv/lib/python3.10…
line 55, in construct_document
data = self.construct_object(node)
File
"/home/greg/dev/marieai/marie-ai/venv/lib/python3.10…
line 100, in construct_object
data = constructor(self, node)
File
"/home/greg/dev/marieai/marie-ai/marie/jaml/__init__…
line 582, in _from_yaml
return get_parser(cls,
version=data.get('version', None)).parse(
File
"/home/greg/dev/marieai/marie-ai/marie/jaml/parsers/…
line 46, in parse
obj = cls(
File
"/home/greg/dev/marieai/marie-ai/marie/serve/executo…
line 58, in arg_wrapper
f = func(self, *args, **kwargs)
File
"/home/greg/dev/marieai/marie-ai/marie/serve/helper.…
line 74, in arg_wrapper
f = func(self, *args, **kwargs)
File
"/home/greg/dev/marieai/marie-ai/marie/executor/text…
line 98, in __init__
self.pipeline =
ExtractPipeline(pipeline_config=pipeline,
cuda=use_cuda)
File
"/home/greg/dev/marieai/marie-ai/marie/pipe/extract_…
line 94, in __init__
self.overlay_processor = OverlayProcessor(
File
"/home/greg/dev/marieai/marie-ai/marie/overlay/overl…
line 44, in __init__
self.opt, self.model = self.__setup(cuda,
checkpoint_dir)
File
"/home/greg/dev/marieai/marie-ai/marie/overlay/overl…
line 109, in __setup
model = create_model(opt)
File
"/home/greg/dev/marieai/marie-ai/marie/models/pix2pi…
line 75, in create_model
instance = model(opt)
File
"/home/greg/dev/marieai/marie-ai/marie/models/pix2pi…
line 45, in __init__
self.netG = networks.define_G(opt.input_nc,
opt.output_nc, opt.ngf, opt.netG,
File
"/home/greg/dev/marieai/marie-ai/marie/models/pix2pi…
line 271, in define_G
return init_net(net, init_type, init_gain,
gpu_ids)
File
"/home/greg/dev/marieai/marie-ai/marie/models/pix2pi…
line 151, in init_net
net.to("cuda")
File
"/home/greg/dev/marieai/marie-ai/venv/lib/python3.10…
line 1152, in to
return self._apply(convert)
File
"/home/greg/dev/marieai/marie-ai/venv/lib/python3.10…
line 802, in _apply
module._apply(fn)
File
"/home/greg/dev/marieai/marie-ai/venv/lib/python3.10…
line 802, in _apply
module._apply(fn)
File
"/home/greg/dev/marieai/marie-ai/venv/lib/python3.10…
line 825, in _apply
param_applied = fn(param)
File
"/home/greg/dev/marieai/marie-ai/venv/lib/python3.10…
line 1150, in convert
return t.to(device, dtype if
t.is_floating_point() or t.is_complex() else None,
non_blocking)
File
"/home/greg/dev/marieai/marie-ai/venv/lib/python3.10…
line 302, in _lazy_init
torch._C._cuda_init()
RuntimeError: CUDA error: initialization error
Compile with `TORCH_USE_CUDA_DSA` to enable
device-side assertions.
