MONAILabel
MONAILabel copied to clipboard
Project-MONAI
CUDA is not getting find by Monailabel
Describe the bug I am using pc with gpu nvidia rtx 4070 with 12gb memory and is cuda enabeled. I have installed pytorch with cuda option. the Monailabel is still not able to find or work with cuda on the pc. can you please help with that.
Server logs
2023-03-13 23:19:59,209 - USING:: version = False
2023-03-13 23:19:59,209 - USING:: app = C:\Users\keyur\apps\radiology
2023-03-13 23:19:59,209 - USING:: studies = C:\Users\keyur\datasets\training
2023-03-13 23:19:59,209 - USING:: verbose = INFO
2023-03-13 23:19:59,209 - USING:: conf = [['models', 'segmentation']]
2023-03-13 23:19:59,210 - USING:: host = 0.0.0.0
2023-03-13 23:19:59,210 - USING:: port = 8000
2023-03-13 23:19:59,210 - USING:: uvicorn_app = monailabel.app:app
2023-03-13 23:19:59,210 - USING:: ssl_keyfile = None
2023-03-13 23:19:59,210 - USING:: ssl_certfile = None
2023-03-13 23:19:59,210 - USING:: ssl_keyfile_password = None
2023-03-13 23:19:59,211 - USING:: ssl_ca_certs = None
2023-03-13 23:19:59,211 - USING:: workers = None
2023-03-13 23:19:59,211 - USING:: limit_concurrency = None
2023-03-13 23:19:59,211 - USING:: access_log = False
2023-03-13 23:19:59,211 - USING:: log_config = None
2023-03-13 23:19:59,211 - USING:: dryrun = False
2023-03-13 23:19:59,212 - USING:: action = start_server
2023-03-13 23:19:59,212 - ENV SETTINGS:: MONAI_LABEL_API_STR =
2023-03-13 23:19:59,212 - ENV SETTINGS:: MONAI_LABEL_PROJECT_NAME = MONAILabel
2023-03-13 23:19:59,212 - ENV SETTINGS:: MONAI_LABEL_APP_DIR =
2023-03-13 23:19:59,212 - ENV SETTINGS:: MONAI_LABEL_STUDIES =
2023-03-13 23:19:59,212 - ENV SETTINGS:: MONAI_LABEL_AUTH_ENABLE = False
2023-03-13 23:19:59,212 - ENV SETTINGS:: MONAI_LABEL_AUTH_DB =
2023-03-13 23:19:59,213 - ENV SETTINGS:: MONAI_LABEL_APP_CONF = '{}'
2023-03-13 23:19:59,213 - ENV SETTINGS:: MONAI_LABEL_TASKS_TRAIN = True
2023-03-13 23:19:59,213 - ENV SETTINGS:: MONAI_LABEL_TASKS_STRATEGY = True
2023-03-13 23:19:59,213 - ENV SETTINGS:: MONAI_LABEL_TASKS_SCORING = True
2023-03-13 23:19:59,213 - ENV SETTINGS:: MONAI_LABEL_TASKS_BATCH_INFER = True
2023-03-13 23:19:59,213 - ENV SETTINGS:: MONAI_LABEL_DATASTORE =
2023-03-13 23:19:59,213 - ENV SETTINGS:: MONAI_LABEL_DATASTORE_URL =
2023-03-13 23:19:59,213 - ENV SETTINGS:: MONAI_LABEL_DATASTORE_USERNAME =
2023-03-13 23:19:59,214 - ENV SETTINGS:: MONAI_LABEL_DATASTORE_PASSWORD =
2023-03-13 23:19:59,214 - ENV SETTINGS:: MONAI_LABEL_DATASTORE_API_KEY =
2023-03-13 23:19:59,214 - ENV SETTINGS:: MONAI_LABEL_DATASTORE_CACHE_PATH =
2023-03-13 23:19:59,214 - ENV SETTINGS:: MONAI_LABEL_DATASTORE_PROJECT =
2023-03-13 23:19:59,214 - ENV SETTINGS:: MONAI_LABEL_DATASTORE_ASSET_PATH =
2023-03-13 23:19:59,214 - ENV SETTINGS:: MONAI_LABEL_DATASTORE_DSA_ANNOTATION_GROUPS =
2023-03-13 23:19:59,214 - ENV SETTINGS:: MONAI_LABEL_DICOMWEB_USERNAME =
2023-03-13 23:19:59,214 - ENV SETTINGS:: MONAI_LABEL_DICOMWEB_PASSWORD =
2023-03-13 23:19:59,214 - ENV SETTINGS:: MONAI_LABEL_DICOMWEB_CACHE_PATH =
2023-03-13 23:19:59,214 - ENV SETTINGS:: MONAI_LABEL_QIDO_PREFIX = None
2023-03-13 23:19:59,214 - ENV SETTINGS:: MONAI_LABEL_WADO_PREFIX = None
2023-03-13 23:19:59,214 - ENV SETTINGS:: MONAI_LABEL_STOW_PREFIX = None
2023-03-13 23:19:59,214 - ENV SETTINGS:: MONAI_LABEL_DICOMWEB_FETCH_BY_FRAME = False
2023-03-13 23:19:59,214 - ENV SETTINGS:: MONAI_LABEL_DICOMWEB_CONVERT_TO_NIFTI = True
2023-03-13 23:19:59,214 - ENV SETTINGS:: MONAI_LABEL_DICOMWEB_SEARCH_FILTER = '{"Modality": "CT"}'
2023-03-13 23:19:59,214 - ENV SETTINGS:: MONAI_LABEL_DICOMWEB_CACHE_EXPIRY = 180
2023-03-13 23:19:59,214 - ENV SETTINGS:: MONAI_LABEL_DICOMWEB_PROXY_TIMEOUT = 30.0
2023-03-13 23:19:59,214 - ENV SETTINGS:: MONAI_LABEL_DICOMWEB_READ_TIMEOUT = 5.0
2023-03-13 23:19:59,214 - ENV SETTINGS:: MONAI_LABEL_DATASTORE_AUTO_RELOAD = True
2023-03-13 23:19:59,215 - ENV SETTINGS:: MONAI_LABEL_DATASTORE_READ_ONLY = False
2023-03-13 23:19:59,215 - ENV SETTINGS:: MONAI_LABEL_DATASTORE_FILE_EXT = '[".nii.gz", ".nii", ".nrrd", ".jpg", ".png", ".tif", ".svs", ".xml"]'
2023-03-13 23:19:59,215 - ENV SETTINGS:: MONAI_LABEL_SERVER_PORT = 8000
2023-03-13 23:19:59,215 - ENV SETTINGS:: MONAI_LABEL_CORS_ORIGINS = '[]'
2023-03-13 23:19:59,215 - ENV SETTINGS:: MONAI_LABEL_SESSIONS = True
2023-03-13 23:19:59,215 - ENV SETTINGS:: MONAI_LABEL_SESSION_PATH =
2023-03-13 23:19:59,215 - ENV SETTINGS:: MONAI_LABEL_SESSION_EXPIRY = 3600
2023-03-13 23:19:59,215 - ENV SETTINGS:: MONAI_LABEL_INFER_CONCURRENCY = -1
2023-03-13 23:19:59,215 - ENV SETTINGS:: MONAI_LABEL_INFER_TIMEOUT = 600
2023-03-13 23:19:59,216 - ENV SETTINGS:: MONAI_LABEL_TRACKING_ENABLED = True
2023-03-13 23:19:59,216 - ENV SETTINGS:: MONAI_LABEL_TRACKING_URI =
2023-03-13 23:19:59,216 - ENV SETTINGS:: MONAI_ZOO_SOURCE = github
2023-03-13 23:19:59,216 - ENV SETTINGS:: MONAI_ZOO_REPO = Project-MONAI/model-zoo/hosting_storage_v1
2023-03-13 23:19:59,219 - ENV SETTINGS:: MONAI_ZOO_AUTH_TOKEN =
2023-03-13 23:19:59,219 - ENV SETTINGS:: MONAI_LABEL_AUTO_UPDATE_SCORING = True
2023-03-13 23:19:59,219 -
Allow Origins: ['']
[2023-03-13 23:20:14,054] [3404] [MainThread] [INFO] (uvicorn.error:75) - Started server process [3404]
[2023-03-13 23:20:14,056] [3404] [MainThread] [INFO] (uvicorn.error:45) - Waiting for application startup.
[2023-03-13 23:20:14,058] [3404] [MainThread] [INFO] (monailabel.interfaces.utils.app:37) - Initializing App from: C:\Users\keyur\apps\radiology; studies: C:\Users\keyur\datasets\training; conf: {'models': 'segmentation'}
[2023-03-13 23:20:14,274] [3404] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for MONAILabelApp Found: <class 'main.MyApp'>
[2023-03-13 23:20:14,290] [3404] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for TaskConfig Found: <class 'lib.configs.deepedit.DeepEdit'>
[2023-03-13 23:20:14,292] [3404] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for TaskConfig Found: <class 'lib.configs.deepgrow_2d.Deepgrow2D'>
[2023-03-13 23:20:14,293] [3404] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for TaskConfig Found: <class 'lib.configs.deepgrow_3d.Deepgrow3D'>
[2023-03-13 23:20:14,296] [3404] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for TaskConfig Found: <class 'lib.configs.localization_spine.LocalizationSpine'>
[2023-03-13 23:20:14,300] [3404] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for TaskConfig Found: <class 'lib.configs.localization_vertebra.LocalizationVertebra'>
[2023-03-13 23:20:14,324] [3404] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for TaskConfig Found: <class 'lib.configs.segmentation.Segmentation'>
[2023-03-13 23:20:14,333] [3404] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for TaskConfig Found: <class 'lib.configs.segmentation_spleen.SegmentationSpleen'>
[2023-03-13 23:20:14,348] [3404] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for TaskConfig Found: <class 'lib.configs.segmentation_vertebra.SegmentationVertebra'>
[2023-03-13 23:20:14,362] [3404] [MainThread] [INFO] (main:93) - +++ Adding Model: segmentation => lib.configs.segmentation.Segmentation
[2023-03-13 23:20:14,451] [3404] [MainThread] [INFO] (main:96) - +++ Using Models: ['segmentation']
[2023-03-13 23:20:14,459] [3404] [MainThread] [INFO] (monailabel.interfaces.app:134) - Init Datastore for: C:\Users\keyur\datasets\training
[2023-03-13 23:20:14,472] [3404] [MainThread] [INFO] (monailabel.datastore.local:130) - Auto Reload: True; Extensions: ['.nii.gz', '.nii', '.nrrd', '.jpg', '.png', '.tif', '.svs', '.xml']
[2023-03-13 23:20:14,507] [3404] [MainThread] [INFO] (monailabel.datastore.local:577) - Invalidate count: 0
[2023-03-13 23:20:14,519] [3404] [MainThread] [INFO] (monailabel.datastore.local:151) - Start observing external modifications on datastore (AUTO RELOAD)
[2023-03-13 23:20:14,538] [3404] [MainThread] [INFO] (main:126) - +++ Adding Inferer:: segmentation => <lib.infers.segmentation.Segmentation object at 0x000001F12698C2E0>
[2023-03-13 23:20:14,539] [3404] [MainThread] [INFO] (main:191) - {'segmentation': <lib.infers.segmentation.Segmentation object at 0x000001F12698C2E0>, 'Histogram+GraphCut': <monailabel.scribbles.infer.HistogramBasedGraphCut object at 0x000001F12AB54580>, 'GMM+GraphCut': <monailabel.scribbles.infer.GMMBasedGraphCut object at 0x000001F12AB54550>}
[2023-03-13 23:20:14,540] [3404] [MainThread] [INFO] (main:206) - +++ Adding Trainer:: segmentation => <lib.trainers.segmentation.Segmentation object at 0x000001F12AB7E6A0>
[2023-03-13 23:20:14,542] [3404] [MainThread] [INFO] (monailabel.utils.sessions:51) - Session Path: C:\Users\keyur.cache\monailabel\sessions
[2023-03-13 23:20:14,543] [3404] [MainThread] [INFO] (monailabel.utils.sessions:52) - Session Expiry (max): 3600
[2023-03-13 23:20:14,543] [3404] [MainThread] [INFO] (monailabel.interfaces.app:468) - App Init - completed
[2023-03-13 23:20:14,544] [timeloop] [INFO] Starting Timeloop..
[2023-03-13 23:20:14,544] [3404] [MainThread] [INFO] (timeloop:60) - Starting Timeloop..
[2023-03-13 23:20:14,549] [timeloop] [INFO] Registered job <function MONAILabelApp.on_init_complete.
Looks an issue with your python or torch version.. irrespective of monailabel try to import torch and see if cuda is enabled
Thank you for your response #SachidanandAlle . It is an issue with torch as you have described. I tried to install the latest version of torch in the conda environment and showing the torch==2.0.0, torchaudio==2.0.1, torchvision==0.15.1. When I install the monailabel it is uninstalling torch 2.0 and torchvision 0.15.1 where torchaudio 2.0.1 is remaining and at the end showing the error :"ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. torchaudio 2.0.1 requires torch==2.0.0, but you have torch 1.13.1 which is incompatible."
can someone please guide me which torch version and cuda version is compatible with monail label?
I have installed torch from pip and I didn't have any such dependency issue. But otherwise monailabel doesn't restrict to use specific torch version.. however monai is tested and verified for certain torch versions only..
https://github.com/Project-MONAI/MONAI/blob/dev/requirements.txt#L1
Possibly try a new venv and try from beginning.. It mostly looks an environment issue at your end.
Thank you for your response,
I did try with new venv but it was showing the same error. Then I did follow the youtube instruction from* https://www.youtube.com/watch?v=8y1OBQs2wis&t=141s https://www.youtube.com/watch?v=8y1OBQs2wis&t=141s* and it did work at the end. I have also noted that monailabel installation process did uninstall the torch 2.0 with torchvision 0.15. Beste Hilsen / Kind regards,
"ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. torchaudio 2.0.1+cu117 requires torch==2.0.0, but you have torch 1.13.1 which is incompatible."
then I used the command to install pytorch again and it did uninstall the torchaudio 2.0 and installed 1.3 as are compatible.
pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu117 Looking in indexes: https://download.pytorch.org/whl/cu117 Requirement already satisfied: torch in c:\users\keyur.conda\envs\monai\lib\site-packages (1.13.1) Requirement already satisfied: torchvision in c:\users\keyur.conda\envs\monai\lib\site-packages (0.14.1) Requirement already satisfied: torchaudio in c:\users\keyur.conda\envs\monai\lib\site-packages (2.0.1+cu117) Requirement already satisfied: typing-extensions in c:\users\keyur.conda\envs\monai\lib\site-packages (from torch) (4.4.0) Requirement already satisfied: numpy in c:\users\keyur.conda\envs\monai\lib\site-packages (from torchvision) (1.23.5) Requirement already satisfied: requests in c:\users\keyur.conda\envs\monai\lib\site-packages (from torchvision) (2.28.1) Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in c:\users\keyur.conda\envs\monai\lib\site-packages (from torchvision) (9.3.0) Collecting torchaudio Downloading https://download.pytorch.org/whl/cu117/torchaudio-2.0.0%2Bcu117-cp39-cp39-win_amd64.whl (2.5 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.5/2.5 MB 11.2 MB/s eta 0:00:00 Downloading https://download.pytorch.org/whl/cu117/torchaudio-0.13.1%2Bcu117-cp39-cp39-win_amd64.whl (2.3 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.3/2.3 MB 14.6 MB/s eta 0:00:00 Requirement already satisfied: certifi>=2017.4.17 in c:\users\keyur.conda\envs\monai\lib\site-packages (from requests->torchvision) (2022.12.7) Requirement already satisfied: idna<4,>=2.5 in c:\users\keyur.conda\envs\monai\lib\site-packages (from requests->torchvision) (3.4) Requirement already satisfied: urllib3<1.27,>=1.21.1 in c:\users\keyur.conda\envs\monai\lib\site-packages (from requests->torchvision) (1.26.13) Requirement already satisfied: charset-normalizer<3,>=2 in c:\users\keyur.conda\envs\monai\lib\site-packages (from requests->torchvision) (2.1.1) Installing collected packages: torchaudio Attempting uninstall: torchaudio Found existing installation: torchaudio 2.0.1+cu117 Uninstalling torchaudio-2.0.1+cu117: Successfully uninstalled torchaudio-2.0.1+cu117 Successfully installed torchaudio-0.13.1+cu117 (monai) PS C:\Users\keyur\monailabel> .monailabel/scripts/monailabel apps
Keyur Radiya
On Mon, Mar 20, 2023 at 7:16 AM SACHIDANAND ALLE @.***> wrote:
I have installed torch from pip and I didn't have any such dependency issue. But otherwise monailabel doesn't restrict to use specific torch version.. however monai is tested and verified for certain torch versions only..
https://github.com/Project-MONAI/MONAI/blob/dev/requirements.txt#L1
Possibly try a new venv and try from beginning.. It mostly looks an environment issue at your end.
— Reply to this email directly, view it on GitHub https://github.com/Project-MONAI/MONAILabel/issues/1337#issuecomment-1475681804, or unsubscribe https://github.com/notifications/unsubscribe-auth/AZHCD2AGXWDXH5E7QIBEY53W47Y33ANCNFSM6AAAAAAVZVWEOM . You are receiving this because you authored the thread.Message ID: @.***>