isl-org
Trying to use for metric depth - ZoeDepth sanity check complains
Hello,
I am trying to use ZoeDepth for metric depth. When I try to use the ZoeDepth sanity check, it complains of... "xformers not available" as well as RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format( RuntimeError: Error(s) in loading state_dict for ZoeDepth: Missing key(s) in state_dict: "core.core.pretrained.cls_token" and various other missing keys.
What am I missing, to fully enable the transformer code?
Best wishes, Michel
Hello, Thanks for your kind reply. I've just done as much, and while there is progress, there still seems to be a mismatch when I try to run the ZoeDepth sanity check as well as python train_mono.py -m zoedepth -d kitti --pretrained_resource="local::./checkpoints/depth_anything_vitl14.pth" . RuntimeError: Error(s) in loading state_dict for ZoeDepth: Missing key(s) in state_dict: "core.core.pretrained.cls_token", "core.core.pretrained.pos_embed", "core.core.pretrained.mask_token", "core.core.pretrained.patch_embed.proj.weight",
name: zoe channels:
- pytorch
- nvidia
- conda-forge dependencies:
- cuda=11.7.1
- h5py=3.7.0
- hdf5=1.12.2
- matplotlib=3.6.2
- matplotlib-base=3.6.2
- mkl==2024.0
- numpy=1.24.1
- opencv=4.6.0
- pip=22.3.1
- python=3.9.7
- pytorch=1.13.1
- pytorch-cuda=11.7
- pytorch-mutex=1.0
- scipy=1.10.0
- torchaudio=0.13.1
- torchvision=0.14.1
- pip:
- huggingface-hub==0.11.1
- timm==0.6.7
- tqdm==4.64.1
- wandb==0.13.9
Cheers, Michel
P.S.: I don't know if this is relevant or not, but I also see the message: "xFormers not available" as this runs.
I hard-coded a few parameters to make it easier to run on the debugger, and I'm further along. No idea why they were not input properly. parser.add_argument("-m", "--model", type=str, default="zoedepth") #"synunet") parser.add_argument("-d", "--dataset", type=str, default='kitti') #'nyu') parser.add_argument("--trainer", type=str, default=None) parser.add_argument("--pretrained_resource=", type=str, default="local::./checkpoints/depth_anything_vitl14.pth")
ZoeDepth seems better initialized now. I'll follow up on this bug later, but for now, will keep it hardcoded.
ZoeDepth( (core): DepthAnythingCore( (core): DPT_DINOv2( (pretrained): DinoVisionTransformer( (patch_embed): PatchEmbed( (proj): Conv2d(3, 1024, kernel_size=(14, 14), stride=(14, 14)) (norm): Identity() ) (blocks): ModuleList( (0): NestedTensorBlock( (norm1): LayerNorm((1024,), eps=1e-06, elementwise_affine=True) (attn): MemEffAttention( (qkv): Linear(in_features=1024, out_features=3072, bias=True) (attn_drop): Dropout(p=0.0, inplace=False) (proj): Linear(in_features=1024, out_features=1024, bias=True) (proj_drop): Dropout(p=0.0, inplace=False) ...
I now get as far as this: ERROR api_key not configured (no-tty). call wandb.login(key=[your_api_key]) .
Thanks. I will add it to the environment.yml file. I'm afraid of breaking my Conda build proceeding otherwise.
To fix your error: “ ERROR api_key not configured (no-tty). call wandb.login(key=[your_api_key]) .” go to wandb.ai then create account after that writing command:” wandb login “ your id was created” . Before training
Thanks for the info. I'll work on it now. Progress. It fails a bit further on.
...mpr/mpr/DepthAnything/metric_depth/train_mono.py", line 173, in
Woops found something else. Could be related. Failed to load image Python extension: libtorch_cuda_cu...
I think that there are some limitations in the current environment.yml file. I noticed that the pytorch 1.13.1 does not overwrite an existing pytorch library on conda, say 2.2.1. if the latter is already present. The environment loads incompatible torchvision and torchaudio libraries as a result. I'm playing with competing options for this environment file.
I'll make it available to the community if there is an interest.
I'm going to use this post to track some of the other problems that I encountered so far. I've been getting a socket error as well. This added option seems to address that: --master_port=(some number) e.g. --master_port=25678 . https://github.com/Lightning-AI/pytorch-lightning/discussions/13264
Hi Dung. I've solved that one actually, but I'm using this thread as a document to track what I am seeing, in case anyone in the community wants to re-use the code, and also for my colleagues who will build on metric Depth Anything. The one thing that I am dealing with now is that the code imposes a path for finding the training data, and I have to ascertain where that is and overwrite those instructions. I'll let you know if I need to interact with you. Cheers, Michel
My last obstacle is making sure that I train on the files of my choosing. This established in DepthAnything/metric_depth/zoedepth/utils/config.py . This config file includes the following code for kitti, which assumes a different file structure than I have... DATASETS_CONFIG = { "kitti": { "dataset": "kitti", "min_depth": 0.001, "max_depth": 80, "data_path": os.path.join(HOME_DIR, "Kitti/raw_data"), "gt_path": os.path.join(HOME_DIR, "Kitti/data_depth_annotated_zoedepth"), "filenames_file": "./train_test_inputs/kitti_eigen_train_files_with_gt.txt", "input_height": 352, "input_width": 1216, # 704 "data_path_eval": os.path.join(HOME_DIR, "Kitti/raw_data"), "gt_path_eval": os.path.join(HOME_DIR, "Kitti/data_depth_annotated_zoedepth"), "filenames_file_eval": "./train_test_inputs/kitti_eigen_test_files_with_gt.txt",
"min_depth_eval": 1e-3,
"max_depth_eval": 80,
"do_random_rotate": True,
"degree": 1.0,
"do_kb_crop": True,
"garg_crop": True,
"eigen_crop": False,
"use_right": False
},
The filenames_file is particularly relevant, as ar data_path and gt_path. I need to overwrite that filenames_file with something else. The structure of that txt file is like this:
2011_09_26/2011_09_26_drive_0051_sync/image_02/data/0000000093.png 2011_09_26_drive_0051_sync/proj_depth/groundtruth/image_02/0000000093.png 721.5377
2011_09_30/2011_09_30_drive_0028_sync/image_02/data/0000002714.png 2011_09_30_drive_0028_sync/proj_depth/groundtruth/image_02/0000002714.png 707.0912
2011_09_26/2011_09_26_drive_0061_sync/image_02/data/0000000045.png 2011_09_26_drive_0061_sync/proj_depth/groundtruth/image_02/0000000045.png 721.5377
The first file is probably the raw image, while the second is the groundtruth, obviously, but there is also third entry, a number of some kind. Any idea what that represents?
I am progressing... I managed to format my datafile as ZoeDepth expects to see it, but CUDA complains about memory...
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 1.03 GiB (GPU 0; 15.70 GiB total capacity; 11.81 GiB already allocated; 167.81 MiB free; 12.17 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
Will look into reducing my training set...
For the community... Mr Dung and I have been interacting on Instragram. He suggested that I play with the batch size in config_zoedepth.json. I tried with values of bs = 16 (original value), 4, 2 and 1. The value of 1 is the only setting that allowed train_mono.py to run properly with my Kitti data. Now my GPU is humming as it trains. I'll report back on test studies.
Edit: GPU hummed for a while but eventually the training process still ran out of memory, with arguably the lowest bs=1 setting...
Following up with export 'PYTORCH_CUDA_ALLOC_CONF=max_split_size_mb:512' https://stackoverflow.com/questions/73747731/runtimeerror-cuda-out-of-memory-how-can-i-set-max-split-size-mb
Send me messages through instagram. My id is dung26032000
Hi Dung, I met some problem for model training, can you help me to have a look? I can send my problem through Instagram to you.
Send me messages through instagram. My id is dung26032000
Hi Dung, I met some problem for model training, can you help me to have a look? I can send my problem through Instagram to you.
Send me now
Send me messages through instagram. My id is dung26032000
Hi Dung, I met some problem for model training, can you help me to have a look? I can send my problem through Instagram to you.
Send me now
Thanks, I've just sent the following application on instagram to you
Send me messages through instagram. My id is dung26032000
Hi Dung, I met some problem for model training, can you help me to have a look? I can send my problem through Instagram to you.
Send me now
https://github.com/isl-org/ZoeDepth/issues/125#issue-2421225302 here is my current problem, and I've also sent it to you on instagram
I am seeing PYTORCH_CUDA_ALLOC_CONF memory errors. I'm trying this.... export 'PYTORCH_CUDA_ALLOC_CONF=garbage_collection_threshold:0.6,max_split_size_mb:32' It failed at export 'PYTORCH_CUDA_ALLOC_CONF=garbage_collection_threshold:0.6,max_split_size_mb:128' which I obtained from https://github.com/CompVis/stable-diffusion/issues/39 . A second thread mentioned a value of 32 https://blog.gopenai.com/how-to-resolve-runtimeerror-cuda-out-of-memory-d48995452a0 Also this: https://medium.com/@soumensardarintmain/manage-cuda-cores-ultimate-memory-management-strategy-with-pytorch-2bed30cab1
Is it possible to use gradient accumulation to deal with this? Can I invoke train_mono with extra parameters like --gradient_accumulation_steps and expect it to address my memory error?