yerfor
Script Reports Lack of GPU Support Despite Correct Environment Setup
Environment:
Operating System: Ubuntu on GCP Python Version: 3.9.19 PyTorch Version: 2.0.1 CUDA Version: 11.7 GPU Model: Tesla T4 (as verified by torch.cuda.get_device_name(0))
Issue Description:
I'm encountering an issue where the fit_3dmm_landmark.py script reports that it is "Not compiled with GPU support" despite having a fully supported and accessible GPU environment. My CUDA installation is verified, and PyTorch recognizes the GPU correctly (as evidenced by torch.cuda.is_available() returning True and torch.cuda.get_device_name(0) successfully reporting the GPU model).
I have ensured that the environment variables are correctly set, including CUDA_VISIBLE_DEVICES, and my PyTorch and related libraries are aligned with the CUDA version installed.
Steps to Reproduce:
Set environment variables (including VIDEO_ID, PYTHONPATH, and CUDA_VISIBLE_DEVICES). Execute the script using: python data_gen/utils/process_video/fit_3dmm_landmark.py --ds_name=nerf --vid_dir=data/raw/videos/${VIDEO_ID}.mp4 --reset --debug --id_mode=global. The process initiates but eventually reports that it was not compiled with GPU support, falling back to CPU processing. Expected Behavior: The script utilizes the GPU for processing, as the environment is correctly configured for CUDA and PyTorch GPU operations.
Actual Behavior:
The script fails to recognize GPU support and defaults to CPU processing, significantly affecting performance and feasibility for my use case.
Additional Context:
All diagnostic checks for CUDA and PyTorch GPU capabilities pass without issue. Environment variables are set to explicitly target GPU usage. A restart of the system/server has been performed to ensure all environment variables and configurations are correctly loaded. Is there a specific compilation step or configuration that might be missing for enabling GPU support within this script? Any guidance or assistance on how to resolve this would be greatly appreciated.
Terminal Command:
export VIDEO_ID=Robert export PYTHONPATH=./ export CUDA_VISIBLE_DEVICES=0 python data_gen/utils/process_video/fit_3dmm_landmark.py --ds_name=nerf --vid_dir=data/raw/videos/${VIDEO_ID}.mp4 --reset --debug --id_mode=global
Terminal Output:
args Namespace(vid_dir='data/raw/videos/Robert.mp4', ds_name='nerf', seed=0, process_id=0, total_process=1, id_mode='global', keypoint_mode='mediapipe', large_yaw_threshold=9999999.9, debug=True, reset=True, load_names=False) ['data/raw/videos/Robert.mp4'] process 0: fitting 3dmm ...: 0%| | 0/1 [00:00<?, ?it/s]batch 0 | loss_lan: 0.93, loss_reg_id: 0.75,loss_reg_exp: 0.27,loss_lap_ldm:0.0041 |--------loss_vel_id: 0.0000, loss_vel_exp: 0.0680, loss_vel_euler: 0.0047, loss_vel_trans: 0.0171 batch 1 | loss_lan: 0.81, loss_reg_id: 0.75,loss_reg_exp: 0.14,loss_lap_ldm:0.0028 |--------loss_vel_id: 0.0000, loss_vel_exp: 0.0550, loss_vel_euler: 0.0030, loss_vel_trans: 0.0135 batch 2 | loss_lan: 0.76, loss_reg_id: 0.75,loss_reg_exp: 0.12,loss_lap_ldm:0.0027 |--------loss_vel_id: 0.0000, loss_vel_exp: 0.0521, loss_vel_euler: 0.0032, loss_vel_trans: 0.0133 batch 3 | loss_lan: 0.77, loss_reg_id: 0.75,loss_reg_exp: 0.17,loss_lap_ldm:0.0015 |--------loss_vel_id: 0.0000, loss_vel_exp: 0.0447, loss_vel_euler: 0.0035, loss_vel_trans: 0.0176 batch 4 | loss_lan: 0.84, loss_reg_id: 0.75,loss_reg_exp: 0.15,loss_lap_ldm:0.0034 |--------loss_vel_id: 0.0000, loss_vel_exp: 0.0635, loss_vel_euler: 0.0042, loss_vel_trans: 0.0166 batch 5 | loss_lan: 0.78, loss_reg_id: 0.75,loss_reg_exp: 0.14,loss_lap_ldm:0.0042 |--------loss_vel_id: 0.0000, loss_vel_exp: 0.0437, loss_vel_euler: 0.0022, loss_vel_trans: 0.0112 batch 6 | loss_lan: 0.72, loss_reg_id: 0.75,loss_reg_exp: 0.10,loss_lap_ldm:0.0010 |--------loss_vel_id: 0.0000, loss_vel_exp: 0.0359, loss_vel_euler: 0.0028, loss_vel_trans: 0.0093 batch 7 | loss_lan: 0.91, loss_reg_id: 0.75,loss_reg_exp: 0.14,loss_lap_ldm:0.0041 |--------loss_vel_id: 0.0000, loss_vel_exp: 0.0727, loss_vel_euler: 0.0055, loss_vel_trans: 0.0154 batch 8 | loss_lan: 0.82, loss_reg_id: 0.75,loss_reg_exp: 0.12,loss_lap_ldm:0.0020 |--------loss_vel_id: 0.0000, loss_vel_exp: 0.0478, loss_vel_euler: 0.0036, loss_vel_trans: 0.0081 batch 9 | loss_lan: 0.88, loss_reg_id: 0.75,loss_reg_exp: 0.15,loss_lap_ldm:0.0039 |--------loss_vel_id: 0.0000, loss_vel_exp: 0.0630, loss_vel_euler: 0.0037, loss_vel_trans: 0.0128 batch 10 | loss_lan: 0.75, loss_reg_id: 0.75,loss_reg_exp: 0.13,loss_lap_ldm:0.0011 |--------loss_vel_id: 0.0000, loss_vel_exp: 0.0390, loss_vel_euler: 0.0039, loss_vel_trans: 0.0118 batch 11 | loss_lan: 0.82, loss_reg_id: 0.75,loss_reg_exp: 0.14,loss_lap_ldm:0.0031 |--------loss_vel_id: 0.0000, loss_vel_exp: 0.0543, loss_vel_euler: 0.0044, loss_vel_trans: 0.0120 batch 12 | loss_lan: 0.80, loss_reg_id: 0.75,loss_reg_exp: 0.17,loss_lap_ldm:0.0041 |--------loss_vel_id: 0.0000, loss_vel_exp: 0.0667, loss_vel_euler: 0.0054, loss_vel_trans: 0.0192 batch 13 | loss_lan: 0.72, loss_reg_id: 0.75,loss_reg_exp: 0.11,loss_lap_ldm:0.0033 |--------loss_vel_id: 0.0000, loss_vel_exp: 0.0478, loss_vel_euler: 0.0021, loss_vel_trans: 0.0118 batch 14 | loss_lan: 0.83, loss_reg_id: 0.75,loss_reg_exp: 0.15,loss_lap_ldm:0.0030 |--------loss_vel_id: 0.0000, loss_vel_exp: 0.0589, loss_vel_euler: 0.0043, loss_vel_trans: 0.0139 batch 15 | loss_lan: 0.87, loss_reg_id: 0.75,loss_reg_exp: 0.14,loss_lap_ldm:0.0017 |--------loss_vel_id: 0.0000, loss_vel_exp: 0.0377, loss_vel_euler: 0.0021, loss_vel_trans: 0.0081 batch 16 | loss_lan: 0.93, loss_reg_id: 0.75,loss_reg_exp: 0.18,loss_lap_ldm:0.0031 |--------loss_vel_id: 0.0000, loss_vel_exp: 0.0583, loss_vel_euler: 0.0052, loss_vel_trans: 0.0158 batch 17 | loss_lan: 0.81, loss_reg_id: 0.75,loss_reg_exp: 0.17,loss_lap_ldm:0.0042 |--------loss_vel_id: 0.0000, loss_vel_exp: 0.0566, loss_vel_euler: 0.0022, loss_vel_trans: 0.0151 batch 18 | loss_lan: 0.83, loss_reg_id: 0.75,loss_reg_exp: 0.12,loss_lap_ldm:0.0048 |--------loss_vel_id: 0.0000, loss_vel_exp: 0.0777, loss_vel_euler: 0.0046, loss_vel_trans: 0.0163 batch 19 | loss_lan: 0.78, loss_reg_id: 0.75,loss_reg_exp: 0.12,loss_lap_ldm:0.0024 |--------loss_vel_id: 0.0000, loss_vel_exp: 0.0507, loss_vel_euler: 0.0050, loss_vel_trans: 0.0100 batch 20 | loss_lan: 0.91, loss_reg_id: 0.75,loss_reg_exp: 0.15,loss_lap_ldm:0.0032 |--------loss_vel_id: 0.0000, loss_vel_exp: 0.0534, loss_vel_euler: 0.0044, loss_vel_trans: 0.0145 batch 21 | loss_lan: 0.78, loss_reg_id: 0.75,loss_reg_exp: 0.12,loss_lap_ldm:0.0033 |--------loss_vel_id: 0.0000, loss_vel_exp: 0.0604, loss_vel_euler: 0.0034, loss_vel_trans: 0.0147 batch 22 | loss_lan: 0.81, loss_reg_id: 0.75,loss_reg_exp: 0.14,loss_lap_ldm:0.0053 |--------loss_vel_id: 0.0000, loss_vel_exp: 0.0659, loss_vel_euler: 0.0035, loss_vel_trans: 0.0163 batch 23 | loss_lan: 0.74, loss_reg_id: 0.75,loss_reg_exp: 0.13,loss_lap_ldm:0.0036 |--------loss_vel_id: 0.0000, loss_vel_exp: 0.0595, loss_vel_euler: 0.0019, loss_vel_trans: 0.0113 batch 24 | loss_lan: 0.79, loss_reg_id: 0.75,loss_reg_exp: 0.12,loss_lap_ldm:0.0044 |--------loss_vel_id: 0.0000, loss_vel_exp: 0.0600, loss_vel_euler: 0.0023, loss_vel_trans: 0.0137 batch 25 | loss_lan: 0.85, loss_reg_id: 0.75,loss_reg_exp: 0.15,loss_lap_ldm:0.0039 |--------loss_vel_id: 0.0000, loss_vel_exp: 0.0645, loss_vel_euler: 0.0048, loss_vel_trans: 0.0142 batch 26 | loss_lan: 0.81, loss_reg_id: 0.75,loss_reg_exp: 0.16,loss_lap_ldm:0.0042 |--------loss_vel_id: 0.0000, loss_vel_exp: 0.0624, loss_vel_euler: 0.0051, loss_vel_trans: 0.0133 batch 27 | loss_lan: 0.72, loss_reg_id: 0.75,loss_reg_exp: 0.12,loss_lap_ldm:0.0018 |--------loss_vel_id: 0.0000, loss_vel_exp: 0.0447, loss_vel_euler: 0.0029, loss_vel_trans: 0.0063 batch 28 | loss_lan: 0.71, loss_reg_id: 0.75,loss_reg_exp: 0.10,loss_lap_ldm:0.0011 |--------loss_vel_id: 0.0000, loss_vel_exp: 0.0354, loss_vel_euler: 0.0017, loss_vel_trans: 0.0067 batch 29 | loss_lan: 0.84, loss_reg_id: 0.75,loss_reg_exp: 0.15,loss_lap_ldm:0.0017 |--------loss_vel_id: 0.0000, loss_vel_exp: 0.0435, loss_vel_euler: 0.0042, loss_vel_trans: 0.0149 batch 30 | loss_lan: 0.74, loss_reg_id: 0.75,loss_reg_exp: 0.11,loss_lap_ldm:0.0027 |--------loss_vel_id: 0.0000, loss_vel_exp: 0.0487, loss_vel_euler: 0.0022, loss_vel_trans: 0.0107 batch 31 | loss_lan: 0.82, loss_reg_id: 0.75,loss_reg_exp: 0.16,loss_lap_ldm:0.0024 |--------loss_vel_id: 0.0000, loss_vel_exp: 0.0587, loss_vel_euler: 0.0055, loss_vel_trans: 0.0169 batch 32 | loss_lan: 0.80, loss_reg_id: 0.75,loss_reg_exp: 0.15,loss_lap_ldm:0.0067 |--------loss_vel_id: 0.0000, loss_vel_exp: 0.0752, loss_vel_euler: 0.0048, loss_vel_trans: 0.0172 batch 33 | loss_lan: 0.69, loss_reg_id: 0.75,loss_reg_exp: 0.09,loss_lap_ldm:0.0007 |--------loss_vel_id: 0.0000, loss_vel_exp: 0.0271, loss_vel_euler: 0.0021, loss_vel_trans: 0.0065 batch 34 | loss_lan: 0.78, loss_reg_id: 0.75,loss_reg_exp: 0.12,loss_lap_ldm:0.0029 |--------loss_vel_id: 0.0000, loss_vel_exp: 0.0569, loss_vel_euler: 0.0034, loss_vel_trans: 0.0098 batch 35 | loss_lan: 0.69, loss_reg_id: 0.75,loss_reg_exp: 0.09,loss_lap_ldm:0.0006 |--------loss_vel_id: 0.0000, loss_vel_exp: 0.0273, loss_vel_euler: 0.0018, loss_vel_trans: 0.0057 batch 36 | loss_lan: 0.77, loss_reg_id: 0.75,loss_reg_exp: 0.12,loss_lap_ldm:0.0016 |--------loss_vel_id: 0.0000, loss_vel_exp: 0.0496, loss_vel_euler: 0.0031, loss_vel_trans: 0.0092 batch 37 | loss_lan: 0.73, loss_reg_id: 0.75,loss_reg_exp: 0.16,loss_lap_ldm:0.0027 |--------loss_vel_id: 0.0000, loss_vel_exp: 0.0551, loss_vel_euler: 0.0039, loss_vel_trans: 0.0187 batch 38 | loss_lan: 0.83, loss_reg_id: 0.75,loss_reg_exp: 0.15,loss_lap_ldm:0.0027 |--------loss_vel_id: 0.0000, loss_vel_exp: 0.0586, loss_vel_euler: 0.0045, loss_vel_trans: 0.0145 batch 39 | loss_lan: 0.82, loss_reg_id: 0.75,loss_reg_exp: 0.14,loss_lap_ldm:0.0032 |--------loss_vel_id: 0.0000, loss_vel_exp: 0.0591, loss_vel_euler: 0.0048, loss_vel_trans: 0.0142 batch 40 | loss_lan: 0.83, loss_reg_id: 0.75,loss_reg_exp: 0.14,loss_lap_ldm:0.0030 |--------loss_vel_id: 0.0000, loss_vel_exp: 0.0550, loss_vel_euler: 0.0047, loss_vel_trans: 0.0134 batch 41 | loss_lan: 0.74, loss_reg_id: 0.75,loss_reg_exp: 0.11,loss_lap_ldm:0.0007 |--------loss_vel_id: 0.0000, loss_vel_exp: 0.0303, loss_vel_euler: 0.0018, loss_vel_trans: 0.0077 batch 42 | loss_lan: 0.75, loss_reg_id: 0.75,loss_reg_exp: 0.11,loss_lap_ldm:0.0015 |--------loss_vel_id: 0.0000, loss_vel_exp: 0.0385, loss_vel_euler: 0.0029, loss_vel_trans: 0.0087 batch 43 | loss_lan: 0.81, loss_reg_id: 0.75,loss_reg_exp: 0.12,loss_lap_ldm:0.0013 |--------loss_vel_id: 0.0000, loss_vel_exp: 0.0423, loss_vel_euler: 0.0034, loss_vel_trans: 0.0116 batch 44 | loss_lan: 0.78, loss_reg_id: 0.75,loss_reg_exp: 0.11,loss_lap_ldm:0.0021 |--------loss_vel_id: 0.0000, loss_vel_exp: 0.0456, loss_vel_euler: 0.0024, loss_vel_trans: 0.0082 batch 45 | loss_lan: 0.86, loss_reg_id: 0.75,loss_reg_exp: 0.14,loss_lap_ldm:0.0024 |--------loss_vel_id: 0.0000, loss_vel_exp: 0.0575, loss_vel_euler: 0.0053, loss_vel_trans: 0.0123 batch 46 | loss_lan: 0.75, loss_reg_id: 0.75,loss_reg_exp: 0.11,loss_lap_ldm:0.0009 |--------loss_vel_id: 0.0000, loss_vel_exp: 0.0328, loss_vel_euler: 0.0027, loss_vel_trans: 0.0072 batch 47 | loss_lan: 0.85, loss_reg_id: 0.75,loss_reg_exp: 0.16,loss_lap_ldm:0.0023 |--------loss_vel_id: 0.0000, loss_vel_exp: 0.0559, loss_vel_euler: 0.0051, loss_vel_trans: 0.0145 batch 48 | loss_lan: 0.81, loss_reg_id: 0.75,loss_reg_exp: 0.13,loss_lap_ldm:0.0027 |--------loss_vel_id: 0.0000, loss_vel_exp: 0.0414, loss_vel_euler: 0.0022, loss_vel_trans: 0.0079 batch 49 | loss_lan: 0.80, loss_reg_id: 0.75,loss_reg_exp: 0.09,loss_lap_ldm:0.0017 |--------loss_vel_id: 0.0000, loss_vel_exp: 0.0333, loss_vel_euler: 0.0022, loss_vel_trans: 0.0070 batch 50 | loss_lan: 0.80, loss_reg_id: 0.75,loss_reg_exp: 0.11,loss_lap_ldm:0.0034 |--------loss_vel_id: 0.0000, loss_vel_exp: 0.0552, loss_vel_euler: 0.0045, loss_vel_trans: 0.0127 batch 51 | loss_lan: 0.83, loss_reg_id: 0.75,loss_reg_exp: 0.13,loss_lap_ldm:0.0025 |--------loss_vel_id: 0.0000, loss_vel_exp: 0.0535, loss_vel_euler: 0.0037, loss_vel_trans: 0.0107 batch 52 | loss_lan: 0.82, loss_reg_id: 0.75,loss_reg_exp: 0.14,loss_lap_ldm:0.0045 |--------loss_vel_id: 0.0000, loss_vel_exp: 0.0713, loss_vel_euler: 0.0039, loss_vel_trans: 0.0135 batch 53 | loss_lan: 0.81, loss_reg_id: 0.75,loss_reg_exp: 0.13,loss_lap_ldm:0.0028 |--------loss_vel_id: 0.0000, loss_vel_exp: 0.0566, loss_vel_euler: 0.0041, loss_vel_trans: 0.0139 batch 54 | loss_lan: 0.73, loss_reg_id: 0.75,loss_reg_exp: 0.12,loss_lap_ldm:0.0012 |--------loss_vel_id: 0.0000, loss_vel_exp: 0.0390, loss_vel_euler: 0.0016, loss_vel_trans: 0.0089 batch 55 | loss_lan: 0.67, loss_reg_id: 0.75,loss_reg_exp: 0.09,loss_lap_ldm:0.0009 |--------loss_vel_id: 0.0000, loss_vel_exp: 0.0265, loss_vel_euler: 0.0014, loss_vel_trans: 0.0071 batch 56 | loss_lan: 0.73, loss_reg_id: 0.75,loss_reg_exp: 0.10,loss_lap_ldm:0.0027 |--------loss_vel_id: 0.0000, loss_vel_exp: 0.0558, loss_vel_euler: 0.0025, loss_vel_trans: 0.0145 batch 57 | loss_lan: 0.75, loss_reg_id: 0.75,loss_reg_exp: 0.13,loss_lap_ldm:0.0018 |--------loss_vel_id: 0.0000, loss_vel_exp: 0.0459, loss_vel_euler: 0.0030, loss_vel_trans: 0.0138 batch 58 | loss_lan: 0.73, loss_reg_id: 0.75,loss_reg_exp: 0.12,loss_lap_ldm:0.0023 |--------loss_vel_id: 0.0000, loss_vel_exp: 0.0505, loss_vel_euler: 0.0028, loss_vel_trans: 0.0149 batch 59 | loss_lan: 0.84, loss_reg_id: 0.75,loss_reg_exp: 0.15,loss_lap_ldm:0.0028 |--------loss_vel_id: 0.0000, loss_vel_exp: 0.0602, loss_vel_euler: 0.0058, loss_vel_trans: 0.0208 batch 60 | loss_lan: 0.73, loss_reg_id: 0.75,loss_reg_exp: 0.12,loss_lap_ldm:0.0010 |--------loss_vel_id: 0.0000, loss_vel_exp: 0.0365, loss_vel_euler: 0.0020, loss_vel_trans: 0.0093 batch 61 | loss_lan: 0.69, loss_reg_id: 0.75,loss_reg_exp: 0.09,loss_lap_ldm:0.0023 |--------loss_vel_id: 0.0000, loss_vel_exp: 0.0489, loss_vel_euler: 0.0031, loss_vel_trans: 0.0103 batch 62 | loss_lan: 0.81, loss_reg_id: 0.75,loss_reg_exp: 0.15,loss_lap_ldm:0.0018 |--------loss_vel_id: 0.0000, loss_vel_exp: 0.0537, loss_vel_euler: 0.0040, loss_vel_trans: 0.0195 batch 63 | loss_lan: 0.70, loss_reg_id: 0.75,loss_reg_exp: 0.10,loss_lap_ldm:0.0022 |--------loss_vel_id: 0.0000, loss_vel_exp: 0.0461, loss_vel_euler: 0.0015, loss_vel_trans: 0.0086 batch 64 | loss_lan: 0.74, loss_reg_id: 0.75,loss_reg_exp: 0.11,loss_lap_ldm:0.0020 |--------loss_vel_id: 0.0000, loss_vel_exp: 0.0375, loss_vel_euler: 0.0009, loss_vel_trans: 0.0075 batch 65 | loss_lan: 0.83, loss_reg_id: 0.75,loss_reg_exp: 0.16,loss_lap_ldm:0.0042 |--------loss_vel_id: 0.0000, loss_vel_exp: 0.0630, loss_vel_euler: 0.0044, loss_vel_trans: 0.0158 batch 66 | loss_lan: 0.75, loss_reg_id: 0.75,loss_reg_exp: 0.12,loss_lap_ldm:0.0032 |--------loss_vel_id: 0.0000, loss_vel_exp: 0.0653, loss_vel_euler: 0.0045, loss_vel_trans: 0.0138 batch 67 | loss_lan: 0.80, loss_reg_id: 0.75,loss_reg_exp: 0.13,loss_lap_ldm:0.0038 |--------loss_vel_id: 0.0000, loss_vel_exp: 0.0692, loss_vel_euler: 0.0067, loss_vel_trans: 0.0137 batch 68 | loss_lan: 0.71, loss_reg_id: 0.75,loss_reg_exp: 0.11,loss_lap_ldm:0.0016 |--------loss_vel_id: 0.0000, loss_vel_exp: 0.0417, loss_vel_euler: 0.0023, loss_vel_trans: 0.0091 batch 69 | loss_lan: 0.72, loss_reg_id: 0.75,loss_reg_exp: 0.10,loss_lap_ldm:0.0028 |--------loss_vel_id: 0.0000, loss_vel_exp: 0.0441, loss_vel_euler: 0.0017, loss_vel_trans: 0.0078 batch 70 | loss_lan: 0.74, loss_reg_id: 0.75,loss_reg_exp: 0.10,loss_lap_ldm:0.0013 |--------loss_vel_id: 0.0000, loss_vel_exp: 0.0411, loss_vel_euler: 0.0027, loss_vel_trans: 0.0099 batch 71 | loss_lan: 0.88, loss_reg_id: 0.75,loss_reg_exp: 0.19,loss_lap_ldm:0.0064 |--------loss_vel_id: 0.0000, loss_vel_exp: 0.0808, loss_vel_euler: 0.0078, loss_vel_trans: 0.0201 batch 72 | loss_lan: 0.73, loss_reg_id: 0.75,loss_reg_exp: 0.12,loss_lap_ldm:0.0021 |--------loss_vel_id: 0.0000, loss_vel_exp: 0.0484, loss_vel_euler: 0.0032, loss_vel_trans: 0.0109 batch 73 | loss_lan: 0.75, loss_reg_id: 0.75,loss_reg_exp: 0.12,loss_lap_ldm:0.0022 |--------loss_vel_id: 0.0000, loss_vel_exp: 0.0469, loss_vel_euler: 0.0048, loss_vel_trans: 0.0107 batch 74 | loss_lan: 0.72, loss_reg_id: 0.75,loss_reg_exp: 0.10,loss_lap_ldm:0.0020 |--------loss_vel_id: 0.0000, loss_vel_exp: 0.0516, loss_vel_euler: 0.0023, loss_vel_trans: 0.0106 batch 75 | loss_lan: 0.74, loss_reg_id: 0.75,loss_reg_exp: 0.13,loss_lap_ldm:0.0018 |--------loss_vel_id: 0.0000, loss_vel_exp: 0.0506, loss_vel_euler: 0.0044, loss_vel_trans: 0.0167 batch 76 | loss_lan: 0.76, loss_reg_id: 0.75,loss_reg_exp: 0.14,loss_lap_ldm:0.0028 |--------loss_vel_id: 0.0000, loss_vel_exp: 0.0518, loss_vel_euler: 0.0032, loss_vel_trans: 0.0089 batch 77 | loss_lan: 0.71, loss_reg_id: 0.75,loss_reg_exp: 0.09,loss_lap_ldm:0.0002 |--------loss_vel_id: 0.0000, loss_vel_exp: 0.0163, loss_vel_euler: 0.0006, loss_vel_trans: 0.0033 batch 78 | loss_lan: 0.78, loss_reg_id: 0.75,loss_reg_exp: 0.14,loss_lap_ldm:0.0031 |--------loss_vel_id: 0.0000, loss_vel_exp: 0.0602, loss_vel_euler: 0.0046, loss_vel_trans: 0.0129 batch 79 | loss_lan: 0.81, loss_reg_id: 0.75,loss_reg_exp: 0.12,loss_lap_ldm:0.0038 |--------loss_vel_id: 0.0000, loss_vel_exp: 0.0684, loss_vel_euler: 0.0068, loss_vel_trans: 0.0191 batch 80 | loss_lan: 0.74, loss_reg_id: 0.75,loss_reg_exp: 0.11,loss_lap_ldm:0.0020 |--------loss_vel_id: 0.0000, loss_vel_exp: 0.0460, loss_vel_euler: 0.0028, loss_vel_trans: 0.0104 batch 81 | loss_lan: 0.74, loss_reg_id: 0.75,loss_reg_exp: 0.11,loss_lap_ldm:0.0023 |--------loss_vel_id: 0.0000, loss_vel_exp: 0.0481, loss_vel_euler: 0.0045, loss_vel_trans: 0.0115 batch 82 | loss_lan: 0.74, loss_reg_id: 0.75,loss_reg_exp: 0.16,loss_lap_ldm:0.0027 |--------loss_vel_id: 0.0000, loss_vel_exp: 0.0579, loss_vel_euler: 0.0040, loss_vel_trans: 0.0115 batch 83 | loss_lan: 0.69, loss_reg_id: 0.75,loss_reg_exp: 0.11,loss_lap_ldm:0.0029 |--------loss_vel_id: 0.0000, loss_vel_exp: 0.0521, loss_vel_euler: 0.0026, loss_vel_trans: 0.0089 batch 84 | loss_lan: 0.67, loss_reg_id: 0.75,loss_reg_exp: 0.12,loss_lap_ldm:0.0032 |--------loss_vel_id: 0.0000, loss_vel_exp: 0.0453, loss_vel_euler: 0.0018, loss_vel_trans: 0.0109 batch 85 | loss_lan: 0.90, loss_reg_id: 0.75,loss_reg_exp: 0.20,loss_lap_ldm:0.0034 |--------loss_vel_id: 0.0000, loss_vel_exp: 0.0728, loss_vel_euler: 0.0059, loss_vel_trans: 0.0183 batch 86 | loss_lan: 0.73, loss_reg_id: 0.75,loss_reg_exp: 0.11,loss_lap_ldm:0.0025 |--------loss_vel_id: 0.0000, loss_vel_exp: 0.0535, loss_vel_euler: 0.0027, loss_vel_trans: 0.0081 batch 87 | loss_lan: 0.73, loss_reg_id: 0.75,loss_reg_exp: 0.11,loss_lap_ldm:0.0024 |--------loss_vel_id: 0.0000, loss_vel_exp: 0.0502, loss_vel_euler: 0.0015, loss_vel_trans: 0.0083 batch 88 | loss_lan: 0.81, loss_reg_id: 0.75,loss_reg_exp: 0.17,loss_lap_ldm:0.0024 |--------loss_vel_id: 0.0000, loss_vel_exp: 0.0599, loss_vel_euler: 0.0055, loss_vel_trans: 0.0165 batch 89 | loss_lan: 0.88, loss_reg_id: 0.75,loss_reg_exp: 0.15,loss_lap_ldm:0.0041 |--------loss_vel_id: 0.0000, loss_vel_exp: 0.0695, loss_vel_euler: 0.0050, loss_vel_trans: 0.0186 batch 90 | loss_lan: 0.81, loss_reg_id: 0.75,loss_reg_exp: 0.14,loss_lap_ldm:0.0024 |--------loss_vel_id: 0.0000, loss_vel_exp: 0.0595, loss_vel_euler: 0.0075, loss_vel_trans: 0.0147 batch 91 | loss_lan: 0.76, loss_reg_id: 0.75,loss_reg_exp: 0.11,loss_lap_ldm:0.0026 |--------loss_vel_id: 0.0000, loss_vel_exp: 0.0544, loss_vel_euler: 0.0038, loss_vel_trans: 0.0108 batch 92 | loss_lan: 0.78, loss_reg_id: 0.75,loss_reg_exp: 0.12,loss_lap_ldm:0.0032 |--------loss_vel_id: 0.0000, loss_vel_exp: 0.0642, loss_vel_euler: 0.0060, loss_vel_trans: 0.0175 batch 93 | loss_lan: 0.78, loss_reg_id: 0.75,loss_reg_exp: 0.12,loss_lap_ldm:0.0015 |--------loss_vel_id: 0.0000, loss_vel_exp: 0.0389, loss_vel_euler: 0.0028, loss_vel_trans: 0.0095 batch 94 | loss_lan: 0.78, loss_reg_id: 0.75,loss_reg_exp: 0.15,loss_lap_ldm:0.0030 |--------loss_vel_id: 0.0000, loss_vel_exp: 0.0597, loss_vel_euler: 0.0053, loss_vel_trans: 0.0123 batch 95 | loss_lan: 0.75, loss_reg_id: 0.75,loss_reg_exp: 0.11,loss_lap_ldm:0.0017 |--------loss_vel_id: 0.0000, loss_vel_exp: 0.0441, loss_vel_euler: 0.0017, loss_vel_trans: 0.0065 batch 96 | loss_lan: 0.72, loss_reg_id: 0.75,loss_reg_exp: 0.10,loss_lap_ldm:0.0017 |--------loss_vel_id: 0.0000, loss_vel_exp: 0.0435, loss_vel_euler: 0.0021, loss_vel_trans: 0.0119 batch 97 | loss_lan: 0.74, loss_reg_id: 0.75,loss_reg_exp: 0.11,loss_lap_ldm:0.0020 |--------loss_vel_id: 0.0000, loss_vel_exp: 0.0437, loss_vel_euler: 0.0018, loss_vel_trans: 0.0108 batch 98 | loss_lan: 0.74, loss_reg_id: 0.75,loss_reg_exp: 0.13,loss_lap_ldm:0.0022 |--------loss_vel_id: 0.0000, loss_vel_exp: 0.0509, loss_vel_euler: 0.0027, loss_vel_trans: 0.0097 batch 99 | loss_lan: 0.73, loss_reg_id: 0.75,loss_reg_exp: 0.11,loss_lap_ldm:0.0025 |--------loss_vel_id: 0.0000, loss_vel_exp: 0.0423, loss_vel_euler: 0.0019, loss_vel_trans: 0.0084 batch 100 | loss_lan: 0.81, loss_reg_id: 0.75,loss_reg_exp: 0.13,loss_lap_ldm:0.0055 |--------loss_vel_id: 0.0000, loss_vel_exp: 0.0742, loss_vel_euler: 0.0038, loss_vel_trans: 0.0121 batch 101 | loss_lan: 0.79, loss_reg_id: 0.75,loss_reg_exp: 0.11,loss_lap_ldm:0.0026 |--------loss_vel_id: 0.0000, loss_vel_exp: 0.0540, loss_vel_euler: 0.0042, loss_vel_trans: 0.0101 batch 102 | loss_lan: 0.80, loss_reg_id: 0.75,loss_reg_exp: 0.14,loss_lap_ldm:0.0023 |--------loss_vel_id: 0.0000, loss_vel_exp: 0.0530, loss_vel_euler: 0.0039, loss_vel_trans: 0.0099 batch 103 | loss_lan: 0.79, loss_reg_id: 0.75,loss_reg_exp: 0.10,loss_lap_ldm:0.0017 |--------loss_vel_id: 0.0000, loss_vel_exp: 0.0465, loss_vel_euler: 0.0018, loss_vel_trans: 0.0086 batch 104 | loss_lan: 0.83, loss_reg_id: 0.75,loss_reg_exp: 0.14,loss_lap_ldm:0.0043 |--------loss_vel_id: 0.0000, loss_vel_exp: 0.0665, loss_vel_euler: 0.0043, loss_vel_trans: 0.0105 batch 105 | loss_lan: 0.79, loss_reg_id: 0.75,loss_reg_exp: 0.12,loss_lap_ldm:0.0043 |--------loss_vel_id: 0.0000, loss_vel_exp: 0.0561, loss_vel_euler: 0.0036, loss_vel_trans: 0.0155 batch 106 | loss_lan: 0.77, loss_reg_id: 0.75,loss_reg_exp: 0.11,loss_lap_ldm:0.0031 |--------loss_vel_id: 0.0000, loss_vel_exp: 0.0535, loss_vel_euler: 0.0021, loss_vel_trans: 0.0100 batch 107 | loss_lan: 0.80, loss_reg_id: 0.75,loss_reg_exp: 0.13,loss_lap_ldm:0.0021 |--------loss_vel_id: 0.0000, loss_vel_exp: 0.0506, loss_vel_euler: 0.0048, loss_vel_trans: 0.0153 batch 108 | loss_lan: 0.83, loss_reg_id: 0.75,loss_reg_exp: 0.24,loss_lap_ldm:0.0021 |--------loss_vel_id: 0.0000, loss_vel_exp: 0.0517, loss_vel_euler: 0.0046, loss_vel_trans: 0.0103 batch 109 | loss_lan: 0.74, loss_reg_id: 0.75,loss_reg_exp: 0.20,loss_lap_ldm:0.0019 |--------loss_vel_id: 0.0000, loss_vel_exp: 0.0419, loss_vel_euler: 0.0025, loss_vel_trans: 0.0074 batch 110 | loss_lan: 0.79, loss_reg_id: 0.75,loss_reg_exp: 0.12,loss_lap_ldm:0.0020 |--------loss_vel_id: 0.0000, loss_vel_exp: 0.0576, loss_vel_euler: 0.0038, loss_vel_trans: 0.0110 batch 111 | loss_lan: 0.76, loss_reg_id: 0.75,loss_reg_exp: 0.11,loss_lap_ldm:0.0028 |--------loss_vel_id: 0.0000, loss_vel_exp: 0.0562, loss_vel_euler: 0.0027, loss_vel_trans: 0.0105 batch 112 | loss_lan: 0.77, loss_reg_id: 0.75,loss_reg_exp: 0.12,loss_lap_ldm:0.0023 |--------loss_vel_id: 0.0000, loss_vel_exp: 0.0503, loss_vel_euler: 0.0033, loss_vel_trans: 0.0108 batch 113 | loss_lan: 0.72, loss_reg_id: 0.75,loss_reg_exp: 0.10,loss_lap_ldm:0.0016 |--------loss_vel_id: 0.0000, loss_vel_exp: 0.0432, loss_vel_euler: 0.0017, loss_vel_trans: 0.0058 batch 114 | loss_lan: 0.75, loss_reg_id: 0.75,loss_reg_exp: 0.15,loss_lap_ldm:0.0018 |--------loss_vel_id: 0.0000, loss_vel_exp: 0.0506, loss_vel_euler: 0.0046, loss_vel_trans: 0.0159 batch 115 | loss_lan: 0.81, loss_reg_id: 0.75,loss_reg_exp: 0.12,loss_lap_ldm:0.0054 |--------loss_vel_id: 0.0000, loss_vel_exp: 0.0729, loss_vel_euler: 0.0034, loss_vel_trans: 0.0143 batch 116 | loss_lan: 0.74, loss_reg_id: 0.75,loss_reg_exp: 0.10,loss_lap_ldm:0.0025 |--------loss_vel_id: 0.0000, loss_vel_exp: 0.0480, loss_vel_euler: 0.0042, loss_vel_trans: 0.0132 batch 117 | loss_lan: 0.72, loss_reg_id: 0.75,loss_reg_exp: 0.10,loss_lap_ldm:0.0016 |--------loss_vel_id: 0.0000, loss_vel_exp: 0.0444, loss_vel_euler: 0.0028, loss_vel_trans: 0.0099 batch 118 | loss_lan: 0.73, loss_reg_id: 0.75,loss_reg_exp: 0.13,loss_lap_ldm:0.0036 |--------loss_vel_id: 0.0000, loss_vel_exp: 0.0596, loss_vel_euler: 0.0051, loss_vel_trans: 0.0155 batch 119 | loss_lan: 0.84, loss_reg_id: 0.75,loss_reg_exp: 0.28,loss_lap_ldm:0.0014 |--------loss_vel_id: 0.0000, loss_vel_exp: 0.0432, loss_vel_euler: 0.0030, loss_vel_trans: 0.0138 batch 120 | loss_lan: 0.85, loss_reg_id: 0.75,loss_reg_exp: 0.31,loss_lap_ldm:0.0011 |--------loss_vel_id: 0.0000, loss_vel_exp: 0.0413, loss_vel_euler: 0.0025, loss_vel_trans: 0.0137 create rasterizer on device cuda:0cessed/videos/Robert/debug_fit_3dmm.mp4..: 0%| | 0/250 [00:00<?, ?it/s] rendering debug video to data/processed/videos/Robert/debug_fit_3dmm.mp4..: 0%| | 0/250 [00:00<?, ?it/s] data/raw/videos/Robert.mp4 Not compiled with GPU support finished 1 / 1 = 1.0000, failed 1 / 1 = 1.0000 process 0: fitting 3dmm ...: 100%|██████████████████████████████████████████████████████████████████████████| 1/1 [00:59<00:00, 59.58s/it] all failed image names: ['data/raw/videos/Robert.mp4'] All finished!
please check the memory space available, try to remove the files which were created in the failed attempts of precprocessing data/training, also remove the trash files so that you can make the space available Also another thing for the error of failed images please check that if you have done the complete preprocessing of data step by step rightly.
