Mask3D
Mask3D copied to clipboard
JonasSchult
hydra.errors.HydraException: Error calling 'datasets.semseg.SemanticSegmentationDataset' : not available number labels, select from: 200, 200
when I run the flow command, I got the issue: hydra.errors.HydraException: Error calling 'datasets.semseg.SemanticSegmentationDataset' : not available number labels, select from: 200, 200
main_instance_segmentation.py general.experiment_name=test1_scannet_val_query_150_topk_500_dbscan_0.95 general.project_name=scannet_eval general.checkpoint='checkpoints/scannet/scannet_val.ckpt' general.train_mode=false general.eval_on_segments=true general.train_on_segments=true model.num_queries=150 general.topk_per_image=500 general.use_dbscan=true general.dbscan_eps=0.95
/root/anaconda3/envs/mask3d_cuda113/lib/python3.10/site-packages/pytorch_lightning/utilities/seed.py:55: UserWarning: No seed found, seed set to 2801433411
rank_zero_warn(f"No seed found, seed set to {seed}")
Global seed set to 2801433411
EXPERIMENT ALREADY EXIST
{'target': 'pytorch_lightning.loggers.WandbLogger', 'project': '${general.project_name}', 'name': '${general.experiment_name}', 'save_dir': '${general.save_dir}', 'entity': 'manjusaka_labs', 'resume': 'allow', 'id': '${general.experiment_name}'}
wandb: Currently logged in as: bh_c (manjusaka_labs). Use wandb login --relogin to force relogin
wandb: wandb version 0.15.4 is available! To upgrade, please run:
wandb: $ pip install wandb --upgrade
wandb: Tracking run with wandb version 0.15.0
wandb: Run data is saved locally in saved/test/test1_scannet_val_query_150_topk_500_dbscan_0.95/wandb/run-20230609_012749-test1_scannet_val_query_150_topk_500_dbscan_0.95
wandb: Run wandb offline to turn off syncing.
wandb: Resuming run test1_scannet_val_query_150_topk_500_dbscan_0.95
wandb: ⭐️ View project at https://wandb.ai/manjusaka_labs/scannet_eval
wandb: 🚀 View run at https://wandb.ai/manjusaka_labs/scannet_eval/runs/test1_scannet_val_query_150_topk_500_dbscan_0.95
2023-06-09 01:27:54.018 | WARNING | utils.utils:load_checkpoint_with_missing_or_exsessive_keys:91 - Key not found, it will be initialized randomly: model.scene_min
2023-06-09 01:27:54.019 | WARNING | utils.utils:load_checkpoint_with_missing_or_exsessive_keys:91 - Key not found, it will be initialized randomly: model.scene_max
2023-06-09 01:27:54.145 | WARNING | utils.utils:load_checkpoint_with_missing_or_exsessive_keys:100 - criterion.empty_weight not in loaded checkpoint
2023-06-09 01:27:54.149 | WARNING | utils.utils:load_checkpoint_with_missing_or_exsessive_keys:115 - excessive key: model.scene_min
2023-06-09 01:27:54.149 | WARNING | utils.utils:load_checkpoint_with_missing_or_exsessive_keys:115 - excessive key: model.scene_max
[2023-06-09 01:27:54,238][main][INFO] - {'general_train_mode': False, 'general_task': 'instance_segmentation', 'general_seed': None, 'general_checkpoint': 'checkpoints/scannet/scannet_val.ckpt', 'general_backbone_checkpoint': None, 'general_freeze_backbone': False, 'general_linear_probing_backbone': False, 'general_train_on_segments': True, 'general_eval_on_segments': True, 'general_filter_out_instances': False, 'general_save_visualizations': False, 'general_visualization_point_size': 20, 'general_decoder_id': -1, 'general_export': False, 'general_use_dbscan': True, 'general_ignore_class_threshold': 100, 'general_project_name': 'scannet_eval', 'general_workspace': 'jonasschult', 'general_experiment_name': 'test1_scannet_val_query_150_topk_500_dbscan_0.95', 'general_num_targets': 19, 'general_add_instance': True, 'general_dbscan_eps': 0.95, 'general_dbscan_min_points': 1, 'general_export_threshold': 0.0001, 'general_reps_per_epoch': 1, 'general_on_crops': False, 'general_scores_threshold': 0.0, 'general_iou_threshold': 1.0, 'general_area': 5, 'general_eval_inner_core': -1, 'general_topk_per_image': 500, 'general_ignore_mask_idx': [], 'general_max_batch_size': 99999999, 'general_save_dir': 'saved/test/test1_scannet_val_query_150_topk_500_dbscan_0.95', 'general_gpus': 1, 'data_train_mode': 'train', 'data_validation_mode': 'validation', 'data_test_mode': 'validation', 'data_ignore_label': 255, 'data_add_raw_coordinates': True, 'data_add_colors': True, 'data_add_normals': False, 'data_in_channels': 3, 'data_num_labels': 20, 'data_add_instance': True, 'data_task': 'instance_segmentation', 'data_pin_memory': False, 'data_num_workers': 4, 'data_batch_size': 5, 'data_test_batch_size': 1, 'data_cache_data': False, 'data_voxel_size': 0.02, 'data_reps_per_epoch': 1, 'data_cropping': False, 'data_cropping_args_min_points': 30000, 'data_cropping_args_aspect': 0.8, 'data_cropping_args_min_crop': 0.5, 'data_cropping_args_max_crop': 1.0, 'data_crop_min_size': 20000, 'data_crop_length': 6.0, 'data_cropping_v1': True, 'data_train_dataloader__target_': 'torch.utils.data.DataLoader', 'data_train_dataloader_shuffle': True, 'data_train_dataloader_pin_memory': False, 'data_train_dataloader_num_workers': 4, 'data_train_dataloader_batch_size': 5, 'data_validation_dataloader__target_': 'torch.utils.data.DataLoader', 'data_validation_dataloader_shuffle': False, 'data_validation_dataloader_pin_memory': False, 'data_validation_dataloader_num_workers': 4, 'data_validation_dataloader_batch_size': 1, 'data_test_dataloader__target_': 'torch.utils.data.DataLoader', 'data_test_dataloader_shuffle': False, 'data_test_dataloader_pin_memory': False, 'data_test_dataloader_num_workers': 4, 'data_test_dataloader_batch_size': 1, 'data_train_dataset__target_': 'datasets.semseg.SemanticSegmentationDataset', 'data_train_dataset_dataset_name': 'scannet', 'data_train_dataset_data_dir': 'data/processed/scannet', 'data_train_dataset_image_augmentations_path': 'conf/augmentation/albumentations_aug.yaml', 'data_train_dataset_volume_augmentations_path': 'conf/augmentation/volumentations_aug.yaml', 'data_train_dataset_label_db_filepath': 'data/processed/scannet/label_database.yaml', 'data_train_dataset_color_mean_std': 'data/processed/scannet/color_mean_std.yaml', 'data_train_dataset_data_percent': 1.0, 'data_train_dataset_mode': 'train', 'data_train_dataset_ignore_label': 255, 'data_train_dataset_num_labels': 20, 'data_train_dataset_add_raw_coordinates': True, 'data_train_dataset_add_colors': True, 'data_train_dataset_add_normals': False, 'data_train_dataset_add_instance': True, 'data_train_dataset_instance_oversampling': 0.0, 'data_train_dataset_place_around_existing': False, 'data_train_dataset_point_per_cut': 0, 'data_train_dataset_max_cut_region': 0, 'data_train_dataset_flip_in_center': False, 'data_train_dataset_noise_rate': 0, 'data_train_dataset_resample_points': 0, 'data_train_dataset_add_unlabeled_pc': False, 'data_train_dataset_cropping': False, 'data_train_dataset_cropping_args_min_points': 30000, 'data_train_dataset_cropping_args_aspect': 0.8, 'data_train_dataset_cropping_args_min_crop': 0.5, 'data_train_dataset_cropping_args_max_crop': 1.0, 'data_train_dataset_is_tta': False, 'data_train_dataset_crop_min_size': 20000, 'data_train_dataset_crop_length': 6.0, 'data_train_dataset_filter_out_classes': [0, 1], 'data_train_dataset_label_offset': 2, 'data_validation_dataset__target_': 'datasets.semseg.SemanticSegmentationDataset', 'data_validation_dataset_dataset_name': 'scannet', 'data_validation_dataset_data_dir': 'data/processed/scannet', 'data_validation_dataset_image_augmentations_path': None, 'data_validation_dataset_volume_augmentations_path': None, 'data_validation_dataset_label_db_filepath': 'data/processed/scannet/label_database.yaml', 'data_validation_dataset_color_mean_std': 'data/processed/scannet/color_mean_std.yaml', 'data_validation_dataset_data_percent': 1.0, 'data_validation_dataset_mode': 'validation', 'data_validation_dataset_ignore_label': 255, 'data_validation_dataset_num_labels': 20, 'data_validation_dataset_add_raw_coordinates': True, 'data_validation_dataset_add_colors': True, 'data_validation_dataset_add_normals': False, 'data_validation_dataset_add_instance': True, 'data_validation_dataset_cropping': False, 'data_validation_dataset_is_tta': False, 'data_validation_dataset_crop_min_size': 20000, 'data_validation_dataset_crop_length': 6.0, 'data_validation_dataset_filter_out_classes': [0, 1], 'data_validation_dataset_label_offset': 2, 'data_test_dataset__target_': 'datasets.semseg.SemanticSegmentationDataset', 'data_test_dataset_dataset_name': 'scannet', 'data_test_dataset_data_dir': 'data/processed/scannet', 'data_test_dataset_image_augmentations_path': None, 'data_test_dataset_volume_augmentations_path': None, 'data_test_dataset_label_db_filepath': 'data/processed/scannet/label_database.yaml', 'data_test_dataset_color_mean_std': 'data/processed/scannet/color_mean_std.yaml', 'data_test_dataset_data_percent': 1.0, 'data_test_dataset_mode': 'validation', 'data_test_dataset_ignore_label': 255, 'data_test_dataset_num_labels': 20, 'data_test_dataset_add_raw_coordinates': True, 'data_test_dataset_add_colors': True, 'data_test_dataset_add_normals': False, 'data_test_dataset_add_instance': True, 'data_test_dataset_cropping': False, 'data_test_dataset_is_tta': False, 'data_test_dataset_crop_min_size': 20000, 'data_test_dataset_crop_length': 6.0, 'data_test_dataset_filter_out_classes': [0, 1], 'data_test_dataset_label_offset': 2, 'data_train_collation__target_': 'datasets.utils.VoxelizeCollate', 'data_train_collation_ignore_label': 255, 'data_train_collation_voxel_size': 0.02, 'data_train_collation_mode': 'train', 'data_train_collation_small_crops': False, 'data_train_collation_very_small_crops': False, 'data_train_collation_batch_instance': False, 'data_train_collation_probing': False, 'data_train_collation_task': 'instance_segmentation', 'data_train_collation_ignore_class_threshold': 100, 'data_train_collation_filter_out_classes': [0, 1], 'data_train_collation_label_offset': 2, 'data_train_collation_num_queries': 150, 'data_validation_collation__target_': 'datasets.utils.VoxelizeCollate', 'data_validation_collation_ignore_label': 255, 'data_validation_collation_voxel_size': 0.02, 'data_validation_collation_mode': 'validation', 'data_validation_collation_batch_instance': False, 'data_validation_collation_probing': False, 'data_validation_collation_task': 'instance_segmentation', 'data_validation_collation_ignore_class_threshold': 100, 'data_validation_collation_filter_out_classes': [0, 1], 'data_validation_collation_label_offset': 2, 'data_validation_collation_num_queries': 150, 'data_test_collation__target_': 'datasets.utils.VoxelizeCollate', 'data_test_collation_ignore_label': 255, 'data_test_collation_voxel_size': 0.02, 'data_test_collation_mode': 'validation', 'data_test_collation_batch_instance': False, 'data_test_collation_probing': False, 'data_test_collation_task': 'instance_segmentation', 'data_test_collation_ignore_class_threshold': 100, 'data_test_collation_filter_out_classes': [0, 1], 'data_test_collation_label_offset': 2, 'data_test_collation_num_queries': 150, 'logging': [{'target': 'pytorch_lightning.loggers.WandbLogger', 'project': 'scannet_eval', 'name': 'test1_scannet_val_query_150_topk_500_dbscan_0.95', 'save_dir': 'saved/test/test1_scannet_val_query_150_topk_500_dbscan_0.95', 'entity': 'manjusaka_labs', 'resume': 'allow', 'id': 'test1_scannet_val_query_150_topk_500_dbscan_0.95'}], 'model__target_': 'models.Mask3D', 'model_hidden_dim': 128, 'model_dim_feedforward': 1024, 'model_num_queries': 150, 'model_num_heads': 8, 'model_num_decoders': 3, 'model_dropout': 0.0, 'model_pre_norm': False, 'model_use_level_embed': False, 'model_normalize_pos_enc': True, 'model_positional_encoding_type': 'fourier', 'model_gauss_scale': 1.0, 'model_hlevels': [0, 1, 2, 3], 'model_non_parametric_queries': True, 'model_random_query_both': False, 'model_random_normal': False, 'model_random_queries': False, 'model_use_np_features': False, 'model_sample_sizes': [200, 800, 3200, 12800, 51200], 'model_max_sample_size': False, 'model_shared_decoder': True, 'model_num_classes': 19, 'model_train_on_segments': True, 'model_scatter_type': 'mean', 'model_voxel_size': 0.02, 'model_config_backbone__target_': 'models.Res16UNet34C', 'model_config_backbone_config_dialations': [1, 1, 1, 1], 'model_config_backbone_config_conv1_kernel_size': 5, 'model_config_backbone_config_bn_momentum': 0.02, 'model_config_backbone_in_channels': 3, 'model_config_backbone_out_channels': 20, 'model_config_backbone_out_fpn': True, 'metrics__target_': 'models.metrics.ConfusionMatrix', 'metrics_num_classes': 20, 'metrics_ignore_label': 255, 'optimizer__target_': 'torch.optim.AdamW', 'optimizer_lr': 0.0001, 'scheduler_scheduler__target_': 'torch.optim.lr_scheduler.OneCycleLR', 'scheduler_scheduler_max_lr': 0.0001, 'scheduler_scheduler_epochs': 601, 'scheduler_scheduler_steps_per_epoch': -1, 'scheduler_pytorch_lightning_params_interval': 'step', 'trainer_deterministic': False, 'trainer_max_epochs': 601, 'trainer_min_epochs': 1, 'trainer_resume_from_checkpoint': 'saved/test/test1_scannet_val_query_150_topk_500_dbscan_0.95/last-epoch.ckpt', 'trainer_check_val_every_n_epoch': 50, 'trainer_num_sanity_val_steps': 2, 'callbacks': [{'target': 'pytorch_lightning.callbacks.ModelCheckpoint', 'monitor': 'val_mean_ap_50', 'save_last': True, 'save_top_k': 1, 'mode': 'max', 'dirpath': 'saved/test/test1_scannet_val_query_150_topk_500_dbscan_0.95', 'filename': '{epoch}-{val_mean_ap_50:.3f}', 'every_n_epochs': 1}, {'target': 'pytorch_lightning.callbacks.LearningRateMonitor'}], 'matcher__target_': 'models.matcher.HungarianMatcher', 'matcher_cost_class': 2.0, 'matcher_cost_mask': 5.0, 'matcher_cost_dice': 2.0, 'matcher_num_points': -1, 'loss__target_': 'models.criterion.SetCriterion', 'loss_num_classes': 19, 'loss_eos_coef': 0.1, 'loss_losses': ['labels', 'masks'], 'loss_num_points': -1, 'loss_oversample_ratio': 3.0, 'loss_importance_sample_ratio': 0.75, 'loss_class_weights': -1}
/root/anaconda3/envs/mask3d_cuda113/lib/python3.10/site-packages/pytorch_lightning/trainer/connectors/accelerator_connector.py:446: LightningDeprecationWarning: Setting Trainer(gpus=1) is deprecated in v1.7 and will be removed in v2.0. Please use Trainer(accelerator='gpu', devices=1) instead.
rank_zero_deprecation(
/root/anaconda3/envs/mask3d_cuda113/lib/python3.10/site-packages/pytorch_lightning/trainer/connectors/checkpoint_connector.py:52: LightningDeprecationWarning: Setting Trainer(resume_from_checkpoint=) is deprecated in v1.5 and will be removed in v1.7. Please pass Trainer.fit(ckpt_path=) directly instead.
rank_zero_deprecation(
/root/anaconda3/envs/mask3d_cuda113/lib/python3.10/site-packages/pytorch_lightning/trainer/connectors/callback_connector.py:57: LightningDeprecationWarning: Setting Trainer(weights_save_path=) has been deprecated in v1.6 and will be removed in v1.8. Please pass dirpath directly to the ModelCheckpoint callback
rank_zero_deprecation(
GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
/home/mylabs/Mask3D/datasets/semseg.py:696: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
file = yaml.load(f)
Traceback (most recent call last):
File "/root/anaconda3/envs/mask3d_cuda113/lib/python3.10/site-packages/hydra/utils.py", line 63, in call
return _instantiate_class(type_or_callable, config, *args, **kwargs)
File "/root/anaconda3/envs/mask3d_cuda113/lib/python3.10/site-packages/hydra/_internal/utils.py", line 500, in _instantiate_class
return clazz(*args, **final_kwargs)
File "/home/mylabs/Mask3D/datasets/semseg.py", line 218, in init
self._labels = self._select_correct_labels(labels, num_labels)
File "/home/mylabs/Mask3D/datasets/semseg.py", line 724, in _select_correct_labels
raise ValueError(msg)
ValueError: not available number labels, select from:
200, 200
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/root/anaconda3/envs/mask3d_cuda113/lib/python3.10/site-packages/hydra/_internal/utils.py", line 198, in run_and_report
return func()
File "/root/anaconda3/envs/mask3d_cuda113/lib/python3.10/site-packages/hydra/_internal/utils.py", line 347, in
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/mylabs/Mask3D/main_instance_segmentation.py", line 114, in
Can you tell me how to solve the problem, thank you very much!
Hi!
Is it possible that your generated dataset is ScanNet200 and not ScanNet?
Best, Jonas
Your reply and help are greatly appreciated!
However,
I regenrated the scannet data by " python python -m datasets.preprocessing.scannet_preprocessing preprocess
--data_dir="./data/raw/scannet/scannet"
--save_dir="./data/processed/scannet"
--git_repo="./data/raw/scannet/ScanNet"
--scannet200=False"
the result of data structure like flow:
ScanNet200
├── instance_gt /
│ ├── train /
│ │ ├── scene0000_00.txt
│ │ ├── ...
│ │ └── scene0706_00.txt
│ └── validation /
│ ├── scene0011_00.txt
│ ├── ...
│ └── scene0704_01.txt
├── train /
│ ├── 0000_00.npy
│ ├── ...
│ └── 0706_00.npy
├── validation /
│ ├── 0011_00.npy
│ ├── ...
│ └── 0704_01.npy
├── test /
│ ├── 0707_00.npy
│ ├── ...
│ └── 0806_00.npy
├── color_mean_std.yaml 137B
├── label_database.yaml 15KB
├── test_database.yaml 24KB
├── train_database.yaml 1.19M
├── validation_database.yaml 319KB
└── train_validation_database.yaml 1.50M
and regenerated the scannet200 data by
"python -m datasets.preprocessing.scannet_preprocessing preprocess
--data_dir="./data/raw/scannet/scannet"
--save_dir="./data/processed/scannet200"
--git_repo="./data/raw/scannet/ScanNet"
--scannet200=true"
the result of data structure like flow:
ScanNet200
├── instance_gt /
│ ├── train /
│ │ ├── scene0000_00.txt
│ │ ├── ...
│ │ └── scene0706_00.txt
│ └── validation /
│ ├── scene0011_00.txt
│ ├── ...
│ └── scene0704_01.txt
├── train /
│ ├── 0000_00.npy
│ ├── ...
│ └── 0706_00.npy
├── validation /
│ ├── 0011_00.npy
│ ├── ...
│ └── 0704_01.npy
├── test /
│ ├── 0707_00.npy
│ ├── ...
│ └── 0806_00.npy
├── color_mean_std.yaml 137B
├── label_database.yaml 15KB
├── test_database.yaml 23KB
├── train_database.yaml 1.17M
├── validation_database.yaml 319KB
└── train_validation_database.yaml 1.14M
I used the same raw dataset "--data_dir="./data/raw/scannet/scannet" ",and just make the different command like " --scannet200=False" or " --scannet200=true".
In the end, I got the same structure output data, and there just have the different on the size of data.
So, if I used a wrong command? I not very understand clearly.
Looking forward to your reply and help,best wish for you!
Hi!
Is it possible that your generated dataset is ScanNet200 and not ScanNet?
Best, Jonas
And when I run the train command as "python main_instance_segmentation.py" or "python main_instance_segmentation.py \ general.checkpoint='/home/mylabs/Mask3D/checkpoints/s3dis/from_scratch/area1.ckpt' general.train_mode=false" there was the same error.
I have the similar problem and hope the author could help to explain it. Many thanks! @JonasSchult
Mask3D/main_instance_segmentation.py", line 79, in train runner = Trainer( File "/usr2/.local/lib/python3.10/site-packages/pytorch_lightning/utilities/argparse.py", line 345, in insert_env_defaults return fn(self, **kwargs)
File "/usr2/.local/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 485, in init
self._callback_connector.on_trainer_init(
File "/usr2/.local/lib/python3.10/site-packages/pytorch_lightning/trainer/connectors/callback_connector.py", line 89, in on_trainer_init
self.trainer.callbacks.extend(_configure_external_callbacks())
File "/usr2/.local/lib/python3.10/site-packages/pytorch_lightning/trainer/connectors/callback_connector.py", line 265, in _configure_external_callbacks
factories = entry_points(group=group) # type: ignore[call-arg]
File "/opt/conda/envs/mask3d/lib/python3.10/importlib/metadata/init.py", line 1021, in entry_points return SelectableGroups.load(eps).select(**params) File "/opt/conda/envs/mask3d/lib/python3.10/importlib/metadata/init.py", line 459, in load ordered = sorted(eps, key=by_group) File "/opt/conda/envs/mask3d/lib/python3.10/importlib/metadata/init.py", line 1018, in
During handling of the above exception, another exception occurred:
Mask3D/main_instance_segmentation.py", line 116, in
File "/opt/conda/envs/mask3d/lib/python3.10/site-packages/hydra/main.py", line 32, in decorated_main
_run_hydra(
File "/opt/conda/envs/mask3d/lib/python3.10/site-packages/hydra/_internal/utils.py", line 346, in _run_hydra
run_and_report(
File "/opt/conda/envs/mask3d/lib/python3.10/site-packages/hydra/_internal/utils.py", line 267, in run_and_report
print_exception(etype=None, value=ex, tb=final_tb) # type: ignore
TypeError: print_exception() got an unexpected keyword argument 'etype'
It seems that the issue results from https://github.com/JonasSchult/Mask3D/blob/3db966df2c021c3361bd6eed56121428b3e7a21d/datasets/semseg.py#L699-L720. The label_database.yaml is the same for both ScanNet and ScanNet200 after preprocessing.
Thank you for your reply, but I don't understand how to change the code to solve this problem. May I get some advice from you?
You just need to rerun scannet data preprocessing with --scannet200=False/True instead of --scannet200=false/true.
See my pull request: #111
Thank you very much, I have solved the issue. However, I have got another problem with the flowing:
"pytorch_lightning.utilities.exceptions.MisconfigurationException: ModelCheckpoint(monitor='val_mean_ap_50') could not find the monitored key in the returned metrics: ['train_loss_ce', 'train_loss_mask', 'train_loss_dice', 'train_loss_ce_0', 'train_loss_mask_0', 'train_loss_dice_0', 'train_loss_ce_1', 'train_loss_mask_1', 'train_loss_dice_1', 'train_loss_ce_2', 'train_loss_mask_2', 'train_loss_dice_2', 'train_loss_ce_3', 'train_loss_mask_3', 'train_loss_dice_3', 'train_loss_ce_4', 'train_loss_mask_4', 'train_loss_dice_4', 'train_loss_ce_5', 'train_loss_mask_5', 'train_loss_dice_5', 'train_loss_ce_6', 'train_loss_mask_6', 'train_loss_dice_6', 'train_loss_ce_7', 'train_loss_mask_7', 'train_loss_dice_7', 'train_loss_ce_8', 'train_loss_mask_8', 'train_loss_dice_8', 'train_loss_ce_9', 'train_loss_mask_9', 'train_loss_dice_9', 'train_loss_ce_10', 'train_loss_mask_10', 'train_loss_dice_10', 'train_loss_ce_11', 'train_loss_mask_11', 'train_loss_dice_11', 'train_mean_loss_ce', 'train_mean_loss_mask', 'train_mean_loss_dice', 'epoch', 'step']. HINT: Did you call log('val_mean_ap_50', value) in the LightningModule?
"
Epoch 49: 100%|████| 1513/1513 [1:56:39<00:00, 4.63s/it, loss=43.9, v_num=TION]
Traceback (most recent call last):
File "/root/anaconda3/envs/mask3d_cuda113/lib/python3.10/site-packages/hydra/_internal/utils.py", line 198, in run_and_report
return func()
File "/root/anaconda3/envs/mask3d_cuda113/lib/python3.10/site-packages/hydra/_internal/utils.py", line 347, in ModelCheckpoint(monitor='val_mean_ap_50') could not find the monitored key in the returned metrics: ['train_loss_ce', 'train_loss_mask', 'train_loss_dice', 'train_loss_ce_0', 'train_loss_mask_0', 'train_loss_dice_0', 'train_loss_ce_1', 'train_loss_mask_1', 'train_loss_dice_1', 'train_loss_ce_2', 'train_loss_mask_2', 'train_loss_dice_2', 'train_loss_ce_3', 'train_loss_mask_3', 'train_loss_dice_3', 'train_loss_ce_4', 'train_loss_mask_4', 'train_loss_dice_4', 'train_loss_ce_5', 'train_loss_mask_5', 'train_loss_dice_5', 'train_loss_ce_6', 'train_loss_mask_6', 'train_loss_dice_6', 'train_loss_ce_7', 'train_loss_mask_7', 'train_loss_dice_7', 'train_loss_ce_8', 'train_loss_mask_8', 'train_loss_dice_8', 'train_loss_ce_9', 'train_loss_mask_9', 'train_loss_dice_9', 'train_loss_ce_10', 'train_loss_mask_10', 'train_loss_dice_10', 'train_loss_ce_11', 'train_loss_mask_11', 'train_loss_dice_11', 'train_mean_loss_ce', 'train_mean_loss_mask', 'train_mean_loss_dice', 'epoch', 'step']. HINT: Did you call log('val_mean_ap_50', value) in the LightningModule?
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/mylabs/Mask3D/main_instance_segmentation.py", line 114, in
I am not very clear about the problem above, may I get some help from you, if so, it would my best greatest.
Hey, I just found that the function traceback.print_exception() has been changed in python v3.10, where the 'etype' parameter has been renamed to 'exc' and is now positional-only.
So I rewrite the file "/Users/.../python3.10/site-packages/hydra/_internal/utils.py", line 267 from
print_exception(etype=None, value=ex, tb=final_tb)
to
print_exception(None, value=ex, tb=final_tb)
And things go well. The package's change log can be found in website).
