open-mmlab
Error while running SegFormer (segformer_mit-b3_8xb1-160k_gta5-1024x1024.py) on synthetic(GTA5) dataset for semantic segmentation
Hi, I was trying to run the SegFormer with MiTB3 backbone with all the default parameters and commands mentioned in the docs on GTA5 dataset for semantic segmentation. But I am getting some errors. I will be highly obliged if anyone can resolve this issue. Before this experiment, I have successfully reproduced results on cityscapes and bdd100k dataset with segformer.
I am putting the related error w.r.t GTA5 below:
(openmmlab) sayandebroy.csmi@gnode019:~/sayandebroy1/mmsegmentation$ bash tools/dist_train.sh configs/segformer/segformer_mit-b3_8xb1-160k_gta5-1024x1024.py 4 --work-dir /ssd_scratch/cvit/sayandebroy/segformer_m
itb3_gta5
/home2/sayandebroy.csmi/miniconda3/envs/openmmlab/lib/python3.8/site-packages/torch/distributed/launch.py:181: FutureWarning: The module torch.distributed.launch is deprecated
and will be removed in future. Use torchrun.
Note that --use-env is set by default in torchrun.
If your script expects --local-rank argument to be set, please
change it to read from os.environ['LOCAL_RANK'] instead. See
https://pytorch.org/docs/stable/distributed.html#launch-utility for
further instructions
warnings.warn( WARNING:torch.distributed.run:
Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
Traceback (most recent call last):
File "tools/train.py", line 104, in
File "tools/train.py", line 104, in
File "tools/train.py", line 58, in main cfg = Config.fromfile(args.config) File "/home2/sayandebroy.csmi/miniconda3/envs/openmmlab/lib/python3.8/site-packages/mmengine/config/config.py", line 457, in fromfile main() File "tools/train.py", line 58, in main cfg = Config.fromfile(args.config) File "/home2/sayandebroy.csmi/miniconda3/envs/openmmlab/lib/python3.8/site-packages/mmengine/config/config.py", line 457, in fromfile cfg_dict, cfg_text, env_variables = Config._file2dict(cfg_dict, cfg_text, env_variables = Config._file2dict(
File "/home2/sayandebroy.csmi/miniconda3/envs/openmmlab/lib/python3.8/site-packages/mmengine/config/config.py", line 941, in _file2dict
cfg_dict, cfg_text, env_variables = Config._file2dict(cfg_dict, cfg_text, env_variables = Config._file2dict( File "/home2/sayandebroy.csmi/miniconda3/envs/openmmlab/lib/python3.8/site-packages/mmengine/config/c onfig.py", line 941, in _file2dict
File "/home2/sayandebroy.csmi/miniconda3/envs/openmmlab/lib/python3.8/site-packages/mmengine/config/config.py", line 941, in _file2dict File "/home2/sayandebroy.csmi/miniconda3/envs/openmmlab/lib/python3.8/site-packages/mmengine/config/config.py", line 941, in _file2dict raise e File "/home2/sayandebroy.csmi/miniconda3/envs/openmmlab/lib/python3.8/site-packages/mmengine/config/config.py", line 883, in _file2dict raise e File "/home2/sayandebroy.csmi/miniconda3/envs/openmmlab/lib/python3.8/site-packages/mmengine/config/config.py", line 883, in _file2dict raise e raise e File "/home2/sayandebroy.csmi/miniconda3/envs/openmmlab/lib/python3.8/site-packages/mmengine/config/config.py", line 883, in _file2dict File "/home2/sayandebroy.csmi/miniconda3/envs/openmmlab/lib/python3.8/site-packages/mmengine/config/config.py", line 883, in _file2dict _cfg_dict, _cfg_text, _env_variables = Config._file2dict( _cfg_dict, _cfg_text, _env_variables = Config._file2dict( File "/home2/sayandebroy.csmi/miniconda3/envs/openmmlab/lib/python3.8/site-packages/mmengine/config/config.py", line 941, in _file2dict
File "/home2/sayandebroy.csmi/miniconda3/envs/openmmlab/lib/python3.8/site-packages/mmengine/config/config.py", line 941, in _file2dict _cfg_dict, _cfg_text, _env_variables = Config._file2dict( _cfg_dict, _cfg_text, _env_variables = Config._file2dict( File "/home2/sayandebroy.csmi/miniconda3/envs/openmmlab/lib/python3.8/site-packages/mmengine/config/config.py", line 941, in _file2dict
File "/home2/sayandebroy.csmi/miniconda3/envs/openmmlab/lib/python3.8/site-packages/mmengine/config/config.py", line 941, in _file2dict
raise e
File "/home2/sayandebroy.csmi/miniconda3/envs/openmmlab/lib/python3.8/site-packages/mmengine/config/config.py", line 883, in _file2dict
raise e
File "/home2/sayandebroy.csmi/miniconda3/envs/openmmlab/lib/python3.8/site-packages/mmengine/config/config.py", line 883, in _file2dict
raise e
File "/home2/sayandebroy.csmi/miniconda3/envs/openmmlab/lib/python3.8/site-packages/mmengine/config/config.py", line 883, in _file2dict
raise e
File "/home2/sayandebroy.csmi/miniconda3/envs/openmmlab/lib/python3.8/site-packages/mmengine/config/config.py", line 883, in _file2dict
_cfg_dict, _cfg_text, _env_variables = Config._file2dict(
File "/home2/sayandebroy.csmi/miniconda3/envs/openmmlab/lib/python3.8/site-packages/mmengine/config/config.py", line 839, in _file2dict
_cfg_dict, _cfg_text, _env_variables = Config._file2dict(
File "/home2/sayandebroy.csmi/miniconda3/envs/openmmlab/lib/python3.8/site-packages/mmengine/config/config.py", line 839, in _file2dict
_cfg_dict, _cfg_text, _env_variables = Config._file2dict(
File "/home2/sayandebroy.csmi/miniconda3/envs/openmmlab/lib/python3.8/site-packages/mmengine/config/config.py", line 839, in _file2dict
_cfg_dict, _cfg_text, _env_variables = Config._file2dict(
File "/home2/sayandebroy.csmi/miniconda3/envs/openmmlab/lib/python3.8/site-packages/mmengine/config/config.py", line 839, in _file2dict
if lazy_import is None and Config._is_lazy_import(filename):
File "/home2/sayandebroy.csmi/miniconda3/envs/openmmlab/lib/python3.8/site-packages/mmengine/config/config.py", line 1657, in _is_lazy_import
if lazy_import is None and Config._is_lazy_import(filename):
File "/home2/sayandebroy.csmi/miniconda3/envs/openmmlab/lib/python3.8/site-packages/mmengine/config/config.py", line 1657, in _is_lazy_import
if lazy_import is None and Config._is_lazy_import(filename):
File "/home2/sayandebroy.csmi/miniconda3/envs/openmmlab/lib/python3.8/site-packages/mmengine/config/config.py", line 1657, in _is_lazy_import
if lazy_import is None and Config._is_lazy_import(filename):
File "/home2/sayandebroy.csmi/miniconda3/envs/openmmlab/lib/python3.8/site-packages/mmengine/config/config.py", line 1657, in _is_lazy_import
parsed_codes = ast.parse(codes_str)
File "/home2/sayandebroy.csmi/miniconda3/envs/openmmlab/lib/python3.8/ast.py", line 47, in parse
parsed_codes = ast.parse(codes_str)
File "/home2/sayandebroy.csmi/miniconda3/envs/openmmlab/lib/python3.8/ast.py", line 47, in parse
parsed_codes = ast.parse(codes_str)
File "/home2/sayandebroy.csmi/miniconda3/envs/openmmlab/lib/python3.8/ast.py", line 47, in parse
parsed_codes = ast.parse(codes_str)
File "/home2/sayandebroy.csmi/miniconda3/envs/openmmlab/lib/python3.8/ast.py", line 47, in parse
return compile(source, filename, mode, flags,
File "
return compile(source, filename, mode, flags, File "<unknown>", line 21
patch_sizes=[7, 3, 3, 3],
^
File "", line 21
SyntaxError : patch_sizes=[7, 3, 3, 3],
invalid syntax
^
SyntaxError: invalid syntax
return compile(source, filename, mode, flags,
File "", line 21
patch_sizes=[7, 3, 3, 3],
^
SyntaxError: invalid syntax
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 15728) of binary: /home2/sayandebroy.csmi/miniconda3/envs/openmmlab/bin/python
Traceback (most recent call last):
File "/home2/sayandebroy.csmi/miniconda3/envs/openmmlab/lib/python3.8/runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/home2/sayandebroy.csmi/miniconda3/envs/openmmlab/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/home2/sayandebroy.csmi/miniconda3/envs/openmmlab/lib/python3.8/site-packages/torch/distributed/launch.py", line 196, in
main()
File "/home2/sayandebroy.csmi/miniconda3/envs/openmmlab/lib/python3.8/site-packages/torch/distributed/launch.py", line 192, in main
launch(args)
File "/home2/sayandebroy.csmi/miniconda3/envs/openmmlab/lib/python3.8/site-packages/torch/distributed/launch.py", line 177, in launch
run(args)
File "/home2/sayandebroy.csmi/miniconda3/envs/openmmlab/lib/python3.8/site-packages/torch/distributed/run.py", line 785, in run
elastic_launch(
File "/home2/sayandebroy.csmi/miniconda3/envs/openmmlab/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 134, in call
return launch_agent(self._config, self._entrypoint, list(args))
File "/home2/sayandebroy.csmi/miniconda3/envs/openmmlab/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 250, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
File "/home2/sayandebroy.csmi/miniconda3/envs/openmmlab/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 250, in launch_agent raise ChildFailedError( torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
tools/train.py FAILED
Failures: [1]: time : 2023-11-16_20:06:12 host : gnode019.local rank : 1 (local_rank: 1) exitcode : 1 (pid: 15729) error_file: <N/A> traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html [2]: time : 2023-11-16_20:06:12 host : gnode019.local rank : 2 (local_rank: 2) exitcode : 1 (pid: 15730) error_file: <N/A> traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html [3]: time : 2023-11-16_20:06:12 host : gnode019.local rank : 3 (local_rank: 3) exitcode : 1 (pid: 15731) error_file: <N/A> traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
Root Cause (first observed failure): [0]: time : 2023-11-16_20:06:12 host : gnode019.local rank : 0 (local_rank: 0) exitcode : 1 (pid: 15728) error_file: <N/A> traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
Hi, could you please release the SegFormer model pretrained on GTA5 datasets?
