OFA
OFA copied to clipboard
Published
20 hours ago •
OFA-Sys
OFA-Sys
run_scripts /caption /evaluate_caption.sh ?
First of all, Thanks for your amazing work.👍
I'm very surprised at the results you've made.
I wonder how to get the last line( ../.. /results/caption/test_predict.json) of the file(run_scripts /caption /evaluate_caption.sh)
I want know how to get ../.. /results/caption/test_predict.json
I can not generate this json file,
I'll be waiting for the reply.
Thank you! @logicwong
@cxy990729 Did you run evaluate_caption.sh successfully? If you run successfully, the JSON file should be generated in ${result_path} (default to ../../results/caption/ relative to the path of the shell script).
Before running evaluate_caption.sh, you should download the data (see datasets.md) and models (see checkpoints.md) and put them in the correct directory. These files should be organized like this:
OFA/
checkpoints/
caption_large_best_clean.pt
criterions/
data/
dataset/
caption_data/
cider_cached_tokens/
coco-train-words.p
coco-valid-words.p
caption_stage1_train.tsv
caption_stage2_train.tsv
caption_test.tsv
caption_val.tsv
test_caption_coco_format.json
examples/
fairseq/
...
@logicwong I haven't run evaluate_caption.sh successfully yet .I encountered the following error when I ran it，when i creat ../../result/caption, the caption file is empty.
Traceback (most recent call last):
File "coco_eval.py", line 38, in
when i creat ../../result/caption/test_predicted.json, I create an empty JSON file in the Caption folder and get the following error.
Traceback (most recent call last):
File "coco_eval.py", line 38, in
@cxy990729 Can you provide the complete log? A normal log should like this:
2022-03-19 21:20:01 | INFO | ofa.evaluate | loading model(s) from ../../checkpoints/caption_large_best_clean.pt
2022-03-19 21:20:01 | INFO | ofa.evaluate | loading model(s) from ../../checkpoints/caption_large_best_clean.pt
2022-03-19 21:20:01 | INFO | ofa.evaluate | loading model(s) from ../../checkpoints/caption_large_best_clean.pt
2022-03-19 21:20:04 | INFO | tasks.ofa_task | source dictionary: 59457 types
2022-03-19 21:20:04 | INFO | tasks.ofa_task | target dictionary: 59457 types
2022-03-19 21:20:04 | INFO | tasks.ofa_task | source dictionary: 59457 types
2022-03-19 21:20:04 | INFO | tasks.ofa_task | target dictionary: 59457 types
2022-03-19 21:20:04 | INFO | tasks.ofa_task | source dictionary: 59457 types
2022-03-19 21:20:04 | INFO | tasks.ofa_task | target dictionary: 59457 types
2022-03-19 21:20:04 | INFO | tasks.ofa_task | source dictionary: 59457 types
2022-03-19 21:20:04 | INFO | tasks.ofa_task | target dictionary: 59457 types
local datafile ../../dataset/caption_data/caption_test.tsv slice_id 2 begin to initialize row_count and line_idx-to-offset mapping
local datafile ../../dataset/caption_data/caption_test.tsv slice_id 0 begin to initialize row_count and line_idx-to-offset mapping
local datafile ../../dataset/caption_data/caption_test.tsv slice_id 1 begin to initialize row_count and line_idx-to-offset mapping
local datafile ../../dataset/caption_data/caption_test.tsv slice_id 3 begin to initialize row_count and line_idx-to-offset mapping
local datafile ../../dataset/caption_data/caption_test.tsv slice_id 2 finished initializing row_count and line_idx-to-offset mapping
file ../../dataset/caption_data/caption_test.tsv slice_id 2 row count 1250 total row count 5000
/root/anaconda3/envs/ofa/lib/python3.7/site-packages/torchvision/transforms/transforms.py:258: UserWarning: Argument interpolation should be of type InterpolationMode instead of int. Please, use InterpolationMode enum.
"Argument interpolation should be of type InterpolationMode instead of int. "
local datafile ../../dataset/caption_data/caption_test.tsv slice_id 0 finished initializing row_count and line_idx-to-offset mapping
file ../../dataset/caption_data/caption_test.tsv slice_id 0 row count 1250 total row count 5000
/root/anaconda3/envs/ofa/lib/python3.7/site-packages/torchvision/transforms/transforms.py:258: UserWarning: Argument interpolation should be of type InterpolationMode instead of int. Please, use InterpolationMode enum.
"Argument interpolation should be of type InterpolationMode instead of int. "
local datafile ../../dataset/caption_data/caption_test.tsv slice_id 1 finished initializing row_count and line_idx-to-offset mapping
file ../../dataset/caption_data/caption_test.tsv slice_id 1 row count 1250 total row count 5000
/root/anaconda3/envs/ofa/lib/python3.7/site-packages/torchvision/transforms/transforms.py:258: UserWarning: Argument interpolation should be of type InterpolationMode instead of int. Please, use InterpolationMode enum.
"Argument interpolation should be of type InterpolationMode instead of int. "
local datafile ../../dataset/caption_data/caption_test.tsv slice_id 3 finished initializing row_count and line_idx-to-offset mapping
file ../../dataset/caption_data/caption_test.tsv slice_id 3 row count 1250 total row count 5000
/root/anaconda3/envs/ofa/lib/python3.7/site-packages/torchvision/transforms/transforms.py:258: UserWarning: Argument interpolation should be of type InterpolationMode instead of int. Please, use InterpolationMode enum.
"Argument interpolation should be of type InterpolationMode instead of int. "
2022-03-19 21:20:34 | INFO | fairseq.logging.progress_bar | : 11 / 79 sentences=16
2022-03-19 21:20:34 | INFO | fairseq.logging.progress_bar | : 11 / 79 sentences=16
2022-03-19 21:20:34 | INFO | fairseq.logging.progress_bar | : 11 / 79 sentences=16
2022-03-19 21:20:34 | INFO | fairseq.logging.progress_bar | : 11 / 79 sentences=16
2022-03-19 21:20:48 | INFO | fairseq.logging.progress_bar | : 21 / 79 sentences=16
2022-03-19 21:20:48 | INFO | fairseq.logging.progress_bar | : 21 / 79 sentences=16
2022-03-19 21:20:48 | INFO | fairseq.logging.progress_bar | : 21 / 79 sentences=16
2022-03-19 21:20:48 | INFO | fairseq.logging.progress_bar | : 21 / 79 sentences=16
2022-03-19 21:21:01 | INFO | fairseq.logging.progress_bar | : 31 / 79 sentences=16
2022-03-19 21:21:01 | INFO | fairseq.logging.progress_bar | : 31 / 79 sentences=16
2022-03-19 21:21:01 | INFO | fairseq.logging.progress_bar | : 31 / 79 sentences=16
2022-03-19 21:21:01 | INFO | fairseq.logging.progress_bar | : 31 / 79 sentences=16
2022-03-19 21:21:15 | INFO | fairseq.logging.progress_bar | : 41 / 79 sentences=16
2022-03-19 21:21:15 | INFO | fairseq.logging.progress_bar | : 41 / 79 sentences=16
2022-03-19 21:21:15 | INFO | fairseq.logging.progress_bar | : 41 / 79 sentences=16
2022-03-19 21:21:15 | INFO | fairseq.logging.progress_bar | : 41 / 79 sentences=16
2022-03-19 21:21:28 | INFO | fairseq.logging.progress_bar | : 51 / 79 sentences=16
2022-03-19 21:21:28 | INFO | fairseq.logging.progress_bar | : 51 / 79 sentences=16
2022-03-19 21:21:29 | INFO | fairseq.logging.progress_bar | : 51 / 79 sentences=16
2022-03-19 21:21:29 | INFO | fairseq.logging.progress_bar | : 51 / 79 sentences=16
2022-03-19 21:21:42 | INFO | fairseq.logging.progress_bar | : 61 / 79 sentences=16
2022-03-19 21:21:42 | INFO | fairseq.logging.progress_bar | : 61 / 79 sentences=16
2022-03-19 21:21:42 | INFO | fairseq.logging.progress_bar | : 61 / 79 sentences=16
2022-03-19 21:21:43 | INFO | fairseq.logging.progress_bar | : 61 / 79 sentences=16
2022-03-19 21:21:56 | INFO | fairseq.logging.progress_bar | : 71 / 79 sentences=16
2022-03-19 21:21:56 | INFO | fairseq.logging.progress_bar | : 71 / 79 sentences=16
2022-03-19 21:21:56 | INFO | fairseq.logging.progress_bar | : 71 / 79 sentences=16
2022-03-19 21:21:56 | INFO | fairseq.logging.progress_bar | : 71 / 79 sentences=16
2022-03-19 21:22:06 | INFO | root | Added key: store_based_barrier_key:2 to store for rank: 3
2022-03-19 21:22:06 | INFO | root | Added key: store_based_barrier_key:2 to store for rank: 1
2022-03-19 21:22:06 | INFO | root | Added key: store_based_barrier_key:2 to store for rank: 2
2022-03-19 21:22:06 | INFO | root | Added key: store_based_barrier_key:2 to store for rank: 0
loading annotations into memory...
Done (t=0.03s)
creating index...
index created!
Loading and preparing results...
DONE (t=0.01s)
creating index...
index created!
tokenization...
PTBTokenizer tokenized 307086 tokens at 1549244.40 tokens per second.
PTBTokenizer tokenized 52998 tokens at 495845.46 tokens per second.
setting up scorers...
computing Bleu score...
{'testlen': 47999, 'reflen': 47660, 'guess': [47999, 42999, 37999, 32999], 'correct': [40764, 25119, 13533, 6840]}
ratio: 1.0071128829206673
Bleu_1: 0.849
Bleu_2: 0.704
Bleu_3: 0.561
Bleu_4: 0.437
computing METEOR score...
METEOR: 0.317
computing Rouge score...
ROUGE_L: 0.624
computing CIDEr score...
CIDEr: 1.496
computing SPICE score...
Parsing reference captions
Parsing test captions
Initiating Stanford parsing pipeline
[main] INFO edu.stanford.nlp.pipeline.StanfordCoreNLP - Adding annotator tokenize
[main] INFO edu.stanford.nlp.pipeline.TokenizerAnnotator - TokenizerAnnotator: No tokenizer type provided. Defaulting to PTBTokenizer.
[main] INFO edu.stanford.nlp.pipeline.StanfordCoreNLP - Adding annotator ssplit
[main] INFO edu.stanford.nlp.pipeline.StanfordCoreNLP - Adding annotator parse
[main] INFO edu.stanford.nlp.parser.common.ParserGrammar - Loading parser from serialized file edu/stanford/nlp/models/lexparser/englishPCFG.ser.gz ...
done [0.3 sec].
[main] INFO edu.stanford.nlp.pipeline.StanfordCoreNLP - Adding annotator lemma
[main] INFO edu.stanford.nlp.pipeline.StanfordCoreNLP - Adding annotator ner
Loading classifier from edu/stanford/nlp/models/ner/english.all.3class.distsim.crf.ser.gz ... done [1.3 sec].
Loading classifier from edu/stanford/nlp/models/ner/english.muc.7class.distsim.crf.ser.gz ... done [0.5 sec].
Loading classifier from edu/stanford/nlp/models/ner/english.conll.4class.distsim.crf.ser.gz ... done [0.5 sec].
Threads( StanfordCoreNLP ) [0.756 seconds]
SPICE evaluation took: 10.57 s
SPICE: 0.258
{'Bleu_1': 0.8492676930769215, 'Bleu_2': 0.7043593708902992, 'Bleu_3': 0.5611386323297256, 'Bleu_4': 0.437463131655471, 'METEOR': 0.3173761444795704, 'ROUGE_L': 0.6235654088134712, 'CIDEr': 1.496288395188934, 'SPICE': 0.2584915992123773}
Hi! I'm having the same issue. This is the log I receive after running cd run_scripts/caption ; sh evaluate_caption.sh I also want to let you know that I'm running this on a custom dataset (already converted image to base64 string and saved it as a tsv file), but the data path is the same (../../dataset/caption_data).
2022-08-22 22:25:20 | INFO | torch.distributed.nn.jit.instantiator | Writing /var/folders/wz/0gn4fmmd69lf50pfbjn9yrv00000gn/T/tmp684j5wqu/_remote_module_non_scriptable.py
2022-08-22 22:25:23 | INFO | ofa.evaluate | {'_name': None, 'common': {'_name': None, 'no_progress_bar': False, 'log_interval': 10, 'log_format': 'simple', 'log_file': None, 'aim_repo': None, 'aim_run_hash': None, 'tensorboard_logdir': None, 'wandb_project': None, 'azureml_logging': False, 'seed': 7, 'cpu': False, 'tpu': False, 'bf16': False, 'memory_efficient_bf16': False, 'fp16': True, 'memory_efficient_fp16': False, 'fp16_no_flatten_grads': False, 'fp16_init_scale': 128, 'fp16_scale_window': None, 'fp16_scale_tolerance': 0.0, 'on_cpu_convert_precision': False, 'min_loss_scale': 0.0001, 'threshold_loss_scale': None, 'amp': False, 'amp_batch_retries': 2, 'amp_init_scale': 128, 'amp_scale_window': None, 'user_dir': '../../ofa_module', 'empty_cache_freq': 0, 'all_gather_list_size': 16384, 'model_parallel_size': 1, 'quantization_config_path': None, 'profile': False, 'reset_logging': False, 'suppress_crashes': False, 'use_plasma_view': False, 'plasma_path': '/tmp/plasma'}, 'common_eval': {'_name': None, 'path': '../../checkpoints/caption_base_best.pt', 'post_process': None, 'quiet': False, 'model_overrides': '{"data":"../../dataset/caption_data/clip1.tsv","bpe_dir":"../../utils/BPE","eval_cider":False,"selected_cols":"1,4,2"}', 'results_path': '../../results/caption'}, 'distributed_training': {'_name': None, 'distributed_world_size': 1, 'distributed_num_procs': 0, 'distributed_rank': 0, 'distributed_backend': 'nccl', 'distributed_init_method': None, 'distributed_port': -1, 'device_id': 0, 'distributed_no_spawn': False, 'ddp_backend': 'pytorch_ddp', 'ddp_comm_hook': 'none', 'bucket_cap_mb': 25, 'fix_batches_to_gpus': False, 'find_unused_parameters': False, 'gradient_as_bucket_view': False, 'fast_stat_sync': False, 'heartbeat_timeout': -1, 'broadcast_buffers': False, 'slowmo_momentum': None, 'slowmo_base_algorithm': 'localsgd', 'localsgd_frequency': 3, 'nprocs_per_node': 1, 'pipeline_model_parallel': False, 'pipeline_balance': None, 'pipeline_devices': None, 'pipeline_chunks': 0, 'pipeline_encoder_balance': None, 'pipeline_encoder_devices': None, 'pipeline_decoder_balance': None, 'pipeline_decoder_devices': None, 'pipeline_checkpoint': 'never', 'zero_sharding': 'none', 'fp16': True, 'memory_efficient_fp16': False, 'tpu': False, 'no_reshard_after_forward': False, 'fp32_reduce_scatter': False, 'cpu_offload': False, 'use_sharded_state': False, 'not_fsdp_flatten_parameters': False}, 'dataset': {'_name': None, 'num_workers': 0, 'skip_invalid_size_inputs_valid_test': False, 'max_tokens': None, 'batch_size': 16, 'required_batch_size_multiple': 8, 'required_seq_len_multiple': 1, 'dataset_impl': None, 'data_buffer_size': 10, 'train_subset': 'train', 'valid_subset': 'valid', 'combine_valid_subsets': None, 'ignore_unused_valid_subsets': False, 'validate_interval': 1, 'validate_interval_updates': 0, 'validate_after_updates': 0, 'fixed_validation_seed': None, 'disable_validation': False, 'max_tokens_valid': None, 'batch_size_valid': 16, 'max_valid_steps': None, 'curriculum': 0, 'gen_subset': 'test', 'num_shards': 1, 'shard_id': 0, 'grouped_shuffling': False, 'update_epoch_batch_itr': False, 'update_ordered_indices_seed': False}, 'optimization': {'_name': None, 'max_epoch': 0, 'max_update': 0, 'stop_time_hours': 0.0, 'clip_norm': 0.0, 'sentence_avg': False, 'update_freq': [1], 'lr': [0.25], 'stop_min_lr': -1.0, 'use_bmuf': False, 'skip_remainder_batch': False}, 'checkpoint': {'_name': None, 'save_dir': 'checkpoints', 'restore_file': 'checkpoint_last.pt', 'continue_once': None, 'finetune_from_model': None, 'reset_dataloader': False, 'reset_lr_scheduler': False, 'reset_meters': False, 'reset_optimizer': False, 'optimizer_overrides': '{}', 'save_interval': 1, 'save_interval_updates': 0, 'keep_interval_updates': -1, 'keep_interval_updates_pattern': -1, 'keep_last_epochs': -1, 'keep_best_checkpoints': -1, 'no_save': False, 'no_epoch_checkpoints': False, 'no_last_checkpoints': False, 'no_save_optimizer_state': False, 'best_checkpoint_metric': 'loss', 'maximize_best_checkpoint_metric': False, 'patience': -1, 'checkpoint_suffix': '', 'checkpoint_shard_count': 1, 'load_checkpoint_on_all_dp_ranks': False, 'write_checkpoints_asynchronously': False, 'model_parallel_size': 1}, 'bmuf': {'_name': None, 'block_lr': 1.0, 'block_momentum': 0.875, 'global_sync_iter': 50, 'warmup_iterations': 500, 'use_nbm': False, 'average_sync': False, 'distributed_world_size': 1}, 'generation': {'_name': None, 'beam': 5, 'nbest': 1, 'max_len_a': 0.0, 'max_len_b': 16, 'min_len': 1, 'match_source_len': False, 'unnormalized': False, 'no_early_stop': False, 'no_beamable_mm': False, 'lenpen': 1.0, 'unkpen': 0.0, 'replace_unk': None, 'sacrebleu': False, 'score_reference': False, 'prefix_size': 0, 'no_repeat_ngram_size': 3, 'sampling': False, 'sampling_topk': -1, 'sampling_topp': -1.0, 'constraints': None, 'temperature': 1.0, 'diverse_beam_groups': -1, 'diverse_beam_strength': 0.5, 'diversity_rate': -1.0, 'print_alignment': None, 'print_step': False, 'lm_path': None, 'lm_weight': 0.0, 'iter_decode_eos_penalty': 0.0, 'iter_decode_max_iter': 10, 'iter_decode_force_max_iter': False, 'iter_decode_with_beam': 1, 'iter_decode_with_external_reranker': False, 'retain_iter_history': False, 'retain_dropout': False, 'retain_dropout_modules': None, 'decoding_format': None, 'no_seed_provided': False, 'eos_token': None}, 'eval_lm': {'_name': None, 'output_word_probs': False, 'output_word_stats': False, 'context_window': 0, 'softmax_batch': 9223372036854775807}, 'interactive': {'_name': None, 'buffer_size': 0, 'input': '-'}, 'model': None, 'task': {'_name': 'caption', 'data': '../../dataset/caption_data/clip1.tsv', 'selected_cols': None, 'bpe_dir': None, 'max_source_positions': 1024, 'max_target_positions': 1024, 'max_src_length': 128, 'max_tgt_length': 30, 'code_dict_size': 8192, 'patch_image_size': 480, 'num_bins': 1000, 'imagenet_default_mean_and_std': False, 'constraint_range': None, 'eval_bleu': False, 'eval_cider': False, 'eval_args': '{}', 'eval_print_samples': False, 'eval_cider_cached_tokens': None, 'scst': False, 'scst_args': '{}'}, 'criterion': {'_name': 'cross_entropy', 'sentence_avg': True}, 'optimizer': None, 'lr_scheduler': {'_name': 'fixed', 'force_anneal': None, 'lr_shrink': 0.1, 'warmup_updates': 0, 'lr': [0.25]}, 'scoring': {'_name': 'bleu', 'pad': 1, 'eos': 2, 'unk': 3}, 'bpe': None, 'tokenizer': None, 'ema': {'_name': None, 'store_ema': False, 'ema_decay': 0.9999, 'ema_start_update': 0, 'ema_seed_model': None, 'ema_update_freq': 1, 'ema_fp32': False}}
2022-08-22 22:25:23 | INFO | ofa.evaluate | loading model(s) from ../../checkpoints/caption_base_best.pt
2022-08-22 22:25:26 | INFO | tasks.ofa_task | source dictionary: 59457 types
2022-08-22 22:25:26 | INFO | tasks.ofa_task | target dictionary: 59457 types
/Users/muna/Downloads/OFA/run_scripts/caption/ofa_cap/lib/python3.9/site-packages/torch/functional.py:478: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at /Users/runner/work/pytorch/pytorch/pytorch/aten/src/ATen/native/TensorShape.cpp:2895.)
return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]
local datafile ../../dataset/caption_data/clip1.tsv slice_id 0 begin to initialize row_count and line_idx-to-offset mapping
local datafile ../../dataset/caption_data/clip1.tsv slice_id 0 finished initializing row_count and line_idx-to-offset mapping
file ../../dataset/caption_data/clip1.tsv slice_id 0 row count 1 total row count 1
/Users/muna/Downloads/OFA/run_scripts/caption/ofa_cap/lib/python3.9/site-packages/torchvision/transforms/transforms.py:332: UserWarning: Argument 'interpolation' of type int is deprecated since 0.13 and will be removed in 0.15. Please use InterpolationMode enum.
warnings.warn(
Traceback (most recent call last):
File "/Users/muna/Downloads/OFA/run_scripts/caption/../../evaluate.py", line 142, in <module>
cli_main()
File "/Users/muna/Downloads/OFA/run_scripts/caption/../../evaluate.py", line 138, in cli_main
distributed_utils.call_main(cfg, main, ema_eval=args.ema_eval, beam_search_vqa_eval=args.beam_search_vqa_eval)
File "/Users/muna/Downloads/OFA/run_scripts/caption/ofa_cap/lib/python3.9/site-packages/fairseq/distributed/utils.py", line 369, in call_main
main(cfg, **kwargs)
File "/Users/muna/Downloads/OFA/run_scripts/caption/../../evaluate.py", line 115, in main
score_sum = torch.FloatTensor([0]).cuda()
File "/Users/muna/Downloads/OFA/run_scripts/caption/ofa_cap/lib/python3.9/site-packages/torch/cuda/__init__.py", line 211, in _lazy_init
raise AssertionError("Torch not compiled with CUDA enabled")
AssertionError: Torch not compiled with CUDA enabled
loading annotations into memory...
Traceback (most recent call last):
File "/Users/muna/Downloads/OFA/run_scripts/caption/coco_eval.py", line 38, in <module>
evaluate_on_coco_caption(sys.argv[1], sys.argv[2])
File "/Users/muna/Downloads/OFA/run_scripts/caption/coco_eval.py", line 15, in evaluate_on_coco_caption
coco = COCO(label_file)
File "/Users/muna/Downloads/OFA/run_scripts/caption/ofa_cap/lib/python3.9/site-packages/pycocotools/coco.py", line 81, in __init__
with open(annotation_file, 'r') as f:
FileNotFoundError: [Errno 2] No such file or directory: '../../dataset/caption_data/test_caption_coco_format.json'
Thank you so much for the help and especially for the amazing code!
@Sultanax Torch not compiled with CUDA enabled. Did you install torch with the correct CUDA version?In addition, If you use a custom dataset. This line python coco_eval.py ../../results/caption/test_predict.json ../../dataset/caption_data/test_caption_coco_format.json is useless, you can remove it.
