sagemaker-debugger icon indicating copy to clipboard operation
sagemaker-debugger copied to clipboard
verified publisher icon awslabs

Check why this log is repeated

Open Vikas-kum opened this issue 5 years ago • 1 comments

In CI : https://console.aws.amazon.com/cloudwatch/home?region=us-east-1#logEventViewer:group=DO-NOT-DELETE-smdebug_rules-LOGS-ONE-REPO;stream=codebuild/c3bda538-9277-42db-931a-de5984013923;filter=%22Loaded%20Index%20Files:%20upload/20200106_221841/c33ae10/s3_trials/trial_loss_not_decreasing_tf_true_parallel_mode_1578351365.7939517/index/000000000/000000000070_worker_0.json%22

Why is this line repeated so many times: "Loaded Index Files: upload/20200106_221841/c33ae10/s3_trials/trial_loss_not_decreasing_tf_true_parallel_mode_1578351365.7939517/index/000000000/000000000070_worker_0.json"

Are we reloading index files again and again ? @NihalHarish Please check and confirm

Vikas-kum avatar Jan 06 '20 23:01 Vikas-kum

This log is repeated because if range steps is defined, s3 list calls are still made but the contents are filtered. Can we not make list calls if range steps have been reached ?

Vikas-kum avatar Jan 10 '20 03:01 Vikas-kum