EasyParallelLibrary icon indicating copy to clipboard operation
EasyParallelLibrary copied to clipboard

Problem of Data Parallel Model, program didn't end when reached global step

Open Jimmy-jin opened this issue 3 years ago • 1 comments

I got a problem when using EPL data parallel Model. The num worker is set to 3 and each worker had its own TF data record input and Model save_dir. The global step is set to 3500 for each worker. It seems normal when global step was below 3500, but the program not end when reached 3500. Seems like the chief worker 0 didn't know other worker was end。

image

Jimmy-jin avatar Aug 04 '22 08:08 Jimmy-jin

Could you provide a example and your script?

adoda avatar Aug 23 '22 07:08 adoda