Dávid Majerčák

According to what I have seen in the dataset, these labels are to large extent wrong and it would be really awesome if you would consider re-assigning them. Examples of...

There are many more similar to these, I could not find all of the previously mentioned ones but these are the ones I have found: - 089587.jpg  - 080658.jpg...

@geyuying This means though that the plot in the paper cannot be reproduced using the data. What was the reason for abandoning the 4 class split?

@tjruwase surely, this is log for 0-th process [std_log_process_0.txt](https://github.com/microsoft/DeepSpeed/files/10876054/std_log_process_0.txt)

@tjruwase unfortunately yes. After I did checkpointing for the forward pass I still get OOM error for backward pass. Let me attach the logs: [std_log_process_0 (2).txt](https://github.com/microsoft/DeepSpeed/files/10961782/std_log_process_0.2.txt)

@tjruwase if I use `fp16` I can use 96x96x96, however I get NaN for loss. If I use `bfloat16` I get loss values and can use 64x64x64 tensor as input...

@tjruwase sorry for late response: [std_log_process_0 (4).txt](https://github.com/microsoft/DeepSpeed/files/11116377/std_log_process_0.4.txt)