mmpose icon indicating copy to clipboard operation
mmpose copied to clipboard

Final saved model

Open YuktiADY opened this issue 2 years ago • 14 comments

Hallo,

I have a general questions i have trained several models, i would like to know for each of the trained model where are final saved weights stored ? where I can see them

because in the work directory folder created during each of training of models there are only logs generated.

Please help.

YuktiADY avatar Aug 04 '22 20:08 YuktiADY

Usually, the checkpoints are also stored in the work directory folder. Could you please share the config with this issue?

ly015 avatar Aug 04 '22 22:08 ly015

Yes, these .pth files are stored checkpoints. Usually best and final checkpoints should have the same size. If their sizes are different, you may need to check if they are stored in the same experiment, or different experiments using the same work directory.

ly015 avatar Aug 05 '22 02:08 ly015

Yes, these .pth files are stored checkpoints. Usually best and final checkpoints should have the same size. If their sizes are different, you may need to check if they are stored in the same experiment, or different experiments using the same work directory.

latest.pth is the final checkpoint ?? I have checked for the experiments done the size of latest.pth = 12 bytes in most of the cases. but size of best epoch = is not 12 bytes .Some models have 400 bytes, some 300 bytes . Is it problem if they have different sizes ??

YuktiADY avatar Aug 05 '22 02:08 YuktiADY

The model size doesn't seem right. It's too small that I'm afraid the checkpoint has not been stored correctly. Is there any error information in the log? And could you please check the content of the .pth file (can be loaded by torch.load())?

ly015 avatar Aug 05 '22 03:08 ly015

There is no error in the log file. In the latest.pth it contains the skeleton info, weight,biases values, sigmas and train annotations paths.but still the size is 64 KB

YuktiADY avatar Aug 05 '22 04:08 YuktiADY

latest.pth is a symlink, isn't it?

jin-s13 avatar Aug 05 '22 05:08 jin-s13

Yes, it should be a symlink. My bad.

ly015 avatar Aug 05 '22 06:08 ly015

latest.pth is a symlink, isn't it?

Then which are final saved weights for the trained models ?

YuktiADY avatar Aug 05 '22 09:08 YuktiADY

latest.pth is a symlink itself. It points to the real checkpoint file. Please use command ll to see the target.

jin-s13 avatar Aug 05 '22 09:08 jin-s13

I meant to ask that for each of the trained models where i can get the final saved weights? is it only best AP.pth in the working directory?

Awaiting for your reply.

YuktiADY avatar Aug 05 '22 09:08 YuktiADY

Inside the work_dir, there should be .log, .log.json, epoch_.pth, best_AP_epoch_.pth, and latest.pth.

latest.pth is a symlink, pointing to a specific epoch_xx.pth (which is the real checkpoint file). best_AP_epoch_xx.pth is a copy of the best checkpoint file.

jin-s13 avatar Aug 05 '22 09:08 jin-s13

Inside the work_dir, there should be .log, .log.json, epoch.pth, best_AP_epoch__.pth, and latest.pth.

latest.pth is a symlink, pointing to a specific epoch_xx.pth (which is the real checkpoint file). best_AP_epoch_xx.pth is a copy of the best checkpoint file.

I want to see for each of the trained models for the experiments done where i can get the final saved weights? Where i can get these ?

YuktiADY avatar Aug 05 '22 09:08 YuktiADY

Inside the work_dir, there should be .log, .log.json, epoch.pth, best_AP_epoch__.pth, and latest.pth. latest.pth is a symlink, pointing to a specific epoch_xx.pth (which is the real checkpoint file). best_AP_epoch_xx.pth is a copy of the best checkpoint file.

I want to see for each of the trained models for the experiments done where i can get the final saved weights? Where i can get these ?

I have done several training experiments . So for each of the trained models I want to know that what are the final saved weights ?? Thats my question.

YuktiADY avatar Aug 05 '22 09:08 YuktiADY

I have a general questions i have trained several models, i would like to know for each of the trained model where are final saved weights stored ? where I can see them

FYR I have a general questions i have trained several models, i would like to know for each of the trained model where are final saved weights stored ? where I can see them

Awaiting for your reply.

YuktiADY avatar Aug 05 '22 09:08 YuktiADY