running lighting-pose on CalMS21 dataset

[egg-benedict]

hello.

I'm trying to run lightning-pose for CalMS21 and test how it perfroms. I configured a new config .yaml file for this specific dataset, but the predictions on the video isn't really working. The new config file for the calm21 is mostly the same with crim13 config file, but added a video to leverage the unsupervised losses (pca_singleview & temporal), and also tested up to 1000 training epochs. The predicted keypoints stay at the top left corner of the video. Since, there was a crim13 config file as default, I wonder how the model performed on crim13 dataset, since these two datasets share a lot of features.

Below is the frame result of the predicted video, hoping it will help. Thank you in advance.

Best regards

[themattinthehatt]

Hi @egg-benedict, we should definitely be able to get this working on the CalMS dataset. Would you mind copy/pasting your config file here?

[themattinthehatt]

Also, it would be useful to use tensorboard to look at the training performance of your models. From the command line you just need to run (inside the LP conda env):

tensorboard --logdir=/path/to/your/models

Does the train/val RMSE go down to a reasonable number?

If not then there's some problem with training. If so, then training is fine but inference is the issue.

[themattinthehatt]

@egg-benedict just checking in on this again, would love to help you get up and running with this dataset