Add non regression tests

[rom1504]

To make it easier to merge PRs in the future, we need automated non regression tests

• In GitHub action
• on CPU

Tests to have:

• [x] exact inference result before/after
  • [x] non crashing inference
  • [x] non reg : wip at 199
• [ ] can still load with torch.load
• [ ] training works the same before after, for one step, same weights and results
  • [x] non crashing training
  • [ ] non reg

[rom1504]

@Iopho if you want to help here, it would be appreciated

[lopho]

I made some time tomorrow and will start writing tests.

[lopho]

@rom1504 I opened a PR draft for inference testing #199

[rom1504]

non crashing training test done at #203, non reg still to do

[rom1504]

Non reg inference tests done, thanks @lopho

Next step here is non reg test for training

[rom1504]

Still very much interested by the non reg for training if you feel like continuing work here @lopho Develoment is still active and if we can make sure we don't break things going forward it will be great

[lopho]

Yes, I'm still working on it. Before training I still have to look into making testing more stable regarding randomness between runs, as I think this will be worse with training tests.

[rom1504]

Looks like inference testing is working very well with your new strategy @lopho

Do you want to look into training non reg as well ?

We have a few PRs with significant changes and it would be very helpful if we can be confident that base clip training is still doing exactly the same thing

[lopho]

Definetly, but I won't have time to until after new years.

[lopho]

I will start working on this again within the next week, unless there is already someone working on it that I am not aware of.

[rom1504]

@lopho would be great. We're still in need of more reliability for merging PRs! Currently doing manual training for validation which is painful