[WIP] Implement SSL-EY

[jameschapman19]

[guarin]

Oh wow, thanks for the PR! This looks interesting, will have a look ASAP.

[guarin]

Just to make sure, is this already ready for review?

[jameschapman19]

Hi,

Still WIP but close and will let you know.

Need to run the test suite locally. Will also run all the benchmarks I am able to (I have written the scripts in the right format though). I put a PR in as I thought it might run the CI and help me fix things up before review.

In many ways the pitch of the original work is VICReg without any hyperparameters in the objective so it plugs in quite easily.

James

[guarin]

Awesome, let use know once it is ready :)

[guarin]

Btw. we can also run some benchmarks on our side if that helps :)

[codecov[bot]]

Codecov Report

Attention: 5 lines in your changes are missing coverage. Please review.

Comparison is base (534cdf4) 85.53% compared to head (f88efd0) 85.58%.

Files	Patch %	Lines
lightly/loss/ssley_loss.py	87.09%	4 Missing :warning:
lightly/utils/dist.py	93.33%	1 Missing :warning:

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1443      +/-   ##
==========================================
+ Coverage   85.53%   85.58%   +0.05%     
==========================================
  Files         135      137       +2     
  Lines        5655     5703      +48     
==========================================
+ Hits         4837     4881      +44     
- Misses        818      822       +4     

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[guarin]

The failing codecov upload is unrelated to the PR :) Could you run make format to apply the autoformatting? See https://github.com/lightly-ai/lightly?tab=readme-ov-file#development for more info.

[guarin]

@jameschapman19 I fixed merged conflicts and some typing issues. I also added a SSLEYProjectionHead changed the examples to use the SSLEYTransform. I'll now try training the model on imagenet and then try to get this PR merged :)