integrations: How to manage versions

[daavoo]

trafficstars

@daavoo I think that this is quite a challenge for us to decide.

1. On one hand we don't want to restrict the dependencies, because that could interfere with users workspace, and dvclive is probably not that important to force user to install particular version of the DL framework.

2. On the other hand, we need to maintain our project and the wide range of supported frameworks makes it hard to be able to know which versions of particular frameworks we support. Solution to that would be restricting the supported versions, but that contradicts with 1. We can discuss this on our next meeting.

Originally posted by @pared in https://github.com/iterative/dvclive/pull/220#issuecomment-1039077791

[daavoo]

Another option would be to try to maintain integrations compatible across different versions by adding conditionals in our code. For example (#219):

        if self.model_file:
            if catalyst.version <= 0.22.0:
                checkpoint = runner.engine.pack_checkpoint(
                     model=runner.model,
                    criterion=runner.criterion,
                    optimizer=runner.optimizer,
                    scheduler=runner.scheduler,
                )
            else:
                # code compatible with >= 0.22.0
                . . .

This could potentially become hard to maintain

[daavoo]

We should get rid of integrations and transfer to the framework itself when possible. As mentioned in the first point of #223

[daavoo]

After #220 we realized that there is no easy way to maintain different integrations - on one hand, dvclive is not that important to make users specific version of a particular framework. On the other, just having non-restricted versions of frameworks in our setup is just asking for trouble (again #220). It seems that quite a popular approach is to have the integrations not in the logging tool itself, but rather in the framework (Tensorboard x MLFlow example). But this only makes sense once the API is more or less defined. In our case, we are still developing the functionalities, and trying to do that might cause maintenance problems (not only for us). For now, we will take the approach of restricting the minimal or maximal version of a particular integrated library. Once we have the dvclive API defined, we will consider how we want to proceed.

Originally Posted by @pared

[dberenbaum]

Would this require a major version bump? I think it's time we need to think about doing this.

[daavoo]

Would this require a major version bump? I think it's time we need to think about doing this.

🤔 Unless we do some hacky stuff to keep the imports working with the same syntax as today, yes (probably not worth the effort)

[dberenbaum]

Opened https://github.com/huggingface/transformers/pull/27352 for hf transformers

[dberenbaum]

Opened https://github.com/huggingface/accelerate/pull/2139 for hf accelerate

[dberenbaum]

Lightning no longer accepts new callbacks 😦. Wish I had seen this before I wrote the PR 😅 :

It was decided a while ago that we no longer add new experiment tracking frameworks to the core of Lightning due to maintenance overhead.

https://github.com/Lightning-AI/lightning/issues/14137#issuecomment-1210433778

Once the huggingface callbacks are released and documented, we can lower the priority on this issue.