evaluate icon indicating copy to clipboard operation
evaluate copied to clipboard

add versioning the `HubEvaluationModuleFactory`

Open lvwerra opened this issue 2 years ago • 2 comments

This PR couples the module repositories on the hub with the main repository in evaluate, based on tags that are propagated from evaluate to each space when they are pushed to the hub via actions (see evaluate/.github/hub/push_evaluations_to_hub.py).

The logic is:

  • if user provides revision, this is used.
  • else: there is no revision is provided then,
    • if evaluate has a dev tag main is used
    • else version tag of evaluate is used (e.g. v0.2.0)

This addresses the massive issues when merging #188. This will not solve the issues for existing versions but at least for newer versions this should not be an issue anymore.

lvwerra avatar Oct 07 '22 14:10 lvwerra

The documentation is not available anymore as the PR was closed or merged.

Alternatively it can use main by default, but if it contains a requirements.txt with an old version of evaluate it can check the tags for release-* ?

(tbh I was thinking of something like that for datasets)

This way we stay consistent with datasets and models: use main by default + no need to make a release to update something

lhoestq avatar Oct 07 '22 18:10 lhoestq

The issue with that is that we'll never be able to have a breaking change between the hub modules and the github repository without breaking every older version of the library.

this is what's happening with #188: since changed the module structure a bit all the older version of evaluate will try to load them.

lvwerra avatar Oct 10 '22 08:10 lvwerra

If we use release tags release-* we can resolve to the latest tag with a requirements.txt that is compatible with the user's local version of evaluate no ?

lhoestq avatar Oct 10 '22 16:10 lhoestq