tokenizers icon indicating copy to clipboard operation
tokenizers copied to clipboard
verified publisher icon huggingface

Missing Version info in build from source installation instruction

Open leo-du opened this issue 3 years ago • 5 comments

In the build from source instructions: https://huggingface.co/docs/tokenizers/python/latest/installation/main.html#installation-from-sources , if one follows this instruction exactly, they will install the latest commit in master branch. Probably should add a warning or tell user to git checkout their desired version.

leo-du avatar Feb 01 '22 11:02 leo-du

I beg to differ, if you want to install from source you probably are a bit knowledgeable.

How sources and git works is beyond the simple commands which focus on making the thing buildable.

May I ask what surprised you/didn't work when you installed from sources ? Maybe the issue you encountered is the actual issue.

Narsil avatar Feb 01 '22 12:02 Narsil

There is nothing surprising or didn't work. I just noticed that the built version is different from the previous version used in another research project. I simply don't know what's changed in between the versions and don't have the time to track them down, and I tend to make a reasonable effort in making research code reproducible which is why I like to follow exact version setup. Since researchers are a main user base of this project I thought they probably want to know which version they're using when they build from the source. That's why I thought a notice in the guide would be nice.

leo-du avatar Feb 01 '22 12:02 leo-du

Okay, I might be too heavily biased.

Would like to do a PR ? The file that should be changed is here: https://github.com/huggingface/tokenizers/blob/master/docs/source/installation/python.inc

I think limiting this a simple comment (with optionnally a link) would do, wdyt ?

Narsil avatar Feb 01 '22 16:02 Narsil

Sure, a PR sounds good. Maybe a simple comment in the bottom like

Note that this will install the latest commit. You will need to git checkout if you would like to install your desired version.

This doesn't give the entire detail but will speak to those who cares about it. Does that work?

leo-du avatar Feb 01 '22 23:02 leo-du

Seems perfectly fine to me.

Narsil avatar Feb 02 '22 08:02 Narsil

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

github-actions[bot] avatar Mar 01 '24 01:03 github-actions[bot]