huggingface
Set throttle limit?
While there is no denying that HF Transfer is an awesome tool, it works a bit too well for me. I've found that it is maxing out our network, leading to slower experiences for my colleagues. Is there a way to throttle the max download speed to e.g. 200MBps with an environment variable or something? If not, it would be a great addition to the library!
I would also like to limit the max concurrent requests since currently it will cause the server to refuse responding.
Or max the speed. I am also experiencing network saturation causing DNS errors because the throughput is too important for download.
#[pyo3(signature = (url, filename, max_files, chunk_size, parallel_failures=0, max_retries=0, headers=None, callback=None))]
With max_files and chunk_size you should be able to throttle this. default is 100 files and 10MB chunk size
Will try thanks ! I think the problem we have might not even be linked to this hf_transfer anyways... It might be because of the new hf xet backend for storing files since hf_transfer is written as deprecated on the huggingface docs.
hf_xet uses a transfer protocol based on hf_transfer so the control should be the same, but yes please report to hf_xet if you're using it.