hf_transfer icon indicating copy to clipboard operation
hf_transfer copied to clipboard

fix: use simple_error instead of PyException

Open lx200916 opened this issue 1 year ago • 1 comments

I experienced this tools hanging without any error like #37 #30 and it happened more frequently in poor Internet situation. It appears that the PyException type is experiencing performance issues during string conversion, which may lead to blocking of Tokio asynchronous tasks.

let parallel_failure_permit = parallel_failures_semaphore.clone().try_acquire_owned().map_err(|err| {
    PyException::new_err(format!(
    "Failed too many failures in parallel ({parallel_failures}): {dlerr} ({err})"  // Here `dlerr` is PyException type
     ))
   })?;

This blocking can prevent the timely dropping of semaphores, causing the semaphore pool associated with max_files to be quickly exhausted, ultimately leading to a deadlock in the entire program. This PR addresses the issue by changing the return type of the download_chunk function to simple_error to circumvent this problem.

lx200916 avatar Aug 28 '24 12:08 lx200916

@McPatate @Narsil hi sorry to bother but could you review this? cause the hanging is really annoying😭

lx200916 avatar Aug 30 '24 06:08 lx200916

I don't think this is the solution in anyway.

Adding a dependency for something so simple is not necessary and should really be avoided. Can you reproduce the behavior in any consistent way ?

Narsil avatar Dec 23 '24 15:12 Narsil