fix: use simple_error instead of PyException
I experienced this tools hanging without any error like #37 #30 and it happened more frequently in poor Internet situation.
It appears that the PyException type is experiencing performance issues during string conversion, which may lead to blocking of Tokio asynchronous tasks.
let parallel_failure_permit = parallel_failures_semaphore.clone().try_acquire_owned().map_err(|err| {
PyException::new_err(format!(
"Failed too many failures in parallel ({parallel_failures}): {dlerr} ({err})" // Here `dlerr` is PyException type
))
})?;
This blocking can prevent the timely dropping of semaphores, causing the semaphore pool associated with max_files to be quickly exhausted, ultimately leading to a deadlock in the entire program. This PR addresses the issue by changing the return type of the download_chunk function to simple_error to circumvent this problem.
@McPatate @Narsil hi sorry to bother but could you review this? cause the hanging is really annoying😭
I don't think this is the solution in anyway.
Adding a dependency for something so simple is not necessary and should really be avoided. Can you reproduce the behavior in any consistent way ?