s3transfer

s3transfer copied to clipboard

In TransferManager.shutdown's call to TransferManager._shutdown(), it passes the same boolean parameter twice, thus setting the string param to a bool and exception_type to a string instead of a class

https://github.com/boto/s3transfer/blob/develop/s3transfer/manager.py#L546-L548

    def shutdown(self, cancel=False, cancel_msg=''):
        ...
        self._shutdown(cancel, cancel, cancel_msg)

    def _shutdown(self, cancel, cancel_msg, exc_type=CancelledError):

After fixing the params locally, attempting to cancel an in-progress s3 file transfer using shutdown(cancel=True) will result in deadlock

sample client code

s3client = boto3.client('s3')
transfer_config = boto3.s3.transfer.TransferConfig(max_concurrency=s3concurrency, multipart_threshold=multipart_threshold, use_threads=use_threads)
manager = boto3.s3.transfer.create_transfer_manager(client=s3, config=transfer_config)
transfer = boto3.s3.transfer.S3Transfer(manager=manager)

class AbortableProgress():
  def __init__(self, s3manager, progress_cb):
    self.manager = s3manager
    self.cb      = progress_cb
    self.start   = datetime.datetime.now()

  def __call__(self, *args, **kwargs):
     if self.cb:
       self.cb(*args, **kwargs)
     if self.manager:
       current = datetime.datetime.now()
       elapsed = (current-self.start).seconds
       if elapsed > 3:
         print("ABORT")
         self.manager.shutdown(cancel=True, cancel_msg="time to abort!")

transfer.download_file(sts['bucket'], sts['key'], local_filename, callback=AbortableProgress(manager, progress_callback))

I have fixed the thread deadlock in #179. Hope it gets merged in soon.

I stumbled on this because today pylint is giving us this error in our code calling shutdown():

Unexpected keyword argument 'cancel_msg' in method call (unexpected-keyword-arg)

Pylint is wrong to call out our code but the internal call to _shutdown() with cancel being passed twice is obviously broken.

This bug has been opened since 2019 with a PR for it, @nateprewitt is this something you can take a look at?