A TypeError is raised while token expires during writing to S3

[ShawnData]

This issue occurs when a STS token expires in the middle of writing to S3. An OSError: Write failed: TypeError("'NoneType' object is not subscriptable",) is raised instead of a PermissionError.

OSError: Write failed: TypeError("'NoneType' object is not subscriptable",) Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/pyarrow/parquet.py", line 1450, in write_to_dataset write_table(subtable, f, **kwargs)
File "/usr/local/lib/python3.6/site-packages/pyarrow/parquet.py", line 1344, in write_table writer.write_table(table, row_group_size=row_group_size)
File "/usr/local/lib/python3.6/site-packages/pyarrow/parquet.py", line 474, in write_table self.writer.write_table(table, row_group_size=row_group_size)
File "pyarrow/_parquet.pyx", line 1375, in pyarrow._parquet.ParquetWriter.write_table File "pyarrow/error.pxi", line 80, in pyarrow.lib.check_statuspyarrow.lib.ArrowIOError: Arrow error: IOError: The provided token has expired.. Detail: Python exception: PermissionError During handling of the above exception, another exception occurred: Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/s3fs/core.py", line 1096, in _upload_chunk PartNumber=part, UploadId=self.mpu['UploadId'],TypeError: 'NoneType' object is not subscriptable

environment is: s3fs==0.4.0 boto3==1.10.27 botocore==1.13.27 pyarrow==0.15.1

[martindurant]

Is it self.mpu which is None?

[ShawnData]

Yes, I think so. Since

self.mpu = self._call_s3(
                self.fs.s3.create_multipart_upload,
                Bucket=self.bucket, Key=self.key, ACL=self.acl)

There might be cases when self._call_s3 returns None

[martindurant]

There might be cases when self._call_s3 returns None

Obviously those cases should have raised the error - presumably indeed a PermissionsError

[chenliu0831]

+1, this happens when the credential is missing/mis-configured. A better exception type or message would be very helpful

[ghost]

Running into the same issue

[martindurant]

Would someone like to put a PR together to add checks in _call_s3 and raise the PermissionsError when appropriate? This is probably not something we can explicitly test.