iterative
s3 remote: handle invalid auth information properly
Bug Report
If S3 auth token is invalid, DVC throws unexpected error
Description
I have MFA (2FA) setup. When temporary token is expired, and I try to push/pull, I'm getting an unexpected error from DVC:
ERROR: unexpected error - [Errno 22] The provided token is malformed or otherwise invalid.: An error occurred (InvalidToken) when calling the ListObjectsV2 operation: The provided token is malformed or otherwise invalid.
Quick fix is to update the token with AWS STS tools.
Reproduce
Use MFA Wait token expire Try to push something to S3
Expected
It's not an unexpected error. It should be handled properly with some useful message on what is wrong. The provided token is malformed or otherwise invalid. is not useful.
Environment information
Output of dvc doctor:
DVC version: 2.5.4+6c8673
---------------------------------
Platform: Python 3.8.9 on macOS-10.15.6-x86_64-i386-64bit
Supports:
azure (adlfs = 2021.7.0, knack = 0.8.2, azure-identity = 1.5.0),
gdrive (pydrive2 = 1.8.2),
gs (gcsfs = 2021.7.0),
hdfs (pyarrow = 4.0.0),
webhdfs (hdfs = 2.5.8),
http (requests = 2.25.1),
https (requests = 2.25.1),
s3 (s3fs = 2021.7.0, boto3 = 1.17.49),
ssh (paramiko = 2.7.2),
oss (ossfs = 2021.7.3),
webdav (webdav4 = 0.8.2),
webdavs (webdav4 = 0.8.2)
Cache types: reflink, hardlink, symlink
Cache directory: apfs on /dev/disk1s1
Caches: local
Remotes: s3
Workspace directory: apfs on /dev/disk1s1
Repo: dvc, git
Stack trace with `-v`
2021-07-15 08:34:10,784 ERROR: unexpected error - [Errno 22] The provided token is malformed or otherwise invalid.: An error occurred (InvalidToken) when calling the ListObjectsV2 operation: The provided token is malformed or otherwise invalid.
------------------------------------------------------------
Traceback (most recent call last):
File "/Users/ivan/Projects/example-get-started/.env/lib/python3.8/site-packages/s3fs/core.py", line 246, in _call_s3
out = await method(**additional_kwargs)
File "/Users/ivan/Projects/example-get-started/.env/lib/python3.8/site-packages/aiobotocore/client.py", line 154, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (InvalidToken) when calling the ListObjectsV2 operation: The provided token is malformed or otherwise invalid.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/Users/ivan/Projects/dvc/dvc/main.py", line 55, in main
ret = cmd.do_run()
File "/Users/ivan/Projects/dvc/dvc/command/base.py", line 50, in do_run
return self.run()
File "/Users/ivan/Projects/dvc/dvc/command/data_sync.py", line 57, in run
processed_files_count = self.repo.push(
File "/Users/ivan/Projects/dvc/dvc/repo/__init__.py", line 51, in wrapper
return f(repo, *args, **kwargs)
File "/Users/ivan/Projects/dvc/dvc/repo/push.py", line 44, in push
pushed += self.cloud.push(objs, jobs, remote=remote)
File "/Users/ivan/Projects/dvc/dvc/data_cloud.py", line 79, in push
return remote_obj.push(
File "/Users/ivan/Projects/dvc/dvc/remote/base.py", line 57, in wrapper
return f(obj, *args, **kwargs)
File "/Users/ivan/Projects/dvc/dvc/remote/base.py", line 488, in push
ret = self._process(
File "/Users/ivan/Projects/dvc/dvc/remote/base.py", line 345, in _process
dir_status, file_status, dir_contents = self._status(
File "/Users/ivan/Projects/dvc/dvc/remote/base.py", line 189, in _status
remote_exists.update(self._indexed_dir_hashes(dir_objs))
File "/Users/ivan/Projects/dvc/dvc/remote/base.py", line 252, in _indexed_dir_hashes
self.odb.list_hashes_exists(indexed_dirs)
File "/Users/ivan/Projects/dvc/dvc/objects/db/base.py", line 419, in list_hashes_exists
ret = list(itertools.compress(hashes, in_remote))
File "/usr/local/opt/[email protected]/Frameworks/Python.framework/Versions/3.8/lib/python3.8/concurrent/futures/_base.py", line 619, in result_iterator
yield fs.pop().result()
File "/usr/local/opt/[email protected]/Frameworks/Python.framework/Versions/3.8/lib/python3.8/concurrent/futures/_base.py", line 444, in result
return self.__get_result()
File "/usr/local/opt/[email protected]/Frameworks/Python.framework/Versions/3.8/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
raise self._exception
File "/usr/local/opt/[email protected]/Frameworks/Python.framework/Versions/3.8/lib/python3.8/concurrent/futures/thread.py", line 57, in run
result = self.fn(*self.args, **self.kwargs)
File "/Users/ivan/Projects/dvc/dvc/objects/db/base.py", line 410, in exists_with_progress
ret = self.fs.exists(path_info)
File "/Users/ivan/Projects/dvc/dvc/fs/fsspec_wrapper.py", line 83, in exists
return self.fs.exists(self._with_bucket(path_info))
File "/Users/ivan/Projects/example-get-started/.env/lib/python3.8/site-packages/fsspec/asyn.py", line 88, in wrapper
return sync(self.loop, func, *args, **kwargs)
File "/Users/ivan/Projects/example-get-started/.env/lib/python3.8/site-packages/fsspec/asyn.py", line 69, in sync
raise result[0]
File "/Users/ivan/Projects/example-get-started/.env/lib/python3.8/site-packages/fsspec/asyn.py", line 25, in _runner
result[0] = await coro
File "/Users/ivan/Projects/example-get-started/.env/lib/python3.8/site-packages/s3fs/core.py", line 804, in _exists
await self._info(path, bucket, key, version_id=version_id)
File "/Users/ivan/Projects/example-get-started/.env/lib/python3.8/site-packages/s3fs/core.py", line 1064, in _info
out = await self._simple_info(path)
File "/Users/ivan/Projects/example-get-started/.env/lib/python3.8/site-packages/s3fs/core.py", line 977, in _simple_info
out = await self._call_s3(
File "/Users/ivan/Projects/example-get-started/.env/lib/python3.8/site-packages/s3fs/core.py", line 265, in _call_s3
raise translate_boto_error(err)
OSError: [Errno 22] The provided token is malformed or otherwise invalid.
The provided token is malformed or otherwise invalid.is not useful.
@shcheklein Could you elaborate on why it is not useful, please? It precisely tells you what's wrong and that error is searchable. Just wondering what kind of error you expect here.
The provided token is malformed or otherwise invalid.
To be precise - this is a second-order issue. First issue here is that ERROR is unexpected while it's not.
Now, why do I think it's not useful, since it's not clear what token it is. At least modifying it to MFA (2FA) token, or use the same terminology as AWS has - "session token". We need something that would connect me to the actual sub-system, part of the auth flow that is failing.
It would be even better to say something - "Looks like your MFA (2FA) session token is expired or invalid"
To be precise - this is a second-order issue. First issue here is that ERROR is unexpected while it's not.
Agreed, this is a generally well-known issue with non-dvc exceptions. Def could reconsider among other things.
It would be even better to say something - "Looks like your MFA (2FA) session token is expired or invalid"
It pretty much says the same thing as The provided token is malformed or otherwise invalid., but is no longer searchable on google, so you'll likely get less good explanations and instructions. Though we could just include a link for https://dvc.org/doc/user-guide/troubleshooting#troubleshooting , where we could give a more useful verbose explanation and recipe on how to fix it and freely play with wording and stuff. A link with a very brief message like Authentication error (see link-to-troubleshooting) (instead of rephrasing in our code base like in your suggestion) is probably best and the easiest to maintain.
But need to take a closer look to say for sure.
Agreed, this is a generally well-known issue with non-dvc exceptions. Def could reconsider among other things.
Yep, to me it looks like a bug I think. Unexpected error to me means that something actually critical error (bug) happened.
Though we could just include a link for https://dvc.org/doc/user-guide/troubleshooting#troubleshooting
Good point. Do one generic "Auth failed " vs being more specific in the message for me that would depend on how bad those libraries that we use. If it's hard to get that information, then generic message with a link is enough I think.
instead of rephrasing in our code base like in your suggestion
It's not what I was suggesting to be precise :) I was only stating that it's not a very good message from the user perspective and can be improved. How? Ideally in the libraries that we use if they take that part of auth, to the extent that is possible. I also don't like parsing exception texts with regexps and wrapping them up. I would though go even that path if it's needed (depends on the availability of other options, etc, etc).
I also felt like we could have better error messages.
$ dvc checkout
WARNING: No file hash info found for '../../iterative.ai/static/uploads'. It won't be created.
ERROR: Checkout failed for following targets: ../../iterative.ai/static/uploads
Is your cache up to date?
<https://error.dvc.org/missing-files>
It's again due to the AWS MFA session token expired and couldn't checkout.
Also on dvc pull:
It would be better if these messages had some hint of authentication error. I don't see any on both of them.
