aws_completer doesn't complete filenames

[Malvineous]

If you use a command like this:

aws s3 cp a<tab>

Then normally the <tab> will result in autocompleting a file in the current directory starting with a. However this does not happen, and pressing tab has no effect.

Can the autocompleter be updated to allow completing of filenames at the appropriate times?

[kyleknap]

Marking as a feature request. I think the difficult part about this is determining if a user wants to autocomplete a parameter/command or autocomplete a local file.

[ahuwaa]

@Malvineous what OS and terminal are you using? I'm able to autocomplete local filenames using tab on Mac OSX in iTerm2 and regular Terminal

[Malvineous]

@ahuwaa I'm using bash under Linux with aws-cli/bin/aws_bash_completer. Have you enabled an autocomplete script for your shell like this one? You might be using your shell's default autocomplete process which will only complete filenames but not aws-cli command parameters.

[ahuwaa]

Sorry, I misunderstood. I did not know there was an autocomplete feature but have now reproduced the issue.

I'd like to take a shot at this, if someone hasn't already started. Any advice/tips appreciated.

[ahuwaa]

Going to need help with this due to time constraints. So far I thought it was simplest to just default to the shell's autocomplete e.g. this works in bash

import subprocess
import pipes
# ...
def complete(cmdline, point):
    choices = Completer().complete(cmdline, point)
    if len(choices) > 0:
        print(' \n'.join(choices))
    else:
        cmd = cmdline.split(" ").pop()
        try:
            out = subprocess.check_output("compgen -o default " + pipes.quote(cmd), shell=True)
        except subprocess.CalledProcessError:
            out = ""
        print out

but of course this relies on something like compgen being available. If anyone has a better suggestion or would like to take over, feel free

[nhed]

@kyleknap remote filename completion is the primary thing I hoped for when I learned that there is a complete module, calling it a feature-request is somewhat of an understatement. Even scp in bash completion goes to the remote server over ssh to obtain list of files/paths. Having the path start with s3:// should give you the hint (as its the hint that aws cp uses to figure out which side of the copy is remote which is local)

(also should note that this should also complete aws s3 ls a<TAB> with the caveat that aws s3 ls is forgiving about the s3:// prefix and one could say that this is required for completion to work)

[CBribiescas]

Is there any ETA on this feature? Now with the new AWS S3 interface its a pain to have to type out paths. I have no idea why they took out the full path from the to of the page.

[ASayre]

Good Morning!

We're closing this issue here on GitHub, as part of our migration to UserVoice for feature requests involving the AWS CLI.

This will let us get the most important features to you, by making it easier to search for and show support for the features you care the most about, without diluting the conversation with bug reports.

As a quick UserVoice primer (if not already familiar): after an idea is posted, people can vote on the ideas, and the product team will be responding directly to the most popular suggestions.

We’ve imported existing feature requests from GitHub - Search for this issue there!

And don't worry, this issue will still exist on GitHub for posterity's sake. As it’s a text-only import of the original post into UserVoice, we’ll still be keeping in mind the comments and discussion that already exist here on the GitHub issue.

GitHub will remain the channel for reporting bugs.

Once again, this issue can now be found by searching for the title on: https://aws.uservoice.com/forums/598381-aws-command-line-interface

-The AWS SDKs & Tools Team

This entry can specifically be found on UserVoice at: https://aws.uservoice.com/forums/598381-aws-command-line-interface/suggestions/33168376-aws-completer-doesn-t-complete-filenames

[jamesls]

Based on community feedback, we have decided to return feature requests to GitHub issues.

[excenter]

Recently installed aws --version > aws-cli/1.15.60 Python/2.7.15 Darwin/17.7.0 botocore/1.10.59 still no tab completion on local files, which makes aws s3 cp meaningfully harder to use than it has to be. If this isn't going to be fixed I'd rather uninstall tab completion for the cli and have file completion back, than have cli completion and not local file names.

[nhed]

[brunogasperini]

Any updates on this feature request? It would be extremely useful to have tabcompletion for s3 buckets and files.

[polarizeme]

Just chiming in with another voice on this request. I honestly thought my AWS CLI was just broken this entire time. 😬

[alisade]

really need this to work pretty please!

[erankor]

Hi all,

I patched the AWS completer to complete s3 paths, for example - typing: aws s3 cp s3://mybucket/a(TAB) will complete all prefixes starting with /a under mybucket.

I'm not submitting it as pull request since it's a bit hacky, but if anyone's interested - here is the patch - completer-s3-paths.zip

Eran

[ragulpr]

@erankor, thanks for sharing! Haven't tried it yet but I inlined erankors patch below for those wary about downloading zips 🙂

completer-s3-paths.zip
        completer-s3-paths.patch
--- completer.py	2019-08-12 11:01:24.536333508 +0000
+++ /usr/lib/python3/dist-packages/awscli/completer.py	2019-08-12 11:00:00.152687298 +0000
@@ -14,6 +14,13 @@
 import logging
 import copy
 
+try:
+	import botocore
+	session = botocore.session.Session()
+	s3 = session.create_client('s3')
+except ImportError:
+	s3 = None
+
 LOG = logging.getLogger(__name__)
 
 
@@ -61,9 +68,38 @@
                 command_help.command_table, current_arg)
         return []
 
+    def _complete_s3_path(self, current_arg, depth = 10):
+        if depth <= 0:
+            return [current_arg]
+        splitted_arg = current_arg.split('/', 1)
+        if len(splitted_arg) < 2:
+            return []
+        bucketName, prefix = splitted_arg
+        paginator = s3.get_paginator('list_objects')
+        page_iterator = paginator.paginate(Bucket=bucketName,Prefix=prefix,Delimiter='/')
+
+        result = []
+        for page in page_iterator:
+            if 'CommonPrefixes' in page:
+                for item in page['CommonPrefixes']:
+                    result.append('%s/%s' % (bucketName, item['Prefix']))
+            if 'Contents' in page:
+                for item in page['Contents']:
+                    result.append('%s/%s' % (bucketName, item['Key']))
+
+        if len(result) == 1:
+            return self._complete_s3_path(result[0], depth - 1)
+        return result
+
     def _complete_subcommand(self, subcmd_name, subcmd_help, current_arg, opts):
         if current_arg != subcmd_name and current_arg.startswith('-'):
             return self._find_possible_options(current_arg, opts, subcmd_help)
+        elif current_arg.startswith('s3://') and s3 is not None:
+            current_arg = current_arg[len('s3://'):]
+            possibilities = self._complete_s3_path(current_arg)
+            if possibilities == [current_arg]:  # avoid duplication when clicking tab after reaching a leaf node
+                return []
+            return ['//' + n for n in possibilities]
         return []
 
     def _complete_option(self, option_name):

[diesal11]

Do any solutions for local files exist yet?

[max0r]

+1 for local file completion.

[erhhung]

@erankor I haven't tried your patch, but do you know if the list_objects call uses the preceding --profile myprofile option to obtain the proper credentials? I have lots of profiles for different accounts, so this would be a must.

[erankor]

Probably won't work, since I'm creating the s3 client directly in my patch, not using something provided by the aws cli.

[cervantek]

This feature request effects v2 as well.

The rules for when to drop through to client side file completion seem straightforward. This is not a complete list, but here are some examples that would be really useful.

For any command, look for file:// to do client side as in... aws cloudformation create-stack --cli-input-json file://<tab for client-side completion of folders/files>

A few special cases that don't follow that rule of needing/expecting file://. aws s3 cp aws s3 cp s3://<tab for server-side completion, i.e. buckets> aws s3 cp <tab for client-side completion folders/files>, do client-side

aws cloudformation package --template-file <tab for client-side completion of folders/files>

[gareth-li]

I also assumed this would be already built into aws_completer.

Making this a standard feature would be immensely useful.

[jaklan]

The issue was created more than 3 years ago, and such a completely fundamental functionality is still missing? No words...

[shugybugy-assaf]

It will be super useful - please implement This one aws s3 cp s3://<tab for server-side completion, i.e. buckets> is the critical feature

[erankor]

After more than a year, I circled back to this open feature request... Opened a PR this time #5849, it passed all checks now, hope it will be merged soon

[shawnzam]

Until this is fixed, you can use https://rclone.org/

[kdaily]

Hi @erankor,

I'm going to take a look at your PR. This is a pretty completx feature, especially for the S3 file name completion. I'll comment some more over there. Thanks!

[erankor]

@kdaily, did you check it out?

[erankor]

So, two months passed since I opened the PR... is this going to happen at some point?

[nikhilweee]

One word: Please. 🥺