AWS CLI does not accept valid file URIs

[pedrolamarao]

Describe the bug

Consider the following command line executed with PowerShell 7 on a Windows system:

aws ecs register-task-definition --container-definitions "file:///e:/tmp/file" --family "foo" --network-mode awsvpc
Error parsing parameter '--container-definitions': Unable to load paramfile file:///e:/tmp/file: [Errno 22] Invalid argument: '/e:/tmp/file'

It is currently not possible to automate AWS CLI on Windows with correct URI generators because of this.

Expected Behavior

I expect that AWS CLI accepts valid file URIs.

Current Behavior

AWS CLI does not accept valid file URIs in Windows systems running PowerShell 7.

Reproduction Steps

On a Windows system with PowerShell 7:

0. create a file at e:/tmp/file.json
1. run the command line: aws ecs register-task-definition --container-definitions "file:///c:/tmp/file.json" --family "foo" --network-mode awsvpc

Possible Solution

Fix file URI parsing on Windows systems.

It is not true that the following is an appropriate file URI on Windows systems: file://\e:\tmp\file.json

Additional Information/Context

https://www.rfc-editor.org/rfc/rfc8089

CLI version used

aws-cli/2.9.0 Python/3.9.11 Windows/10 exe/AMD64 prompt/off

Environment details (OS name and version, etc.)

windows, powershell

[RyanFitzSimmonsAK]

Hi @pedrolamarao, thanks for reaching out. This is not actually a bug with the CLI, but rather just a result of Python not natively supporting URIs. Marking this as a feature request for this functionality to be added. Other users who'd like to see this happen can react 👍 and comment with their use case.

[pedrolamarao]

Thanks for your response! I cannot agree with your assessment. The aws cli tool has a documented interface. The documentation for this interface states that it accepts file URIs as arguments. Such URIs are defined by stable technical standards. The tool interprets such URIs incorrecly in real world cases. I am not requesting a feature, I am pointing out that the tool does not work as advertised.

[kellertk]

Hi there. You're right, such URIs are defined by standards - and in our documentation. I tried this out and it worked for me:

aws ec2 run-instances --image-id ami-017fecd1353bcc96e --instance-type t3.micro --key-name id_rsa --security-groups my-testing-group --user-data file://d:\userdata.txt

This is for the D:\userdata.txt file on my machine. My aws --version:

aws-cli/2.5.8 Python/3.9.11 Windows/10 exe/AMD64 prompt/off

You can also use the Windows DOS device path naming scheme to specify all kinds of paths, not just mapped to a drive letter, like UNC paths and maybe also the internal WSL2 filesystem. You would specify something like file://\\.\D:\userdata.txt.

[pedrolamarao]

file://d:\userdata.txt is not a valid file URI. file:///d:/userdata.txt is a valid file URI. aws cli accepts the invalid URI and does not accept the valid URI.

[kellertk]

NB: The slashes can be done either way, so you can use file://d:/userdata.txt if you choose.

OK, paging through RFC8089 I think it is correct that we're not strictly following the RFC syntax, specifically the "file-hier-part" component, which is /// for local systems. Instead, we're assuming that it's file:// followed by some local file component, so an extra / is misinterpreted as part of the file path.

The optional minimal representation talked about in Section 2 of the RFC is supported - file:d:/userdata.txt - which omits the authority part entirely, but the traditional file URI for a local file with an empty authority (file:///) is incorrectly interpreted as a leading / due what we're doing here https://github.com/aws/aws-cli/blob/665514e7496a6487962dd2a8cef47804bf56d384/awscli/paramfile.py#L261.

We also support the relative resolution semantics in Section E.2.1, and the vertical line character in Section E.2.2. I didn't check the sharing semantics by specifying a different host, but I believe that would work as well.

At any rate, I'm comfortable calling this a bug, but I expect that it will be a low priority for the team as there is a workaround available: use the minimal syntax described elsewhere in the RFC.

[pedrolamarao]

aws cli does not seem to support the "optional minimal representation" with sufficient consistency. In the exercise below, the representation finally accept is incorrect.

PS E:\prodist\git\pix-qrc-http> aws iam create-role --assume-role-policy-document "file:e:/tmp/task-execution-policy" --role-name pix-qrc-http-verifier

An error occurred (MalformedPolicyDocument) when calling the CreateRole operation: This policy contains invalid Json
PS E:\prodist\git\pix-qrc-http> aws iam create-role --assume-role-policy-document "file:e:\tmp\task-execution-policy" --role-name pix-qrc-http-verifier

An error occurred (MalformedPolicyDocument) when calling the CreateRole operation: This policy contains invalid Json
PS E:\prodist\git\pix-qrc-http> aws iam create-role --assume-role-policy-document "file:/e:\tmp\task-execution-policy" --role-name pix-qrc-http-verifier

An error occurred (MalformedPolicyDocument) when calling the CreateRole operation: This policy contains invalid Json
PS E:\prodist\git\pix-qrc-http> aws iam create-role --assume-role-policy-document "file:/e:/tmp/task-execution-policy" --role-name pix-qrc-http-verifier

An error occurred (MalformedPolicyDocument) when calling the CreateRole operation: This policy contains invalid Json
PS E:\prodist\git\pix-qrc-http> aws iam create-role --assume-role-policy-document "file://e:/tmp/task-execution-policy" --role-name pix-qrc-http-verifier
{
    "Role": {
        "Path": "/",
        "RoleName": "pix-qrc-http-verifier",
        "RoleId": "AROAUOAN5LAIO4ZCIUR26",
        "Arn": "arn:aws:iam::304971864080:role/pix-qrc-http-verifier",
        "CreateDate": "2022-12-12T17:28:32+00:00",
        "AssumeRolePolicyDocument": {
            "Statement": [
                {
                    "Sid": "",
                    "Effect": "Allow",
                    "Principal": {
                        "Service": "ecs-tasks.amazonaws.com"
                    },
                    "Action": "sts:AssumeRole"
                }
            ]
        }
    }
}