aws-toolkit-azure-devops
aws-toolkit-azure-devops copied to clipboard
aws
How to install AWS CLI on VSTS Hosted windows agent or will that be a default feature of hosted agent in the future.
ran into this issue when running aws cli task on hosted windows agent "2017-09-11T20:45:32.5563058Z ##[error]AWS CLI is not installed on this machine." it would make more sense that it comes by default. I'm wondering if it will be released in the future?
Given the different environments and ways to install the CLI, and the requirement to install Python etc (at least I think that still needs to be done) we're thinking about documenting the install process as part of the task reference, as a script that can be copied into a shell command task. I don't think we are going to try and do the install automatically (but you never know).
Just had this issue on Linux Hosted agent.
##[error]AWS CLI is not installed on this machine.
Must the 'AWS CLI' step be dependent on a previous manual installation task on the remote agent?
@steveataws what is the roadmap plan for adding this task you mentioned?
Here is your solution https://docs.aws.amazon.com/cli/latest/userguide/installing.html
@renatorcosta the issue here is not how to install the AWS CLI. we've all done that. The issue is that we would expect the official VSTS AWS integration to include the AWS CLI installation by default.
@steveataws This all makes me wonder what use case you were even trying to address with this Task. There's no way to get the return values from a get-* or list-* operation; we'd be better off writing a custom script to do that and write the output to a temporary file. And now I find that the AWS CLI doesn't come even come pre-installed, so we have to write a separate script to install it first. So again, what is even the point of having this Task?
Sorry for sounding harsh, I love all your other AWS Tools Tasks. This one in particular just seems like it was a waste of man hours...
I couldnt get this to work either on hosted VSTS windows box. I tried numerous ways of installing without luck. My alternative was to just use the AWS SDK in node and create lamda functions. Used the invoke lamda functions to do various devops tasks. Bit of a long winded approach, but its a workaround until there is a proper solution :)
Huge apologies, I wasn't aware of any follow up correspondence on this issue and have been mostly focused elsewhere.
I totally agree we haven't done a great job of addressing the 'why' of this task, together with finding some convenient way to make it work better in a hosted environment, and integration with the environment itself. Our integration test setup uses a Team Foundation Server setup, so we have the CLI pre-installed for the build agents. This is possible with VSTS too, but we don't document it (I did find a useful blog post on this while discussing it with a colleague yesterday, which also reminded me I'd not recalled any activity on the issue for a while).
The task was intended (like the PowerShell task) as a catch-all for scenarios for working with AWS that our other tasks didn't fulfill at the time. We can't bundle the cli with the tools as there is a size limit in the marketplace for extensions therefore our only options are either to document the process (which we also haven't done a great job on) or find some way to do an automatic install, like we do with the AWS Tools for Windows PowerShell module - although that too has caused its fair share of issues and now we have users wanting us to install PowerShell 6 and the corresponding AWS module for use on Linux hosts too.
The cli with its Python dependencies requires a slightly more involved install process but, if we were to be sure that Python was available pre-installed on all hosted agents, it might be possible for us to run the 'pip' install automatically - I don't know whether Python is in fact always available on all hosts however. Is someone more familiar with all the different hosts able to more quickly confirm or deny this is the case?
Regarding the output issue from the task, my current suggestion is for us to extend the task so that the output can be captured into a build variable and thus processed further downstream in builds. Alternatively, the AWS shell script task that we recently added could be used to install the cli (for now, if necessary), run the command(s) and do the output processing, using the formatted console output to inject the data you need into VSTS build variables for downstream tasks.
Wish I had better news at this stage but the task isn't 'dead', we're just not at all sure of the best way to address the auto-install issue on multiple platforms with the Python dependency and are certainly open to suggestions.
@steveataws perhaps a discussion with the azure pipelines team for them to increase the size limit would all that would be required? They are obviously pushing their service far and are embracing the market place and perhaps would be open to an exception to the file size limitation?
Even if we managed to get the size limit raised we'd then find ourselves in a bind on the CLI version bundled with the tools. The AWS SDKs and the CLI update almost daily. The VSTS tools rev much more slowly. So if we bundled the CLI we'd then potentially have users confused as to why recent service APIs or new services were not present.
This (plus size) is why we set the PowerShell task to auto-install instead of bundling (plus it was much easier since we know the PowerShellGet commands to perform installs are present on the Windows hosts). A similar process which would always pull down the latest CLI if not installed would avoid bundled version issues and is the approach I think we need to follow (perhaps with an option to try to do an update if already installed).
@steveataws Great points, thanks for the update. In reference to whether Python is always available, there is a Microsoft Docs article that describes the software available on hosted VSTS agents. Unfortunately, it seems to only describe the operating systems. Maybe one of the authors of that topic can give more specifics on pre-installed tools/frameworks?
I've heard from others inside AWS that apparently Python is available on all hosts (but haven't confirmed it) but that 'pip install' errors out about a new version of pip being available. My hunch is that this is not actually an error, just a misinterpretation of an exit code - I see the same message when installing the CLI on my Windows laptop.
If that's the case, it should be possible for the task to be updated to trigger an install (or update if the CLI is found) and squelch the pip output, but it needs some testing to confirm.
This worked for me, with a Linux build agent:
pip install setuptools pip install wheel pip install awscli --upgrade
The pip version output is just a notice, not an error.
@jasonwhetton How did you get this to work. I have a command line task to run those commands, but even directly after that I still get aws command not found The log shows the cli getting installed ok, but it still doesn't work.
@vinla I was using the Hosted Linux Preview, it's not working on the Ubuntu 16.04 agent. Still trying to get it working there (hopefully before they remove the preview!).
@jasonwhetton I got this working on Ubuntu 1604 by prefixing the 3 commands with sudo
@vinla thanks for coming back and posting - you just saved my afternoon :)
I had trouble getting aws cli installed on the agent also but it wasnt an aws issue. I will try Vinia's suggestion. I did work around this already by using my own container on the build agent. I find it to be more consistent and predictable that the vsts environment. I find vsts is alot of slow and painful trial and error and the documentation doesn't really cover real build scenarios just small pieces
docker pull your-container docker run --name container-name -dt image-name docker exec aws-cli aws sts get-caller-identity docker stop container-name
This is broken now again, it used to work until yesterday on CMD for me, just says: Script contents: pip install awscli ##[command]"C:\windows\system32\cmd.exe" /D /E:ON /V:OFF /S /C "CALL "D:\a_temp\XXXXXXXXXXXXXXXXXXXXX.cmd"" ##[error]Cmd.exe exited with code '1'. ##[section]Finishing: Add AWS CLI
This is fixed now with 'Use Python 3.X' https://developercommunity.visualstudio.com/content/problem/470321/cmd-pip-install-awscli-doesnt-work-anymore.html