kitchen-ec2
kitchen-ec2 copied to clipboard
Published
20 hours ago •
test-kitchen
test-kitchen
The request must contain the parameter AWSAccessKeyId (Aws::EC2::Errors::MissingParameter)
Hello,
I've discover an issue on kitchen-ec2 version 2.4.0. My kitchen are run in a container when i'm running kitchen diagnose on alpine:3.7, everything is ok.
I've moved my container to alpine:3.8 since that change, I'm having the following issue when running kitchen diagnose:
Traceback (most recent call last):
40: from /usr/bin/kitchen:23:in `<main>'
39: from /usr/bin/kitchen:23:in `load'
38: from /usr/lib/ruby/gems/2.5.0/gems/test-kitchen-1.24.0/bin/kitchen:13:in `<top (required)>'
37: from /usr/lib/ruby/gems/2.5.0/gems/test-kitchen-1.24.0/lib/kitchen/errors.rb:171:in `with_friendly_errors'
36: from /usr/lib/ruby/gems/2.5.0/gems/test-kitchen-1.24.0/bin/kitchen:13:in `block in <top (required)>'
35: from /usr/lib/ruby/gems/2.5.0/gems/thor-0.20.3/lib/thor/base.rb:466:in `start'
34: from /usr/lib/ruby/gems/2.5.0/gems/thor-0.20.3/lib/thor.rb:387:in `dispatch'
33: from /usr/lib/ruby/gems/2.5.0/gems/thor-0.20.3/lib/thor/invocation.rb:126:in `invoke_command'
32: from /usr/lib/ruby/gems/2.5.0/gems/thor-0.20.3/lib/thor/command.rb:27:in `run'
31: from /usr/lib/ruby/gems/2.5.0/gems/test-kitchen-1.24.0/lib/kitchen/cli.rb:142:in `diagnose'
30: from /usr/lib/ruby/gems/2.5.0/gems/test-kitchen-1.24.0/lib/kitchen/cli.rb:52:in `perform'
29: from /usr/lib/ruby/gems/2.5.0/gems/test-kitchen-1.24.0/lib/kitchen/command/diagnose.rb:38:in `call'
28: from /usr/lib/ruby/gems/2.5.0/gems/test-kitchen-1.24.0/lib/kitchen/diagnostic.rb:52:in `read'
27: from /usr/lib/ruby/gems/2.5.0/gems/test-kitchen-1.24.0/lib/kitchen/diagnostic.rb:125:in `prepare_instances'
26: from /usr/lib/ruby/gems/2.5.0/gems/test-kitchen-1.24.0/lib/kitchen/diagnostic.rb:125:in `each'
25: from /usr/lib/ruby/gems/2.5.0/gems/test-kitchen-1.24.0/lib/kitchen/diagnostic.rb:125:in `block in prepare_instances'
24: from /usr/lib/ruby/gems/2.5.0/gems/test-kitchen-1.24.0/lib/kitchen/instance.rb:257:in `diagnose'
23: from /usr/lib/ruby/gems/2.5.0/gems/test-kitchen-1.24.0/lib/kitchen/instance.rb:257:in `each'
22: from /usr/lib/ruby/gems/2.5.0/gems/test-kitchen-1.24.0/lib/kitchen/instance.rb:259:in `block in diagnose'
21: from /usr/lib/ruby/gems/2.5.0/gems/test-kitchen-1.24.0/lib/kitchen/configurable.rb:120:in `diagnose'
20: from /usr/lib/ruby/gems/2.5.0/gems/test-kitchen-1.24.0/lib/kitchen/configurable.rb:120:in `each'
19: from /usr/lib/ruby/gems/2.5.0/gems/test-kitchen-1.24.0/lib/kitchen/configurable.rb:120:in `block in diagnose'
18: from /usr/lib/ruby/gems/2.5.0/gems/test-kitchen-1.24.0/lib/kitchen/lazy_hash.rb:76:in `[]'
17: from /usr/lib/ruby/gems/2.5.0/gems/test-kitchen-1.24.0/lib/kitchen/lazy_hash.rb:143:in `proc_or_val'
16: from /usr/lib/ruby/gems/2.5.0/gems/kitchen-ec2-2.4.0/lib/kitchen/driver/ec2.rb:83:in `block in <class:Ec2>'
15: from /usr/lib/ruby/gems/2.5.0/gems/kitchen-ec2-2.4.0/lib/kitchen/driver/ec2.rb:381:in `default_ami'
14: from /usr/lib/ruby/gems/2.5.0/gems/kitchen-ec2-2.4.0/lib/kitchen/driver/aws/standard_platform.rb:106:in `find_image'
13: from /usr/lib/ruby/gems/2.5.0/gems/kitchen-ec2-2.4.0/lib/kitchen/driver/aws/standard_platform.rb:210:in `sort_images'
12: from /usr/lib/ruby/gems/2.5.0/gems/kitchen-ec2-2.4.0/lib/kitchen/driver/aws/standard_platform.rb:210:in `sort_by'
11: from /usr/lib/ruby/gems/2.5.0/gems/aws-sdk-resources-2.11.197/lib/aws-sdk-resources/collection.rb:18:in `each'
10: from /usr/lib/ruby/gems/2.5.0/gems/aws-sdk-resources-2.11.197/lib/aws-sdk-resources/collection.rb:18:in `each'
9: from /usr/lib/ruby/gems/2.5.0/gems/aws-sdk-resources-2.11.197/lib/aws-sdk-resources/operations.rb:139:in `all_batches'
8: from /usr/lib/ruby/gems/2.5.0/gems/aws-sdk-resources-2.11.197/lib/aws-sdk-resources/request.rb:24:in `call'
7: from /usr/lib/ruby/gems/2.5.0/gems/aws-sdk-core-2.11.197/lib/seahorse/client/base.rb:207:in `block (2 levels) in define_operation_methods'
6: from /usr/lib/ruby/gems/2.5.0/gems/aws-sdk-core-2.11.197/lib/seahorse/client/request.rb:70:in `send_request'
5: from /usr/lib/ruby/gems/2.5.0/gems/aws-sdk-core-2.11.197/lib/seahorse/client/plugins/response_target.rb:21:in `call'
4: from /usr/lib/ruby/gems/2.5.0/gems/aws-sdk-core-2.11.197/lib/aws-sdk-core/plugins/response_paging.rb:26:in `call'
3: from /usr/lib/ruby/gems/2.5.0/gems/aws-sdk-core-2.11.197/lib/aws-sdk-core/plugins/param_converter.rb:20:in `call'
2: from /usr/lib/ruby/gems/2.5.0/gems/aws-sdk-core-2.11.197/lib/aws-sdk-core/plugins/idempotency_token.rb:18:in `call'
1: from /usr/lib/ruby/gems/2.5.0/gems/aws-sdk-core-2.11.197/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:20:in `call'
/usr/lib/ruby/gems/2.5.0/gems/aws-sdk-core-2.11.197/lib/seahorse/client/plugins/raise_response_errors.rb:15:in `call': The request must contain the parameter AWSAccessKeyId (Aws::EC2::Errors::MissingParameter)
The difference I've found is the ruby version which 2.4 on alpine 3.7
ruby --version
ruby 2.4.5p335 (2018-10-18 revision 65137) [x86_64-linux-musl]
gem list | grep aws
aws-sdk (2.11.197)
aws-sdk-core (2.11.197)
aws-sdk-resources (2.11.197)
aws-sigv4 (1.0.3)
gem list | grep kitchen
kitchen-docker (2.7.0)
kitchen-ec2 (2.4.0)
test-kitchen (1.24.0)
and 2.5 on alpine 3.8:
ruby --version
ruby 2.5.2p104 (2018-10-18 revision 65133) [x86_64-linux-musl]
gem list | grep aws
aws-sdk (2.11.197)
aws-sdk-core (2.11.197)
aws-sdk-resources (2.11.197)
aws-sigv4 (1.0.3)
I've tried to enable the debug of kitchen but output is the same on both. And also enable aws-sdk the wire trace. And I can see that the request done on aws is not the same.
On version alpine 3.7:
kitchen diagnose centos-7
opening connection to ec2.eu-west-1.amazonaws.com:443...
opened
starting SSL for ec2.eu-west-1.amazonaws.com:443...
SSL established
<- "POST / HTTP/1.1\r\nContent-Type: application/x-www-form-urlencoded; charset=utf-8\r\nAccept-Encoding: \r\nUser-Agent: aws-sdk-ruby2/2.11.197 ruby/2.4.5 x86_64-linux-musl resources\r\nX-Amz-Date: 20181231T123525Z\r\nHost: ec2.eu-west-1.amazonaws.com\r\nX-Amz-Content-Sha256: cb0da5743aeabc7994f1ab91253b55cbf743d09b2188af9ab8a56f8\r\nAuthorization: AWS4-HMAC-SHA256 Credential=AKIAJULWORCMDGA/20181231/eu-west-1/ec2/aws4_request, SignedHeaders=content-type;host;x-amz-content-sha256;x-amz-date, Signature=0b812979d942b4f78021d90d96fead277fd35c0f28c0fb66c2d64f3e\r\nContent-Length: 193\r\nAccept: */*\r\n\r\n"
-> "HTTP/1.1 200 OK\r\n"
-> "Content-Type: text/xml;charset=UTF-8\r\n"
-> "Transfer-Encoding: chunked\r\n"
-> "Vary: Accept-Encoding\r\n"
-> "Date: Mon, 31 Dec 2018 12:35:27 GMT\r\n"
-> "Server: AmazonEC2\r\n"
-> "\r\n"
-> "2000\r\n"
reading 8192 bytes...
-> "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<DescribeImagesResponse xmlns=\"http://ec2.amazonaws.com/doc/2016-11-15/\">\n <requestId>c8fda463-4a8b-4569-85ea-341bf7bf8a8a</requestId>\n <imagesSet>\n <item>\n <imageId>ami-192a9460</imageId>\n <imageLocation>aws-marketplace/CentOS Linux 7 x86_64 HVM EBS 1708_11.01-b7ee8a69-ee97-4a49-9e68-afaee216db2e-ami-95096eef.4</imageLocation>\n <imageState>available</imageState>\n <imageOwnerId>679593333241</imageOwnerId>\n <creationDate>20
On 3.8:
kitchen diagnose centos-7
opening connection to ec2.eu-west-1.amazonaws.com:443...
opened
starting SSL for ec2.eu-west-1.amazonaws.com:443...
SSL established
<- "POST / HTTP/1.1\r\nContent-Type: application/x-www-form-urlencoded; charset=utf-8\r\nAccept-Encoding: \r\nUser-Agent: aws-sdk-ruby2/2.11.197 ruby/2.5.2 x86_64-linux-musl resources\r\nContent-Length: 193\r\nAccept: */*\r\nHost: ec2.eu-west-1.amazonaws.com\r\n\r\n"
-> "HTTP/1.1 400 Bad Request\r\n"
-> "Transfer-Encoding: chunked\r\n"
-> "Date: Mon, 31 Dec 2018 12:35:33 GMT\r\n"
-> "Connection: close\r\n"
-> "Server: AmazonEC2\r\n"
-> "\r\n"
-> "fc\r\n"
reading 252 bytes...
-> "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Response><Errors><Error><Code>MissingParameter</Code><Message>The request must contain the parameter AWSAccessKeyId</Message></Error></Errors><RequestID>92781b98-7ea6-4e82-8ef3-c0b1317b9243</RequestID></Response>"
read 252 bytes
reading 2 bytes...
-> "\r\n"
read 2 bytes
-> "0\r\n"
-> "\r\n"
Conn close
I don't have any clue if the issue is on kitchen-ec2 side or on aws-sdk ?
How are you setting credentials? This might be something as silly/easy as the AWS Access Key ID env variable not being set (if that is the auth mechanism)
Set with environment variables and properly set in both case the same way.
The reason I ask is that the request lines seem to hint at that:
3.7 (working), appears to be signing the request properly and passing credentials
<- "POST / HTTP/1.1\r\nContent-Type: application/x-www-form-urlencoded; charset=utf-8\r\nAccept-Encoding: \r\nUser-Agent: aws-sdk-ruby2/2.11.197 ruby/2.4.5 x86_64-linux-musl resources\r\nX-Amz-Date: 20181231T123525Z\r\nHost: ec2.eu-west-1.amazonaws.com\r\nX-Amz-Content-Sha256: cb0da5743aeabc7994f1ab91253b55cbf743d09b2188af9ab8a56f8\r\nAuthorization: AWS4-HMAC-SHA256 Credential=AKIAJULWORCMDGA/20181231/eu-west-1/ec2/aws4_request, SignedHeaders=content-type;host;x-amz-content-sha256;x-amz-date, Signature=0b812979d942b4f78021d90d96fead277fd35c0f28c0fb66c2d64f3e\r\nContent-Length: 193\r\nAccept: */*\r\n\r\n"
3.8 clearly isn't signing the requests
<- "POST / HTTP/1.1\r\nContent-Type: application/x-www-form-urlencoded; charset=utf-8\r\nAccept-Encoding: \r\nUser-Agent: aws-sdk-ruby2/2.11.197 ruby/2.5.2 x86_64-linux-musl resources\r\nContent-Length: 193\r\nAccept: */*\r\nHost: ec2.eu-west-1.amazonaws.com\r\n\r\n"
Could we get the kitchen.yml by chance?
That why i've opened the issue because I don't get why it's sign in one case and why it's not in the other with same kitchen.yml, same env variable and same version of kitchen and aws gem only ruby and os version changed.
Here is my kitchen file:
---
driver:
name: ec2
aws_ssh_key_id: key-chef-ssh
security_group_ids: ["sg-08b19275"]
region: eu-west-1
availability_zone: a
subnet_id: subnet-bd6415e6
instance_type: t2.micro
interface: private
transport:
ssh_key: /home/key-chef-ssh.pem
provisioner:
name: chef_zero
platforms:
- name: centos-7
driver:
tags:
Name: "kitchen-ci-centos7"
- name: centos-6
driver:
tags:
Name: "kitchen-ci-centos6"
- name: ubuntu-14-04
driver:
tags:
Name: "kitchen-ci-ubuntu1404"
image_id: ami-754e4a0c
- name: ubuntu-16-04
driver:
tags:
Name: "kitchen-ci-ubuntu1604"
image_id: ami-f5596e8c
- name: ubuntu-18-04
driver:
tags:
Name: "kitchen-ci-ubuntu1804"
image_id: ami-00035f41c82244dab
- name: debian-8
driver:
tags:
Name: "kitchen-ci-debian8"
- name: debian-9
driver:
tags:
Name: "kitchen-ci-debian9"
suites:
- name: default
run_list:
- 'recipe[postfix]'
To reproduce the issue you can follow those steps:
Working version with alpine 3.7, use the following Dockerfile and the .kitchen.yml given previously:
FROM alpine:3.7
RUN apk update && apk upgrade && apk --update add \
build-base ruby ruby-dev gmp ruby-irb ruby-rake ruby-io-console ruby-bigdecimal ruby-json ruby-bundler \
libstdc++ tzdata bash ca-certificates \
&& echo 'gem: --no-document' > /etc/gemrc
# Add chef and kitchen
RUN gem install json:2.0.4 chef-dk kitchen-docker test-kitchen kitchen-ec2 berkshelf
WORKDIR /mnt
CMD ["bash"]
docker build -t alpine-37 .
docker run -e AWS_ACCESS_KEY_ID=my_aws_id -e AWS_SECRET_ACCESS_KEY=my_aws_secret -v $(pwd):/mnt/ -it alpine-37 kitchen diagnose
---
timestamp: 2019-01-02 02:58:18 UTC
kitchen_version: 1.24.0
instances:
default-centos-7:
platform:
os_type: unix
shell_type: bourne
... working properly
The not working case (only change in the Docker file the alpine version to 3.8 that will use ruby 2.5 instead of 2.4):
FROM alpine:3.8
RUN apk update && apk upgrade && apk --update add \
build-base ruby ruby-dev gmp ruby-irb ruby-rake ruby-io-console ruby-bigdecimal ruby-json ruby-bundler \
libstdc++ tzdata bash ca-certificates \
&& echo 'gem: --no-document' > /etc/gemrc
# Add chef and kitchen
RUN gem install json:2.0.4 chef-dk kitchen-docker test-kitchen kitchen-ec2 berkshelf
WORKDIR /mnt
CMD ["bash"]
docker build -t alpine-37 .
docker run -e AWS_ACCESS_KEY_ID=my_aws_id -e AWS_SECRET_ACCESS_KEY=my_aws_secret -v $(pwd):/mnt/ -it alpine-38 kitchen diagnose
Traceback (most recent call last):
40: from /usr/bin/kitchen:23:in `<main>'
39: from /usr/bin/kitchen:23:in `load'
38: from /usr/lib/ruby/gems/2.5.0/gems/test-kitchen-1.24.0/bin/kitchen:13:in `<top (required)>'
37: from /usr/lib/ruby/gems/2.5.0/gems/test-kitchen-1.24.0/lib/kitchen/errors.rb:171:in `with_friendly_errors'
36: from /usr/lib/ruby/gems/2.5.0/gems/test-kitchen-1.24.0/bin/kitchen:13:in `block in <top (required)>'
35: from /usr/lib/ruby/gems/2.5.0/gems/thor-0.20.3/lib/thor/base.rb:466:in `start'
34: from /usr/lib/ruby/gems/2.5.0/gems/thor-0.20.3/lib/thor.rb:387:in `dispatch'
33: from /usr/lib/ruby/gems/2.5.0/gems/thor-0.20.3/lib/thor/invocation.rb:126:in `invoke_command'
32: from /usr/lib/ruby/gems/2.5.0/gems/thor-0.20.3/lib/thor/command.rb:27:in `run'
31: from /usr/lib/ruby/gems/2.5.0/gems/test-kitchen-1.24.0/lib/kitchen/cli.rb:142:in `diagnose'
30: from /usr/lib/ruby/gems/2.5.0/gems/test-kitchen-1.24.0/lib/kitchen/cli.rb:52:in `perform'
29: from /usr/lib/ruby/gems/2.5.0/gems/test-kitchen-1.24.0/lib/kitchen/command/diagnose.rb:38:in `call'
28: from /usr/lib/ruby/gems/2.5.0/gems/test-kitchen-1.24.0/lib/kitchen/diagnostic.rb:52:in `read'
27: from /usr/lib/ruby/gems/2.5.0/gems/test-kitchen-1.24.0/lib/kitchen/diagnostic.rb:125:in `prepare_instances'
26: from /usr/lib/ruby/gems/2.5.0/gems/test-kitchen-1.24.0/lib/kitchen/diagnostic.rb:125:in `each'
25: from /usr/lib/ruby/gems/2.5.0/gems/test-kitchen-1.24.0/lib/kitchen/diagnostic.rb:125:in `block in prepare_instances'
24: from /usr/lib/ruby/gems/2.5.0/gems/test-kitchen-1.24.0/lib/kitchen/instance.rb:257:in `diagnose'
23: from /usr/lib/ruby/gems/2.5.0/gems/test-kitchen-1.24.0/lib/kitchen/instance.rb:257:in `each'
22: from /usr/lib/ruby/gems/2.5.0/gems/test-kitchen-1.24.0/lib/kitchen/instance.rb:259:in `block in diagnose'
21: from /usr/lib/ruby/gems/2.5.0/gems/test-kitchen-1.24.0/lib/kitchen/configurable.rb:120:in `diagnose'
20: from /usr/lib/ruby/gems/2.5.0/gems/test-kitchen-1.24.0/lib/kitchen/configurable.rb:120:in `each'
19: from /usr/lib/ruby/gems/2.5.0/gems/test-kitchen-1.24.0/lib/kitchen/configurable.rb:120:in `block in diagnose'
18: from /usr/lib/ruby/gems/2.5.0/gems/test-kitchen-1.24.0/lib/kitchen/lazy_hash.rb:76:in `[]'
17: from /usr/lib/ruby/gems/2.5.0/gems/test-kitchen-1.24.0/lib/kitchen/lazy_hash.rb:143:in `proc_or_val'
16: from /usr/lib/ruby/gems/2.5.0/gems/kitchen-ec2-2.4.0/lib/kitchen/driver/ec2.rb:83:in `block in <class:Ec2>'
15: from /usr/lib/ruby/gems/2.5.0/gems/kitchen-ec2-2.4.0/lib/kitchen/driver/ec2.rb:381:in `default_ami'
14: from /usr/lib/ruby/gems/2.5.0/gems/kitchen-ec2-2.4.0/lib/kitchen/driver/aws/standard_platform.rb:106:in `find_image'
13: from /usr/lib/ruby/gems/2.5.0/gems/kitchen-ec2-2.4.0/lib/kitchen/driver/aws/standard_platform.rb:210:in `sort_images'
12: from /usr/lib/ruby/gems/2.5.0/gems/kitchen-ec2-2.4.0/lib/kitchen/driver/aws/standard_platform.rb:210:in `sort_by'
11: from /usr/lib/ruby/gems/2.5.0/gems/aws-sdk-resources-2.11.197/lib/aws-sdk-resources/collection.rb:18:in `each'
10: from /usr/lib/ruby/gems/2.5.0/gems/aws-sdk-resources-2.11.197/lib/aws-sdk-resources/collection.rb:18:in `each'
9: from /usr/lib/ruby/gems/2.5.0/gems/aws-sdk-resources-2.11.197/lib/aws-sdk-resources/operations.rb:139:in `all_batches'
8: from /usr/lib/ruby/gems/2.5.0/gems/aws-sdk-resources-2.11.197/lib/aws-sdk-resources/request.rb:24:in `call'
7: from /usr/lib/ruby/gems/2.5.0/gems/aws-sdk-core-2.11.197/lib/seahorse/client/base.rb:207:in `block (2 levels) in define_operation_methods'
6: from /usr/lib/ruby/gems/2.5.0/gems/aws-sdk-core-2.11.197/lib/seahorse/client/request.rb:70:in `send_request'
5: from /usr/lib/ruby/gems/2.5.0/gems/aws-sdk-core-2.11.197/lib/seahorse/client/plugins/response_target.rb:21:in `call'
4: from /usr/lib/ruby/gems/2.5.0/gems/aws-sdk-core-2.11.197/lib/aws-sdk-core/plugins/response_paging.rb:26:in `call'
3: from /usr/lib/ruby/gems/2.5.0/gems/aws-sdk-core-2.11.197/lib/aws-sdk-core/plugins/param_converter.rb:20:in `call'
2: from /usr/lib/ruby/gems/2.5.0/gems/aws-sdk-core-2.11.197/lib/aws-sdk-core/plugins/idempotency_token.rb:18:in `call'
1: from /usr/lib/ruby/gems/2.5.0/gems/aws-sdk-core-2.11.197/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:20:in `call'
/usr/lib/ruby/gems/2.5.0/gems/aws-sdk-core-2.11.197/lib/seahorse/client/plugins/raise_response_errors.rb:15:in `call': The request must contain the parameter AWSAccessKeyId (Aws::EC2::Errors::MissingParameter)
The only thing that stands out on first pass is the usage of chef gem install - given that you're not installing a built binary of ChefDK there is no reason to use chef gem install over gem install. The former command is only intended for installing to the custom omnibus ruby install that comes as part of a ChefDK or Workstation package, otherwise it's functionally meaningless.
i.e.
RUN chef gem install kitchen-docker test-kitchen kitchen-ec2 berkshelf
I'll attempt to repro this later but my hunch is that it's specific to the Ruby build, a bug in the AWS SDK version, or something silly like a missing ~/.aws/config.
The only thing that stands out on first pass is the usage of
chef gem install- given that you're not installing a built binary of ChefDK there is no reason to usechef gem installovergem install. The former command is only intended for installing to the custom omnibus ruby install that comes as part of a ChefDK or Workstation package, otherwise it's functionally meaningless.i.e.
RUN chef gem install kitchen-docker test-kitchen kitchen-ec2 berkshelf
I fixed that to not use the useless chef gem install
Let's confirm that this is still occurring under the latest Kitchen, with the given repro case in the comments.
