Support for EC2 instance profiles when running Packer on an EC2 instance

[darrenob]

I'm running Packer on an EC2 instance (triggered by Jenkins but that's irrelevant)

The instance has access to the s3 bucket because it is assigned an instance profile. However I get an error when running Packer that vagrant-s3 can not find the AWS credentials in the env vars or credentials files. This is because there ARE none.

For example: I can list the bucket:

bash-4.2$ aws s3 ls s3://whatever-bucket/my-box/
                           PRE 0.1.0/
                           PRE 0.2.0/
                           PRE 0.3.0/
                           PRE 0.4.0/
                           PRE 0.5.0/
                           PRE 0.6.0/
                           PRE 0.7.0/
2018-03-07 16:20:22       1893 my-box.json

But when I run Packer I get this

bash-4.2$ packer build -only=amazon-ebs "my-box-build.json"
amazon-ebs output will be in this color.

1 error(s) occurred:

* Unable to access the bucket whatever-bucket:
EnvAccessKeyNotFound: AWS_ACCESS_KEY_ID or AWS_ACCESS_KEY not found in environment
Make sure your credentials are valid and have sufficient permissions

Would it be possible to check for the credentials of the EC2 instance and use them if they exist?

[Jalle19]

Can you paste your Packer template? You can replace any sensitive information with dummy values.

[darrenob]

Sure, here it is:

{
    "variables": {
        "vagrant_box_name": "{{ env `vagrant_box_name`}}",
        "aws_ami_name": "app-{{timestamp}}",
        "environment_aws": "{{ env `environment_aws`}}",
        "environment_virtualbox": "{{ env `environment_virtualbox`}}",
        "ansible_inventory_file_aws": "{{env `ansible_inventory_file_aws`}}",
        "ansible_inventory_file_virtualbox": "{{env `ansible_inventory_file_virtualbox`}}",
        "vaultpass": "{{ env `vaultpass`}}",
        "ansible_playbook_aws": "{{env `ansible_playbook_aws`}}",
        "ansible_playbook_virtualbox": "{{env `ansible_playbook_virtualbox`}}",
        "role": "{{ env `role`}}",
        "region": "{{ env `region`}}",
        "Type": "{{ env `Type`}}",
        "aws_access_key": "{{env `AWS_ACCESS_KEY_ID`}}",
        "aws_secret_key": "{{env `AWS_SECRET_ACCESS_KEY`}}",
        "aws_session_token": "{{env `AWS_SESSION_TOKEN`}}",
        "virtualbox_disk_size": "{{env `virtualbox_disk_size`}}"
    },

    "builders": [{
        "type": "virtualbox-iso",
        "virtualbox_version_file": ".vbox_version",
        "headless": false,
        "keep_registered": false,
        "guest_os_type": "RedHat_64",
        "disk_size": "{{ user `virtualbox_disk_size` }}",
        "output_directory": "virtualbox/build-{{ user `vagrant_box_name` }}-{{timestamp}}",

        "iso_url": "http://mirror.airenetworks.es/CentOS/7/isos/x86_64/CentOS-7-x86_64-Minimal-1708.iso",
        "iso_checksum": "bba314624956961a2ea31dd460cd860a77911c1e0a56e4820a12b9c5dad363f5",
        "iso_checksum_type": "sha256",
        "http_directory": "virtualbox",
        "boot_command": [
          "<tab> net.ifnames=0 biosdevname=0 text ks=http://{{ .HTTPIP }}:{{ .HTTPPort }}/kickstart-http/centos-7.0-x86_64/ks.cfg<enter><wait>"
        ],
        "boot_wait": "4s",

        "guest_additions_path": "VBoxGuestAdditions_{{.Version}}.iso",

        "ssh_username": "vagrant",
        "ssh_password": "vagrant",
        "ssh_port": 22,
        "ssh_wait_timeout": "10000s",

        "vboxmanage": [
          ["modifyvm", "{{.Name}}", "--memory", "1024"],
          ["modifyvm", "{{.Name}}", "--cpus", "1"]
        ],

        "shutdown_command": "echo 'vagrant'|sudo -S /sbin/halt -h -p",
        "post_shutdown_delay": "2m"
  },
  {
        "type": "amazon-ebs",
        "region": "{{user `region`}}",
        "instance_type": "t2.micro",
        "user_data_file": "cloud-init.yaml",
        "ssh_pty": "true",
        "ssh_username": "centos",
        "ami_name": "{{ user `environment_aws` }}-{{ user `role` }}-{{timestamp}}",
        "ami_description": "Customized Amazon Linux",
        "associate_public_ip_address": "false",
        "iam_instance_profile": "r_jenkins_packer",
        "tags": {
          "environment": "{{ user `environment_aws` }}",
          "role": "{{ user `role` }}",
          "Type": "{{ user `Type` }}"
        },
        "run_tags": {
          "Name": "packer-{{ user `environment_aws` }}-{{ user `role` }}-{{timestamp}}",
          "environment": "{{ user `environment_aws` }}",
          "role": "{{ user `role` }}",
          "Type": "{{ user `Type` }}"
        },
        "launch_block_device_mappings": [
          {
            "device_name": "/dev/sda1",
            "volume_size": 15,
            "volume_type": "gp2",
            "delete_on_termination": true
          }
        ],
        "ami_users":[
          "9486356xxxxxxx",
          "0806905xxxxxxx",
          "7905975xxxxxxx",
          "8875424xxxxxxx"
        ],
        "source_ami": "ami-061b1560"
    }
  ],

    "provisioners": [
      {
          "type": "shell",
          "only": ["virtualbox-iso"],
          "script": "virtualbox/virtualbox-init.sh",
          "environment_vars": [
            "AWS_ACCESS_KEY_ID={{ user `aws_access_key` }}",
            "AWS_SECRET_ACCESS_KEY={{ user `aws_secret_key` }}",
            "AWS_SESSION_TOKEN={{ user `aws_session_token` }}"
          ],
          "execute_command": "echo 'vagrant' | sudo -S sh -c '{{ .Vars }} {{ .Path }}'"
      },
      {
          "type": "shell",
          "only": ["amazon-ebs"],
          "inline": [
              "echo starting provisioner amazon-ebs as user `whoami`",
              "/root/Automation/sites/{{ user `environment_aws` }}/ec2.py --refresh",
              "env",
              "IP_ADDRESS=`hostname -I`",
              "ANSIBLE_CMD=\"ansible-playbook -i /root/Automation/sites/{{ user `environment_aws` }}/{{ user `ansible_inventory_file_aws` }} /root/Automation/{{ user `ansible_playbook_aws` }} -e'packer_build=true' -c local --vault-password-file=/root/.vaultpass --limit $IP_ADDRESS -vv\"",
              "ls -l /root/Automation/sites/aws-staging/",
              "echo Running ansible command: \"$ANSIBLE_CMD\"",
              "$ANSIBLE_CMD"
          ],
          "pause_before": "140s",
          "execute_command": "echo 'centos' | sudo -S sh -c '{{ .Vars }} {{ .Path }}'"
      },
      {
          "type": "shell",
          "only": ["virtualbox-iso"],
          "inline": [
              "sudo -E echo starting provisioner virtualbox-iso as user `whoami`",
              "sudo -E echo Using Ansible inventory /root/Automation/sites/{{ user `environment_virtualbox` }}/{{ user `ansible_inventory_file_virtualbox` }}",
              "sudo -E env",
              "sudo -E ansible-playbook -i /root/Automation/sites/{{ user `environment_virtualbox` }}/{{ user `ansible_inventory_file_virtualbox` }} /root/Automation/{{ user `ansible_playbook_virtualbox` }} -e'packer_build=true' -c local --vault-password-file=/root/.vaultpass --limit `hostname -I` -vv"
          ],
          "environment_vars": [
            "AWS_ACCESS_KEY_ID={{ user `aws_access_key` }}",
            "AWS_SECRET_ACCESS_KEY={{ user `aws_secret_key` }}",
            "AWS_SESSION_TOKEN={{ user `aws_session_token` }}"
          ],
          "pause_before": "10s"
      }
    ],
    "post-processors": [
      [
        {
          "type": "vagrant",
          "keep_input_artifact": false,
           "output": "virtualbox/boxes/{{ user `vagrant_box_name` }}.box"
        },
        {
          "type": "vagrant-s3",
          "region": "eu-west-1",
          "bucket": "my-comapnys-virtual-machines",
          "manifest": "vagrant/{{ user `vagrant_box_name` }}/{{ user `vagrant_box_name` }}.json",
          "box_dir": "vagrant/{{ user `vagrant_box_name` }}",
          "box_name": "vendo/{{ user `vagrant_box_name` }}"
        }
      ]
    ]

}

[Jalle19]

Can you try removing this line: https://github.com/lmars/packer-post-processor-vagrant-s3/blob/master/post-processor.go#L125 ?

[Jalle19]

Now that I look at this again, I think it fails because in your template you specify AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY. Try removing every configuration related to AWS credentials and see if it works then.

[disaac]

I believe there should be support added for the use of the credentials.ec2rolecreds

• Here is an example of trying in succession based on err when calling credentials.Get()
• Here is a workaround for storing the credentials in exported environment variables so that the environment variables will be used in the existing logic.

  # Used jq and curl to export the environment variables prior to executing packer so that the instance profile credentials are present there working around the lack of support for ec2rolecreds
  this_role_name=$(curl -s http://169.254.169.254/latest/meta-data/iam/security-credentials/)
  for temp_creds in $(curl -s http://169.254.169.254/latest/meta-data/iam/security-credentials/${this_role_name} | jq -r '{"AWS_ACCESS_KEY_ID": .AccessKeyId,"AWS_SECRET_ACCESS_KEY": .SecretAccessKey,"AWS_SESSION_TOKEN": .Token} | to_entries | map("\(.key)=\(.value|tostring)")|.[]')
  do
    export "${temp_creds?}" 
  done