AWS SpotFleet support - very basic

[fprochazka]

I know this repo is in maintenance - this is here for other users that might want to pick this up. If this interests the maintainers of this repo, I'd be willing to invest some more time to make this mergeable (this was my first encounter with Go).

---

Problem: Basic aws spot instances cannot automatically (to my knowledge) use different availability zone, based on what spot pools are depleted.

Solution: Introduce basic support for AWS SpotFleet. This is a minimal implementation, that solves the problem for me, but anyone can use it as inspiration or directly.

I will not be supporting this fork. Use at your own risk. I will maintain the fork only for my own needs.

Compiling my fork:

• setup the repo like the Contributing guide says
• cd src/github.com/docker/machine
• make sure you know how to compile the main repo, before continuing
• git remote add fprochazka [email protected]:fprochazka/docker-machine.git
• git fetch fprochazka
• git checkout fp-allow-multiple-availability-zones
• make build

I have build & tested this with

• go version go1.12.4 linux/amd64
• dep version devel (installed today)

Usage: basic

I'm using the docker-machine for my gitlab runners.

1. test in EC2 Console, that you're able to request and cancel SpotFleet (this should initialize the required role, if your account doesn't already have it)
2. find out your vpc-id and all the subnet-id's of availability zones, that you want to use.
3. find out your "IAM fleet role" (the EC2 SpotFleet dialog shows it to you)
4. profit

docker-machine create --driver amazonec2 \
      --amazonec2-access-key "AKIA" \
      --amazonec2-secret-key "secret" \
      --amazonec2-region "eu-central-1" \
      --amazonec2-vpc-id "vpc-abc123" \
      --amazonec2-subnet-id "subnet-abc1" \
      --amazonec2-subnet-id "subnet-abc2" \
      --amazonec2-subnet-id "subnet-abc3" \
      --amazonec2-instance-type "t3.small" \
      --amazonec2-tags "Name,Gitlab CI Runner,user:Application,Gitlab,user:Stack,production,gitlab,true,gitlab-runner-autoscale,true" \
      --amazonec2-security-group "gitlab-runners" \
      --amazonec2-security-group-readonly \
      --amazonec2-request-spot-fleet \
      --amazonec2-iam-fleet-role "arn:aws:iam::123:role/aws-ec2-spot-fleet-tagging-role" \
      --amazonec2-use-private-address \
      spot-fleet-test

Usage: with gitlab-runner

  [runners.machine]
    IdleCount = 0
    IdleTime = 1800
    OffPeakIdleCount = 0
    OffPeakIdleTime = 0
    MachineDriver = "amazonec2"
    MachineName = "gitlab-runner-micro-%s"
    MachineOptions = [
      "amazonec2-access-key=AKIA",
      "amazonec2-secret-key=secret",
      "amazonec2-instance-type=t3.micro",
      "amazonec2-tags=Name,Gitlab CI Runner,user:Application,Gitlab,user:Stack,production,gitlab,true,gitlab-runner-autoscale,true",
      "amazonec2-region=eu-central-1", 
      "amazonec2-vpc-id=vpc-abc123", 
      "amazonec2-subnet-id=subnet-abc1", 
      "amazonec2-subnet-id=subnet-abc2",
      "amazonec2-subnet-id=subnet-abc3",
      "amazonec2-use-private-address=true",
      "amazonec2-security-group=gitlab-runners", 
      "amazonec2-security-group-readonly=true",
      "amazonec2-request-spot-fleet=true",
      "amazonec2-iam-fleet-role=arn:aws:iam::123:role/aws-ec2-spot-fleet-tagging-role"
    ]

[GordonTheTurtle]

Please sign your commits following these rules: https://github.com/moby/moby/blob/master/CONTRIBUTING.md#sign-your-work The easiest way to do this is to amend the last commit:

$ git clone -b "fp-allow-multiple-availability-zones" [email protected]:fprochazka/docker-machine.git somewhere
$ cd somewhere
$ git rebase -i HEAD~842359068536
editor opens
change each 'pick' to 'edit'
save the file and quit
$ git commit --amend -s --no-edit
$ git rebase --continue # and repeat the amend for each commit
$ git push -f

Amending updates the existing PR. You DO NOT need to open a new one.

[declension]

Any progress on this? It'd be great to have Spot Fleet support!

[fprochazka]

@declension what kind of progress do you expect? :) The maintainers already said no more features.

[declension]

The maintainers already said no more features.

Oh :disappointed: Can't find any evidence of this on the home page or contributing guides though, and haven't followed.

Could this become a plugin or even fork, somehow?

[fprochazka]

@declension https://github.com/docker/machine/issues/4537

You actually can use my fork with this feature very easily, I've tried to describe it in the description. I'm using it on our production. But at your own risk ;-)

[mohag]

It might be useful to get into rancher/machine, which is used by Rancher (and seems to be getting more features than this...) (If they are willing to diverge from upstream...)

[ghost]

I tried this on gitlab-runner v12.9.1. but dose not work.

• spot fleet request was created

• But Docker machine provisioning fails on SSH connection

• log text

Checking for jobs... received                       job=8766 repo_url=http://xxxx/xxxx.git runner=bdc652f8
Creating CA: /root/.docker/machine/certs/ca.pem     driver=amazonec2 name=runner-bdc652f8-runner-spottest-1592280235-612023db operation=create
Creating client certificate: /root/.docker/machine/certs/cert.pem  driver=amazonec2 name=runner-bdc652f8-runner-spottest-1592280235-612023db operation=create
Running pre-create checks...                        driver=amazonec2 name=runner-bdc652f8-runner-spottest-1592280235-612023db operation=create
Creating machine...                                 driver=amazonec2 name=runner-bdc652f8-runner-spottest-1592280235-612023db operation=create
(runner-bdc652f8-runner-spottest-1592280235-612023db) Launching instance...  driver=amazonec2 name=runner-bdc652f8-runner-spottest-1592280235-612023db operation=create
(runner-bdc652f8-runner-spottest-1592280235-612023db) Created spot fleet request sfr-a64862c1-dc27-44dc-a2ee-153cbb5e9598 ...  driver=amazonec2 name=runner-bdc652f8-runner-spottest-1592280235-612023db operation=create
WARNING: Failed to parse "X-GitLab-Trace-Update-Interval" header  error=strconv.Atoi: parsing "": invalid syntax header-value= job=8766 runner=bdc652f8

If you know anything, please

[fprochazka]

@kubokkey I have the same warnings as you, but my runners are working fine, sorry I have no idea how to fix this...

Although, I was considering forking the Gitlab-maintained docker-machine and adding my changes there, because last time I've tried rebasing this and updating dependencies my runners stopped working and I had to revert the binary.

[krzysztof-miemiec]

If you came here because you'd like to find a relatively simple way to create an instance using spot fleet, check out custom docker-machine-driver-terraform and usage example for Gitlab runner. You can also check out this PR: https://github.com/npalm/terraform-aws-gitlab-runner/pull/249. :wink: