Changing backends orphans running machines

[stealthybox]

For this to behave any differently, we would either:

check for equivalently named/labelled machines in every backend (could be slow with a cluster or other networked backend)

or

store some local state about what backends are being used for which groups or machines. (adds imperative state / could be really buggy)

It might just be better to admit that footloose configs are declarative, but footloose commands for machine-management are imperative. I believe docker-compose has some similar behavior limitations.

Here's a detailed example:

---

If I create two groups of machines using the docker backend like so:

cluster:
  name: cluster
  privateKey: cluster-key
machines:
- count: 1
  spec:
    image: quay.io/footloose/ubuntu18.04
    name: a-%d
- count: 1
  spec:
    image: quay.io/footloose/ubuntu18.04
    name: b-%d

footloose create
INFO[0000] Docker Image: quay.io/footloose/ubuntu18.04 present locally 
INFO[0000] Docker Image: quay.io/footloose/ubuntu18.04 present locally 
INFO[0000] Creating machine: cluster-a-0 ...            
INFO[0001] Creating machine: cluster-b-0 ... 

footloose show
NAME          HOSTNAME   PORTS   IP           IMAGE                           CMD          STATE     BACKEND
cluster-a-0   a-0                172.17.0.2   quay.io/footloose/ubuntu18.04   /sbin/init   Running   
cluster-b-0   b-0                172.17.0.3   quay.io/footloose/ubuntu18.04   /sbin/init   Running   

Then I edit one of the groups to use the ignite backend:

cluster:
  name: cluster
  privateKey: cluster-key
machines:
- count: 1
  spec:
    image: quay.io/footloose/ubuntu18.04
    name: a-%d
- count: 1
  spec:
    image: quay.io/footloose/ubuntu18.04
    name: b-%d
    backend: ignite

As soon as I've edited the file, footloose will orphan the machines running for that spec:

sudo footloose show
NAME          HOSTNAME   PORTS   IP           IMAGE                           CMD          STATE         BACKEND
cluster-a-0   a-0                172.17.0.2   quay.io/footloose/ubuntu18.04   /sbin/init   Running       
cluster-b-0   b-0                             quay.io/footloose/ubuntu18.04                Not created   ignite
# notice the state for b-0 is 'Not created'

Let's try creating again in this state:

sudo footloose create
INFO[0000] Docker Image: quay.io/footloose/ubuntu18.04 present locally 
INFO[0000] Docker Image: quay.io/footloose/ubuntu18.04 present locally 
INFO[0000] Creating machine: cluster-a-0 ...            
INFO[0000] Machine cluster-a-0 is already created...    
INFO[0000] Creating machine: cluster-b-0 ...            
# b-0 is re-created as an ignite vm -- the running docker container is ignored

# now we have 2 containers (1 orphaned) + 1 ignite vm
docker ps
CONTAINER ID        IMAGE                           COMMAND                  CREATED             STATUS              PORTS               NAMES
919da6ca901c        weaveworks/ignite:v0.5.2        "/usr/local/bin/igni…"   28 seconds ago      Up 27 seconds                           ignite-8abc2902767aff2b
ea479c46c743        quay.io/footloose/ubuntu18.04   "/sbin/init"             4 minutes ago       Up 4 minutes        22/tcp              cluster-b-0
6c83e1eb0aa9        quay.io/footloose/ubuntu18.04   "/sbin/init"             4 minutes ago       Up 4 minutes        22/tcp              cluster-a-0

ignite vm
VM ID			IMAGE					KERNEL					SIZE	CPUS	MEMORY		CREATED	STATUS	IPS		PORTS	NAME
8abc2902767aff2b	quay.io/footloose/ubuntu18.04:latest	weaveworks/ignite-kernel:4.19.47	4.0 GB	2	1024.0 MB	38s ago	Up 38s	172.17.0.4		cluster-b-0

and deleting everything:

sudo footloose delete
INFO[0000] Machine cluster-a-0 is started, stopping and deleting machine... 
INFO[0000] Deleting machine: 919da6ca901c ...           

ignite vm
VM ID	IMAGE	KERNEL	SIZE	CPUS	MEMORY	CREATED	STATUS	IPS	PORTS	NAME

docker ps
CONTAINER ID        IMAGE                           COMMAND             CREATED             STATUS              PORTS               NAMES
ea479c46c743        quay.io/footloose/ubuntu18.04   "/sbin/init"        5 minutes ago       Up 5 minutes        22/tcp              cluster-b-0
# deleting everything leaves behind the orphaned container

If we switch back to our first file that was only using docker, we can delete the orphaned container:

footloose delete
INFO[0000] Machine cluster-a-0 hasn't been created...   
INFO[0000] Machine cluster-b-0 is started, stopping and deleting machine... 

---

Should we do anything to change this?

Again, this may be fine for the constraints of our design, but we should document this behavior. TODO: Find best spot in the docs for this info.

[dlespiau]

Thanks for the well written issue!

Currently footloose only acts on a single cluster, the one described in the config file and that's indeed the main limitation here.

I can see a world where footloose queries back some state from either docker or a CRI runtime (the CRI interface has some provision to store metadata with each container). This way we would be able to list the clusters that have been created in the past and act on them.

This seems like a desirable behaviour in the future!