Cloudstor plugin not enabled in newly provisioned swarm

[vesylapp]

Expected behavior

swarm-manager000000:~$ docker plugin ls

ID                  NAME                DESCRIPTION                       ENABLED
6e2f0b697f8a        cloudstor:azure     cloud storage plugin for Docker   true

Actual behavior

swarm-manager000000:~$ docker plugin ls

ID                  NAME                DESCRIPTION                       ENABLED
6e2f0b697f8a        cloudstor:azure     cloud storage plugin for Docker   false

swarm-manager000000:~$ docker plugin enable 6e2f0b697f8a

Error response from daemon: dial unix /run/docker/plugins/6e2f0b697f8a38279d0d9cca2016dfda53134c04a9481f0fc7d0698676febb98/cloudstor.sock: connect: no such file or directory

Information

• Full output of the diagnostics from "docker-diagnose" ran from one of the instance

swarm-manager000000:~$ docker-diagnose

OK hostname=swarm-manager000000 session=1525366729-Y8ByfB4c7yD0gZaJ59FGrwu8Y24l7SAv
OK hostname=swarm-manager000001 session=1525366729-Y8ByfB4c7yD0gZaJ59FGrwu8Y24l7SAv
OK hostname=swarm-manager000002 session=1525366729-Y8ByfB4c7yD0gZaJ59FGrwu8Y24l7SAv
OK hostname=swarm-worker000000 session=1525366729-Y8ByfB4c7yD0gZaJ59FGrwu8Y24l7SAv
OK hostname=swarm-worker000001 session=1525366729-Y8ByfB4c7yD0gZaJ59FGrwu8Y24l7SAv
OK hostname=swarm-worker000002 session=1525366729-Y8ByfB4c7yD0gZaJ59FGrwu8Y24l7SAv
Done requesting diagnostics.
Your diagnostics session ID is 1525366729-Y8ByfB4c7yD0gZaJ59FGrwu8Y24l7SAv
Please provide this session ID to the maintainer debugging your issue.

• More info

swarm-manager000000:~$ docker logs $(docker ps -a | grep init | cut -d' ' -f 1)

## snip ##
Connected to leader, NODE_ID=wdgclpg8pmxxsrn36ewp6epdt , SWARM_ID=443wkxmnjzj7os7s5nmzobxlj.
  Successfully joined as a Swarm Manager.
Install cloudstor ....
Install storage plugin.
No handlers could be found for logger "msrestazure.azure_active_directory".
Traceback (most recent call last):.
  File "/usr/bin/sakey.py", line 9, in <module>.
    from azure.storage.table import TableService, Entity.
ImportError: No module named table.
18.03.0-ce-azure1: Pulling from docker4x/cloudstor.
.
.[1A.[1K.[K.8bb80f59b17d: Downloading  178.1kB/17.77MB..[1B.[1A.[1K.[K.8bb80f59b17d: Downloading  2.537MB/17.77MB..[1B.[1A.[1K.[K.8bb80f59b17d: Downloading  6.371MB/17.77MB..[1B.[1A.[1K.[K.8bb80f5
9b17d: Downloading  15.33MB/17.77MB..[1B.[1A.[1K.[K.8bb80f59b17d: Downloading  17.77MB/17.77MB..[1B.[1A.[1K.[K.8bb80f59b17d: Verifying Checksum ..[1B.[1A.[1K.[K.8bb80f59b17d: Download complete ..[
1BDigest: sha256:84cb62d9fd8904f69d681af000fe82d7555944a566349c651ae7b65dc36900db.
Status: Downloaded newer image for docker4x/cloudstor:18.03.0-ce-azure1.
Error response from daemon: dial unix /run/docker/plugins/6e2f0b697f8a38279d0d9cca2016dfda53134c04a9481f0fc7d0698676febb98/cloudstor.sock: connect: no such file or directory.

Steps to reproduce the behavior

1. Go to https://docs.docker.com/docker-for-azure/
2. Create a swarm (stable channel)
3. SSH into the swarm
4. Issue docker plugin ls

[worldspawn]

I am also encountering this on a brand new, from template swarm. Attempts to enable the plugin fail with the same error.

[lidorcg]

same here

[dnataraj]

We have the same issue - just created a swarm from template (https://portal.azure.com/#create/Microsoft.Template/uri/https%3A%2F%2Fdownload.docker.com%2Fazure%2Fstable%2FDocker.tmpl), and docker plugin ls shows the plugin is disabled

swarm-manager000000:~$ docker plugin ls
ID                  NAME                DESCRIPTION                       ENABLED
3a0d83e40286        cloudstor:azure     cloud storage plugin for Docker   false

Docker logs -

Swarm initialized: current node (bfz88rwov2kwdvrrdcl7j2y9a) is now a manager.

To add a worker to this swarm, run the following command:

    docker swarm join --token SWMTKN-1-4lce75w7k2bmrv9hjbll62stswj30m16zenj3zupnhqumzqfz5-300rorjl1cusrq1wt0s0wuvmq 10.0.0.5:2377

To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions.

SWARM_ID: pnfn82z3u1k2gfd1x4ragd03b
NODE: bfz88rwov2kwdvrrdcl7j2y9a
No handlers could be found for logger "msrestazure.azure_active_directory"
successfully inserted/replaced tokens
   Leader init complete
Install cloudstor ...
Install storage plugin
No handlers could be found for logger "msrestazure.azure_active_directory"
Traceback (most recent call last):
  File "/usr/bin/sakey.py", line 9, in <module>
    from azure.storage.table import TableService, Entity
ImportError: No module named table
18.03.0-ce-azure1: Pulling from docker4x/cloudstor
8bb80f59b17d: Download complete 
Digest: sha256:84cb62d9fd8904f69d681af000fe82d7555944a566349c651ae7b65dc36900db
Status: Downloaded newer image for docker4x/cloudstor:18.03.0-ce-azure1
Error response from daemon: dial unix /run/docker/plugins/3a0d83e4028608831648bed7c03094979eb4fa651f5908f08e1351ff151b0d74/cloudstor.sock: connect: no such file or directory

EDIT: And trying to enable it -

swarm-manager000000:~$ docker plugin enable cloudstor:azure
Error response from daemon: dial unix /run/docker/plugins/3a0d83e4028608831648bed7c03094979eb4fa651f5908f08e1351ff151b0d74/cloudstor.sock: connect: no such file or directory

[gmsantos]

@dnataraj I noticed running docker plugin inspect cloudstor:azure that the file storage access key is missing. After setting the storage key in every node it worked.

docker plugin set cloudstor:azure AZURE_STORAGE_ACCOUNT_KEY=<your_access_key>
docker plugin enable cloudstor:azure

docker plugin inspect tells you what is the correct storage in use.

[alexsandro-xpt]

Same problem here! I found It appeared that we had a firewall between the nodes and azure files at https://forums.docker.com/t/cannot-enable-cloudstor-plugin/38953 but I don't known what firewall is that.

[walmon]

Hello! No answer on this one? Anybody could fix the issue?

[walmon]

I found a fix here: https://github.com/docker/for-azure/issues/55#issuecomment-385687421

docker plugin rm cloudstor:azure || true &&
    docker ps -a | \
    grep init-azure | \
    ( read ID OTHER; docker restart $ID; docker exec $ID sed -ire 's,from azure.storage.table ,from azure.cosmosdb.table ,' /usr/bin/azureleader.py; docker exec $ID sed -ire 's,from azure.storage.table ,from azure.cosmosdb.table ,' /usr/bin/sakey.py ) &&
    docker logs -f $(docker ps -a | grep init-azure | awk '{print $1}')

[alexsandro-xpt]

Nick @walmon I test it in another cluster but not in Docker for Azure Swarm.

[walmon]

Yeah, the way we worked around this bug:

@gmsantos was right, for some reason the template doesn't set the 'AZURE_STORAGE_ACCOUNT_KEY' for the plugin's disk.

What we did: On the manager, run:

docker plugin rm cloudstor:azure || true &&
    docker ps -a | \
    grep init-azure | \
    ( read ID OTHER; docker restart $ID; docker exec $ID sed -ire 's,from azure.storage.table ,from azure.cosmosdb.table ,' /usr/bin/azureleader.py; docker exec $ID sed -ire 's,from azure.storage.table ,from azure.cosmosdb.table ,' /usr/bin/sakey.py ) &&
    docker logs -f $(docker ps -a | grep init-azure | awk '{print $1}')

It will setup the disk on your manager.

And then on the workers run the script that @gmsantos posted here (you need to connect to them by forwarding your ssh when connecting to the worker, if you don't know how, more details at the end). First, you will need the access key, so you can go to the azure console and get it from there, or you can copy it from doing a:

 docker plugin ls
 docker plugin inspect <the id of the plugin>

After docker plugin inspect you get something like this:

docker plugin set cloudstor:azure AZURE_STORAGE_ACCOUNT_KEY=<your_access_key>
docker plugin enable cloudstor:azure

And by applying that to every worker node, you can access the same disk.

---

To be able to connect to the worker nodes:

• On your computer run: ssh-add -K
• While connecting to the Swarm master: ssh **-A** -i <the path to your private> -p 50000 docker@<ip> pay extra attention to the -A param.
• Then, connect to the worker nodes. To see their host names run docker node ls, proceed with ssh -A docker@<name of the worker>

[swoldanski]

I have another workaround for Docker4Azure Swarm:

• login to your Swarm manager

• inspect plugin and notice which storage account is used docker plugin inspect cloudstor:azure | grep AZURE_STORAGE_ACCOUNT

• look for AZURE_STORAGE_ACCOUNT key

• login to Azure Portal, find above listed storage account and write down one of Access keys

• on Swarm Manager node create global service with your key value

  docker service create --mode global --restart-condition none --mount type=bind,source=/var/run/docker.sock,target=/var/run/docker.sock --name enable-cloudstor-azure-plugin docker:latest sh -c "echo Set up and enable cloudstor:azure plugin. Be patient... && sleep 120 && docker plugin set cloudstor:azure AZURE_STORAGE_ACCOUNT_KEY=PUT-YOUR-AZURE-STORAGE-KEY-HERE && docker plugin enable cloudstor:azure && docker plugin ls && echo Done."

This global service will set up and enable cloudstor:azure plugin on every new node. For already existing node you have to wait for around 2 minutes and the plugin will be enabled. It is a very convenient way when you are using autoscaling and you add or remove nodes on demand.

[alexvy86]

With regards to the Error response from daemon: dial unix /run/docker/plugins/<plugin_id>/cloudstor.sock: connect: no such file or directory error, in my case it was happening because the storage account had the "Secure Transfer Enabled" setting turned on, forcing connections to be through https. As soon as I turned it off, I was immediately able to install the driver without errors. Looks the cloudstor plugin uses an http to reach the storage account.