Swarm mode Preferences bug

[AlexDyakonov]

Before you start please confirm the following.

• [x] Yes, I've searched similar issues on GitHub.
• [x] Yes, I've checked whether this issue is covered in the Portainer documentation or knowledge base.

Problem Description

I am running Portainer Community Edition 2.27.3 LTS with 3 nodes, each tagged as prod, dev, and services. For example, in a production environment, I use the following configuration for deployment:

x-common-deploy: &common-deploy
  restart_policy:
    condition: on-failure
  mode: replicated
  placement:
    constraints:
      - "node.labels.TAG==prod"
  update_config:
    parallelism: 1
    delay: 5s
    order: start-first

This configuration ensures that the service is deployed on the correct node, specifically on a node with the tag prod. However, I need the service to automatically be deployed on other nodes in case the prod node goes down. According to the Docker documentation for deploy, it is possible to use both constraints for restrictions and preferences for preferences in the deployment configuration.

Expected Behavior

I expect Portainer to allow both constraints and preferences to be used together in a Docker Compose file. This would allow me to define the primary placement constraint on the prod node and a fallback preference for other nodes in case the prod node is unavailable, allowing for automatic failover and proper distribution.

Actual Behavior

When I attempt to use both constraints and preferences in the placement section of my Docker Compose file, Portainer loses the connection to the Docker environment, and I see the following error:

Failure
Unable to retrieve stacks

Additionally, I get this error:

Failed loading environment
Unable to connect to the Docker environment

Despite these errors, I am still able to manage the Docker environment from the manager node, but cannot retrieve or deploy stacks until I resolve the issue.

Steps to Reproduce

1. Create a Docker Compose file with both constraints and preferences under the placement section.
2. Use node.labels.TAG==prod for the constraint and spread: "node.labels.TAG" for preferences.
3. Deploy the stack using Portainer.
4. Observe that the environment becomes disconnected with the error message mentioned above.
5. Disable the stack and reboot Docker to temporarily fix the issue.

Portainer logs or screenshots

No response

Portainer version

2.27.3

Portainer Edition

Community Edition (CE)

Platform and Version

Swarm

OS and Architecture

Ubuntu 22.04

Browser

No response

What command did you use to deploy Portainer?

services:
  agent:
    image: portainer/agent
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - /var/lib/docker/volumes:/var/lib/docker/volumes
    networks:
      - traefik
    deploy:
      mode: global
      placement:
        constraints: [node.platform.os == linux]

  portainer:
    image: portainer/portainer-ce:latest
    command: -H tcp://tasks.agent:9001 --tlsskipverify
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - ./portainer_data:/data
      - ./portainer_logs:/portainer_logs
    environment:
      - LOG_LEVEL=info
      - LOGS_LOCATION=/portainer_logs/portainer.log
    logging:
      driver: "json-file"
      options:
        max-size: "10m"
        max-file: "3"
    networks:
      - traefik
    deploy:
      labels:
        - traefik.enable=true
        - traefik.http.routers.portainer.entrypoints=https
        - traefik.http.routers.portainer.rule=Host(``)
        - traefik.http.routers.portainer.tls.certresolver=letsEncrypt
        - traefik.http.services.portainer.loadbalancer.server.port=9000
      mode: replicated
      placement:
        constraints:
          - "node.labels.TAG==services"

volumes:
  portainer_data:
    driver: local

networks:
  traefik:
    name: traefik
    external: true

Additional Information

No response

[AlexDyakonov]

Also tried this:

  placement:
    preferences:
      - spread: "node.labels.TAG==prod"
      - spread: "node.labels.TAG==dev"
      - spread: "node.labels.TAG==services"

docs I used: https://docs.docker.com/reference/cli/docker/service/create/#placement-pref

[Nick-Portainer]

It sounds like you are having a Swarm Overlay network issue, can you please try a service update of your agents and see if that helps?

For example:

docker service update --force portainer_agent