[BUG] `docker compose config --ouput=json` omits `x-*` fields - generally some differences in marshalled YAML vs. JSON forms

[Code0x58]

Description

The JSON config output omits x-* entries. Diff from the actual generated JSON to the JSON-ified YAML of example below:

--- actual.json
+++ expect.json
@@ -2,18 +2,16 @@
     "name": "COMPOSE_PROJECT_NAME",
     "networks": {
         "default": {
-            "name": "COMPOSE_PROJECT_NAME_default",
-            "ipam": {}
+            "name": "COMPOSE_PROJECT_NAME_default"
         }
     },
     "services": {
         "some_service": {
-            "command": null,
-            "entrypoint": null,
             "image": "ubuntu:latest",
             "networks": {
                 "default": null
-            }
+            },
+            "x-something": "hello"
         }
     }
 }

My main issue is the absence of the x-* fields, but can see there's at least some additional difference as the default network's ipam, command, and entrypoint entries are omitted in YAML whereas it is an empty objects/nulls in the JSON.

I imagine in an ideal world that diff <(docker compose config --output=yaml | yq -output-format=json) <(docker compose config --output=json) would always have a 0 exit code/no diff, and that x-* fields would be included.

Steps To Reproduce

# docker-compose.yml
version: "4"
services:
  some_service:
    image: ubuntu:latest
    x-something: hello

docker compose config output

name: COMPOSE_PROJECT_NAME
services:
  some_service:
    image: ubuntu:latest
    networks:
      default: null
    x-something: hello
networks:
  default:
    name: COMPOSE_PROJECT_NAME_default

docker compose config --format=json | python3 -m json.tool output

{
    "name": "COMPOSE_PROJECT_NAME",
    "networks": {
        "default": {
            "name": "COMPOSE_PROJECT_NAME_default",
            "ipam": {}
        }
    },
    "services": {
        "some_service": {
            "command": null,
            "entrypoint": null,
            "image": "ubuntu:latest",
            "networks": {
                "default": null
            }
        }
    }
}

Compose Version

Docker Compose version v2.24.5

[ndeloof]

This issue is due to limitation in the golang json binding library: this one doesn't ditsinguish empty vs null (so "ipam": {} missing from output) and doesn't allow to restore x-* as inlined attributes (https://github.com/golang/go/issues/6213)

We should be able soon-ish to workaround this limitation as compose-go is now able to manage the compose file parsing without having to convert input into go structs.

[ndeloof]

Can you please let us know which usage you have for compose config output as a json stream ?

[Code0x58]

Thanks @ndeloof.

I'm using docker-compose.yml manifests to store additional configuration/information that is required for the deployment environment I'm working with, x-* feels like a natural place for this, particularly as I use things like service.extends. Its fairly similar in concept to how extra config is added for the ecs docker context, but not built in with go. This allows me to have things like compose['x-my-manifest-extension'] and compose['services']['some-service']['x-my-service-extension'].

In my case, I'm using python and prefer to avoid extra dependencies where possible, so am relying on the built in json parsing module. I've worked around the issue for now by piping the YAML output through yq --output-format=json.

[ndeloof]

fixed by https://github.com/docker/compose/pull/11556