Using extends I got duplicated env variables in the env hash

[awsmsrc]

Hey folks,

really looking forward to using libcompose. I noticed one issue whereby env variables were duplicated incorrectly in the env hash when using extends. This case also seems to be missing from merge_test.go.

Would love to look into it further but I'm a little busy right now so i provided my test program in case any one wants to repro and look into it

base.yml

web:
  image: ubuntu
  evironment:
    - FOO=BAR

docker-compose.yml

web:
  extends: ./base.yml
  image: ubuntu
  links:
    - redis
  environment:
    - COMPUTERS=COOL

redis:
  image: redis

test program

package main

import (
    "fmt"

    "github.com/docker/libcompose/docker"
    "github.com/docker/libcompose/project"
)

func main() {
    project, err := docker.NewProject(&docker.Context{
        Context: project.Context{
            ComposeFiles: []string{"docker-compose.yml"},
            ProjectName:  "my-compose",
        },
    })

    if err != nil {
        fmt.Println(err.Error())
        return
    }

    err = project.Parse()
    if err != nil {
        fmt.Println(err.Error())
        return
    }

    for _, c := range project.Configs {
        fmt.Printf("%#v\n", c)
        fmt.Printf("%#v\n", c.Environment)
    }

}

output

&project.ServiceConfig{Build:"", CapAdd:[]string(nil), CapDrop:[]string(nil), CgroupParent:"", CPUQuota:0, CPUSet:"", CPUShares:0, Command:project.Command{parts:[]string(nil)}, ContainerName:"", Devices:[]string(nil), DNS:project.Stringorslice{parts:[]string(nil)}, DNSSearch:project.Stringorslice{parts:[]string(nil)}, Dockerfile:"", DomainName:"", Entrypoint:project.Command{parts:[]string(nil)}, EnvFile:project.Stringorslice{parts:[]string(nil)}, Environment:project.MaporEqualSlice{parts:[]string(nil)}, Hostname:"", Image:"redis", Labels:project.SliceorMap{parts:map[string]string{}}, Links:project.MaporColonSlice{parts:[]string(nil)}, LogDriver:"", MacAddress:"", MemLimit:0, MemSwapLimit:0, Name:"", Net:"", Pid:"", Uts:"", Ipc:"", Ports:[]string(nil), Privileged:false, Restart:"", ReadOnly:false, StdinOpen:false, SecurityOpt:[]string(nil), Tty:false, User:"", VolumeDriver:"", Volumes:[]string(nil), VolumesFrom:[]string(nil), WorkingDir:"", Expose:[]string(nil), ExternalLinks:[]string(nil), LogOpt:map[string]string(nil), ExtraHosts:[]string(nil), Ulimits:project.Ulimits{Elements:[]project.Ulimit(nil)}}
project.MaporEqualSlice{parts:[]string(nil)}
&project.ServiceConfig{Build:"", CapAdd:[]string(nil), CapDrop:[]string(nil), CgroupParent:"", CPUQuota:0, CPUSet:"", CPUShares:0, Command:project.Command{parts:[]string(nil)}, ContainerName:"", Devices:[]string(nil), DNS:project.Stringorslice{parts:[]string(nil)}, DNSSearch:project.Stringorslice{parts:[]string(nil)}, Dockerfile:"", DomainName:"", Entrypoint:project.Command{parts:[]string(nil)}, EnvFile:project.Stringorslice{parts:[]string(nil)}, Environment:project.MaporEqualSlice{parts:[]string{"COMPUTERS=COOL", "COMPUTERS=COOL"}}, Hostname:"", Image:"ubuntu", Labels:project.SliceorMap{parts:map[string]string{}}, Links:project.MaporColonSlice{parts:[]string{"redis", "redis"}}, LogDriver:"", MacAddress:"", MemLimit:0, MemSwapLimit:0, Name:"", Net:"", Pid:"", Uts:"", Ipc:"", Ports:[]string(nil), Privileged:false, Restart:"", ReadOnly:false, StdinOpen:false, SecurityOpt:[]string(nil), Tty:false, User:"", VolumeDriver:"", Volumes:[]string(nil), VolumesFrom:[]string(nil), WorkingDir:"", Expose:[]string(nil), ExternalLinks:[]string(nil), LogOpt:map[string]string(nil), ExtraHosts:[]string(nil), Ulimits:project.Ulimits{Elements:[]project.Ulimit(nil)}}
project.MaporEqualSlice{parts:[]string{"COMPUTERS=COOL", "COMPUTERS=COOL"}}

[vdemeester]

Hi @awsmsrc, thanks for the report — will try to investigate and fix it for the 0.2.0 release :wink:

[vdemeester]

After digging a bit this issue, it's because docker.NewProject(…) already calls the Parse method on the project. This means running Parse is not idenpotent..

[awsmsrc]

So this will work as expected if I remove that call?

[vdemeester]

@awsmsrc I would guess so, but I'm gonna still list this as a bug as it feels weird that calling twice Parse append environment variables.

[awsmsrc]

yeah and it looks like it's still not getting the env var from the extended file

[vdemeester]

ah, and that too :sweat_smile:

[vdemeester]

@awsmsrc hum actually looking back at the documentation, the extends should look more like

webapp:
  extends:
    file: ./base.yml
    service: webapp

[awsmsrc]

Right you are, in that case there should probably be some error handling there?

[joshwget]

@awsmsrc Are you running the latest version? Validation was merged recently, and it should (hopefully) catch this.

[awsmsrc]

not updated since adding this ticket

[awsmsrc]

it looks like my bad extends format is still accepted

[joshwget]

It looks like a string is actually valid for the extends key (https://github.com/docker/libcompose/blob/master/script/config_schema_v1.json#L61). I think it's meant to be the name of a service in the same file, and not the name of a separate file as you're expecting.

[awsmsrc]

yip, but should there be an error if that service is not found?

[joshwget]

Yeah, that's certainly a bug!

[awsmsrc]

well my report was a little off but im glad we've found a couple things here :) this project is gonna be very useful for me

[joshwget]

Actually, the absence of an error isn't the full issue either. It looks like libcompose doesn't support passing service names directly to extends at all. Docker Compose does, but according to their docs the extends key must be a map. I'll try to figure out what the plan is for Docker Compose before implementing it.

[vdemeester]

@joshwget @awsmsrc Ok I fixed the extends duplicate environment, but now the multiple compose file is broken :sweat_smile: (the code path is not the same and.. somehow I broke it :sweat: ).

[awsmsrc]

hey @vdemeester what is the status on this issue now?