sh.Run and sh.Output sh.OutCmd fail on a go build command but running it directly on local machine works

[varunturlapati]

I setup my mage target to build locally and it worked fine. Then I wanted to introduce ldflags to get build info embedded as well. The build target errors out.

Expected: Build succeeds

Actual:

go: finding github.myorg.com/myrepo/heyo/version.BuildSha=4accbfe latest
go: finding github.myorg.com/myrepo/heyo/version.ReleaseType=localdev' latest
go: finding github.myorg.com/myrepo/heyo/version.BuildVersion=local latest
can't load package: package github.myorg.com/myrepo/heyo/version.BuildDate=20200505: no matching versions for query "latest"
can't load package: package -X: malformed module path "-X": leading dash
can't load package: package github.myorg.com/myrepo/heyo/version.BuildSha=4accbfe: no matching versions for query "latest"
can't load package: package github.myorg.com/myrepo/heyo/version.BuildVersion=local: no matching versions for query "latest"
can't load package: package github.myorg.com/myrepo/heyo/version.ReleaseType=localdev': no matching versions for query "latest"
can't load package: package -os: malformed module path "-os": leading dash
can't load package: package darwin: malformed module path "darwin": missing dot in first path element
can't load package: package -arch: malformed module path "-arch": leading dash
can't load package: package amd64: malformed module path "amd64": missing dot in first path element
can't load package: package -output: malformed module path "-output": leading dash
can't load package: package targets/ocs_heyo_darwin_amd64: malformed module path "targets/ocs_heyo_darwin_amd64": missing dot in first path element

Error: running "gox -ldflags '-X github.myorg.com/myrepo/heyo/version.BuildDate=20200505 -X github.myorg.com/myrepo/heyo/version.BuildSha=4accbfe -X github.myorg.com/myrepo/heyo/version.BuildVersion=local -X github.myorg.com/myrepo/heyo/version.ReleaseType=localdev' -os darwin -arch amd64 -output targets/ocs_heyo_darwin_amd64 github.myorg.com/myrepo/heyo/cmd" failed with exit code 1

Then as an experiment I simply copied the whole command that it says it ran in the last line.

gox -ldflags '-X github.myorg.com/myrepo/heyo/version.BuildDate=20200505 -X github.myorg.com/myrepo/heyo/version.BuildSha=4accbfe -X github.myorg.com/myrepo/heyo/version.BuildVersion=local -X github.myorg.com/myrepo/heyo/version.ReleaseType=localdev' -os darwin -arch amd64 -output targets/ocs_heyo_darwin_amd64 github.myorg.com/myrepo/heyo/cmd

And this worked perfectly. Got the build and it reflected all the build attrs correctly.

Build function is:

func build(b *BuildOpts) error {
	if b == nil {
		fmt.Println("No BuildOpts specified. Will use defaults.")
		b = DefaultBuildOpts
	}
	if _, ok := Os2BuildArgsMap[b.Goos]; !ok {
		return fmt.Errorf("your local machine os %s is not supported in builds. Add it in constants.go or see why the OS is not"+
			"recognized", b.Goos)
	}
	var ldPrefixedArgs = strings.Split(PrepareLdFlagsFromBuildOpts(b), " ")
	ldPrefixedArgs = append(ldPrefixedArgs, Os2BuildArgsMap[b.Goos]...)
	fmt.Printf("%d --> %v", len(ldPrefixedArgs), ldPrefixedArgs)
	//os.Exit(1)
	out, err := sh.Output(Gox, ldPrefixedArgs...) // TODO looping to create all in a BuildAll target
	if err != nil {
		fmt.Println(out)
		return err
	}
	return nil
}

func PrepareLdFlagsFromBuildOpts(b *BuildOpts) string {
    var flags = []string{
        fmt.Sprintf("-X github.myorg.com/myrepo/heyo/version.BuildDate=%v", b.Timestamp),
        fmt.Sprintf("-X github.myorg.com/myrepo/heyo/version.BuildSha=%v", b.Sha),
        fmt.Sprintf("-X github.myorg.com/myrepo/heyo/version.BuildVersion=%v", b.Version),
        fmt.Sprintf("-X github.myorg.com/myrepo/heyo/version.ReleaseType=%v", b.Reltype),
    }
    return fmt.Sprintf("-ldflags '%s'", strings.Join(flags, " "))
}

[natefinch]

formatting ldflags with the correct quoting is super annoying and hard. Now add that to the fact that os/exec and thus sh are not actual shells (despite the misleading name for sh)... and things get tricky.

The trick is that the value for -ldflags has to exist in a single string passed as an argument to sh.Output/Run. You can't split it up by spaces.

You can see how I format it here: https://github.com/magefile/mage/blob/master/magefile.go#L55

[shouldsee]

Just ran to the same issue when inspecting the exec: message. It is super annoying that os/exec is not an actual shell!!!! This bascially means all shell keywords like time and exit would have to be implemented in the golang way...

[egonbraun]

I ran into the same problem and I can't figure it out how to fix it. Here is my Build function:

	args := []string{
		`-arch="` + SupportedArch + `"`,
		`-osarch="` + UnsupportedOsArch + `"`,
		`-os="` + SupportedOs + `"`,
		`-output="` + path.Join(PkgDir, "{{.OS}}_{{.Arch}}", ProgramName) + `"`,
		`-ldflags="` + ldFlags + `"`,
		mainFilePath,
	}

	return sh.RunWithV(env, "gox", args...)

The error:

No valid platforms to build for. If you specified a value
for the 'os', 'arch', or 'osarch' flags, make sure you're
using a valid value.
Error: running "gox -arch="amd64 arm64" -osarch="!windows/arm64" -os="darwin linux windows" -output="pkg/{{.OS}}_{{.Arch}}/template-golang-project" -ldflags="-s -w" cmd/template-golang-project/main.go" failed with exit code 1

Then if I ran the same manually, everything works fine.

gox -arch="amd64 arm64" -osarch="!windows/arm64" -os="darwin linux windows" -output="pkg/{{.OS}}_{{.Arch}}/template-golang-project" -ldflags="-s -w" cmd/template-golang-project/main.go

Any ideas are welcome. :)