os/exec: Cannot execute command with space in the name on Windows, when there are parameters

[calmh]

Go 1.7.1 on windows-amd64, Windows 10 latest.

Consider a test project:

src/github.com/calmh/wincmdtest/
    main.go
    folder name/
        test.bat

main.go contents:

package main

import (
    "fmt"
    "os/exec"
    "strings"
)

func main() {
    execCmd("./folder name/test.bat")
    execCmd("./folder name/test.bat", "one param")
}

func execCmd(path string, args ...string) {
    fmt.Printf("Running: %q %q\n", path, strings.Join(args, " "))
    cmd := exec.Command(path, args...)
    bs, err := cmd.CombinedOutput()
    fmt.Printf("Output: %s", bs)
    fmt.Printf("Error: %v\n\n", err)
}

folder name/test.bat contents:

@echo off
echo Success

Expected output is two runs with "Success" in them.

Actual:

C:\Users\jb\Go\src\github.com\calmh\wincmdtest>go run main.go
Running: "./folder name/test.bat" ""
Output: Success
Error: <nil>

Running: "./folder name/test.bat" "one param"
Output: '.' is not recognized as an internal or external command,
operable program or batch file.
Error: exit status 1

It appears that having params on a command, where the command contains a space, breaks the parsing of it. I haven't been able to work around this by experimenting with various ways of quoting the command, using backslashes or slashes, etc.

[calmh]

I started following this down through syscall.StartProcess, but not really being a Windows programmer I ran into unacceptable levels of nope almost immediately so probably won't try to fix this myself, sorry. :(

[alexbrainman]

It appears that having params on a command, where the command contains a space, breaks the parsing of it.

Go encodes child process parameters in a way that is understood by most programs. Go uses rules similar to what CommandLineToArgvW implements.

Unfortunately, your child process is cmd.exe (cmd.exe is called to execute the batch file you've requested). And cmd.exe parses its input parameters differently.

You can read this very long post http://daviddeley.com/autohotkey/parameters/parameters.htm#WIN about it all.

I haven't been able to work around this by experimenting with various ways of quoting the command, using backslashes or slashes, etc.

You should stop using exec.Command to build your child process command line, and build it yourself (as per rules described in the doco I mentioned). Then you can pass raw command line to your child process by setting exec.Cmd.SysProcAttr to syscall.SysProcAttr with CmdLine set appropriately.

Maybe we could fix this problem by changing Go encoding algorithm to use cmd.exe rules every time we execute batch file.

There is more of the same discussion on issue #15566.

Alex

[mattn]

I guess this should be fixed by small change.

diff --git a/src/syscall/exec_windows.go b/src/syscall/exec_windows.go
index cafce1e..a0e1f56 100644
--- a/src/syscall/exec_windows.go
+++ b/src/syscall/exec_windows.go
@@ -91,6 +91,9 @@ func makeCmdLine(args []string) string {
        }
        s += EscapeArg(v)
    }
+   if s != "" && s[0] == '"' && len(args) > 1 {
+       s = `"` + s + `"`
+   }
    return s
 }

If the first argument contain spaces, and it have arguments (no matter if the argument not contains spaces), whole of string should be quoted.

c:\dev\go-sandbox\space>go run main.go
Running: ".\\folder name\\test.bat" ""
Output: Success
Error: <nil>

Running: ".\\folder name\\test.bat" "one param"
Output: Success
Error: <nil>

[gopherbot]

CL https://golang.org/cl/32490 mentions this issue.

[rsc]

Postponing decisions about this to Go 1.9.

[basepack]

+1 This is still an issue here, also see: https://github.com/docker/machine/issues/3152

[alexbrainman]

@cannect docker/machine#3152 title mentions "powershell", but this issue is about running batch files. How do you know that if we make running batch file work, that it will fix docker/machine#3152 ? Is there a simple way to reproduce docker/machine#3152 ?

Alex

[basepack]

As you can see here: https://github.com/docker/machine/issues/3152#issuecomment-261811531 there is some confidence we think it is related with this issue.

[alexbrainman]

@cannect do you know of a simple way to reproduce docker/machine#3152 ? How did you personally discovered docker/machine#3152 ? Did you encountered that problem yourself?

Alex

[basepack]

Hi @alexbrainman ,

I had exact the same issues as described in the first post of docker/machine#3152. I do not have the knowledge to reproduce that issue in Go. But I can tell you my steps:

• WIndows 10 Pro
• Docker for Windows
• Started two docker machines in Windows 10 bash, listed as active in docker-machine ls
• Looked those docker machines up with: docker-machine ls in Powershell, machines not active. (but they were active of course but not listed as such in Powershell)

Unfortunately I have no experience whatsoever with Go.

[alexbrainman]

@cannect thank you very much for these steps. But I have never used Docker for Windows. How do I install it and use it?

Docker for Windows

How do I install it? What are the steps?

Looked those docker machines up with: docker-machine ls in Powershell, machines not active. (but they were active of course but not listed as such in Powershell)

What are the commands I should run to reproduce this? What did you do? What did you expect to see? What did you see instead?

Thank you

Alex

[rsc]

I agree with https://github.com/golang/go/issues/17149#issuecomment-257518448 that the answer here should be to rewrite the path for argv[0] from using slash to backslash. We should try to fix this early in the Go 1.12 cycle.

[benlye]

Any progress on this issue?

I'd like to add that it only seems to be a problem when both the command and the parameter contain spaces. If only one or the other has a space then everything works. I've expanded on @calmh's example to demonstrate:

main.go:

package main

import (
    "fmt"
    "os/exec"
    "strings"
)

func main() {
    execCmd("C:/Users/blye/Go/src/github.com/benlye/cmdtest/folder name/test.bat")
    execCmd("C:/Users/blye/Go/src/github.com/benlye/cmdtest/folder name/test.bat", "oneparam")
    execCmd("C:/Users/blye/Go/src/github.com/benlye/cmdtest/folder name/test.bat", "one param")

    execCmd("C:/Users/blye/Go/src/github.com/benlye/cmdtest/foldername/test.bat")
    execCmd("C:/Users/blye/Go/src/github.com/benlye/cmdtest/foldername/test.bat", "oneparam")
    execCmd("C:/Users/blye/Go/src/github.com/benlye/cmdtest/foldername/test.bat", "one param")
    }

func execCmd(path string, args ...string) {
    fmt.Printf("Running: %q %q\n", path, strings.Join(args, " "))
    cmd := exec.Command(path, args...)
    bs, err := cmd.CombinedOutput()
    fmt.Printf("Output: %s", bs)
    fmt.Printf("Error: %v\n\n", err)
}

Produces only one failure (the 3rd test):

C:\Users\blye\Go\src\github.com\benlye\cmdtest>go run main.go
Running: "C:/Users/blye/Go/src/github.com/benlye/cmdtest/folder name/test.bat" ""
Output: Success
Error: <nil>

Running: "C:/Users/blye/Go/src/github.com/benlye/cmdtest/folder name/test.bat" "oneparam"
Output: Success
Error: <nil>

Running: "C:/Users/blye/Go/src/github.com/benlye/cmdtest/folder name/test.bat" "one param"
Output: 'C:/Users/blye/Go/src/github.com/benlye/cmdtest/folder' is not recognized as an internal or external command,
operable program or batch file.
Error: exit status 1

Running: "C:/Users/blye/Go/src/github.com/benlye/cmdtest/foldername/test.bat" ""
Output: Success
Error: <nil>

Running: "C:/Users/blye/Go/src/github.com/benlye/cmdtest/foldername/test.bat" "oneparam"
Output: Success
Error: <nil>

Running: "C:/Users/blye/Go/src/github.com/benlye/cmdtest/foldername/test.bat" "one param"
Output: Success
Error: <nil>

[alexbrainman]

Any progress on this issue?

I don't believe anyone working on this.

I'd like to add that it only seems to be a problem when both the command and the parameter contain spaces. ...

I am pretty sure we know what the problem is (see https://github.com/golang/go/issues/17149#issuecomment-248822918 for details). We just have not decided how to fix it.

Alex

[someblue]

For those stuck by this bug:

As https://github.com/golang/go/issues/17149#issuecomment-473976818 mentioned, this problem happened only when both binary path and arguments have space. So you can write a .bat file wrapping all arguments, and then run this .bat directly.

[kot0]

Why not fixed?

[bjg2]

Wow. What a bug. XD

[siradam7th]

It took me a while to figure out what was wrong, I had to use Process Monitor in order to see what Go was passing as arguments to the process, any news on a fix?

[alexbrainman]

any news on a fix?

I am not working on this issue, if that is what you are asking.

Alex

[ItsMattL]

For anyone needing a more concrete example of how to do the manual path construction, it works something like this:

cmd := exec.Command(os.Getenv("SystemRoot")+`\System32\cmd.exe`) 
args :=  []string{`/c`, `""C:\Program Files\echo.bat"`, `"hello friend""`}
cmd.SysProcAttr = &syscall.SysProcAttr{CmdLine: " " + strings.Join(args, " ")}
out, _ := cmd.CombinedOutput()
fmt.Printf("Output: %s", out)

Note that you're on the hook to have everything on the CmdLine escaped/quoted/etc properly, which can be a real chore.

[bertysentry]

Here's an updated version of @ItsMattL's workaround:

// command to execute, may contain quotes, backslashes and spaces
var commandLine = `"C:\Program Files\echo.bat" "hello friend"`

var comSpec = os.Getenv("COMSPEC")
if comSpec == "" {
	comSpec = os.Getenv("SystemRoot") + "\\System32\\cmd.exe"
}
childProcess = exec.Command(comSpec)
childProcess.SysProcAttr = &syscall.SysProcAttr{CmdLine: "/C \"" + commandLine + "\""}

// Then execute and read the output
out, _ := childProcess.CombinedOutput()
fmt.Printf("Output: %s", out)

[fyow93]

For Linux system: input = strings.ReplaceAll(input, " ", "\ ") args := []string{ "ffmpeg", fmt.Sprintf("-i %s", input), fmt.Sprintf("-vf %s", "scale=-1:360"), fmt.Sprintf("-c:v %s", "libx264"), fmt.Sprintf("-crf %s", "18"), fmt.Sprintf("-c:a %s", "copy"), "b.mp4", "-y", } myCmd := strings.Join(args, " ") cmd := exec.Command("bash","-c", myCmd)

[rafd123]

Here's an updated version of @ItsMattL's workaround:

// command to execute, may contain quotes, backslashes and spaces
var commandLine = `"C:\Program Files\echo.bat" "hello friend"`

var comSpec = os.Getenv("COMSPEC")
if comSpec == "" {
	comSpec = os.Getenv("SystemRoot") + "\\System32\\cmd.exe"
}
childProcess = exec.Command(comSpec)
childProcess.SysProcAttr = &syscall.SysProcAttr{CmdLine: "/C \"" + commandLine + "\""}

// Then execute and read the output
out, _ := childProcess.CombinedOutput()
fmt.Printf("Output: %s", out)

At this risk of sounding pedantic 😅, should the executable path be included in the CmdLine, too? Not sure if it makes a difference in practice, but reasons I ask:

• The docs for the underlying Windows API mentions it.
• If you use exec.Command without the SysProcAttr escape hatch, it automatically prepends the executable to the args (see https://github.com/golang/go/blob/master/src/os/exec/exec.go#L379). Those args ultimate are used to generate the cmdLine (see https://github.com/golang/go/blob/master/src/syscall/exec_windows.go#L302)
• If you run the batch file from Windows Run and watch the procmon output, it includes the exe in the command line.
• If you run the example code and watch the procmon output, it doesn't include the exe in the command line.

I'm also wondering if the arguments should be escaped when we create the cmdline ourselves as a more general solution. We might get close by using syscall.EscapeArg on each argument, but I think cmd.exe has its own set of crazy escaping rules.

[bertysentry]

@rafd123 I think you are right! So, the proper code becomes:

// command to execute, may contain quotes, backslashes and spaces
var commandLine = `"C:\Program Files\echo.bat" "hello friend"`

var comSpec = os.Getenv("COMSPEC")
if comSpec == "" {
	comSpec = os.Getenv("SystemRoot") + "\\System32\\cmd.exe"
}
childProcess = exec.Command(comSpec)
childProcess.SysProcAttr = &syscall.SysProcAttr{CmdLine: comSpec + " /C \"" + commandLine + "\""}

// Then execute and read the output
out, _ := childProcess.CombinedOutput()
fmt.Printf("Output: %s", out)

[rafd123]

What do you think about something like this as a general solution?

Caveats:

• syscall.EscapeArg probably needs to be replaced with something that is aware of cmd.exe's escaping rules in order to be airtight; see this.
• I can imagine adding handling for the problems inherent with running msiexec, too.

exec.go

package exec

import (
	"os/exec"
)

// MakeCmd returns the Cmd struct to execute the named program with
// the given arguments.
//
// It handles the case where the program is a Windows batch file
// and the implication it has on argument quoting.
func MakeCmd(name string, args ...string) (*exec.Cmd, error) {
	return makeCmd(name, args...)
}

exec_windows.go

package exec

import (
	"fmt"
	"os"
	"os/exec"
	"path/filepath"
	"strings"
	"syscall"
)

func makeCmd(name string, args ...string) (*exec.Cmd, error) {
	if len(args) == 0 {
		return exec.Command(name), nil
	}

	name, err := exec.LookPath(name)
	if err != nil {
		return nil, err
	}

	if !isBatchFile(name) {
		return exec.Command(name, args...), nil
	}

	argsEscaped := make([]string, len(args)+1)
	argsEscaped[0] = syscall.EscapeArg(name)
	for i, a := range args {
		argsEscaped[i+1] = syscall.EscapeArg(a)
	}

	shell := os.Getenv("COMSPEC")
	if shell == "" {
		shell = "cmd.exe" // Will be expanded by exec.LookPath in exec.Command
	}

	cmd := exec.Command(shell)
	cmd.Args = nil
	cmd.SysProcAttr = &syscall.SysProcAttr{
		CmdLine: fmt.Sprintf(`%s /c "%s"`, syscall.EscapeArg(cmd.Path), strings.Join(argsEscaped, " ")),
	}

	return cmd, nil
}

func isBatchFile(path string) bool {
	ext := filepath.Ext(path)
	return strings.EqualFold(ext, ".bat") || strings.EqualFold(ext, ".cmd")
}

exec_other.go

//go:build !windows

package exec

func makeCmd(name string, args ...string) (*exec.Cmd, error) {
	return exec.Command(name, args...), nil
}

[bertysentry]

@rafd123 Looks like a good improvement. However I don't see why we would need to handle batch files differently. It's important to pass them through CMD.EXE too, so they need the same escaping.

And yeah, you would need to follow the excellent article that you linked to, and escape meta characters with ^. Otherwise, there is a serious security issue where the one could inject commands by using "&malicious.exe in any of the arguments.

We need a Windows-specific escaping mechanism.

[rafd123]

Digging into this a bit further, I noticed that the dotnet core equivalent of exec.Command does not suffer from this problem. Also note that dotnet is cross-platform these days and has a similar abstraction over OS-specific APIs for creating processes.

Specifically, this works fine in dotnet:

using System.Diagnostics;

Process.Start(@"C:\Program Files\echo.bat", new[] {"hello world"});

Debugging how both golang and dotnet ultimately call Windows' CreateProcess API, I noticed one meaningful difference: dotnet passes null for the optional lpApplicationName parameter of CreateProcess while golang does not.

This is how dotnet calls CreateProcess (note the null for lpApplicationName argument).

This is how golang calls CreateProcess (note the use of argv0p...which in practice maps to the name argument of exec.Command...also note that in practice argvp will also contain the name argument of exec.Command).

Empirically, I can also confirm that if I force golang to pass nil for lpApplicationName here instead of argv0p, executing exec.Command(`C:\Program Files\echo.bat`, "hello world") works without resorting to using the SysProcAttr escape hatch.

All that said:

• Specifying lpApplicationName for CreateProcess is cited as a security best-practice as a fail-safe should the caller fail to add quotes around the path of the executable in lpCommandLine. FWIW, dotnet ensures that the executable is quoted for this very reason, and it turns out golang does, too, via makeCmdLine's use of appendEscapeArg.
• I can't find any official documentation around this subtle behavior difference with CreateProcess as to why it works when NOT specifying lpApplicationName, so at this point it's all circumstantial. All I was able to find is someone else pointing out this suble behavioral difference way back in 2001.
• It's entirely possible I'm glossing over some other important detail. I'm stabbing in the dark at this point. 😅

[bertysentry]

@rafd123 Following what is done in .NET Core, and adapting it to the expected behavior in Go is probably the best we can do. Thank you for the excellent research!

In your suggested makeCmd(), you would just need to add proper escaping for CMD.EXE. We probably need to use ^ instead of " to surround the arguments. WDYT?

[bcmills]

(CC @golang/windows)

[iyzyi]

It still exists in golang 1.20, e.g: cmd.exe /c copy/b "input 1.ts"+"input 2.ts" ouput.ts I guess that golang can automatically add double quotes to paths that contain Spaces, but many commands have their own coding rules, such as copy/b. The "+" in copy/b means concatenating the input files, but golang cannot parse it and cannot add double quotes to paths of input files that contain Spaces correctly.