SCons
long (> ~10000 ch) command lines cause failures on win32
This issue was originally created at: 2004-04-21 19:47:14.
This issue was reported by: tobiasjs.
tobiasjs said at 2004-04-21 19:47:14
the os.spawnve() mechanism (maybe in combination with the windows shell) doesn't handle very long command lines. If win32 modules are available, it would be good if scons used win32process.CreateProcess() to spawn subprocesses, as the limit doesn't exist in that case.
I worked around the problem in the scons file as follows. I'm not sure about the return values, but hopefully this should serve as a starting point.
Import("env")
import win32file
import win32event
import win32process
import win32security
def my_spawn(sh, escape, cmd, args, env):
sAttrs = win32security.SECURITY_ATTRIBUTES()
StartupInfo = win32process.STARTUPINFO()
cmdline = cmd + ' ' + string.join(map(escape, args[1:]), ' ')
hProcess, hThread, dwPid, dwTid = win32process.CreateProcess(args[0], cmdline, None, None, 1, 0, env, None, StartupInfo)
win32event.WaitForSingleObject(hProcess)
exit_code = win32process.GetExitCodeProcess(hProcess, win32event.INFINITE)
win32file.CloseHandle(hThread)
win32file.CloseHandle(hProcess)
return exit_code
def my_piped_spawn(sh, escape, cmd, args, env, stdout, stderr,
real):
sAttrs = win32security.SECURITY_ATTRIBUTES()
StartupInfo = win32process.STARTUPINFO()
StartupInfo.hStdOutput = win32file._get_osfhandle(stdout)
StartupInfo.hStdError = win32file._get_osfhandle(stderr)
StartupInfo.hStdInput = 0
StartupInfo.dwFlags = win32process.STARTF_USESTDHANDLES
cmdline = cmd + ' ' + string.join(map(escape, args[1: ' ')
hProcess, hThread, dwPid, dwTid = win32process.CreateProcess(args[0], cmdline, None, None, 1, 0, env, None, StartupInfo)
win32event.WaitForSingleObject(hProcess)
exit_code = win32process.GetExitCodeProcess(hProcess, win32event.INFINITE)
win32file.CloseHandle(hThread)
win32file.CloseHandle(hProcess)
return exit_code
env['SPAWN'] = my_spawn
env['PSPAWN'] = my_piped_spaw
issues@scons said at 2004-04-21 19:47:14
Converted from SourceForge tracker item 939747
stevenknight said at 2006-11-08 15:08:36
Changing to a patch so this code can be preserved and more prominently available.
A change has just been integrated that helps long command lines that don't have redirectionr or pipe character or the like by executing them directly. (cmd is what's responsible for the line length limitation.) That will help, but is not a complete solution.
The code attached here is a possible future direction, but requiring the win32 extensions to run SCons would be a bit of a problem. Perhaps at some point in the future.
stevenknight said at 2006-11-08 15:25:21
Forgot to actual change the type last comment...
garyo said at 2008-03-17 18:14:07
need more info.
gregnoel said at 2008-03-18 18:45:55
Bill said he'd research this one and issue 1727, which is also about long lines.
gregnoel said at 2008-04-18 18:47:32
We shouldn't have UNCONFIRMED issues; changing them to NEW.
gregnoel said at 2010-07-21 16:59:27
Bug party triage. Bump the priority of this issue.
gregnoel said at 2010-11-14 18:41:55
Promote issue so it will be reviewed at next bug party.
bdbaddog said at 2016-01-13 16:05:05
Added Windows keyword.
"The code attached here" - but don't see any attachment. There's been later work on long command lines... is this still an active issue?
"The code attached here" - but don't see any attachment. There's been later work on long command lines... is this still an active issue?
I reformatted the text, you can see the code bit above.
Revisiting this for one more comment - it refers to problems due to os.spawnve, which we still use on the Windows platform. An experimental change to launch processes a different way exists - the Python subprocess module does use the appropriate Windows mechanism: _winapi.CreateProcess(executable, args, .... I suspect we'll still launch using a shell, though (for reasons of having metacharacters like > interpreted) so it may not end up fixing this particular limitation.