chdir + -j confuses scons itself about cwd

[bdbaddog]

This issue was originally created at: 2010-03-26 07:40:26. This issue was reported by: ydirson. ydirson said at 2010-03-26 07:40:27

As part of the chdir/-j interaction, it seems scons has problems restoring the original working dir (probably a problem with saving $PWD after a concurrent job did chdir). As a result, the .sconsign.tmp is created in a wrong place, and .sconsign.dblite cannot be replaced by the new version, giving the error message I mentionned elsewhere, which is reproduced below.

Bug created since this is a separate issue from the chdir one, and can probably be fixed more quickly. However, patch #824 would make this a non-issue altogether...

Here is an excerpt from a build log which exhibits the problem.

Script started on ven 26 mar 2010 14:04:25 CET
+ scons --debug=explain -j2 -f src/tests/SConstruct r=:/Mark dir=../_builds
scons: Reading SConscript files ...
scons: done reading SConscript files.
scons: Building targets ...
scons: building `/home/localadm/_builds/398596d/nodebug/.stamp-build' because it doesn't exist
os.chdir('/home/localadm/_builds/398596d/nodebug')
scons: building `/home/localadm/_builds/398596d/debug/.stamp-build' because it doesn't exist
os.chdir('/home/localadm/_builds/398596d/debug')
os.chdir('/home/localadm/testv1')
os.chdir('/home/localadm/_builds/398596d/nodebug')
printf "DEBUG=OFF\n" >Makefile.runtests
os.chdir('/home/localadm/_builds/398596d/nodebug')
os.chdir('/home/localadm/_builds/398596d/debug')
printf "DEBUG=ON\n" >Makefile.runtests
os.chdir('/home/localadm/_builds/398596d/nodebug')
os.chdir('/home/localadm/_builds/398596d/debug')
make
os.chdir('/home/localadm/testv1')
os.chdir('/home/localadm/_builds/398596d/nodebug')
make
[...]
os.chdir('/home/localadm/_builds/398596d/nodebug')
[...]
scons: done building targets.
OSError: [Errno 2] No such file or directory: '.sconsign.dblite':
  File "/usr/lib/scons/SCons/Script/Main.py", line 1248:
    _exec_main(parser, values)
  File "/usr/lib/scons/SCons/Script/Main.py", line 1213:
    _main(parser)
  File "/usr/lib/scons/SCons/Script/Main.py", line 980:
    nodes = _build_targets(fs, options, targets, target_top)
  File "/usr/lib/scons/SCons/Script/Main.py", line 1172:
    jobs.run(postfunc = jobs_postfunc)
  File "/usr/lib/scons/SCons/Job.py", line 112:
    postfunc()
  File "/usr/lib/scons/SCons/Script/Main.py", line 1169:
    SCons.SConsign.write()
  File "/usr/lib/scons/SCons/SConsign.py", line 109:
    syncmethod()
  File "/usr/lib/scons/SCons/dblite.py", line 104:
    self._os_unlink(self._file_name)
Exception exceptions.OSError: (2, 'No such file or directory',
'.sconsign.dblite') in <bound method dblite.__del__ of <SCons.dblite.dblite
instance at 0x83568ac>> ignored
 
Script done on ven 26 mar 2010 15:03:16 CET

garyo said at 2010-04-01 19:34:08

Steven to either update the doc or ideally generate a warning in this case. The chdir/-j interactions are documented in a couple of places, but since the combination definitely doesn't work we can document it better and/or emit a warning. The problem is that python is a single process and has only one working dir. So -j combined with chdir can cause the main python thread to be in the wrong dir when it does things.

[JonathonReinhart]

This call to os.chdir() affects the entire SCons Python process, including all threads created by the executor when -j is given.

Is there any reason that call can't be replaced with the cwd argument to subprocess.Popen()? It seems like this shouldn't be too hard, at least for the posix Platform. But I kind of wonder why that code has to be platform specific anyway, when subprocess.Popen is cross-platform.

[mwichmann]

So without really answering directly, there's an extremeley long-standing issue with the chdir stuff. See for example #824, which date back... a ways. And yes, IMO, feeding the dir information to the subprocess call seems better than having SCons do the changing, at least on the surface.

[bdbaddog]

The reasons for not doing that are gone. There was a time when we supported pythons which didn't have subprocess (I beleive that was the reason) And then the recommended solution was to have your action something like cd ${TARGET.dir}; gcc....

@JonathonReinhart - would you be interested in putting together a PR?

[JonathonReinhart]

I threw together #3873 just to get my feet wet.

One of the first major stumbling blocks was the fact that both CommandAction and FunctionAction have an execute() which has to handle the chdir (via a new added argument). And this brings up the question: If SCons isn't doing an os.chdir() call before executing a function action, then how should the chdir be handled for function actions? I can imagine this causing grief for a ton of custom actions. But the first that hit me was MkdirFunc to produce the output directories.

[bdbaddog]

Ahh.. Good question. I think least painful way to enable might be to pass a cwd to the FunctionAction() and then require all such to be updated.. Changing to multiprocessing is a huge undertaking.

[bdbaddog]

That said, a PR which addresses only command actions would still move the ball forward right?

[mwichmann]

There seem to be a lot of issues in flight with this and the other chdir bugs. Some stuff should be unplugged from the idea of doing a chdir related to a build, for example the dblite module:

the .sconsign.tmp is created in a wrong place, and .sconsign.dblite cannot be replaced by the new version

that just shouldn't happen, and shouldn't be hard to fix - dblite is just very simple-minded about that.

[bdbaddog]

The problem is since subprocesses are running in threads, and all threads share the same CWD.. if any other command runs from another thread, it'll be in the wrong directory.

There was a plan/idea to use subprocess for -j, then each process would have per job cwd, but other issues complicated that.