SCons
Interactive mode fails on Windows
This issue was originally created at: 2016-05-18 05:05:15.
This issue was reported by: calmer.
calmer said at 2016-05-18 05:05:15
On Windows when using SCons in interactive mode for incrementally building a target then the build step sporadically fails, because SCons doesn't pass the source files to be compiled to the compiler any more.
I've managed to reproduce the bug in a standalone test setup. I'll attach a zip file and log file of the bug reproduction.
The bug triggers only if the target is already built when --interactive mode is entered. Then after cleaning the target and trying to rebuild it, it will fail.
Steps to reproduce:
- run
scons mylibin the test buildsystem - run
scons --interactive, then in the interactive shell:- run
clean mylib - run
build mylib
- run
--> The bug should be triggered, the compiler stating "missing source file name".
This was reproduced using scons 2.5.0. But is also happens in older versions (tested with 2.3.0, too). I've used Windows 7 with Visual Studio 2015 to trigger this. But it also happened with Visual Studio 2012.
It seems to me that in the bug situation the construction variable CHANGED_SOURCES is not correctly refreshed. In Executor.py, Executor.get_lvars(), for example, there is some kind of caching mechanism that returns an empty CHANGED_SOURCES in case of the bug. But that is just a shot in the dark of mine.
calmer said at 2016-05-18 05:06:27 Created an attachment (id=967)
scons setup for triggering the bug
calmer said at 2016-05-18 05:08:05 Created an attachment (id=968)
Console output when reproducing the bug
Votes for this issue: 2.
calmer attached scons_bug.zip at 2016-05-18 05:06:27.
scons setup for triggering the bug
calmer attached bug_session.txt at 2016-05-18 05:08:05.
Console output when reproducing the bug
Bug still present with this configuration :
- scons 3.0.1
- python 3.5.4
- Windows 10
- Visual Studio 2010
I just tried this on my Windows system, and confirm the same failure ("missing source filename") still occurs. Details: SCons git master (4.8.1 + changes since), python 3.13.1, Windows 11, Visual Studio 2022).
The SConstruct from the included reproducer specified MSVC_VERSION=14.0 which I got rid of since that system has 14.3, 14.2 and 14.1 installed but not 14.0.
It was observed on Discord that the difference between Windows/msvc (that sees this problem) and basically all other platform/compiler combos is that on Windows (a) $TEMPFILE is used, and (b) $CHANGED_SOURCES is used instead of $SOURCES, to support batch mode. The latter appears to be the culprit, as surmised in the original posting. The Executor class methods _get_changed_sources and _get_changed_targets, as well as get_lvars, use a kind of memoization, for example _get_changed_sources looks like:
def _get_changed_sources(self, *args, **kw):
try:
return self._changed_sources_list
except AttributeError:
self._get_changes() # side-effect: updates self._{changed,unchanged}_{sources,targets}_list
return self._changed_sources_list
So if the first thing through in interactive mode is to clean, and _changed_sources_list ends up empty as it seems to, that's saved and returned for subsequent operations, without re-calling self._get_changes() to rescan.
There are steps otherwise taken in interactive mode to clear some things out so that stuff is not remembered when not appropriate; perhaps this was something else that needed clearing (at the moment, not sure what that would look like).
A few further notes - there is some "information clearing" in interactive mode, the memo dictionary is dropped after each op to avoid remembering too much stuff between "independent" operations within a single scons invocation. However, the four attributes _changed_sources_list. _changed_targets_list. _unchanged_sources_list and _unchanged_targets_list are just set directly as attributes in the instance (for which there might be reasons - not fully investigated yet), not in the memo dict, so they don't get cleared by this step. Adding an experimental del of the _changed_sources_list in the same clean function seems to clear up the reported problem. Whether that's a viable solution, or whether it's appropriate to move these four (or maybe just the one) into the memo dict is also unknown at this point.
After more conversation with @bdbaddog, here's a patch to Executor.py that moves the four lists of sources/targets from executor attributes to entries in the executor's memo dict, which is already cleared in the appropriate circumstances in interactive mode. All four of the functions now use an executor memo attribute, all four are set by the _get_changes method, whichever one of those functions that fires first will thus trigger the setup and the other three will get the memoized attribute to avoid _get_changes being re-called.
Okay, github refuses to let me attach that file ("failed uploading"). Looking for solutions...
diff --git a/SCons/Executor.py b/SCons/Executor.py
index 53eb5cbb2..7a891f200 100644
--- a/SCons/Executor.py
+++ b/SCons/Executor.py
@@ -26,6 +26,7 @@
from __future__ import annotations
import collections
+from contextlib import suppress
import SCons.Errors
import SCons.Memoize
@@ -163,10 +164,6 @@ class Executor(metaclass=NoSlotsPyPy):
'builder_kw',
'_memo',
'lvars',
- '_changed_sources_list',
- '_changed_targets_list',
- '_unchanged_sources_list',
- '_unchanged_targets_list',
'action_list',
'_do_execute',
'_execute_str')
@@ -205,37 +202,45 @@ class Executor(metaclass=NoSlotsPyPy):
return self.lvars
def _get_changes(self) -> None:
- cs = []
- ct = []
- us = []
- ut = []
+ """Populate all the changed/unchanged lists.
+
+ .. versionchanged:: NEXT_RELEASE
+ The four instance attributes '_changed_sources_list',
+ '_changed_targets_list', '_unchanged_sources_list' and
+ '_unchanged_targets_list' are no longer set here, or used
+ at all, instead entries are added to the memo dict.
+ """
+ changed_sources = []
+ changed_targets = []
+ unchanged_sources = []
+ unchanged_targets = []
for b in self.batches:
# don't add targets marked always build to unchanged lists
# add to changed list as they always need to build
if not b.targets[0].always_build and b.targets[0].is_up_to_date():
- us.extend(list(map(rfile, b.sources)))
- ut.extend(b.targets)
+ unchanged_sources.extend(list(map(rfile, b.sources)))
+ unchanged_targets.extend(b.targets)
else:
- cs.extend(list(map(rfile, b.sources)))
- ct.extend(b.targets)
- self._changed_sources_list = SCons.Util.NodeList(cs)
- self._changed_targets_list = SCons.Util.NodeList(ct)
- self._unchanged_sources_list = SCons.Util.NodeList(us)
- self._unchanged_targets_list = SCons.Util.NodeList(ut)
+ changed_sources.extend(list(map(rfile, b.sources)))
+ changed_targets.extend(b.targets)
+ self._memo["_get_changed_sources"] = changed_sources
+ self._memo["_get_changed_targets"] = changed_targets
+ self._memo["_get_unchanged_sources"] = unchanged_sources
+ self._memo["_get_unchanged_targets"] = unchanged_targets
+ @SCons.Memoize.CountMethodCall
def _get_changed_sources(self, *args, **kw):
- try:
- return self._changed_sources_list
- except AttributeError:
- self._get_changes()
- return self._changed_sources_list
+ with suppress(KeyError):
+ return self._memo["_get_changed_sources"]
+ self._get_changes() # sets the memo entry
+ return self._memo["_get_changed_sources"]
+ @SCons.Memoize.CountMethodCall
def _get_changed_targets(self, *args, **kw):
- try:
- return self._changed_targets_list
- except AttributeError:
- self._get_changes()
- return self._changed_targets_list
+ with suppress(KeyError):
+ return self._memo["_get_changed_targets"]
+ self._get_changes() # sets the memo entry
+ return self._memo["_get_changed_targets"]
def _get_source(self, *args, **kw):
return rfile(self.batches[0].sources[0]).get_subst_proxy()
@@ -249,19 +254,19 @@ class Executor(metaclass=NoSlotsPyPy):
def _get_targets(self, *args, **kw):
return SCons.Util.NodeList([n.get_subst_proxy() for n in self.get_all_targets()])
+ @SCons.Memoize.CountMethodCall
def _get_unchanged_sources(self, *args, **kw):
- try:
- return self._unchanged_sources_list
- except AttributeError:
- self._get_changes()
- return self._unchanged_sources_list
+ with suppress(KeyError):
+ return self._memo["_get_unchanged_sources"]
+ self._get_changes() # sets the memo entry
+ return self._memo["_get_unchanged_sources"]
+ @SCons.Memoize.CountMethodCall
def _get_unchanged_targets(self, *args, **kw):
- try:
- return self._unchanged_targets_list
- except AttributeError:
- self._get_changes()
- return self._unchanged_targets_list
+ with suppress(KeyError):
+ return self._memo["_get_unchanged_targets"]
+ self._get_changes() # sets the memo entry
+ return self._memo["_get_unchanged_targets"]
def get_action_targets(self):
if not self.action_list:
@@ -585,6 +590,9 @@ class Null(metaclass=NoSlotsPyPy):
This might be able to disappear when we refactor things to
disassociate Builders from Nodes entirely, so we're not
going to worry about unit tests for this--at least for now.
+
+ Note the slots have to match :class:`Executor` exactly,
+ or the :meth:`_morph` will fail.
"""
__slots__ = ('pre_actions',
@@ -595,10 +603,6 @@ class Null(metaclass=NoSlotsPyPy):
'builder_kw',
'_memo',
'lvars',
- '_changed_sources_list',
- '_changed_targets_list',
- '_unchanged_sources_list',
- '_unchanged_targets_list',
'action_list',
'_do_execute',
'_execute_str')
Yeah, I spent a little bit of time poking around at this, and your patch looks like the correct fix to me.
Note that the CHANGED_SOURCES computed in _get_changes() only appears to be used sometimes.
I was testing printing out the return values whenever scons_subst_list("${CXXCOM}") is called, and in interactive mode it gets the correct sources list when initially scanning dependencies, and only gets the wrong value later when actually executing the action.
When scanning deps it appears to be getting the CHANGED_SOURCES value from this code here:
https://github.com/SCons/scons/blob/de74f54cf3b55f4966ed0966af73854c1725e7f2/SCons/Subst.py#L316-L321
