scons icon indicating copy to clipboard operation
scons copied to clipboard
verified publisher icon SCons

PCH source file rebuilt every time when certain filenames are used

Open bdbaddog opened this issue 7 years ago • 56 comments

This issue was originally created at: 2008-11-01 23:58:39. This issue was reported by: jchristian.

jchristian said at 2008-11-01 23:58:39

Scons Version: Using "vs_revamp" branch build off of the 1.1.0 release. I suspect this problem happens on the official 1.1.0 release, but I haven't checked it.

Compiler: VS2008

Defect Summary: In attempting to use precompiled headers in building a Windows project, I ran into a fairly obscure bug. Apparently, if the filename specified as the source file for the PCH generation begins with the letter 'c' or 'C', the build regenerates the PCH file every time the build is run...even if no files have changed. If I change that filename to begin with another letter, it works. Note that originally my PCH source file was called "CorePCH.cpp". In trying things, I found it could be shortened to just "C.cpp" and produce the problem.

Expected Behavior: On the second build, it should just return with an "up to date" status.

Below is my directory structure and the files needed to reproduce the issue. Note that once you get this all setup, you can make it work successfully by changing the filename C.cpp in the SConscript to just PCH.cpp or XCorePCH.cpp.

Let me know if you have questions.

Directory structure:

project
    |
    +--SConstruct
    |
    +--include
    |    |
    |    +-- core
    |         |
    |         +-- CorePCH.hpp
    |             Hello.hpp
    |
    +-- lib
         |
         +-- core
              |
              +-- CorePCH.cpp
                  Hello.cpp
                  SConscript

SConscript:

import glob

# get all the build variables we need
Import("env", "buildroot", "project", "mode", "debugcflags", "releasecflags")
staticlibenv = env.Clone()

builddir = buildroot + "/" + project  # holds the build directory for this
project
targetpath = builddir  # holds the path to the executable in the build directory

# append the user's additional compile flags
# assume debugcflags and releasecflags are defined
if mode == "debug":
    staticlibenv.Append(CPPFLAGS=debugcflags)
else:
    staticlibenv.Append(CPPFLAGS=releasecflags)


# get source file list
srclst = map(lambda x: builddir + "/static/" + x, glob.glob("*.cpp"))

# specify the build directory
staticlibenv.BuildDir(builddir + "/static", ".", duplicate=0)

staticlibenv2 = staticlibenv.Clone()
staticlibenv2["PCHSTOP"] = "core/CorePCH.hpp"
# pch.cpp is beside this SConscript
staticlibenv2["PCH"] = staticlibenv2.PCH(builddir + "/static/C.cpp")[0]
print("here 0")
print(staticlibenv2["PCH"])
print("here 1")
additionalcflags = ["-Yl__xge_pch_symbol"]
staticlibenv2.Append(CPPFLAGS=additionalcflags)
staticlib = staticlibenv2.StaticLibrary(targetpath + "/static/core", source=srclst)

SConstruct:

# get the mode flag from the command line
# default to 'release' if the user didn't specify
mode = ARGUMENTS.get("mode", "debug")  # holds current mode
platform = ARGUMENTS.get("platform", "win32")  # holds current platform

# check if the user has been naughty: only 'debug' or 'release' allowed
if not (mode in ["debug", "release"]):
    print("Error: expected 'debug' or 'release', found: " + mymode)
    Exit(1)

# check if the user has been naughty: only 'win32' or 'xbox' allowed
if not (platform in ["win32", "xbox"]):
    print("Error: expected 'win32' or 'release', found: " + platform)
    Exit(1)


# tell the user what we're doing
print("**** Compiling in " + mode + " mode...")

debugcflags = [  # extra compile flags for debug
    "-Od",
    "-MDd",
    "-Ob0",
    "-Z7",
    "-W4",
    "-EHsc",
    "-GR",
    "-D_DEBUG",
    "-DUSE_PCH",
]
releasecflags = [  # extra compile flags for release
    "-O2",
    "-MD",
    "-Ob2",
    "-EHsc",
    "-GR",
    "-DNDEBUG",
    "-DUSE_PCH",
]

env = Environment()

env.Append(CPPPATH=["#include", "#ext/boost-1.34.1"])

if mode == "debug":
    env.Append(LINKFLAGS="/DEBUG")

buildroot = "#build/" + mode  # holds the root of the build directory tree

# make sure the sconscripts can get to the variables
Export("env", "mode", "platform", "debugcflags", "releasecflags")

# put all .sconsign files in one place
env.SConsignFile()

# specify the sconscript for myprogram
project = "lib/core"
SConscript(project + "/SConscript", exports=["buildroot", "project"])

CorePCH.hpp:

#ifndef CORE_PCH_HPP
#define CORE_PCH_HPP
 
#ifdef USE_PCH
#include <string>
#include <iostream>
#include "core/Hello.hpp"
#endif
 
#endif // CORE_PCH_HPP

CorePCH.cpp or C.cpp:

#include "core/CorePCH.hpp"

Hello.hpp:

#ifndef HELLO_HPP
#define HELLO_HPP
 
#ifndef USE_PCH
#include <string>
#endif
 
class Hello
{
    public:
       Hello(const std::string& msg);
       ~Hello();
       void sayIt();
       
   private:
      std::string m_msg; 
};
#endif // HELLO_HPP

Hello.cpp:

#include "core/CorePCH.hpp"
#ifndef USE_PCH
#include "core/Hello.hpp"
#include <iostream>
#endif
 
Hello::Hello(const std::string& msg):m_msg(msg)
{
}
 
Hello::~Hello()
{
}
 
void Hello::sayIt()
{
    std::cout << m_msg << std::endl;
}

jchristian said at 2008-11-02 00:08:53

I made this a P2 because even though it is obscure, it is hella hard to track down what is going on. The workaround is to simply use another filename but this isn't apparent until many hours have been wasted. As well, I suspect the fix will be trivial and from my experience P3 and P4 defects tend to get put off practically forever.

jchristian said at 2008-11-04 00:39:53

Created an attachment (id=531) Zip file that creates the dir structure described in the defect

jchristian said at 2008-11-04 00:49:32

Upon further investigation I have found that the filenames A.cpp thru H.cpp do not work. I have confirmed that I.cpp thru Q.cpp work. And also X.cpp. Other filenames were not tested because I got tired of doing it. To test a different filename, simply rename C.cpp in the lib/core directory and then update the SConscript file in that same directory to use that filename.

My suspicion is that the dependencies are not being generated or kept because of some defect in parsing or concatenating the filename.

jchristian said at 2008-11-04 00:55:43

Hopefully this can be fixed for 1.2

gregnoel said at 2008-12-04 17:41:09

1.2 is frozen; issues that didn't make it into 1.2 are moved to 1.3.

stevenknight said at 2008-12-06 11:22:53

This issue erroneously had a target milestone put on it right away, so it never got discussed and assigned to someone specific. Consequently, it's been languishing in a 1.[23] + issues@scons bucket that no one actually looks at. Change the target milestone to -unspecified- so we can discuss this at the next Bug Party and get it prioritized appropriately.

gregnoel said at 2008-12-24 06:41:53

Bug party triage. Steven suggests that it might be an interaction with the heuristics for determining the drive letter, but otherwise nobody has a clue. Assign to Steven for research as a penalty for suggesting a possibility.

Next time let us assign the milestone and priority. If it doesn't pass through our process, it'll get lost as this one did.

jchristian said at 2008-12-24 12:26:00

Sorry about that, I'll do that in the future. This was my first issue submitted and I wasn't sure what to put on some of the fields :O

stevenknight said at 2009-11-10 18:00:19

stevenknight => issues@scons

gregnoel said at 2010-01-20 02:20:51

Bug party triage. This area has been heavily revamped. Bill to research and see if the problem still happens.

John, can you try your test case with the newest checkpoint (published yesterday) and see if it's been fixed? Add a note to this issue and let us (well, Bill, to be precise) if you still see the problem. Tks.

gregnoel said at 2010-07-21 16:58:11

Bug party triage. Bump this issue to P1 so it will be reviewed at next bug party.

jchristian attached testpch.zip at 2008-11-04 00:39:53.

Zip file that creates the dir structure described in the defect

bdbaddog avatar Jan 02 '18 13:01 bdbaddog

Tagged this with variantdir because that (in the old form BuildDir) appears in the SConscript.

mwichmann avatar Apr 07 '21 14:04 mwichmann

Something about this sounds familiar - wasn't there some ordering issue? #2877 has one of these.

mwichmann avatar Nov 03 '23 17:11 mwichmann

The original filer is using glob.glob('*.cpp') which doesn't guarantee order staying the same between runs... That could have something to do with this..

bdbaddog avatar Nov 04 '23 05:11 bdbaddog

The original filer is using glob.glob('*.cpp') which doesn't guarantee order staying the same between runs... That could have something to do with this..

I've imported the zip file and updated for python3 and BuildDir->VariantDir change. Here: https://github.com/bdbaddog/scons-bugswork/tree/main/2249

This also has the problem where the VariantDir()'s source is a parent of the variantdir..

staticlibenv.VariantDir(builddir+"/static", ".", duplicate=0)

With my updates and one more, it's not finding c.pch in static.

bdbaddog avatar Nov 04 '23 05:11 bdbaddog

Not exactly sure what's the issue.

Here's the generated command line which get's run (after my changes.. I moved variant_dir to the SConscript call so the variantdir wasn't a child of the src dir). When it gets run, it doesn't generated any files? (@jcbrill - any idea? I'm not the PCH expert by far)..

cl /Fobuild\debug\lib\core\static\C.obj /c build\debug\lib\core\static\C.cpp /TP /nologo -Od -MDd -Ob0 -Z7 -W4 -EHsc -GR -D_DEBUG -DUSE_PCH -Yl__xge_pch_symbol /Iinclude /Yucore/CorePCH.hpp /Fpbuild\debug\lib\core\static\C.pch

All my updates are available here: https://github.com/bdbaddog/scons-bugswork/tree/main/2249

bdbaddog avatar Nov 04 '23 06:11 bdbaddog

When it gets run, it doesn't generated any files? (@jcbrill - any idea? I'm not the PCH expert by far)..

I'm not either. When proceeding, keep in mind I'm just trying to help.

The sample files do illustrate a problem though.

Something about this sounds familiar - wasn't there some ordering issue? #2877 has one of these.

There does appear to be a lexical ordering issue.

All my updates are available here: https://github.com/bdbaddog/scons-bugswork/tree/main/2249

Using a recent version of scons with the build files, I had to change one line in the SConscript file due to a "PCHSTOP construction variable must be a string" error.

Original:

staticlibenv2['PCHSTOP'] = File('core/CorePCH.hpp')

Modified:

staticlibenv2['PCHSTOP'] = str(File('core/CorePCH.hpp'))

Working thesis: Any generated pch file name that is lexically ordered before "core" fails while any generated pch file name that is lexically ordered after "core" passes.

File C.cpp (C.pch) is lexically ordered before core and fails. File Z.cpp (Z.pch) is lexically ordered after core and passes.

The results from two builds with and environment dump and command-line options --taskmastertrace - --tree=all are included in the text files corD-fail.txt (source file corD.cpp) and corF-pass.txt. (source file corF.cpp).

corD-fail.txt corF-pass.txt

File corD-fail.txt contains the results when using corD.cpp for the file name (pch_nodes = staticlibenv2.PCH("corD.cpp")) File corF-fail.txt contains the results when using corF.cpp for the file name (pch_nodes = staticlibenv2.PCH("corF.cpp"))

corD-fail.txt

The tree for corD-fail.txt is:

+-.
  +-build
  | +-build\debug
  |   +-build\debug\lib
  |     +-build\debug\lib\core
  |       +-build\debug\lib\core\static
  |         +-build\debug\lib\core\static\corD.cpp
  |         +-build\debug\lib\core\static\corD.obj
  |         | +-build\debug\lib\core\static\corD.cpp
  |         | +-include\core\CorePCH.hpp
  |         | +-include\core\Hello.hpp
  |         | +-C:\Software\MSVS-2022-143-Com\VC\Tools\MSVC\14.37.32822\bin\HostX64\x64\cl.EXE
  |         +-build\debug\lib\core\static\corD.pch
  |         | +-build\debug\lib\core\static\corD.cpp
  |         | +-include\core\CorePCH.hpp
  |         | +-include\core\Hello.hpp
  |         | +-C:\Software\MSVS-2022-143-Com\VC\Tools\MSVC\14.37.32822\bin\HostX64\x64\cl.EXE
  |         +-build\debug\lib\core\static\core
  |         | +-build\debug\lib\core\static\core\CorePCH.hpp
  |         +-build\debug\lib\core\static\core.lib
  |         | +-build\debug\lib\core\static\Hello.obj
  |         | | +-build\debug\lib\core\static\Hello.cpp
  |         | | +-build\debug\lib\core\static\corD.pch
  |         | | | +-build\debug\lib\core\static\corD.cpp
  |         | | | +-include\core\CorePCH.hpp
  |         | | | +-include\core\Hello.hpp
  |         | | | +-C:\Software\MSVS-2022-143-Com\VC\Tools\MSVC\14.37.32822\bin\HostX64\x64\cl.EXE
  |         | | +-include\core\CorePCH.hpp
  |         | | +-include\core\Hello.hpp
  |         | | +-C:\Software\MSVS-2022-143-Com\VC\Tools\MSVC\14.37.32822\bin\HostX64\x64\cl.EXE
  |         | +-build\debug\lib\core\static\corD.obj
  |         | | +-build\debug\lib\core\static\corD.cpp
  |         | | +-include\core\CorePCH.hpp
  |         | | +-include\core\Hello.hpp
  |         | | +-C:\Software\MSVS-2022-143-Com\VC\Tools\MSVC\14.37.32822\bin\HostX64\x64\cl.EXE
  |         | +-C:\Software\MSVS-2022-143-Com\VC\Tools\MSVC\14.37.32822\bin\HostX64\x64\lib.EXE
  |         +-build\debug\lib\core\static\Hello.cpp
  |         +-build\debug\lib\core\static\Hello.obj
  |         | +-build\debug\lib\core\static\Hello.cpp
  |         | +-build\debug\lib\core\static\corD.pch
  |         | | +-build\debug\lib\core\static\corD.cpp
  |         | | +-include\core\CorePCH.hpp
  |         | | +-include\core\Hello.hpp
  |         | | +-C:\Software\MSVS-2022-143-Com\VC\Tools\MSVC\14.37.32822\bin\HostX64\x64\cl.EXE
  |         | +-include\core\CorePCH.hpp
  |         | +-include\core\Hello.hpp
  |         | +-C:\Software\MSVS-2022-143-Com\VC\Tools\MSVC\14.37.32822\bin\HostX64\x64\cl.EXE
  |         +-build\debug\lib\core\static\SConscript
  +-include
  | +-include\core
  |   +-include\core\CorePCH.hpp
  |   +-include\core\Hello.hpp
  +-lib
  | +-lib\core
  |   +-lib\core\corD.cpp
  |   +-lib\core\core
  |   | +-lib\core\core\CorePCH.hpp
  |   +-lib\core\Hello.cpp
  |   +-lib\core\SConscript
  +-SConstruct

corF-pass.txt

The tree for corF-pass.txt is:

+-.
  +-build
  | +-build\debug
  |   +-build\debug\lib
  |     +-build\debug\lib\core
  |       +-build\debug\lib\core\static
  |         +-build\debug\lib\core\static\core
  |         | +-build\debug\lib\core\static\core\CorePCH.hpp
  |         +-build\debug\lib\core\static\core.lib
  |         | +-build\debug\lib\core\static\Hello.obj
  |         | | +-build\debug\lib\core\static\Hello.cpp
  |         | | +-build\debug\lib\core\static\corF.pch
  |         | | | +-build\debug\lib\core\static\corF.cpp
  |         | | | +-include\core\CorePCH.hpp
  |         | | | +-include\core\Hello.hpp
  |         | | | +-C:\Software\MSVS-2022-143-Com\VC\Tools\MSVC\14.37.32822\bin\HostX64\x64\cl.EXE
  |         | | +-include\core\CorePCH.hpp
  |         | | +-include\core\Hello.hpp
  |         | | +-C:\Software\MSVS-2022-143-Com\VC\Tools\MSVC\14.37.32822\bin\HostX64\x64\cl.EXE
  |         | +-build\debug\lib\core\static\corF.obj
  |         | | +-build\debug\lib\core\static\corF.cpp
  |         | | +-include\core\CorePCH.hpp
  |         | | +-include\core\Hello.hpp
  |         | | +-C:\Software\MSVS-2022-143-Com\VC\Tools\MSVC\14.37.32822\bin\HostX64\x64\cl.EXE
  |         | +-C:\Software\MSVS-2022-143-Com\VC\Tools\MSVC\14.37.32822\bin\HostX64\x64\lib.EXE
  |         +-build\debug\lib\core\static\corF.cpp
  |         +-build\debug\lib\core\static\corF.obj
  |         | +-build\debug\lib\core\static\corF.cpp
  |         | +-include\core\CorePCH.hpp
  |         | +-include\core\Hello.hpp
  |         | +-C:\Software\MSVS-2022-143-Com\VC\Tools\MSVC\14.37.32822\bin\HostX64\x64\cl.EXE
  |         +-build\debug\lib\core\static\corF.pch
  |         | +-build\debug\lib\core\static\corF.cpp
  |         | +-include\core\CorePCH.hpp
  |         | +-include\core\Hello.hpp
  |         | +-C:\Software\MSVS-2022-143-Com\VC\Tools\MSVC\14.37.32822\bin\HostX64\x64\cl.EXE
  |         +-build\debug\lib\core\static\Hello.cpp
  |         +-build\debug\lib\core\static\Hello.obj
  |         | +-build\debug\lib\core\static\Hello.cpp
  |         | +-build\debug\lib\core\static\corF.pch
  |         | | +-build\debug\lib\core\static\corF.cpp
  |         | | +-include\core\CorePCH.hpp
  |         | | +-include\core\Hello.hpp
  |         | | +-C:\Software\MSVS-2022-143-Com\VC\Tools\MSVC\14.37.32822\bin\HostX64\x64\cl.EXE
  |         | +-include\core\CorePCH.hpp
  |         | +-include\core\Hello.hpp
  |         | +-C:\Software\MSVS-2022-143-Com\VC\Tools\MSVC\14.37.32822\bin\HostX64\x64\cl.EXE
  |         +-build\debug\lib\core\static\SConscript
  +-include
  | +-include\core
  |   +-include\core\CorePCH.hpp
  |   +-include\core\Hello.hpp
  +-lib
  | +-lib\core
  |   +-lib\core\core
  |   | +-lib\core\core\CorePCH.hpp
  |   +-lib\core\corF.cpp
  |   +-lib\core\Hello.cpp
  |   +-lib\core\SConscript
  +-SConstruct

Hope this points someone in the right direction.

Edit: Does the lexical ordering issue indicate a potential dependency issue?

jcbrill avatar Nov 04 '23 10:11 jcbrill

The original filer is using glob.glob('*.cpp') which doesn't guarantee order staying the same between runs... That could have something to do with this..

I guess we've each "ported" this example in different ways...

Yes the sort order seems to matter, but it's not caused by using glob here (1) - the list ends up (for me at least) in alphabetical order anyway, and the key seems to be that when the source to be fed to the PCH builder is later in the source list it works, and when it's first it doesn't. I ran the example of taking a "working" setup and sorting the list in reverse, and it goes back to failing. That is:

# Working:

DEBUG: ['#build/debug/lib/core/static/Hello.cpp', '#build/debug/lib/core/static/T.cpp']
scons: done reading SConscript files.
scons: Building targets ...
cl /Fobuild\debug\lib\core\static\T.obj /TP /nologo -Od -MDd -Ob0 -Z7 -W4 -EHsc -GR -D_DEBUG -DUSE_PCH -Yl__xge_pch_symbol /Iinclude /c lib\core\T.cpp /Yccore/CorePCH.hpp /Fpbuild\debug\lib\core\static\T.pch T.cpp
cl /Fobuild\debug\lib\core\static\Hello.obj /c lib\core\Hello.cpp /TP /nologo -Od -MDd -Ob0 -Z7 -W4 -EHsc -GR -D_DEBUG -DUSE_PCH -Yl__xge_pch_symbol /Iinclude /Yucore/CorePCH.hpp /Fpbuild\debug\lib\core\static\T.pch Hello.cpp
lib /nologo /OUT:build\debug\lib\core\static\core.lib build\debug\lib\core\static\Hello.obj build\debug\lib\core\static\T.obj
scons: done building targets.

# Not working:

DEBUG: ['#build/debug/lib/core/static/T.cpp', '#build/debug/lib/core/static/Hello.cpp']
scons: done reading SConscript files.
scons: Building targets ...
cl /Fobuild\debug\lib\core\static\T.obj /c lib\core\T.cpp /TP /nologo -Od -MDd -Ob0 -Z7 -W4 -EHsc -GR -D_DEBUG -DUSE_PCH -Yl__xge_pch_symbol /Iinclude /Yucore/CorePCH.hpp /Fpbuild\debug\lib\core\static\T.pch T.cpp
lib\core\T.cpp(1): fatal error C1083: Cannot open precompiled header file: 'build\debug\lib\core\static\T.pch': No such file or directory

What's head-scratching there is the line which should end up building the pch comes out ordered a bit differently, though it appears to have identical contents.

The two dep tree dumps @jcbrill posted have the same contents, as well, accounting for the different filename. The sort order differs, I don't know if that's because we sort the deps when we emit the tree dump, or whether SCons actually has an internal difference that matters.

(1) footnote: I imagine this example uses glob.glob instead of Glob because the author is convinced of needing to do all this path fiddling, Trying to not let that get in the way of thinking about this thing...

mwichmann avatar Nov 04 '23 14:11 mwichmann

Bear in mind that I don't know what I'm talking about.

The runs that pass seem to consider node build\\debug\\lib\\core\\static\\core before any individual file nodes (e.g., build\\debug\\lib\\core\\static\\corF.cpp).

What does the node build\\debug\\lib\\core\\static\\core represent? I'm guessing it is the static library itself.

In runs that fail, the taskmaster evaluates child file nodes before the "root" node.

Passes:

Taskmaster:      adjusted ref count: <pending    1   'build\\debug\\lib\\core'>, child 'build\\debug\\lib\\core\\static'
Taskmaster:     Considering node <no_state   0   'build\\debug\\lib\\core\\static'> and its children:
Taskmaster:        <no_state   0   'build\\debug\\lib\\core\\static\\core'>         # <- root object?
Taskmaster:        <no_state   0   'build\\debug\\lib\\core\\static\\core.lib'>
Taskmaster:        <no_state   0   'build\\debug\\lib\\core\\static\\corF.cpp'>
Taskmaster:        <no_state   0   'build\\debug\\lib\\core\\static\\corF.obj'>
Taskmaster:        <no_state   0   'build\\debug\\lib\\core\\static\\corF.pch'>
Taskmaster:        <no_state   0   'build\\debug\\lib\\core\\static\\Hello.cpp'>
Taskmaster:        <no_state   0   'build\\debug\\lib\\core\\static\\Hello.obj'>
Taskmaster:        <no_state   0   'build\\debug\\lib\\core\\static\\SConscript'>
Taskmaster:      adjusted ref count: <pending    1   'build\\debug\\lib\\core\\static'>, child 'build\\debug\\lib\\core\\static\\core'
Taskmaster:      adjusted ref count: <pending    2   'build\\debug\\lib\\core\\static'>, child 'build\\debug\\lib\\core\\static\\core.lib'
Taskmaster:      adjusted ref count: <pending    3   'build\\debug\\lib\\core\\static'>, child 'build\\debug\\lib\\core\\static\\corF.cpp'
Taskmaster:      adjusted ref count: <pending    4   'build\\debug\\lib\\core\\static'>, child 'build\\debug\\lib\\core\\static\\corF.obj'
Taskmaster:      adjusted ref count: <pending    5   'build\\debug\\lib\\core\\static'>, child 'build\\debug\\lib\\core\\static\\corF.pch'
Taskmaster:      adjusted ref count: <pending    6   'build\\debug\\lib\\core\\static'>, child 'build\\debug\\lib\\core\\static\\Hello.cpp'
Taskmaster:      adjusted ref count: <pending    7   'build\\debug\\lib\\core\\static'>, child 'build\\debug\\lib\\core\\static\\Hello.obj'
Taskmaster:      adjusted ref count: <pending    8   'build\\debug\\lib\\core\\static'>, child 'build\\debug\\lib\\core\\static\\SConscript'
Taskmaster:     Considering node <no_state   0   'build\\debug\\lib\\core\\static\\core'> and its children:
Taskmaster:        <no_state   0   'build\\debug\\lib\\core\\static\\core\\CorePCH.hpp'>
Taskmaster:      adjusted ref count: <pending    1   'build\\debug\\lib\\core\\static\\core'>, child 'build\\debug\\lib\\core\\static\\core\\CorePCH.hpp'
Taskmaster:     Considering node <no_state   0   'build\\debug\\lib\\core\\static\\core\\CorePCH.hpp'> and its children:
Taskmaster: Evaluating <pending    0   'build\\debug\\lib\\core\\static\\core\\CorePCH.hpp'>

Fails:

Taskmaster:     Considering node <no_state   0   'build\\debug\\lib\\core\\static'> and its children:
Taskmaster:        <no_state   0   'build\\debug\\lib\\core\\static\\corD.cpp'>
Taskmaster:        <no_state   0   'build\\debug\\lib\\core\\static\\corD.obj'>
Taskmaster:        <no_state   0   'build\\debug\\lib\\core\\static\\corD.pch'>
Taskmaster:        <no_state   0   'build\\debug\\lib\\core\\static\\core'>            # <-- root object?
Taskmaster:        <no_state   0   'build\\debug\\lib\\core\\static\\core.lib'>
Taskmaster:        <no_state   0   'build\\debug\\lib\\core\\static\\Hello.cpp'>
Taskmaster:        <no_state   0   'build\\debug\\lib\\core\\static\\Hello.obj'>
Taskmaster:        <no_state   0   'build\\debug\\lib\\core\\static\\SConscript'>
Taskmaster:      adjusted ref count: <pending    1   'build\\debug\\lib\\core\\static'>, child 'build\\debug\\lib\\core\\static\\corD.cpp'
Taskmaster:      adjusted ref count: <pending    2   'build\\debug\\lib\\core\\static'>, child 'build\\debug\\lib\\core\\static\\corD.obj'
Taskmaster:      adjusted ref count: <pending    3   'build\\debug\\lib\\core\\static'>, child 'build\\debug\\lib\\core\\static\\corD.pch'
Taskmaster:      adjusted ref count: <pending    4   'build\\debug\\lib\\core\\static'>, child 'build\\debug\\lib\\core\\static\\core'
Taskmaster:      adjusted ref count: <pending    5   'build\\debug\\lib\\core\\static'>, child 'build\\debug\\lib\\core\\static\\core.lib'
Taskmaster:      adjusted ref count: <pending    6   'build\\debug\\lib\\core\\static'>, child 'build\\debug\\lib\\core\\static\\Hello.cpp'
Taskmaster:      adjusted ref count: <pending    7   'build\\debug\\lib\\core\\static'>, child 'build\\debug\\lib\\core\\static\\Hello.obj'
Taskmaster:      adjusted ref count: <pending    8   'build\\debug\\lib\\core\\static'>, child 'build\\debug\\lib\\core\\static\\SConscript'
Taskmaster:     Considering node <no_state   0   'build\\debug\\lib\\core\\static\\corD.cpp'> and its children:
Taskmaster: Evaluating <pending    0   'build\\debug\\lib\\core\\static\\corD.cpp'>

jcbrill avatar Nov 04 '23 14:11 jcbrill

Just a WAG and likely misinformation, but it seems the difference in evaluation order of the DAG could be responsible for the difference in behavior. Maybe?

jcbrill avatar Nov 04 '23 14:11 jcbrill

Bear in mind that I don't know what I'm talking about.

The runs that pass seem to consider node build\\debug\\lib\\core\\static\\core before any individual file nodes (e.g., build\\debug\\lib\\core\\static\\corF.cpp).

In a case-insenstive sort, which I assume SCons thinks it needs to do because it's a Windows file system, "core" is before corF". It's probably not more than that... although this seems the wrong place to be worrying about sort order...

What does the node build\\debug\\lib\\core\\static\\core represent? I'm guessing it is the static library itself.

This is probably because SCons lets you call targets by un-suffixed names, aka StaticLibrary(target="core", source=some-list), and so it wants to keep track of that as well as the produced output file name. At this point /me/ just guessing, though.

mwichmann avatar Nov 04 '23 14:11 mwichmann

We always say "the taskmaster will figure it out", but there have been more than one case where things that are essentially "side effects" which also become sources don't work out right - I think there are several closed PCH bugs about that problem, maybe it wasn't completely stamped out in the past...

mwichmann avatar Nov 04 '23 15:11 mwichmann

What's head-scratching there is the line which should end up building the pch comes out ordered a bit differently, though it appears to have identical contents.

The lines to use (/Yu) the generated pch file are identical in both listings.

The line to create (/Yc) the generated pch only appears in the first listing:

cl /Fobuild\debug\lib\core\static\T.obj /TP /nologo -Od -MDd -Ob0 -Z7 -W4 -EHsc -GR -D_DEBUG -DUSE_PCH -Yl__xge_pch_symbol /Iinclude /c lib\core\T.cpp /Yccore/CorePCH.hpp /Fpbuild\debug\lib\core\static\T.pch T.cpp

jcbrill avatar Nov 04 '23 15:11 jcbrill

You're absolutely right, that is the difference, it fooled my old tired eyes, because after I moved the /c ... stanza in one copy, they were the same length and looked the same, but they're not. Sigh. Thanks for spotting.

/Yccore/CorePCH.hpp  # <- working
/Yucore/CorePCH.hpp  # <- not working

mwichmann avatar Nov 04 '23 15:11 mwichmann

Stupid question time.

In the abstract, it appears that there is a difference when processing the DAG by the taskmaster whether or not the static library node is considered first or not.

Is it possible that the the pch source file and target are "disconnected" from the DAG when evaluated prior to the static library node?

That is, is it possible that the pch source/target is evaluated before a parent/child relationship is established by processing the static library node first?

I don't know anything about how the DAG is constructed. "Construct and then evaluate" is different from "construct and evaluate during discovery". In the latter case, starting/root nodes might matter.

jcbrill avatar Nov 04 '23 15:11 jcbrill

This also has the problem where the VariantDir()'s source is a parent of the variantdir..

staticlibenv.VariantDir(builddir+"/static", ".", duplicate=0)

@bdbaddog you keep saying this, but there are so many testcases that do exactly this, as well as documentation and examples, that I can't help but think it was intended to work... just from a quick search:

SCons/Environment.xml:VariantDir('build', '.', duplicate=0)
SCons/Script/SConscript.xml:SConscript(dirs = 'src', variant_dir = 'build', src_dir = '.')
SCons/Script/SConscript.xml:VariantDir('build', '.') 

SCons/Node/FSTests.py:        fs.VariantDir('build', '.')
test/Configure/VariantDir2.py:SConscript('SConscript', variant_dir='build', src='.')
test/Configure/VariantDir.py:VariantDir('build', '.')
test/Configure/VariantDir-SConscript.py:VariantDir( 'build', '.' )
testing/framework/TestSCons.py:    VariantDir(builddir, '.', duplicate=dup)
test/option/option--duplicate.py:VariantDir('build', '.', duplicate=1)
test/QT/qt3/installed.py:VariantDir('bld', '.')
test/QT/qt3/QTFLAGS.py:    VariantDir('build', '.', duplicate=1)
test/VariantDir/Clean.py:VariantDir('build0', '.', duplicate=0)
test/VariantDir/Clean.py:VariantDir('build1', '.', duplicate=1)
test/VariantDir/guess-subdir.py:VariantDir(c_builddir, '.', duplicate=0)
test/VariantDir/SConscript-variant_dir.py:SConscript('SConscript', variant_dir='Build', src_dir='.', duplicate=0)
test/VariantDir/SConscript-variant_dir.py:SConscript('src/SConscript', variant_dir='build/var9', src_dir='.')
test/VariantDir/SConscript-variant_dir.py:VariantDir('build/var9', '.')
test/VariantDir/VariantDir.py:VariantDir('build', '.')
test/VariantDir/VariantDir.py:VariantDir('build', '.', duplicate=1 )

mwichmann avatar Nov 04 '23 17:11 mwichmann

I've wasted a little more time poking at this, and I'm pretty convinced everything is consistent coming out of the sconscript phase - the PCH builder has been invoked the same way and produced the same result - and that the problem indeed happens at taskmaster time. Won't put any money on it, though :-)

mwichmann avatar Nov 04 '23 21:11 mwichmann

@mwichmann When you have time, would you please review to see if the following makes any sense.

My comment about the commands in your runs is not quite true. In both runs the first compiler command is for the pch file but with different switches for pch (/Yc vs /Yu).

The difference between the builds that fail and the builds that pass is that in the failed builds the first generated compile command is not creating the precompiled header files but instead is generated to use the precompiled header file.

I tried another pair of tests. The file names were changed so they both lexically precede "core".

Test 1 - [pch source file discovered first]: a-pch.cpp [formerly C.cpp] b-hello.cpp [formerly Hello.cpp]

Test 1 - Fails:

cl /Fobuild\debug\lib\core\static\a-pch.obj /c build\debug\lib\core\static\a-pch.cpp /TP /nologo -Od -MDd -Ob0 -Z7 -W4 -EHsc -GR -D_DEBUG -DUSE_PCH -Yl__xge_pch_symbol /Iinclude /Yucore\CorePCH.hpp /Fpbuild\debug\lib\core\static\a-pch.pch

The pre-compiled header source file command-line should be creating (/Yc) the pch header instead of using (/Yu) the pch header.

Test 2 - [pch source file discovered after source file]: b-pch.cpp [formerly C.cpp] a-hellp.cpp [formerly Hello.cpp]

Test 2 - Passes:

cl /Fobuild\debug\lib\core\static\b-pch.obj /TP /nologo -Od -MDd -Ob0 -Z7 -W4 -EHsc -GR -D_DEBUG -DUSE_PCH -Yl__xge_pch_symbol /Iinclude /c build\debug\lib\core\static\b-pch.cpp /Yccore\CorePCH.hpp /Fpbuild\debug\lib\core\static\b-pch.pch
cl /Fobuild\debug\lib\core\static\a-hello.obj /c build\debug\lib\core\static\a-hello.cpp /TP /nologo -Od -MDd -Ob0 -Z7 -W4 -EHsc -GR -D_DEBUG -DUSE_PCH -Yl__xge_pch_symbol /Iinclude /Yucore\CorePCH.hpp /Fpbuild\debug\lib\core\static\b-pch.pch

The precompiled header source file command-line is creating (/Yu) the pch header.

Observation: In the fail build, the pch source file is the first compiled and the command-line is using (/Yu) the compiled header file. In the pass build, the pch source file is the first compiled and the command-line is creating (/Yc) the compiled header file.

When the pch source file is discovered first, the command-line is not generated with the create (/Yc) option.

From your run above, the first compiler calls from both runs are shown (manually adjusted the options order so they line-up visually):

cl /Fobuild\debug\lib\core\static\T.obj /c lib\core\T.cpp /TP /nologo -Od -MDd -Ob0 -Z7 -W4 -EHsc -GR -D_DEBUG -DUSE_PCH -Yl__xge_pch_symbol /Iinclude /Yccore/CorePCH.hpp /Fpbuild\debug\lib\core\static\T.pch T.cpp
cl /Fobuild\debug\lib\core\static\T.obj /c lib\core\T.cpp /TP /nologo -Od -MDd -Ob0 -Z7 -W4 -EHsc -GR -D_DEBUG -DUSE_PCH -Yl__xge_pch_symbol /Iinclude /Yucore/CorePCH.hpp /Fpbuild\debug\lib\core\static\T.pch T.cpp

The only difference is the successful build creates (/Yc) the pch file while the failed build uses (/Yc) the pch file.

jcbrill avatar Nov 05 '23 00:11 jcbrill

Based on the generated argument order and pch switch difference for the pch source file, I'm starting to think that PCHCOM is not called/applied when the pch source file is discovered first:

cl /Fobuild\debug\lib\core\static\b-pch.obj /TP /nologo -Od -MDd -Ob0 -Z7 -W4 -EHsc -GR -D_DEBUG -DUSE_PCH -Yl__xge_pch_symbol /Iinclude /c build\debug\lib\core\static\b-pch.cpp /Yccore\CorePCH.hpp /Fpbuild\debug\lib\core\static\b-pch.pch

cl /Fobuild\debug\lib\core\static\a-pch.obj /c build\debug\lib\core\static\a-pch.cpp /TP /nologo -Od -MDd -Ob0 -Z7 -W4 -EHsc -GR -D_DEBUG -DUSE_PCH -Yl__xge_pch_symbol /Iinclude /Yucore\CorePCH.hpp /Fpbuild\debug\lib\core\static\a-pch.pch

jcbrill avatar Nov 05 '23 01:11 jcbrill

The problem with having the variant dir be a child of the source is that then on the second run the build dir itself is the source for build/build.. and so on iteratively... It's just a BAD idea. This can lead to really difficult to debug situations.. so I always advise not to do it. There's little to gain beside some insistance on a structure which is problematic.

Now take the above and add more than one variant where all the N variants are children of the source dir..

If it happens to work sometimes, that doesn't in any way mean it's a good idea.

bdbaddog avatar Nov 05 '23 09:11 bdbaddog

@mwichmann When you have time, would you please review to see if the following makes any sense.

The difference between the builds that fail and the builds that pass is that in the failed builds the first generated compile command is not creating the precompiled header files but instead is generated to use the precompiled header file.

Yes, this is what I was saying, and also that there doesn't seem to be any difference coming out of the emitter in the PCH builder, so there's some other problem in the plumbing. Not sure how well the PCH logic understands the current model anyway. We have a construction variable (which this example sets):

https://scons.org/doc/production/HTML/scons-man.html#cv-PCH

which is per-environment, but that's not really a requirement:

Although you can use only one precompiled header (.pch) file per source file, you can use multiple .pch files in a project.

mwichmann avatar Nov 05 '23 14:11 mwichmann

Old-school print-style debugging.

The builds that pass generate a command to build the pch and obj file. The builds that fail generate a command to build the obj file.

PASS

Building build\debug\lib\core\static\b-pch.pch and build\debug\lib\core\static\b-pch.obj with action:
  $CXX /Fo${TARGETS[1]} $CXXFLAGS $CCFLAGS $CPPFLAGS $_CPPDEFFLAGS $_CPPINCFLAGS /c $SOURCES /Yc$PCHSTOP /Fp${TARGETS[0]} $CCPDBFLAGS $PCHPDBFLAGS

ACTION.PROCESS 
  $CXX /Fo${TARGETS[1]} $CXXFLAGS $CCFLAGS $CPPFLAGS $_CPPDEFFLAGS $_CPPINCFLAGS /c $SOURCES /Yc$PCHSTOP /Fp${TARGETS[0]} $CCPDBFLAGS $PCHPDBFLAGS 
  [['cl', '/Fobuild\\debug\\lib\\core\\static\\b-pch.obj', '/TP', '/nologo', '-Od', '-MDd', '-Ob0', '-Z7', '-W4', '-EHsc', '-GR', '-D_DEBUG', '-DUSE_PCH', '-Yl__xge_pch_symbol', '/Iinclude', '/c', 'build\\debug\\lib\\core\\static\\b-pch.cpp', '/Yccore\\CorePCH.hpp', '/Fpbuild\\debug\\lib\\core\\static\\b-pch.pch']] 
  None 
  None

cl /Fobuild\debug\lib\core\static\b-pch.obj /TP /nologo -Od -MDd -Ob0 -Z7 -W4 -EHsc -GR -D_DEBUG -DUSE_PCH -Yl__xge_pch_symbol /Iinclude /c build\debug\lib\core\static\b-pch.cpp /Yccore\CorePCH.hpp /Fpbuild\debug\lib\core\static\b-pch.pch

FAIL

Building build\debug\lib\core\static\a-pch.obj with action:
  ${TEMPFILE("$CXX $_MSVC_OUTPUT_FLAG /c $CHANGED_SOURCES $CXXFLAGS $CCFLAGS $_CCCOMCOM","$CXXCOMSTR")}

...

ACTION.PROCESS 
  ${TEMPFILE("$CXX $_MSVC_OUTPUT_FLAG /c $CHANGED_SOURCES $CXXFLAGS $CCFLAGS $_CCCOMCOM","$CXXCOMSTR")} 
  [['cl', '/Fobuild\\debug\\lib\\core\\static\\a-pch.obj', '/c', 'build\\debug\\lib\\core\\static\\a-pch.cpp', '/TP', '/nologo', '-Od', '-MDd', '-Ob0', '-Z7', '-W4', '-EHsc', '-GR', '-D_DEBUG', '-DUSE_PCH', '-Yl__xge_pch_symbol', '/Iinclude', '/Yucore\\CorePCH.hpp', '/Fpbuild\\debug\\lib\\core\\static\\a-pch.pch']] 
  None 
  None

cl /Fobuild\debug\lib\core\static\a-pch.obj /c build\debug\lib\core\static\a-pch.cpp /TP /nologo -Od -MDd -Ob0 -Z7 -W4 -EHsc -GR -D_DEBUG -DUSE_PCH -Yl__xge_pch_symbol /Iinclude /Yucore\CorePCH.hpp /Fpbuild\debug\lib\core\static\a-pch.pch

jcbrill avatar Nov 05 '23 14:11 jcbrill

Edit:

  • change FILENAME.pch to FILENAME.cpp in 3rd paragraph
  • change a-pch.cpp to a-pch.pch in annotation for first listing.

In the builds that fail, the taskmaster is not evaluating the FILENAME.pch node before the object file is created.

In the builds that pass, the taskmaster evaluates one the source files first and this creates a dependency on the FILENAME.pch node which in turn will require the pch node to be evaluated. It appears that evaluating the FILENAME.pch node is what generates the pch creation command-line.

It appears that when the FILENAME.cpp node is evaluated before any other node, there is not a dependency on the FILENAME.pch node which means the FILENAME.pch node is not evaluated which means the pch is not created.

In the runs that fail, the FILENAME.cpp for the precompiled header has no dependency on the FILENAME.pch node so generation of the object file can take place immediately (i.e., before any of the source files or generation of the pch file).

In the runs that pass, the FILENAME.cpp and FILENAME.pch dependencies are established and traversing the DAG works as expected for code generation.

There are a handful of annotations in the output listings.

Fail:

Taskmaster:     Considering node <no_state   0   'build\\debug\\lib\\core\\static'> and its children:
Taskmaster:        <no_state   0   'build\\debug\\lib\\core\\static\\a-pch.cpp'>
Taskmaster:        <no_state   0   'build\\debug\\lib\\core\\static\\a-pch.obj'>
Taskmaster:        <no_state   0   'build\\debug\\lib\\core\\static\\a-pch.pch'>
Taskmaster:        <no_state   0   'build\\debug\\lib\\core\\static\\b-hello.cpp'>
Taskmaster:        <no_state   0   'build\\debug\\lib\\core\\static\\b-hello.obj'>
Taskmaster:        <no_state   0   'build\\debug\\lib\\core\\static\\core'>
Taskmaster:        <no_state   0   'build\\debug\\lib\\core\\static\\core.lib'>
Taskmaster:        <no_state   0   'build\\debug\\lib\\core\\static\\SConscript'>
Taskmaster:      adjusted ref count: <pending    1   'build\\debug\\lib\\core\\static'>, child 'build\\debug\\lib\\core\\static\\a-pch.cpp'
Taskmaster:      adjusted ref count: <pending    2   'build\\debug\\lib\\core\\static'>, child 'build\\debug\\lib\\core\\static\\a-pch.obj'
Taskmaster:      adjusted ref count: <pending    3   'build\\debug\\lib\\core\\static'>, child 'build\\debug\\lib\\core\\static\\a-pch.pch'
Taskmaster:      adjusted ref count: <pending    4   'build\\debug\\lib\\core\\static'>, child 'build\\debug\\lib\\core\\static\\b-hello.cpp'
Taskmaster:      adjusted ref count: <pending    5   'build\\debug\\lib\\core\\static'>, child 'build\\debug\\lib\\core\\static\\b-hello.obj'
Taskmaster:      adjusted ref count: <pending    6   'build\\debug\\lib\\core\\static'>, child 'build\\debug\\lib\\core\\static\\core'
Taskmaster:      adjusted ref count: <pending    7   'build\\debug\\lib\\core\\static'>, child 'build\\debug\\lib\\core\\static\\core.lib'
Taskmaster:      adjusted ref count: <pending    8   'build\\debug\\lib\\core\\static'>, child 'build\\debug\\lib\\core\\static\\SConscript'
Taskmaster:     Considering node <no_state   0   'build\\debug\\lib\\core\\static\\a-pch.cpp'> and its children:
Taskmaster: Evaluating <pending    0   'build\\debug\\lib\\core\\static\\a-pch.cpp'>

!!! a-pch.cpp [NO REFERENCE to a-pch.pch] !!!
Task.make_ready_current(): node <pending    0   'build\\debug\\lib\\core\\static\\a-pch.cpp'>
Task.prepare():      node <up_to_date 0   'build\\debug\\lib\\core\\static\\a-pch.cpp'>
Task.executed_with_callbacks(): node <up_to_date 0   'build\\debug\\lib\\core\\static\\a-pch.cpp'>
Task.postprocess():  node <up_to_date 0   'build\\debug\\lib\\core\\static\\a-pch.cpp'>
Task.postprocess():  removing <up_to_date 0   'build\\debug\\lib\\core\\static\\a-pch.cpp'>
Task.postprocess():  adjusted parent ref count <pending    7   'build\\debug\\lib\\core\\static'>

Taskmaster: Looking for a node to evaluate
!!! a-pch.obj [NO REFERENCE to a-pch.pch] !!!
Taskmaster:     Considering node <no_state   0   'build\\debug\\lib\\core\\static\\a-pch.obj'> and its children:
Taskmaster:        <up_to_date 0   'build\\debug\\lib\\core\\static\\a-pch.cpp'>
Taskmaster:        <no_state   0   'include\\core\\CorePCH.hpp'>
Taskmaster:        <no_state   0   'include\\core\\Hello.hpp'>
Taskmaster:        <no_state   0   'C:\\Software\\MSVS-2022-143-Com\\VC\\Tools\\MSVC\\14.37.32822\\bin\\HostX64\\x64\\cl.EXE'>
Taskmaster:      adjusted ref count: <pending    1   'build\\debug\\lib\\core\\static\\a-pch.obj'>, child 'include\\core\\CorePCH.hpp'
Taskmaster:      adjusted ref count: <pending    2   'build\\debug\\lib\\core\\static\\a-pch.obj'>, child 'include\\core\\Hello.hpp'
Taskmaster:      adjusted ref count: <pending    3   'build\\debug\\lib\\core\\static\\a-pch.obj'>, child 'C:\\Software\\MSVS-2022-143-Com\\VC\\Tools\\MSVC\\14.37.32822\\bin\\HostX64\\x64\\cl.EXE'
Taskmaster:     Considering node <no_state   0   'include\\core\\CorePCH.hpp'> and its children:
Taskmaster: Evaluating <pending    0   'include\\core\\CorePCH.hpp'>

Task.make_ready_current(): node <pending    0   'include\\core\\CorePCH.hpp'>
Task.prepare():      node <up_to_date 0   'include\\core\\CorePCH.hpp'>
Task.executed_with_callbacks(): node <up_to_date 0   'include\\core\\CorePCH.hpp'>
Task.postprocess():  node <up_to_date 0   'include\\core\\CorePCH.hpp'>
Task.postprocess():  removing <up_to_date 0   'include\\core\\CorePCH.hpp'>
Task.postprocess():  adjusted parent ref count <pending    2   'build\\debug\\lib\\core\\static\\a-pch.obj'>

Taskmaster: Looking for a node to evaluate
Taskmaster:     Considering node <no_state   0   'include\\core\\Hello.hpp'> and its children:
Taskmaster: Evaluating <pending    0   'include\\core\\Hello.hpp'>

Task.make_ready_current(): node <pending    0   'include\\core\\Hello.hpp'>
Task.prepare():      node <up_to_date 0   'include\\core\\Hello.hpp'>
Task.executed_with_callbacks(): node <up_to_date 0   'include\\core\\Hello.hpp'>
Task.postprocess():  node <up_to_date 0   'include\\core\\Hello.hpp'>
Task.postprocess():  removing <up_to_date 0   'include\\core\\Hello.hpp'>
Task.postprocess():  adjusted parent ref count <pending    1   'build\\debug\\lib\\core\\static\\a-pch.obj'>

Taskmaster: Looking for a node to evaluate
Taskmaster:     Considering node <no_state   0   'C:\\Software\\MSVS-2022-143-Com\\VC\\Tools\\MSVC\\14.37.32822\\bin\\HostX64\\x64\\cl.EXE'> and its children:
Taskmaster: Evaluating <pending    0   'C:\\Software\\MSVS-2022-143-Com\\VC\\Tools\\MSVC\\14.37.32822\\bin\\HostX64\\x64\\cl.EXE'>

Task.make_ready_current(): node <pending    0   'C:\\Software\\MSVS-2022-143-Com\\VC\\Tools\\MSVC\\14.37.32822\\bin\\HostX64\\x64\\cl.EXE'>
Task.prepare():      node <up_to_date 0   'C:\\Software\\MSVS-2022-143-Com\\VC\\Tools\\MSVC\\14.37.32822\\bin\\HostX64\\x64\\cl.EXE'>
Task.executed_with_callbacks(): node <up_to_date 0   'C:\\Software\\MSVS-2022-143-Com\\VC\\Tools\\MSVC\\14.37.32822\\bin\\HostX64\\x64\\cl.EXE'>
Task.postprocess():  node <up_to_date 0   'C:\\Software\\MSVS-2022-143-Com\\VC\\Tools\\MSVC\\14.37.32822\\bin\\HostX64\\x64\\cl.EXE'>
Task.postprocess():  removing <up_to_date 0   'C:\\Software\\MSVS-2022-143-Com\\VC\\Tools\\MSVC\\14.37.32822\\bin\\HostX64\\x64\\cl.EXE'>
Task.postprocess():  adjusted parent ref count <pending    0   'build\\debug\\lib\\core\\static\\a-pch.obj'>

Taskmaster: Looking for a node to evaluate
Taskmaster:     Considering node <pending    0   'build\\debug\\lib\\core\\static\\a-pch.obj'> and its children:
Taskmaster:        <up_to_date 0   'build\\debug\\lib\\core\\static\\a-pch.cpp'>
Taskmaster:        <up_to_date 0   'include\\core\\CorePCH.hpp'>
Taskmaster:        <up_to_date 0   'include\\core\\Hello.hpp'>
Taskmaster:        <up_to_date 0   'C:\\Software\\MSVS-2022-143-Com\\VC\\Tools\\MSVC\\14.37.32822\\bin\\HostX64\\x64\\cl.EXE'>
Taskmaster: Evaluating <pending    0   'build\\debug\\lib\\core\\static\\a-pch.obj'>

Task.make_ready_current(): node <pending    0   'build\\debug\\lib\\core\\static\\a-pch.obj'>
Task.prepare():      node <executing  0   'build\\debug\\lib\\core\\static\\a-pch.obj'>
Task.execute():      node <executing  0   'build\\debug\\lib\\core\\static\\a-pch.obj'>

!!! NODE a-pch.pch NOT EVALUATED at this point !!!

Building build\debug\lib\core\static\a-pch.obj with action:
  ${TEMPFILE("$CXX $_MSVC_OUTPUT_FLAG /c $CHANGED_SOURCES $CXXFLAGS $CCFLAGS $_CCCOMCOM","$CXXCOMSTR")}

cl /Fobuild\debug\lib\core\static\a-pch.obj /c build\debug\lib\core\static\a-pch.cpp /TP /nologo -Od -MDd -Ob0 -Z7 -W4 -EHsc -GR -D_DEBUG -DUSE_PCH -Yl__xge_pch_symbol /Iinclude /Yucore\CorePCH.hpp /Fpbuild\debug\lib\core\static\a-pch.pch

Pass:

Taskmaster:     Considering node <no_state   0   'build\\debug\\lib\\core\\static'> and its children:
Taskmaster:        <no_state   0   'build\\debug\\lib\\core\\static\\a-hello.cpp'>
Taskmaster:        <no_state   0   'build\\debug\\lib\\core\\static\\a-hello.obj'>
Taskmaster:        <no_state   0   'build\\debug\\lib\\core\\static\\b-pch.cpp'>
Taskmaster:        <no_state   0   'build\\debug\\lib\\core\\static\\b-pch.obj'>
Taskmaster:        <no_state   0   'build\\debug\\lib\\core\\static\\b-pch.pch'>
Taskmaster:        <no_state   0   'build\\debug\\lib\\core\\static\\core'>
Taskmaster:        <no_state   0   'build\\debug\\lib\\core\\static\\core.lib'>
Taskmaster:        <no_state   0   'build\\debug\\lib\\core\\static\\SConscript'>
Taskmaster:      adjusted ref count: <pending    1   'build\\debug\\lib\\core\\static'>, child 'build\\debug\\lib\\core\\static\\a-hello.cpp'
Taskmaster:      adjusted ref count: <pending    2   'build\\debug\\lib\\core\\static'>, child 'build\\debug\\lib\\core\\static\\a-hello.obj'
Taskmaster:      adjusted ref count: <pending    3   'build\\debug\\lib\\core\\static'>, child 'build\\debug\\lib\\core\\static\\b-pch.cpp'
Taskmaster:      adjusted ref count: <pending    4   'build\\debug\\lib\\core\\static'>, child 'build\\debug\\lib\\core\\static\\b-pch.obj'
Taskmaster:      adjusted ref count: <pending    5   'build\\debug\\lib\\core\\static'>, child 'build\\debug\\lib\\core\\static\\b-pch.pch'
Taskmaster:      adjusted ref count: <pending    6   'build\\debug\\lib\\core\\static'>, child 'build\\debug\\lib\\core\\static\\core'
Taskmaster:      adjusted ref count: <pending    7   'build\\debug\\lib\\core\\static'>, child 'build\\debug\\lib\\core\\static\\core.lib'
Taskmaster:      adjusted ref count: <pending    8   'build\\debug\\lib\\core\\static'>, child 'build\\debug\\lib\\core\\static\\SConscript'
Taskmaster:     Considering node <no_state   0   'build\\debug\\lib\\core\\static\\a-hello.cpp'> and its children:
Taskmaster: Evaluating <pending    0   'build\\debug\\lib\\core\\static\\a-hello.cpp'>

Task.make_ready_current(): node <pending    0   'build\\debug\\lib\\core\\static\\a-hello.cpp'>
Task.prepare():      node <up_to_date 0   'build\\debug\\lib\\core\\static\\a-hello.cpp'>
Task.executed_with_callbacks(): node <up_to_date 0   'build\\debug\\lib\\core\\static\\a-hello.cpp'>
Task.postprocess():  node <up_to_date 0   'build\\debug\\lib\\core\\static\\a-hello.cpp'>
Task.postprocess():  removing <up_to_date 0   'build\\debug\\lib\\core\\static\\a-hello.cpp'>
Task.postprocess():  adjusted parent ref count <pending    7   'build\\debug\\lib\\core\\static'>

Taskmaster: Looking for a node to evaluate
Taskmaster:     Considering node <no_state   0   'build\\debug\\lib\\core\\static\\a-hello.obj'> and its children:
Taskmaster:        <up_to_date 0   'build\\debug\\lib\\core\\static\\a-hello.cpp'>
Taskmaster:        <no_state   0   'build\\debug\\lib\\core\\static\\b-pch.pch'>
Taskmaster:        <no_state   0   'include\\core\\CorePCH.hpp'>
Taskmaster:        <no_state   0   'include\\core\\Hello.hpp'>
Taskmaster:        <no_state   0   'C:\\Software\\MSVS-2022-143-Com\\VC\\Tools\\MSVC\\14.37.32822\\bin\\HostX64\\x64\\cl.EXE'>
                 !!! REFERENCE TO b-pch.pch !!!
Taskmaster:      adjusted ref count: <pending    1   'build\\debug\\lib\\core\\static\\a-hello.obj'>, child 'build\\debug\\lib\\core\\static\\b-pch.pch'
Taskmaster:      adjusted ref count: <pending    2   'build\\debug\\lib\\core\\static\\a-hello.obj'>, child 'include\\core\\CorePCH.hpp'
Taskmaster:      adjusted ref count: <pending    3   'build\\debug\\lib\\core\\static\\a-hello.obj'>, child 'include\\core\\Hello.hpp'
Taskmaster:      adjusted ref count: <pending    4   'build\\debug\\lib\\core\\static\\a-hello.obj'>, child 'C:\\Software\\MSVS-2022-143-Com\\VC\\Tools\\MSVC\\14.37.32822\\bin\\HostX64\\x64\\cl.EXE'

!!! EVALUATING b-pch.pch [REFERENCE to b-pch.cpp] !!!
Taskmaster:     Considering node <no_state   0   'build\\debug\\lib\\core\\static\\b-pch.pch'> and its children:
Taskmaster:        <no_state   0   'build\\debug\\lib\\core\\static\\b-pch.cpp'>
Taskmaster:        <no_state   0   'include\\core\\CorePCH.hpp'>
Taskmaster:        <no_state   0   'include\\core\\Hello.hpp'>
Taskmaster:        <no_state   0   'C:\\Software\\MSVS-2022-143-Com\\VC\\Tools\\MSVC\\14.37.32822\\bin\\HostX64\\x64\\cl.EXE'>
                 !!! REFERENCE TO b-pch.cpp !!!
Taskmaster:      adjusted ref count: <pending    1   'build\\debug\\lib\\core\\static\\b-pch.pch'>, child 'build\\debug\\lib\\core\\static\\b-pch.cpp'
Taskmaster:      adjusted ref count: <pending    2   'build\\debug\\lib\\core\\static\\b-pch.pch'>, child 'include\\core\\CorePCH.hpp'
Taskmaster:      adjusted ref count: <pending    3   'build\\debug\\lib\\core\\static\\b-pch.pch'>, child 'include\\core\\Hello.hpp'
Taskmaster:      adjusted ref count: <pending    4   'build\\debug\\lib\\core\\static\\b-pch.pch'>, child 'C:\\Software\\MSVS-2022-143-Com\\VC\\Tools\\MSVC\\14.37.32822\\bin\\HostX64\\x64\\cl.EXE'
Taskmaster:     Considering node <no_state   0   'build\\debug\\lib\\core\\static\\b-pch.cpp'> and its children:
Taskmaster: Evaluating <pending    0   'build\\debug\\lib\\core\\static\\b-pch.cpp'>

!!! EVALUATING b-pch.cpp [REFERENCE to b-pch.pch] !!!
Task.make_ready_current(): node <pending    0   'build\\debug\\lib\\core\\static\\b-pch.cpp'>
Task.prepare():      node <up_to_date 0   'build\\debug\\lib\\core\\static\\b-pch.cpp'>
Task.executed_with_callbacks(): node <up_to_date 0   'build\\debug\\lib\\core\\static\\b-pch.cpp'>
Task.postprocess():  node <up_to_date 0   'build\\debug\\lib\\core\\static\\b-pch.cpp'>
Task.postprocess():  removing <up_to_date 0   'build\\debug\\lib\\core\\static\\b-pch.cpp'>
                     !!! REFERENCE TO b-pch.pch
Task.postprocess():  adjusted parent ref count <pending    3   'build\\debug\\lib\\core\\static\\b-pch.pch'>
Task.postprocess():  adjusted parent ref count <pending    6   'build\\debug\\lib\\core\\static'>

Taskmaster: Looking for a node to evaluate
Taskmaster:     Considering node <no_state   0   'include\\core\\CorePCH.hpp'> and its children:
Taskmaster: Evaluating <pending    0   'include\\core\\CorePCH.hpp'>

Task.make_ready_current(): node <pending    0   'include\\core\\CorePCH.hpp'>
Task.prepare():      node <up_to_date 0   'include\\core\\CorePCH.hpp'>
Task.executed_with_callbacks(): node <up_to_date 0   'include\\core\\CorePCH.hpp'>
Task.postprocess():  node <up_to_date 0   'include\\core\\CorePCH.hpp'>
Task.postprocess():  removing <up_to_date 0   'include\\core\\CorePCH.hpp'>
Task.postprocess():  adjusted parent ref count <pending    2   'build\\debug\\lib\\core\\static\\b-pch.pch'>
Task.postprocess():  adjusted parent ref count <pending    3   'build\\debug\\lib\\core\\static\\a-hello.obj'>

Taskmaster: Looking for a node to evaluate
Taskmaster:     Considering node <no_state   0   'include\\core\\Hello.hpp'> and its children:
Taskmaster: Evaluating <pending    0   'include\\core\\Hello.hpp'>

Task.make_ready_current(): node <pending    0   'include\\core\\Hello.hpp'>
Task.prepare():      node <up_to_date 0   'include\\core\\Hello.hpp'>
Task.executed_with_callbacks(): node <up_to_date 0   'include\\core\\Hello.hpp'>
Task.postprocess():  node <up_to_date 0   'include\\core\\Hello.hpp'>
Task.postprocess():  removing <up_to_date 0   'include\\core\\Hello.hpp'>
Task.postprocess():  adjusted parent ref count <pending    1   'build\\debug\\lib\\core\\static\\b-pch.pch'>
Task.postprocess():  adjusted parent ref count <pending    2   'build\\debug\\lib\\core\\static\\a-hello.obj'>

Taskmaster: Looking for a node to evaluate
Taskmaster:     Considering node <no_state   0   'C:\\Software\\MSVS-2022-143-Com\\VC\\Tools\\MSVC\\14.37.32822\\bin\\HostX64\\x64\\cl.EXE'> and its children:
Taskmaster: Evaluating <pending    0   'C:\\Software\\MSVS-2022-143-Com\\VC\\Tools\\MSVC\\14.37.32822\\bin\\HostX64\\x64\\cl.EXE'>

Task.make_ready_current(): node <pending    0   'C:\\Software\\MSVS-2022-143-Com\\VC\\Tools\\MSVC\\14.37.32822\\bin\\HostX64\\x64\\cl.EXE'>
Task.prepare():      node <up_to_date 0   'C:\\Software\\MSVS-2022-143-Com\\VC\\Tools\\MSVC\\14.37.32822\\bin\\HostX64\\x64\\cl.EXE'>
Task.executed_with_callbacks(): node <up_to_date 0   'C:\\Software\\MSVS-2022-143-Com\\VC\\Tools\\MSVC\\14.37.32822\\bin\\HostX64\\x64\\cl.EXE'>
Task.postprocess():  node <up_to_date 0   'C:\\Software\\MSVS-2022-143-Com\\VC\\Tools\\MSVC\\14.37.32822\\bin\\HostX64\\x64\\cl.EXE'>
Task.postprocess():  removing <up_to_date 0   'C:\\Software\\MSVS-2022-143-Com\\VC\\Tools\\MSVC\\14.37.32822\\bin\\HostX64\\x64\\cl.EXE'>
Task.postprocess():  adjusted parent ref count <pending    0   'build\\debug\\lib\\core\\static\\b-pch.pch'>
Task.postprocess():  adjusted parent ref count <pending    1   'build\\debug\\lib\\core\\static\\a-hello.obj'>

Taskmaster: Looking for a node to evaluate
Taskmaster:     Considering node <pending    0   'build\\debug\\lib\\core\\static\\b-pch.pch'> and its children:
Taskmaster:        <up_to_date 0   'build\\debug\\lib\\core\\static\\b-pch.cpp'>
Taskmaster:        <up_to_date 0   'include\\core\\CorePCH.hpp'>
Taskmaster:        <up_to_date 0   'include\\core\\Hello.hpp'>
Taskmaster:        <up_to_date 0   'C:\\Software\\MSVS-2022-143-Com\\VC\\Tools\\MSVC\\14.37.32822\\bin\\HostX64\\x64\\cl.EXE'>
Taskmaster: Evaluating <pending    0   'build\\debug\\lib\\core\\static\\b-pch.pch'>

Task.make_ready_current(): node <pending    0   'build\\debug\\lib\\core\\static\\b-pch.pch'>
Task.prepare():      node <executing  0   'build\\debug\\lib\\core\\static\\b-pch.pch'>
Task.execute():      node <executing  0   'build\\debug\\lib\\core\\static\\b-pch.pch'>

Building build\debug\lib\core\static\b-pch.pch and build\debug\lib\core\static\b-pch.obj with action:
  $CXX /Fo${TARGETS[1]} $CXXFLAGS $CCFLAGS $CPPFLAGS $_CPPDEFFLAGS $_CPPINCFLAGS /c $SOURCES /Yc$PCHSTOP /Fp${TARGETS[0]} $CCPDBFLAGS $PCHPDBFLAGS

cl /Fobuild\debug\lib\core\static\b-pch.obj /TP /nologo -Od -MDd -Ob0 -Z7 -W4 -EHsc -GR -D_DEBUG -DUSE_PCH -Yl__xge_pch_symbol /Iinclude /c build\debug\lib\core\static\b-pch.cpp /Yccore\CorePCH.hpp /Fpbuild\debug\lib\core\static\b-pch.pch

jcbrill avatar Nov 05 '23 15:11 jcbrill

At this point it's pretty clear that the $PCHCOM isn't being called in the failing case. You can also see this by setting a value for $PCHCOMSTR. Looks like $PCHCOM hasn't evolved with $CCCOM and $CXXCOM and could use some updating, but that's (yet) another story... Edit: this was typed before, but not submitted before, the previous comment.

mwichmann avatar Nov 05 '23 15:11 mwichmann

Excerpts from output (listings above without annotations): scratch-fail.txt scratch-pass.txt

jcbrill avatar Nov 05 '23 15:11 jcbrill

Updated annotations in listings above. I made a mess of them originally. Apologies.

jcbrill avatar Nov 05 '23 15:11 jcbrill

Hmmm, it looks like there were some errors before, and a fix may have caused this particular problem. See #2505. There's a line in particular there:

have the object_emitter() realize that the base names of Source1.pch and Source1.obj match, assume that this means they're actually being generated by the same call to env.PCH(), and to then avoid adding the explicit .obj => .pch dependency. That assumption feels a little risky, but only a little,

So it avoids adding an explicit dep. Isn't that kind of what we're seeing?

mwichmann avatar Nov 05 '23 17:11 mwichmann

So it avoids adding an explicit dep. Isn't that kind of what we're seeing?

It sure looks that way but I can't be certain.

I feel like I've read that paragraph a bazillion times now.

I don't understand this bit:

That assumption feels a little risky, but only a little, because to construct a configuration that breaks the assumption would require something like checking in a source file named Source1.pch and setting env['PCH'] by hand, instead of using the env.PCH() builder.

The example does not explicitly add C.pch to the source list, calls the env.PCH() builder, and sets env['PCH'] manually.

Setting env['PCH'] manually appears to be required otherwise the builds fail (I think).

I don't fully understand yet how the pch file is discovered/added as a dependency when processing the Hello.cpp/Hello.obj nodes.

jcbrill avatar Nov 05 '23 18:11 jcbrill

No, I don't get this bit either. Usually, you find header deps by scanning the source file, but that doesn't happen in the case of Microsoft's PCH, as those only come from the command line (/Yu', and /Fp` in case you don't want the default pch filename). And since we're the ones supplying the command line, how do we tell?

mwichmann avatar Nov 05 '23 18:11 mwichmann

And since we're the ones supplying the command line, how do we tell?

I haven't the foggiest idea (yet).

I tried a handful of ways to manually add a dependency via "Depends". Everything yielded the header missing error or a cycle detected error.

I may not have done it correctly but it makes me wonder if manually added a dependency causes the deferred dependency to be discovered. I'm not confident.

You might have better luck trying for the runs that fail.

jcbrill avatar Nov 05 '23 18:11 jcbrill

If the PCH variables are defined following the StaticLibrary call (i.e., after the source files have been registered) the build succeeds.

I'm not saying this is a good idea and probably fails for some other reason in a real-world scenario, but it definitely illustrates that the DAG is processed/constructed differently between the two cases.

When DEFER_PCH=True the build succeeds. When DEFER_PCH=False the build fails.

SConscript file:

import glob

DEFER_PCH = True

#get all the build variables we need
Import('env', 'buildroot', 'project', 'mode', 'debugcflags', 'releasecflags')
staticlibenv = env.Clone()

builddir = buildroot + '/' + project   #holds the build directory for this project
targetpath = builddir  #holds the path to the executable in the build directory

#append the user's additional compile flags
#assume debugcflags and releasecflags are defined
if mode == 'debug':
   staticlibenv.Append(CPPFLAGS=debugcflags)
else:
   staticlibenv.Append(CPPFLAGS=releasecflags)

#get source file list
srclst = Glob('*.cpp')

staticlibenv2 = staticlibenv.Clone()

if not DEFER_PCH:
    staticlibenv2['PCHSTOP'] = str(File('core/CorePCH.hpp'))
    # pch.cpp is beside this SConscript
    pch_nodes = staticlibenv2.PCH("a-pch.cpp")
    print("PCHN: %s"%pch_nodes)
    staticlibenv2['PCH'] = pch_nodes[0]

additionalcflags = ['-Yl__xge_pch_symbol']
staticlibenv2.Append(CPPFLAGS=additionalcflags)

staticlib = staticlibenv2.StaticLibrary("core", source=srclst)

if DEFER_PCH:
    staticlibenv2['PCHSTOP'] = str(File('core/CorePCH.hpp'))
    # pch.cpp is beside this SConscript
    pch_nodes = staticlibenv2.PCH("a-pch.cpp")
    print("PCHN: %s"%pch_nodes)
    staticlibenv2['PCH'] = pch_nodes[0]

fail.txt pass.txt

jcbrill avatar Nov 05 '23 19:11 jcbrill