VS2012: Allow For the Ignoring of unpredictable and Unnecessary Outputs

[ahicks92]

I just tried to move a simple project to Tup because, hey, why not and it looks great. unfortunately, cl.exe, Microsoft's command line C++ compiler, insists on writing a completely meaningless log file somewhere down in the appdata folder. This file is undocumented, cannot be disabled as far as I can tell, and includes the current user's username in the path (and probably varies depending on version of windows). More ironically yet, it contains 5 or 6 characters, one of which is unprintable. It has no bearing on the current build whatsoever, but stops tup from actually building. The file's name is sqmcpp.log, and no documentation as to its function exists anywhere. Can we add something that will allow me to say "This command's file output is suspect, ignore it because I know better?" As it stands, I cannot use Tup for my project and have had to revert to Cmake. Thank you Microsoft. If this exists and is undocumented, then this issue is for that instead.

[gittup]

On Fri, Apr 25, 2014 at 4:09 PM, Austin Hicks [email protected]:

I just tried to move a simple project to Tup because, hey, why not and it looks great. unfortunately, cl.exe, Microsoft's command line C++ compiler, insists on writing a completely meaningless log file somewhere down in the appdata folder. This file is undocumented, cannot be disabled as far as I can tell, and includes the current user's username in the path (and probably varies depending on version of windows). More ironically yet, it contains 5 or 6 characters, one of which is unprintable. It has no bearing on the current build whatsoever, but stops tup from actually building. The file's name is sqmcpp.log, and no documentation as to its function exists anywhere. Can we add something that will allow me to say "This command's file output is suspect, ignore it because I know better?" As it stands, I cannot use Tup for my project and have had to revert to Cmake. Thank you Microsoft. If this exists and is undocumented, then this issue is for that instead.

Hi Austin,

Can you clarify what command-line you're trying to run that causes this file to be created? I just installed vs2012 and tried to compile a simple file using cl, but no such file was created.

There isn't currently the ability in a Tupfile to list a file that should be ignored, though we do ignore other files similar to this automatically. I think I'd prefer to add this to the list of automatically ignored files, rather than force the user to specify them (otherwise everyone who uses vs2012 would presumably have to have the same list in their Tupfiles).

Thanks, -Mike

[ahicks92]

Actually, looking at it, it appears I was being an idiot when I wrote it-it's vs2013 professional, not 2012. My bad. I'll have to see if I still have the tup build file, but don't think so; I hadn't yet pushed the commits and, after an afternoon of trying, I'm pretty sure I locally reverted them. I'll have to hunt and see if they've been garbage collected yet for the exact command, but it wasn't anything super tricky. If I recall, not even optimizations. I think the issue is more general, however; I need to do other things in my build file for this project, most notably call a cmake project if it isn't built. This causes even more unpredictable outputs, or at least a rather large list of outputs I really don't care about. Really. Forever. I think that this option makes the tool more generally useful, because end-users can get around these kind of problems. I also think that it's my responsibility to tell you about my code's dependencies, not your tool's responsibility to say "You might want to, but can't"-but I think that is part of why Tup was written in the first place, so perhaps don't mind me. Note: I'm not 100% sure I can get Tup and CMake happily interfacing anyway, and it might just be better to give up and use CMake as I care about windows more than anything and consequently am including and building dependencies myself. I'm not sure I can give up CMake and recompile Portaudio and possibly a few other things with Tup, nor am I sure I want to as that makes tracking upstream harder. It still looks like a very nice tool just, now that I've started taking the project somewhere, perhaps not the right one for the job.

[distee]

Just started on using 'tup' and experienced the thing camlorn describes here. Feeding the following command line to the Visual Studio 2013 compiler: > cl /GS /TC /FC /EHsc /Fo"doc_try.o" ..\src\doc_try.c produces > tup error: File 'C:\Users\dirk\AppData\Local\Microsoft\VSCommon\12.0\SQM\sqmcpp.log' was written to, but is not in .tup/db. ...

This makes the tup job fail.

[gittup]

I just installed vs2013 and tried to reproduce the issue, but I was unable to. It sounds like this may have been fixed in "Update 2" - is it possible that I got that since I just installed it? Can you try to upgrade VS and see if it resolves the issue?

See also: http://connect.microsoft.com/VisualStudio/feedback/details/838232/update-kb2932965-creates-sqm-files-in-the-default-user-profile-that-prevent-new-local-user-accounts-to-login-to-windows

Thanks, -Mike

On Fri, Jun 6, 2014 at 1:15 PM, Dirk Steenpass [email protected] wrote:

Just started on using 'tup' and experienced the thing camlorn describes here. Feeding the following command line to the Visual Studio 2013 compiler:

cl /GS /TC /FC /EHsc /Fo"doc_try.o" ..\src\doc_try.c produces tup error: File 'C:\Users\dirk\AppData\Local\Microsoft\VSCommon\12.0\SQM\sqmcpp.log' was written to, but is not in .tup/db. ...

This makes the tup job fail.

— Reply to this email directly or view it on GitHub https://github.com/gittup/tup/issues/182#issuecomment-45361500.

[gittup]

On Tue, Apr 29, 2014 at 3:08 PM, Austin Hicks [email protected] wrote:

I think the issue is more general, however; I need to do other things in my build file for this project, most notably call a cmake project if it isn't built. This causes even more unpredictable outputs, or at least a rather large list of outputs I really don't care about. Really. Forever. I think that this option makes the tool more generally useful, because end-users can get around these kind of problems. I also think that it's my responsibility to tell you about my code's dependencies, not your tool's responsibility to say "You might want to, but can't"-but I think that is part of why Tup was written in the first place, so perhaps don't mind me.

I don't think you'll have much luck trying to run one build system from inside another. If you need to build things with two build systems, you might be better off having a top-level build script that does something like:

cd cmake-proj; cmake cd tup-proj; tup

Note: I'm not 100% sure I can get Tup and CMake happily interfacing anyway, and it might just be better to give up and use CMake as I care about windows more than anything and consequently am including and building dependencies myself. I'm not sure I can give up CMake and recompile Portaudio and possibly a few other things with Tup, nor am I sure I want to as that makes tracking upstream harder. It still looks like a very nice tool just, now that I've started taking the project somewhere, perhaps not the right one for the job.

If you manage dependencies yourself, you're pretty much guaranteed to get them wrong at some point and waste your time. That's the second rule of build systems; I'm not compromising on it.

Thanks, -Mike

[distee]

I was already running on VS2013 Update 2, so it's likely unrelated to "prevent-new-local-user-accounts-to-login-to-windows" (I like that one :).

I looked around a bit and learned that SQM stands for Software Quality Metrics. This data is send to MS if you opt into the "Customer Experience Improvement Program", which I did. I opted out, but the data is still generated, it is just not send.

Right now, I have no idea how to turn writing off. Opting into the "Customer Experience Improvement Program" may trigger the writing for you ... or not.

I used procmon (process monitor, sysinternals) to follow the activity of cl.exe and discovered that link.exe also writes to 'sqmcpp.log'. Sadly, the event log did not convey any clue about the cause for this annoyance to me.

Maybe somebody else has an idea?

[ahicks92]

I have moved on to cmake because I needed to make progress. That is not to say I couldn't move back, however, and I kind of want to simply because cmake is monolithic, archaic, and just painful. I've done the build script thing before for another project.

It is nice that Tup manages dependencies. Software may eventually become as smart as a human, but it isn't currently. Tup protects me, but if I know that I am safe and that the tool is wrong, it should be my option to tell it so. I have had to deal with dependency problems before and get where you're coming from, but being able to say "This command makes a bunch of extra output that I don't use" would fix this case and any cases in future, as well as allowing for stuff like building documentation (the output of doxygen comes to mind here). If we start implementing special rules for specific tools, it starts making the build system specific to those tools-instead, let me loosen the requirements as I need to. I may waste time when I get it wrong, but I did waste time figuring out that Tup wouldn't work on my setup because Tup insists on telling me that I'm wrong even when I know I'm not. This could be fixed in the specific case of visual studio, but I think that a more general fix is applicable; if I'm warned not to use it in the documentation unless I really need it, it's my fault for wasting my time and not the fault of the tool.

Anyhow, that's my two cents and tup isn't my project.

[greneholt]

I am very interested in using tup to manage the build process for extremely large Verilog ASIC designs at my company. Unfortunately, the compilers and toolchains involved in this process (mostly Cadence and Synopsys software) are very guilty of creating lots of extraneous or unpredictably named outputs. I understand that this is an extremely uncommon use case for build systems, and that probably no-one else using tup even touches these tools. As such, I would never expect support for ignoring the unnecessary outputs of these fringe programs to be built into tup, so the ability to manually configure what is ignored is critical.

I love the design of tup, especially the way it will refuse to run if you have implicit dependencies on generated files. That feature combined with the automatic dependency detection would save us so many headaches and bugs in our thousands of lines of makefiles. But the sad fact is that closed-source commercial tools can be stupid and/or do unpredictable things, and without the ability to manually compensate for these idiosyncrasies, many people will never be able to use tup.

[gittup]

On Sun, Jun 8, 2014 at 11:08 AM, Dirk Steenpass [email protected] wrote:

I was already running on VS2013 Update 2, so it's likely unrelated to "prevent-new-local-user-accounts-to-login-to-windows" (I like that one :).

I looked araound a bit and learned that SQM stands for Software Quality Metrics. This data is send to MS if you opt into the "Customer Experience Improvement Program", which I did. I tried opting out, but the data is still generated it is just not send.

Right now, I have no idea how to turn writing off. Opting into the "Customer Experience Improvement Program" may trigger the writing for you ... or not.

Ahh, that was indeed it. I had that turned off, and when I flipped it on I could reproduce it. I've just pushed a patch to ignore this particular file, since I think all normal compilers should work out of the box.

-Mike

[gittup]

On Mon, Jun 9, 2014 at 12:31 AM, Connor McKay [email protected] wrote:

I am very interested in using tup to manage the build process for extremely large Verilog ASIC designs at my company. Unfortunately, the compilers and toolchains involved in this process (mostly Cadence and Synopsys software) are very guilty of creating lots of extraneous or unpredictably named outputs. I understand that this is an extremely uncommon use case for build systems, and that probably no-one else using tup even touches these tools. As such, I would never expect support for ignoring the unnecessary outputs of these fringe programs to be built into tup, so the ability to manually configure what is ignored is critical.

I love the design of tup, especially the way it will refuse to run if you have implicit dependencies on generated files. That feature combined with the automatic dependency detection would save us so many headaches and bugs in our thousands of lines of makefiles. But the sad fact is that closed-source commercial tools can be stupid and/or do unpredictable things, and without the ability to manually compensate for these idiosyncrasies, many people will never be able to use tup.

Connor & Austin - thanks for your feedback. I agree that tup is not as usable in the cases that you describe as it should be. I think these are slightly different from the sqmcpp.log issue - for something common like that, I think it makes sense to ignore it in tup so that we can support standard compilers & tools by default. In other words, no-one would want to track writing to that file, and it's a little silly to require everyone to add it to an "ignore list" in their Tupfile.

As for the more general case that you are describing, where more rarely used or specialist tools do funky things, we definitely need a better way to support that, since obviously I can't keep track of them all inside the tup code itself. However, I'm not entirely sure that just ignoring the files outright is the way to go. Instead, we may want to consider allowing unspecified outputs as long as the command outputs to a group, though we'd need to figure out how to properly handle things like wildcards or overwriting ghost nodes, which are probably solvable but not entirely trivial.

My concern with adding an ignore feature is that rather than properly accounting for dependencies, we are allowing the user to compromise the integrity of the build. As such, users are much more likely to have to clean everything in order to get rid of the "ignored" files and return to an uncorrupted build state.

Thanks, -Mike

[distee]

Just fetched the latest build and run it: the issue is gone.

Thanks, Dirk

[gittup]

On Wed, Jun 11, 2014 at 10:24 AM, Dirk Steenpass [email protected] wrote:

Just fetched the latest build and run it: the issue is gone.

Thanks, Dirk

Cool! Thanks for digging into what SQM was - I wouldn't have figured that out on my own :)

-Mike

[distee]

Well, ..., figuring out something like tup is much more useful!

Kudos. Dirk

[xwizard]

Hi. tup error: File 'C:\Users\Mateusz\AppData\Local\Microsoft\VSCommon\14.0\SQM\VCToolsTelemetry.dat' was written to, but is not in .tup/db. You probably should specify it as an output

New version, new files :(

[petervaro]

@gittup actually @xwizard is right, for example on Windows 10, with the latest VS (2015) it is not working because of the above mentioned problem. (Even if "Customer Experience Improvement Program" is turned off! It looks like, you can't really do anything about this, unless you use something else than cl.exe) Gosh.. I'm so sick of supporting this fudging OS.. miserable :(

[xwizard]

@petervaro Tup works fine with Mingw64 on Windows 10. It's cl.exe's fault in this case.

[petervaro]

@xwizard I know, and I already switched to mingw-w64 as my default windows-compiler, I just wanted to make sure my code-base will also compile via cl.exe as well -- and I know, it's definitely not tup's fault, never mentioned it though :P tup's only fault is the absence of the run directive on windows.. but it's another story, isn't it? :)

[cfillion]

For anyone having the same issue @xwizard had with VS2015's cl.exe writing to VCToolsTelemetry.dat: There's a way to disable this unwanted data collection with a registry edit: https://msdn.microsoft.com/en-us/library/ee225238.aspx#Anchor_5 On my computer the key was in HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Microsoft\VSCommon\14.0\SQM.

[mattiasflodin]

I just ran into this issue too, with the VCToolsTelemetry.dat file. In my humble opinion, hard-coding specific file names is the wrong way to go about this because it will lead to new problems with every new version and compiler brand. It's annoying but possibly solvable when the developer hits it, but what if my users who are already struggling to build my software run into this? What if they use some antivirus or similar that hooks into processes and happen to write to some other file that has nothing to do with the build? It's just too much of a usability problem to be worth it. Besides, from a design perspective tup should be compiler agnostic; hardcoding behavior for specific compiler brands and versions is an ugly hack.

I would suggest that anything that is written outside the source tree is ignored unless a warning is explicitly enabled.

[ahicks92]

I've long since left Tup, but am still subscribed to the thread. I doubt I'll come back as I now have a huge amount of CMake and Python automation (maybe 500-1000 lines and growing) for stuff like automated releasing to Pypi and handling continuous integration and finding libraries. I'm not saying this isn't a good tool, just that I would now have to spend very significant effort to get back to it. But it doesn't matter if I want to, as Tup cannot handle my case.

The scenario in my original comment is no longer hypothetical. I now have code that calls python setup.py bdist_wheel as part of the build process for a primarily C++ project. It also calls sphinx to generate docs. These two tools create hundreds of files at least in the build directory, and I don't care about 99% of them.

Saying "Ignore everything outside the build directory" fixes VC++, but it doesn't fix the actual problem. This is like the unsafe keyword in Rust. Our tools are not smarter than us. It is essential that I am able to tell my tools that I know better than them on issues like this. Otherwise you just keep running up against these cases.

[mattiasflodin]

Agreed, but I think both solutions can and should coexist.

[ahicks92]

yes, both is certainly workable. My point is merely that "ignore everything outside the build directory" is fixing a set of symptoms, not an underlying problem.

[mattiasflodin]

Well, I see it more as implementing what you suggested but with a convenient shortcut for a common use case.

Edit: And, in the short run, it will work as an initial stopgap for a problem that everyone who uses Visual C++ will run into.

[xmclark]

I'm trying tup with cl.exe, visual studio 2017 on my windows 10 x64 machine. I've tried setting the "OptIn" flag in various registry string keys, but I still get the error:

tup error: File 'C:\Users\Me\AppData\Local\Microsoft\VSCommon\15.0\SQM\VCToolsTelemetry.dat' was written to, but is not in .tup/db.

I've tried setting the value "OptIn" string value in:

• \HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Policies\Microsoft\VSCommon\15.0\SQM
• \HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Policies\Microsoft\VSCommon\14.0\SQM
• \HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Policies\Microsoft\VisualStudio\SQM
• \HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Policies\Microsoft\SQMClient
• \HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\VSCommon\15.0\SQM
• \HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\VSCommon\14.0\SQM
• \HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\VisualStudio\SQM
• \HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\SQMClient

No success. I'm hoping somebody else has a work around for this. This is a pretty big blocker for me.

[kalrish]

Try with \HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Microsoft\VSCommon\14.0\SQM or \HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Microsoft\VSCommon\15.0\SQM.

[xmclark]

\HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Microsoft\VSCommon\15.0\SQM worked! Thanks @kalrish.

[datgame]

Updated Visual Studio 2017 yesterday, and now this problem is back. The setting for feedback in the GUI is off, and the registry key above is set to 0. Seems like they don't listen to either anymore. \HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Microsoft\VSCommon\15.0\SQM is what exists here. I don't have the \HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\VSCommon\15.0\SQM entry in my registry, or any of the other examples above. One question is if any of them should be created, by making a new folder and key called "OptIn". (rebooted after modified to verify) Setting works in VS 15.5.6 but not in 15.6.0 and 15.6.1.

[datgame]

I have sent a report via their feedback button in MSVS now, with a link to this thread. Asking for them to fix this regression or provide an alternative.

Vote it up: https://developercommunity.visualstudio.com/content/problem/210627/vctoolstelemetrydat-is-being-written-to-when-not-n.html

[datgame]

I downgraded to 15.5.7 and now it's working again: https://docs.microsoft.com/en-us/visualstudio/productinfo/installing-an-earlier-release-of-vs2017 I have no plans to upgrade for a few years now, until really required. Will try to create an offline installer too, just in case, since old installers are removed after a while from the site.

[manbearian]

Hi, @datgame i'm replying here and I also replied to the dev-community issue you opened. First, thank you for brining this too my attention and second, i'm sorry for the inconvenience this change has caused you.

I have a fix in 15.7 for this issue which will be in 15.7 preview 3 when it's released imminently.

You can immediately unblock usage of 15.6 via this workaround: set VCToolsTelemetry.dat to be read-only so that the VC++ compiler and friends cannot write to it.

Below is a simple CMD script that accomplishes this; feel free to cut-and-paste it into a CMD shell on your machine.

:: first kill any process that might be accessing or will access VCToolsTelemetry.dat
taskkill /f /t /IM vctip.exe /IM cl.exe /IM link.exe /IM msbuild.exe

:: second, create a new, empty VCToolsTelemetry.dat and lock it
set file=%USERPROFILE%\AppData\Local\Microsoft\VisualStudio\15.0\VCToolsTelemetry.dat
echo.>%file%
attrib +r %file%

Please let me know if you have any more issues or have any questions or concerns regarding this.

ian Bearman Principal Software Engineering Manager Microsoft Compiler and Languages Platform Group
/* Making your code faster, smaller, smarter! */