Can not retrieve symlink nodes from cache

[bdbaddog]

trafficstars

This issue was originally created at: 2014-05-13 19:14:25. This issue was reported by: mryan1539.

mryan1539 said at 2014-05-13 19:14:25

I've found that if you have a Command builder where one of the targets is a relative symlink, SCons will store the symlink in the cache, but will never be able to retrieve the node from the cache.

The minimized reproduction case:

CacheDir('.scons_cache')
Command(['real', 'link'], [], [
    'echo 1 > real',
    'ln -s real link',
])

1. Create an SConstruct with that code in an empty directory.
2. Run SCons in that directory. Both targets will end up in the cache.
3. Remove either or both targets.
4. Run SCons again.

I would expect that both targets would be retrieved from the cache. However, only the real file is; the symlink is not, causing the builder to be re-run.

I dug into this, and the short is that the symlink is stored directly as a symlink in the cache. The CacheDir code uses does a symlink-following check when doing a cache lookup, which will fail for this symlink.

In more detail, in SCons/CacheDir.py:

    t = target[0]
    fs = t.fs
    cd = env.get_CacheDir()
    cachedir, cachefile = cd.cachepath(t)
    if not fs.exists(cachefile):
        cd.CacheDebug('CacheRetrieve(%s):  %s not in cache\n', t, cachefile)
        return 1

The target (t) is a SCons.Node.FS.File, and the call to fs.exists() maps to LocalFS.exists(), which in turn is:

    def exists(self, path):
        return os.path.exists(path)

os.path.exists() will follow symlinks, and so return false on dangling symlinks.

I'm not familiar enough with SCons internals to suggest a fix for this, unfortunately.

Votes for this issue: 10.

[DeeeeLAN]

Here is the workaround I am using:

if not os.path.lexists(cachefile):
    cd.CacheDebug('CacheRetrieve(%s):  %s not in cache\n', t, cachefile)
    return 1
# ...
if SCons.Action.execute_action:
    if fs.islink(cachefile):
        fs.symlink('...')
    else:
        cd.copy_from_cache('...')
        #...
        # indent the following lines to be inside the else branch
        st = fs.stat(cachefile)
        fs.chmod(t.get_internal_path(), stat.S_IMODE(st[st.ST_MODE]) | stat.S_IWRITE)
return 0

os.path.lexists will return true for existing files and broken links, so it handles this case well.

Since the file might not exist because of this bypass, the fs.stat can fail. So I moved the fs.stat and fs.chmod calls inside the else branch of the symlink check. If the restored file is a symlink, the chmod doesn't matter because once the regular file is restored it will get the proper permissions, and if the symlink is pointing to a file outside the build tree, the permissions wouldn't have changed anyway because it would not be effected by the cache/uncache process.

I put this code in my own CacheRetrieveFunc and created a subclass of CacheDir to point to this version instead of the original version.