Functions to manipulate filesystem names

[bdbaddog]

This issue was originally created at: 2008-09-09 13:51:56. This issue was reported by: gregnoel.

gregnoel said at 2008-09-09 13:51:56

Issue #2192 asks for optional arguments on Install() to select a subset of the filesystem names returned by a directory scanner. Instead, implement a set of functions to manipulate the names that could then be passed to Install() or any other builder.

Yes, it's pretty simple to write the functions in Python, but that means that everyone ends up implementing the same functionality over and over again.

GNU Make already has a set of text functions for this sort of thing; using those functions as a loose model would make it easier for those users to convert.

I'd suggest implementing them as (list of) string functions, rather than using Nodes, to avoid proliferating Nodes that would never be used otherwise. A Node that is passed in would be converted to its path (via str() so the relative path is used).

Glob() is already too complicated, so we add a function that does a treewalk, call it FilterTree() for the sake of discussion:

• FilterTree(dirs, include=None, exclude=None) where dirs is one or more directories that are the roots of the treewalks, include is zero or more (wildcard) patterns that are selected (None implies select all names), exclude is zero or more (wildcard) patterns that are excluded (None implies all names are selected).

imgs = TreeFilter("build/pix", include=["*.png", "*.jpg"])
all = TreeFilter("build/headers", exclude=["CVS", ".svn"])

• FilterIn(patterns, names) where patterns is one or more (wildcard) patterns and names is zero or more names; it returns a list of names that match any of the patterns.

c_src = FilterIn(['*.c','*.cpp'], sources)

• FilterOut(patterns, names) where patterns is one or more patterns and names is zero or more names; it returns a list of names that do NOT match all of the patterns.

nontext = FilterOut('*.txt', files)

• PatternSubst(pattern, replacement, names) where pattern is a (wildcard!) pattern and replacement is what should replace any pattern that matches. Ideally, it should act like GNU's patsubst function, where the replacement is also a wildcard:

objs = PatternSubst('*.c', '*.o', sources)

• PatternReplace(pattern, replacement, names) where pattern is a pattern and replacement is what should replace any pattern that matches. This one uses regular expressions:

objs = PatternReplace(r'\.c$', '.o', sources)   # same as above
base = PatternReplace('.*/', '', sources)       # get the basenames
PatternReplace('^pix/', dist.Dir('pix'), imgs)  # targets to match sources

Other functions might also make sense; this should be considered open-ended.

garyo said at 2008-09-22 18:55:20

Hi Greg. How would this behave w.r.t. Nodes that don't exist in the FS yet? I presume it would return paths for all those files as well as ones that already exist on disk.

gregnoel said at 2008-09-24 01:20:17

To respond to Gary's comment, yes, the FilterTree() function would, like Glob(), pick up pick up Nodes that hadn't been created yet; I'd imagine that they would share some logic. The idea is that Glob and FilterTree, between them, provide the functionality of the GNU wildcard() function. The rest of the functions just shuffle strings, really; they don't look at the filesystem.

gregnoel said at 2008-09-25 00:35:49

Bug party triage. Try to get it in during 1.x, but if it slips, that's OK.

gregnoel said at 2008-12-26 13:30:37

Adjust triage of issues.

simpleton said at 2009-01-06 01:54:29

I definitely like this idea better. Would filenames returned by FilterTree should only contain the path-components that are under the specified dirs? Sort of such that FilterTree('data/pixmaps', include=['*.png']) would return:

[ 'a.png', 'b.png', 'misc/c.png', etc ]

and not:

[ 'data/pixmaps/a.png', 'data/pixmaps/b.png', 'data/pixmaps/misc/c.pn', etc ]

And would be easy to use:

env.Install('$DESTDATADIR/package/share/pixmaps', FilterTree('data/pixmaps', include='*.png'))

or would it need another function like this?

files = FilterTree('data/pixmaps', include='*.png')
env.Install('$DESTDATADIR/package/share/pixmaps', PatternSubst('data/pixmaps/*', '*', files))

gregnoel said at 2009-01-09 13:04:56

Brian Vanderburg: I'm not quite sure I understand your question, but I believe the answer is yes. The reason I'm puzzled is that it already does what you describe, as names are already relative to the directory of the SConscript (at least at the text level; names may be mapped as you indicate when the path is evaluated for the build command).

gregnoel said at 2010-02-03 14:02:52

*** Issue 2545 has been marked as a duplicate of this issue. ***

garyo said at 2012-09-01 10:04:01

de-assigning all tickets assigned to Greg Noel (no longer working on SCons)

techtonik said this issue blocks #2545 at 2010-05-12 04:18:41.

[mwichmann]

Okay, a little oddity here. Issue #2545 requested adding an exclude= kwarg to Glob. That issue was reclassified as depending on this one, but then was noted as implemented, and closed (and indeed, Glob now takes an exclude). Does that mean the proposal here was rejected? An open issue shouldn't be blocking a closed issue. The dependency isn't a "real" block, since github doesn't have that functionality, it downgraded to just a "mention" linkage when the bugs were migrated, but still....