.gitignore: No support for `!`

[giftig]

Discovered this because I have a gitignore in a golang project which excludes /src/* but commits /src/xantoria.com/*, like this:

(gnotif→) gnotify (master) $ cat .gitignore
.*
!/.gitignore
*.swp
*.swo

# Don't commit binaries
/bin
/pkg

# Third party packages
/src/*
!/src/xantoria.com
# Log files
*.log

# Cache files
/_oauth_cache.json

# Convenience symlink to /etc/gnotify.conf
/dev.yaml

I discovered I couldn't search my code with ag, but found it just fine with ack or grep. I reproduced this in a minimal form in a test repository: https://github.com/giftig/ag-issue

where this demonstrates the problem:

testing $ git clone [email protected]:giftig/ag-issue.git
Cloning into 'ag-issue'...
...
testing $ cd ag-issue/
ag-issue (master) $ ag foo
ag-issue (master) $ ack foo
src/mycode/lala.js
2:  println('foo');

[marcelpaulo]

I'm running ag built from master (5b10c68e0f3454ef96c1b62edf611b980bfff333) on Xubuntu 18.04, and I've just been hit by this issue. Here's a way to reproduce it:

1. Create a git repo

   paulo:~/tmp$ git init deleteme
   Initialized empty Git repository in /home/paulo/tmp/deleteme/.git/
   

2. Create a directory structure

   paulo:~/tmp/deleteme$ mkdir -p foo/bar
   

3. Create a file under each directory

   paulo:~/tmp/deleteme$ echo whatever >foo/afile; echo whatever >foo/bar/afile
   

   Here's the directory structure:

   paulo:~/tmp/deleteme$ tree --charset ascii
   .
   `-- foo
   |-- afile
   `-- bar
       `-- afile
   
   2 directories, 2 files
   

4. Create a .gitignore file that will exclude files under foo/ but will include files under foo/bar/:

   paulo:~/tmp/deleteme$ echo $'foo/*\n!foo/bar/' >.gitignore
   paulo:~/tmp/deleteme$ cat .gitignore
   foo/*
   !foo/bar/
   

5. Search for whatever in the repo

   paulo:~/tmp/deleteme$ ag whatever 
   paulo:~/tmp/deleteme$ 
   

It should have found foo/bar/afile if it had properly interpreted !foo/bar/ in .gitignore

[cevaris]

was able to capture this issue in a test

Setup:

  $ . $TESTDIR/setup.sh
  $ mkdir -p foo/bar
  $ printf whatever >foo/afile;
  $ printf whatever >foo/bar/afile
  $ printf $'foo/*\n!foo/bar/' >.gitignore

Ignore .gitignore patterns but not .ignore patterns:

  $ ag blah
  afile:1:blah1

And when we run it.

cram -v tests/ignore_invert_in_subdirectory.t
tests/ignore_invert_in_subdirectory.t: failed
--- tests/ignore_invert_in_subdirectory.t
+++ tests/ignore_invert_in_subdirectory.t.err
@@ -9,4 +9,4 @@
 Ignore .gitignore patterns but not .ignore patterns:

   $ ag blah
-  afile:1:blah1
+  [1]
# Ran 1 tests, 0 skipped, 1 failed.

[lathiat]

Also ran into this problem today with the "haproxy" git repository, they have !/src and so the search ignores everything.

If we can't "fix" support for this fully, one option might be to ignore the gitignore file if it contains such directives?

[giftig]

Wouldn't the easiest option simply be to get git to interpret it? It looks like the problem is that ag has attempted to naively parse git's gitignore syntax, but it doesn't really need to; git is the authority on which files would be included or excluded by its various gitignores, so why not ast it to list the files for you? git ls-files will do it, I think.

The answer to that may be performance, but, well, it has to actually work to be considered performant.

[Chipe1]

https://git-scm.com/docs/gitignore#_pattern_format mentions "It is not possible to re-include a file if a parent directory of that file is excluded" So the reported issue might not be an actual issue. ag does however fail to parse '!' even when the parent directory is not ignored

[giftig]

@Chipe1 notice that the parent directory in my example isn't excluded; I've excluded src/* and then set src/xantoria.com as an exception to that rule; I didn't exclude /src itself.

I also provided a repository which clearly reproduces the issue.

[TysonAndre]

I see that ag will always include a file if it matches a non-regex pattern (i.e. no *, [, ], ?, etc), regardless of pattern negations.

https://github.com/ggreer/the_silver_searcher/blob/a61f1780b64266587e7bc30f0f5f71c6cca97c0f/src/ignore.c#L226-L284

For example, if you're trying to exclude tags (from ctags) from search, but include tags/ as a directory

# does not work, tags/ is still ignored
tags
!tags/

But this workaround does work - the file is ignored but the files in the directory aren't.

[t]ags
!tags/

[TysonAndre-tmg]

The workaround doesn't actually work, I was mistaken earlier and had [t]ags commented out for the search. That's because there's actually 2 calls to path_ignore_search instead of 1, so the ignore pattern for "!tags/" doesn't affect the first call.

It effectively:

1. First checks if the path in question (e.g. "subdir/tags") matches any file. The arguments to path_ignore_search(ig, path_start, filename) don't indicate if that was a directory (ignore behavior is identical for files and folders for this call). If path_ignore_search is true(non-zero) it returns early to ignore the file
2. If the path was a directory, it checks again with a trailing slash (e.g. calls path_ignore_search with filename of temp="subdir/tags/"). If path_ignore_search is true(non-zero) it returns early to ignore the file

https://github.com/ggreer/the_silver_searcher/blob/a61f1780b64266587e7bc30f0f5f71c6cca97c0f/src/ignore.c#L358-L377

[TysonAndre-tmg]

This ignore file would work with the below patch - this is obviously hackish (I'm just unlucky to have several folders named tags)

A more general workaround would be to check for only the exclusions (literals or patterns) that match the filename and directory name with a trailing slash added without any hardcoding

[t]ags
!tags/

--- a/src/ignore.c
+++ b/src/ignore.c
@@ -356,7 +356,10 @@ int filename_filter(const char *path, const struct dirent *dir, void *baton) {
         }
 
         if (path_ignore_search(ig, path_start, filename)) {
-            return 0;
+            if (strcmp(filename, "tags") != 0 || !is_directory(path, dir)) {
+                return 1;
+            }
+            // Fall through if this is a path to a directory named "tags"
         }
 
         if (is_directory(path, dir)) {

---

Separately, with git, gitignore seems to allow a/tags/foo.txt but exclude the text file b/tags with the above pattern https://git-scm.com/docs/gitignore#_pattern_format

An optional prefix "!" which negates the pattern; any matching file excluded by a previous pattern will become included again.

• the_silver_searcher does not care about the order of ignore patterns relative to other patterns as seen in https://github.com/ggreer/the_silver_searcher/blob/a61f1780b64266587e7bc30f0f5f71c6cca97c0f/src/ignore.c#L226-L284 (it checks all patterns of a given type at once)

tags
!tags/

[TysonAndre-tmg]

https://github.com/BurntSushi/ripgrep seems to work as an alternative (efficient grep-like alternative) for this use case with a/b/tags and a/tags/foo.txt both containing the string example

# .gitignore
tags
# The negated pattern must be *after* the matching pattern for this to work with ripgrep
!tags/

» rg example
a/tags/foo.txt
2:example

[giftig]

This issue has been open for 6 years and it looks like the last commit to the project was also 4 years ago, so I'll close this; seems like ag is no longer actively maintained.