Directory.list() should allow skipping hidden files

[jmagman]

Use case is to list the contents of a directory, and skip any "hidden" files as defined on the current platform (dot files in Linux and macOS, hidden Windows files, etc).

Directory:

  @override
  Stream<FileSystemEntity> list(
      {bool recursive = false, bool followLinks = true, bool includeHidden = true});

  @override
  List<FileSystemEntity> listSync(
      {bool recursive = false, bool followLinks = true, bool includeHidden = true});

And io.FileSystemEntity should have a corresponding property

bool get hidden;

[jmagman]

For example, to fix https://github.com/flutter/flutter/issues/29052 we would have to explicitly exclude directories prepended with ..

[sortie]

The Unix case is easily implemented in user code by handling files starting with a period specially. The Windows case is the most useful as it's a separate file attribute that I don't believe is otherwise available in Dart.

Adding features to classes is currently considered a breaking change under our breaking change policy as people might be implementing the Directory and FileSystemEntity classes, and adding new named parameters / fields would break people's code as they are no longer proper subtypes. That unfortunately means adding these features is not currently a trivial change as the breakage would need to be assessed and a formal breaking change request needs to be made.

[ShiinaOrez]

This issue must be fixed as soon as possiable... I don't wanna get ALL HIDDEN DIR or FILE on Windows, even though I use "invokeMethod" to invoke golang method(use golang.org/x/sys/windows) to make a syscall on Windows to filter HIDDEN DIR or FILE, but MacOS cannot complie this windows package.... I'm crazy now, f**king hidden file

[hanskokx]

This has been a problem for me for months.

https://github.com/hanskokx/dexa/blob/bfa403304347e645dd58d63995e166a2d67fdbba/bin/dexa.dart#L43

[brianquinlan]

The issue here is that adding options to list and listSync will break a lot of existing code e.g. package:file.

Would it make sense to document that hidden files will be returned? Or is there something smarter that we can do?

[hanskokx]

Ideally, you'd be able to either list only visible files or hidden+visible, and you'd know the difference between each.

[cbenhagen]

When implementing this on macOS, please make sure to check st_flags for UF_HIDDEN in addition to names starting with a dot.

[natebosch]

Or is there something smarter that we can do?

I don't know if it would be smarter but we might be able to solve enough use cases with a new static method. Maybe something like Directory.listUnhidden(String path). I don't like the inconsistency, but since this isn't something that can be worked around in Dart code we might settle for an uglier API. cc @lrhn

[lrhn]

If being "hidden" is a well-defined property of a file system entity on all platforms, then we can build the notion into the platform library.

Is that so?

Being "hidden" is a UI feature, from the file system's perspective, the file is just there. It has some flags. Some UI (which can be the console ls command) then interprets some combination of flags and naming strategies as a suggestion to not include the file in some listings. Which means, deciding whether something is hidden is a strategy, not an inherent property of the entity. There could be other choices for which files to consider hidden. (Woulld you want .config to be hidden on Windows? Sometimes, yes.)

But that's also a position which can reasonably be argued against by pointing to the "HIDDEN" flag on Windows files, which is very much part of the file system definition and has precisely one meaning. But the file is still there, and whether to show a file with the HIDDEN bit set is a UI decision. Some views do, others do not. It's just an otherwise meaningless togglable bit with a consistent interpretation.

If I were to define an API from scratch, I'd provide a way to filter on the list/listSync method, one that allows you to filter the entities close to where they are being listed, before giving them to the user. That's for listSync in particular, filtering before building the list, rather than removing things again afterwards. For the stream, a .where on the returned stream would filter too.

That suggest an approach, that is also the "easiest" (least breaking), which is to add static bool isHidden(FileSystemEntity entity) {...} on FileSystemEntity. It is implemented to figure out whether an entity is considered "hidden" by the current platform's normal strategy.

Then you can just do Directory.list(...).whereNot(FileSystemEntity.isHidden), and it works.

Doesn't work so well for listSync, though.

We could add extension methods listWhere/listWhereSync on Directory which takes a filter, and tries to be more efficient about it. That really only works if the Directory is a platform Directory, then it can call an internal helper _listWhereSync which does the filtering before creating the list. For any other implementation of Directory, it'll have to call listSync and filter the list afterwards. Which will suck, because it'll likely have to do a statSync on every entity to read its HIDDEN bit. Except on Unix, if the only way to be hidden is to start with ., then it can be checked directly on the file name.

(For list we can do something similar, assuming that there is a more efficient way to find the non-hidden/filter out the hidden entities while doing the listing, otherwise it wouldn't buy us anything over just a .whereNot(FSE.isHidden).)

Another option is to have a FileSystemEntityFilter class which can be configured based on some standard flags, name-validation and an optional custom user function. Something like

abstract class FileSystemEntityFilter {
  abstract final bool? hidden;
  abstract final bool? backup;
  abstract final bool? readOnly;
  abstract final Pattern? fileNamePattern;
  abstract final Pattern? fullPathPattern;
  abstract final bool Function(FileSystemEntity)? customFilter;
}

(whatever flags have some sort of general meaning).

Then our listWhere from above can take one of those, and apply any filtering that has been set to non-null. It's more optimizable than just a single bool Function(FileSystemEntity), because the list operation can see which properties of the entity are needed, before it starts listing.

(Maybe we should just create a new file system API. From scratch. Using dart:ffi. What could possibly go wrong!)

[natebosch]

Then you can just do Directory.list(...).whereNot(FileSystemEntity.isHidden), and it works.

Doesn't work so well for listSync, though.

It doesn't work well for recursive: true either. We don't want to spend time crawling the hidden subdirectories at all, filtering them from the stream after the fact is not sufficient.

It is implemented to figure out whether an entity is considered "hidden" by the current platform's normal strategy.

What would we do with non-platform implementations of FileSystemEntity? Treat them as always unhidden? Vary behavior by platform (on linux non-platform file implementations can be hidden by name, but on windows a different class implementation is always unhidden?)

[lrhn]

We don't want to spend time crawling the hidden subdirectories at all,

That's probably the correct behavior, which also means that being hidden is not just a property of the individual entities, it's also part of the process of finding them, being applied to directories being traversed, but presumably not to present directories further out than where the listing started.

So a filter function argument to list could work, but that would be breaking.