skip hidden directories (start with .) picked up by -f

[cppforlife]

there is probably a common case for ignoring hidden directories by default.

common case within carvel tools might be: imgpkg pull results in a directory with .imgpkg directory. if bundle's files arent wrapped within a directory (eg config), then running ytt -f . would include .imgpkg/*.yml which isnt really desired within templates.

another example: currently ytt -f . that includes .git directory actually makes .git files available via data.read and friends. it just happens that there is no .yml files within .git by default.

what other examples of hidden directories ytt may encounter? it might be worth adding a way to control this behaviour?

[danielhelfand]

Running into this same issue with kbld. The bundle pulled using imgpkg has the .imgpkg contents. If running a kbld -f ... | kapp deploy -a simple-app -f-, the following error is thrown:

kapp: Error: Expected to find kind 'imgpkg.carvel.dev/v1alpha1/Bundle', but did not:
- Kubernetes API server did not have matching apiVersion + kind
- No matching CRD was found in given configuration

[pivotaljohn]

For the main use-cases we're targeting, this seems like a sensible default.

Let's add a flag to include hidden files: -a, --all-files Include all files (including hidden files) when scanning for files

-f should obviously allow hidden files, if specified explicitly.

[joaopapereira]

Just to see if I understand clearly

ytt -f config/* This command will only include files that are not hidden

ytt -a -f config/* This command will include all the files in the directory, including the hidden ones

Is this correct?

[joaopapereira]

Can y'll please review these acceptance criteria to ensure they match our view of the feature

---

Scenario: Ignore hidden folders Given I have a folder that contains YAML files And Also contains a hidden folder (sample below) When I invoke ytt -f config Then the output contains only the files that aren't hidden

rendered: always

Scenario: Use hidden folders with a short flag Given I have a folder that contains YAML files And Also contains a hidden folder (sample below) When I invoke ytt -a -f config Then the output contains only the files in all the folders

hidden: content
---
rendered: always

Scenario: Use hidden folders with the full flag Given I have a folder that contains YAML files And Also contains a hidden folder (sample below) When I invoke ytt --a-files -f config Then the output contains only the files in all the folders

hidden: content
---
rendered: always

Scenario: Help displayed Given I have a ytt When I invoke ytt --help Then I should see the output

......
Flags:
-a, --all-files                                                      Include all files (including hidden files) when scanning for files
      --allow-symlink-destination strings           File paths to which symlinks are allowed (can be specified multiple times)
      --bulk-in string                                            Accept files in bulk format
.....

---

Possible folder structure

$ tree -a config
config
├── .hidden
│   └── some-other-file.yml
└── some-file.yml

1 directory, 2 files

Files:

$ cat config/some-file.yml
---
rendered: always

$ cat config/.hidden/some-other-file.yml
---
hidden: content

[danielhelfand]

I wonder if this story should be more general as far as what the idea is (i.e. ignore hidden files by default and support an option to allow hidden files) for all Carvel tools where applicable. I think it's possible that this should apply to kbld and kapp as well. The idea is so similar for each tool that I feel it can be written as a single idea first, and we should also consider where else this is applicable before moving forward. This way, the idea will be consistent where applicable across the tools, which has been a criticism in the past.

[cppforlife]

im still not quite sold on the feature tbh (also it's would be backwards incompat). i think this might be a good issue to discuss in community meeting.

[pivotaljohn]

(will come back, add link and fill out a few more details)

We talked about this in the December 9, 2020 Community Meeting (recording failed).

TL;DR.

For now, we will:

1. recommend that imgpkg users follow the convention of sequestering their configuration files in a directory off the root of the bundle (rather than the root of the bundle itself).
2. for each tool that would want to consider features that also addresses this need, we'll draft a proposal.

High-Level Points

• for each tool in the suite, we want to strike a balance between it working well on its own and working together with other tools in the suite. So, let's avoid Carvel tool-specific features (e.g. ytt somehow knowing about implementation details imgpkg, first-hand).
• @cppforlife reemphasized that he opened this issue in the most public forum for the purpose of discussion, not necessarily advocacy.

[rgeraskin]

It will be great to have the ability to exclude something with .yttignore. Like .gitignore but for ytt.