utilizing child requirement files in caches

[pahwaranger]

Given that you are generating a prod.txt and dev.txt where dev is a child of prod. If you add/remove a dependency from prod.txt then dev.txt is not necessarily modified at all since dev.txt simply includes -r prod.txt. When combining this with env caching, it means you cannot generate the cache key using only dev.txt but instead have to include both prod.txt and dev.txt in order to properly manage the invalidate/regenerate the cache.

For example, in Github actions, because my project utilizes several different envs, I am forced to use a hash of all files in the requirements dir. This ends up with the cache key including ${{ hashFiles('requirements/**.txt') }}. This is imperfect since I am then overly aggressive with invalidation/regeneration since all envs will be purged if any of the envs are changed.

Is there a better solution for this I am unaware of? Would we be open to an arg that tells pip-compile-multi to include all packages within the child .txt (avoid using -r prod.txt)? Or perhaps we could include the SHA1 of all parent files in the header comment?

[matthayes]

Or perhaps we could include the SHA1 of all parent files in the header comment?

This makes sense and it seems it would solve the problem.

[peterdemin]

I'm not too fond of the option to duplicate packages across referenced envs. A flag to include SHA1s from all referenced envs sounds plausible. I'd like to clarify that the individual SHA1 generation logic should remain intact. And the same goes for checksum verification. The referenced SHA1s can go as comments below the header. But they don't need to be parsed back. Would you be open to working on a PR?

[pahwaranger]

@peterdemin yes, agreed. I'll give it a shot. thanks for the quick response

[pahwaranger]

Quick update, I will likely not have bandwidth to take this on right now but may be able to give it a shot in a few months.