Consider adding a PYTHON_CACHE setting, like the nodejs NODE_MODULES_CACHE

[davepeck]

This is a feature request to add support for a PYTHON_CACHE variable, much like the NODE_MODULES_CACHE variable, that clears out the built env entirely on each build.

Our project uses mypy and Heroku CI. Because mypy (and pyright) scan for installed type stubs, and because the build cache is shared even across builds for different branches, it's possible for changes to type stub dependencies on branch A to cause mypy failures on (unaltered) branch B. It seems there are cases where clearing out the cache on every build is necessary.

[davepeck]

If it's helpful, here's some detailed context:

Tests passed on our main branch. We created a feature branch (upgrades) to update mypy and some type stubs dependencies (django-stubs, etc.).

CI passed on the upgrades branch but suddenly started breaking on the main branch, despite the fact that there were no material changes there.

The issue was that django-stubs==1.8.0 had a new transitive dep (django-stubs-ext) that wasn't needed in the main branch and that altered mypy's behavior on that branch in a way that caused it to fail.

[edmorley]

@davepeck Hi! Sorry for the delayed reply. The GitHub issues here are less actively monitored than eg support tickets.

Could you explain more about this use-case/steps to reproduce? At the moment it feels like we're jumping to a potential solution, without having first explored the problem in more detail :-)

Heroku CI has a shared cache across all CI runs, which can be cleared manually, but otherwise it's up to buildpacks to perform cache invalidation the same as normal builds for the same branch. If there are issues with the cache invalidation, then this would affect normal builds as well as CI, so this seems like less of a CI issue, and more of a general caching issue.

This buildpack currently invalidates the cache if the file contents of the repo root requirements.txt changes. Due to the non-deterministic nature of pip installs from a requirements file (compared to say packaging tools that use a full lockfile and sync an environment including package removals etc) the onus is unfortunately somewhat on the user to ensure that the requirements file specifies dependencies as explicitly as possible.

ie: List all dependencies, have them pinned to exact versions, and then use a tool like Dependabot to automatically keep them up to date via PRs.

The benefit of this approach is that after each dependabot PR, the contents of the requirements file changes, so the buildpack knows to invalidate the cache. With this, I don't believe the scenario described in the OP would be able to occur :-)

[edmorley]

Closing since it doesn't seem like there is an active need for this, given the above.