Broken cache layer

[POD666]

Actual behavior Running a build without cache pass successfully. Running with cache has been working for a while but now it fails.

Dockerfile looks like this:

COPY requirements.txt /dist/
RUN pip install -r requirements.txt
COPY mypackage.whl /dist/
RUN pip install mypackage --no-index --find-links=/dist/

Commands 1&2 are cached. But when it comes to the 4th command I get following warnings (only with kaniko cache):

WARNING: Ignoring invalid distribution -wh-wtforms (/usr/local/lib/python3.6/site-packages)
WARNING: Ignoring invalid distribution -wh-werkzeug (/usr/local/lib/python3.6/site-packages)
WARNING: Ignoring invalid distribution -wh-typing-extensions (/usr/local/lib/python3.6/site-packages)
WARNING: Ignoring invalid distribution -wh-text-unidecode (/usr/local/lib/python3.6/site-packages)
WARNING: Ignoring invalid distribution -wh-tenacity (/usr/local/lib/python3.6/site-packages)
WARNING: Ignoring invalid distribution -wh-sqlalchemy (/usr/local/lib/python3.6/site-packages)
...

And then installing of mypackage fails with some missing packages.

It seems to be an issue with kaniko cache because without cache it works as expected and without warnings.

Expected behavior

I miss an easy way to rebuild/cleanup cache. Would be nice if there was a flag to achieve it like --rebulid-cache. In my CI, we build images without cache daily (scheduled pipelines) to detect if some 3rd party becomes broken, so rebuilding cache would be also helpful. Especially, if the new cache is only commited if a build succeeded (otherwise I will need a manual job for it).

To Reproduce

Additional Information

• Kaniko Image (fully qualified with digest)

Digest: sha256:473d6dfb011c69f32192e668d86a47c0235791e7e857c870ad70c5e86ec07e8c
Status: Downloaded newer image for gcr.io/kaniko-project/executor:v1.3.0-debug

Triage Notes for the Maintainers

Description	Yes/No
Please check if this a new feature you are proposing	- [x]
Please check if the build works in docker but not in kaniko	- [ ]
Please check if this error is seen when you use --cache flag	- [x]
Please check if your dockerfile is a multistage dockerfile	- [ ]

[POD666]

Just upgraded to the 1.8.1 kaniko to invalidate cache.

But I guess such issues might still occur so might worth investigating and introducing some workarounds.