build_backup folder growing out of control

[axelson]

Hi, we've been using this buildpack without issue in production for some time now. However recently we've been noticing some oddities. The largest one is that the _build/build_backup folder is growing out of control causing an error of: Compiled slug size: 505.2M is too large (max is 500M). Looking into the error it looks like the culprit is the build_backup folder:

~/_build $ ls /app/_build/build_backup/
build_backup  staging  test
~/_build $ ls /app/_build/build_backup/
build_backup  staging  test
~/_build $ ls /app/_build/build_backup/build_backup/
build_backup  staging  test
~/_build $ ls /app/_build/build_backup/build_backup/build_backup/
build_backup  staging  test
~/_build $ ls /app/_build/build_backup/build_backup/build_backup/build_backup/
build_backup  staging  test
~/_build $ du -hs /app/_build/build_backup/build_backup/build_backup/build_backup/
1.2G    /app/_build/build_backup/build_backup/build_backup/build_backup/
~/_build $ du -hs /app/_build/build_backup/build_backup/build_backup/build_backup/*
1.2G    /app/_build/build_backup/build_backup/build_backup/build_backup/build_backup
39M     /app/_build/build_backup/build_backup/build_backup/build_backup/staging
4.0K    /app/_build/build_backup/build_backup/build_backup/build_backup/test

As you can see the entire folder has grown to over 1.2GB! For some reason each build_backup is being saved within the existing build_backup. It seems to point to some type of problem with deleting the previous backups which appears to be on line 56 of app_funcs.sh

https://github.com/HashNuke/heroku-buildpack-elixir/blob/6251590e3a354d4b8ac99960977daf417d3128db/lib/app_funcs.sh#L56

Is there anything we can do to help debug this issue? Right now we're working around it by adding /app/_build/build_backup/build_backup to our .slugignore file. This brought the compressed size of the slug down from 505MB to 79MB.

[ericmj]

I haven't tested this but my guess is that the bug is here https://github.com/HashNuke/heroku-buildpack-elixir/blob/73df63b9617b44ecd6c90ff8ece959facb7adcb3/lib/app_funcs.sh#L8.

We should add / to the end of the source path so that the contents are copied instead of the whole directory, i.e. cp -pR "$(build_backup_path)/" ${build_path}/_build If you would like to try this out I can set up a branch and give instructions on how to use it.

[axelson]

Yeah that definitely sounds plausible. And we'd be willing to test out a branch of the buildpack.

[ericmj]

I have a branch ready https://github.com/HashNuke/heroku-buildpack-elixir/commit/f480baffdf9f194b25ec5d63487770c635e4e2a9.

If this is the first buildpack in your chain use the test branch by running:

heroku buildpacks:set -a MYAPP -i 1 https://github.com/HashNuke/heroku-buildpack-elixir.git\#emj/backup-path

[axelson]

@ericmj okay I've tested that branch but I'm seeing the same behavior:

Jasons-MacBook-Pro-2% heroku buildpacks -r staging2
=== app-staging2 Buildpack URL
https://github.com/HashNuke/heroku-buildpack-elixir.git#emj/backup-path
Jasons-MacBook-Pro-2% heroku run 'ls /app/_build/build_backup' -r staging2
Running ls /app/_build/build_backup on ⬢ app-staging2... up, run.8622 (Hobby)
build_backup  staging  test

[ericmj]

@axelson Did you test it on a new app or cleaned the cache by setting always_rebuild=true in the buildpack config or by running heroku repo:purge_cache -a MYAPP after heroku plugins:install heroku-repo.

[axelson]

I didn't run purge cache before, but after running heroku repo:purge_cache -a MYAPP I am still seeing the same behavior :(

Jasons-MacBook-Pro-2% heroku run 'ls /app/_build/build_backup' -r staging2
Running ls /app/_build/build_backup on ⬢ app-staging2... up, run.4008 (Hobby)
build_backup  staging  test

[ericmj]

Okay, I will look more into this then.

[rockwood]

@axelson Were you able to find a solution to this issue? We're also seeing the nested build_backup/ directories causing our slug size to grow on each deploy. The weird thing is that it's only happening on our staging instance.

[axelson]

@rockwood as far as I know this is still an issue. We're still working around it with the .slugignore file.

cat .slugignore
/app/_build/build_backup/build_backup

[rockwood]

@axelson got it, that's working for me as well. Thanks!

[axelson]

Great, glad I was able to help!

[HashNuke]

@axelson @rockwood Do you folks have a recommended change to the buildpack to resolve this permanently for all users?

[axelson]

@HashNuke It looks to me like there's an issue calculating the $(build_backup_path) but I haven't looked into it too deeply. If that path was correct and successfully deleted previous backups in function backup_app() then I think it would work great.

[HashNuke]

Ah ok. If anyone can make a PR and if a bunch of folks can try out the PR, I can help merge it

[phlppn]

I found the reason for this.

The Heroku-Bash cp command does not respect the trailing slash of the source folder, to tell cp to copy the content of the folder instead of the folder itself.

So instead of writing cp foo/ bar we have to use cp foo/* bar.

This has to be fixed in multiple lines in the project.

It was hard to debug, because my local cp works with /, and also the local manual of cp tells me that a / is enough.

This is also the reason for stuff like /.hex/.hex/...

[rafaeelaudibert]

This should probably be closed by now, it looks like #152 fixed it