terraform-backend-git icon indicating copy to clipboard operation
terraform-backend-git copied to clipboard

Published 20 hours ago • verified publisher icon plumber-cd

persistence git clone storage

Open rucciva opened this issue 9 months ago • 3 comments

Hi, first of all thanks a lot for creating this.

one thing i noticed is that the cloned git repository is stored in-memory. If we were to run it as standalone server, then as long as the server doesn't exit, the cloned state still persist, meanwhile if using the cli wrapper, it needs to git clone every time it runs.

the problem is, it doesn't take too long for the git repo to grow in size and usually means slow download time. If we were to use cli wrapper, then the execution time will grow.

so is it possible to store the git clone persistently? or is there any workaround?

i did try using LFS but it seems to change the state and make it undecryptable (i'm currently using AES enryption)

rucciva avatar Sep 14 '23 07:09 rucciva

Hi @rucciva - thanks for interest in this project and for the feedback. You are right - this will become a pain on big repositories with long history, I haven't thought of that. The intention was to avoid dealing with cleanup if I were to use real filesystem. Should be easy enough to change that behavior - just a matter of giving it different fs implementation in https://github.com/plumber-cd/terraform-backend-git/blob/v0.1.5/storages/git/git.go#L160-L172. I will try to look into it when I get some free time.

dee-kryvenko avatar Sep 21 '23 20:09 dee-kryvenko

Hi @dee-kryvenko , thanks for the kind respond. Looking forward to it. Regarding the cleanup, might i suggest adding flag or env that signal the executable to delete the cloned directory as cleanup before doing the main job?

rucciva avatar Sep 27 '23 03:09 rucciva

Hi @dee-kryvenko , i've been thinking, what if instead of commiting to $ref each time writing state, the backend do the following:

  1. Copy the existing state to state-$unixtimestamp
  2. Delete old state-* file so that only N latest state-* file remained
  3. Write the new state
  4. Git commit --amend
  5. Git force push

This way, we don't keep appending to git history of the state which in most case is no longer used (unlike the history of the terraform code it self) and save storage.

If you approve the design, i would like to try to contribute

rucciva avatar Oct 02 '23 13:10 rucciva