Large disk space overhead

[domq]

What's wrong

straight.el hasn't installed a single package yet, and it's already consuming north of 300M on disk:

Directions to reproduce

1. rm -rf ~/.emacs.d/straight
2. Apply the installation instructions
3. If you are vacationing out in the sticks like I am with turn-of-the-century Internet connectivity, come back after lunch

Version information

• Emacs version: gccemacs 28.1 (built on June 10th, 2022)
• Operating system: MacOS Monterey 12.4 (21F79)

[progfolio]

I was able to get it under 100M by explicitly adding :depth 1 to the recipe repositories in bootstrap.el.

Test Case

(straight-bug-report
  :user-dir "disk-usage.straight"
  :pre-bootstrap 
  (setq straight-repository-user "progfolio" straight-repository-branch "fix/nongnu-depth")
  :post-bootstrap 
  (message "%S"
           (straight--process-output "du" "-hSc" "/tmp/disk-usage.straight")))

• Test run at: 2022-07-29 14:30:32
• system-type: gnu/linux
• straight-version: prerelease (HEAD -> fix/nongnu-depth, fork/fix/nongnu-depth) e3469e3 2022-07-29
• emacs-version: GNU Emacs 29.0.50 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.34, cairo version 1.17.6) of 2022-07-28

Output

Bootstrapping straight.el...
Bootstrapping straight.el...done
Looking for gnu-elpa-mirror recipe → Cloning melpa...
Looking for gnu-elpa-mirror recipe → Cloning melpa...done
Looking for nongnu-elpa recipe → Cloning gnu-elpa-mirror...
Looking for nongnu-elpa recipe → Cloning gnu-elpa-mirror...done
Looking for emacsmirror-mirror recipe → Cloning nongnu-elpa...
Looking for emacsmirror-mirror recipe → Cloning nongnu-elpa...done
Looking for emacsmirror-mirror recipe → Cloning el-get...
Looking for emacsmirror-mirror recipe → Cloning el-get...done
Looking for straight recipe → Cloning emacsmirror-mirror...
Looking for straight recipe → Cloning emacsmirror-mirror...done
Building straight...
Building straight...done

Test run with version: prerelease (HEAD -> fix/nongnu-depth, origin/fix/nongnu-depth) e3469e3 2022-07-29
"284K	/tmp/disk-usage.straight/straight/build/straight
0	/tmp/disk-usage.straight/straight/build
4.0K	/tmp/disk-usage.straight/straight/repos/emacsmirror-mirror/.git/logs/refs/heads
4.0K	/tmp/disk-usage.straight/straight/repos/emacsmirror-mirror/.git/logs/refs/remotes/origin
0	/tmp/disk-usage.straight/straight/repos/emacsmirror-mirror/.git/logs/refs/remotes
0	/tmp/disk-usage.straight/straight/repos/emacsmirror-mirror/.git/logs/refs
4.0K	/tmp/disk-usage.straight/straight/repos/emacsmirror-mirror/.git/logs
0	/tmp/disk-usage.straight/straight/repos/emacsmirror-mirror/.git/objects/info
44K	/tmp/disk-usage.straight/straight/repos/emacsmirror-mirror/.git/objects/pack
0	/tmp/disk-usage.straight/straight/repos/emacsmirror-mirror/.git/objects
4.0K	/tmp/disk-usage.straight/straight/repos/emacsmirror-mirror/.git/refs/remotes/origin
0	/tmp/disk-usage.straight/straight/repos/emacsmirror-mirror/.git/refs/remotes
0	/tmp/disk-usage.straight/straight/repos/emacsmirror-mirror/.git/refs/tags
4.0K	/tmp/disk-usage.straight/straight/repos/emacsmirror-mirror/.git/refs/heads
0	/tmp/disk-usage.straight/straight/repos/emacsmirror-mirror/.git/refs
60K	/tmp/disk-usage.straight/straight/repos/emacsmirror-mirror/.git/hooks
0	/tmp/disk-usage.straight/straight/repos/emacsmirror-mirror/.git/branches
4.0K	/tmp/disk-usage.straight/straight/repos/emacsmirror-mirror/.git/info
24K	/tmp/disk-usage.straight/straight/repos/emacsmirror-mirror/.git
92K	/tmp/disk-usage.straight/straight/repos/emacsmirror-mirror
4.0K	/tmp/disk-usage.straight/straight/repos/el-get/test/pkgs
248K	/tmp/disk-usage.straight/straight/repos/el-get/test/issues
60K	/tmp/disk-usage.straight/straight/repos/el-get/test
7.3M	/tmp/disk-usage.straight/straight/repos/el-get/recipes
136K	/tmp/disk-usage.straight/straight/repos/el-get/methods
188K	/tmp/disk-usage.straight/straight/repos/el-get/logo
4.0K	/tmp/disk-usage.straight/straight/repos/el-get/.github/workflows
0	/tmp/disk-usage.straight/straight/repos/el-get/.github
4.0K	/tmp/disk-usage.straight/straight/repos/el-get/.git/logs/refs/heads
4.0K	/tmp/disk-usage.straight/straight/repos/el-get/.git/logs/refs/remotes/origin
0	/tmp/disk-usage.straight/straight/repos/el-get/.git/logs/refs/remotes
0	/tmp/disk-usage.straight/straight/repos/el-get/.git/logs/refs
4.0K	/tmp/disk-usage.straight/straight/repos/el-get/.git/logs
0	/tmp/disk-usage.straight/straight/repos/el-get/.git/objects/info
1020K	/tmp/disk-usage.straight/straight/repos/el-get/.git/objects/pack
0	/tmp/disk-usage.straight/straight/repos/el-get/.git/objects
4.0K	/tmp/disk-usage.straight/straight/repos/el-get/.git/refs/remotes/origin
0	/tmp/disk-usage.straight/straight/repos/el-get/.git/refs/remotes
0	/tmp/disk-usage.straight/straight/repos/el-get/.git/refs/tags
4.0K	/tmp/disk-usage.straight/straight/repos/el-get/.git/refs/heads
0	/tmp/disk-usage.straight/straight/repos/el-get/.git/refs
60K	/tmp/disk-usage.straight/straight/repos/el-get/.git/hooks
0	/tmp/disk-usage.straight/straight/repos/el-get/.git/branches
4.0K	/tmp/disk-usage.straight/straight/repos/el-get/.git/info
196K	/tmp/disk-usage.straight/straight/repos/el-get/.git
360K	/tmp/disk-usage.straight/straight/repos/el-get
0	/tmp/disk-usage.straight/straight/repos/nongnu-elpa/packages
4.0K	/tmp/disk-usage.straight/straight/repos/nongnu-elpa/.git/logs/refs/heads
4.0K	/tmp/disk-usage.straight/straight/repos/nongnu-elpa/.git/logs/refs/remotes/origin
0	/tmp/disk-usage.straight/straight/repos/nongnu-elpa/.git/logs/refs/remotes
0	/tmp/disk-usage.straight/straight/repos/nongnu-elpa/.git/logs/refs
4.0K	/tmp/disk-usage.straight/straight/repos/nongnu-elpa/.git/logs
0	/tmp/disk-usage.straight/straight/repos/nongnu-elpa/.git/objects/info
62M	/tmp/disk-usage.straight/straight/repos/nongnu-elpa/.git/objects/pack
0	/tmp/disk-usage.straight/straight/repos/nongnu-elpa/.git/objects
4.0K	/tmp/disk-usage.straight/straight/repos/nongnu-elpa/.git/refs/remotes/origin
0	/tmp/disk-usage.straight/straight/repos/nongnu-elpa/.git/refs/remotes
0	/tmp/disk-usage.straight/straight/repos/nongnu-elpa/.git/refs/tags
4.0K	/tmp/disk-usage.straight/straight/repos/nongnu-elpa/.git/refs/heads
0	/tmp/disk-usage.straight/straight/repos/nongnu-elpa/.git/refs
60K	/tmp/disk-usage.straight/straight/repos/nongnu-elpa/.git/hooks
0	/tmp/disk-usage.straight/straight/repos/nongnu-elpa/.git/branches
4.0K	/tmp/disk-usage.straight/straight/repos/nongnu-elpa/.git/info
40K	/tmp/disk-usage.straight/straight/repos/nongnu-elpa/.git
88K	/tmp/disk-usage.straight/straight/repos/nongnu-elpa
4.0K	/tmp/disk-usage.straight/straight/repos/gnu-elpa-mirror/.git/logs/refs/heads
4.0K	/tmp/disk-usage.straight/straight/repos/gnu-elpa-mirror/.git/logs/refs/remotes/origin
0	/tmp/disk-usage.straight/straight/repos/gnu-elpa-mirror/.git/logs/refs/remotes
0	/tmp/disk-usage.straight/straight/repos/gnu-elpa-mirror/.git/logs/refs
4.0K	/tmp/disk-usage.straight/straight/repos/gnu-elpa-mirror/.git/logs
0	/tmp/disk-usage.straight/straight/repos/gnu-elpa-mirror/.git/objects/info
8.0K	/tmp/disk-usage.straight/straight/repos/gnu-elpa-mirror/.git/objects/pack
0	/tmp/disk-usage.straight/straight/repos/gnu-elpa-mirror/.git/objects
4.0K	/tmp/disk-usage.straight/straight/repos/gnu-elpa-mirror/.git/refs/remotes/origin
0	/tmp/disk-usage.straight/straight/repos/gnu-elpa-mirror/.git/refs/remotes
0	/tmp/disk-usage.straight/straight/repos/gnu-elpa-mirror/.git/refs/tags
4.0K	/tmp/disk-usage.straight/straight/repos/gnu-elpa-mirror/.git/refs/heads
0	/tmp/disk-usage.straight/straight/repos/gnu-elpa-mirror/.git/refs
60K	/tmp/disk-usage.straight/straight/repos/gnu-elpa-mirror/.git/hooks
0	/tmp/disk-usage.straight/straight/repos/gnu-elpa-mirror/.git/branches
4.0K	/tmp/disk-usage.straight/straight/repos/gnu-elpa-mirror/.git/info
48K	/tmp/disk-usage.straight/straight/repos/gnu-elpa-mirror/.git
0	/tmp/disk-usage.straight/straight/repos/gnu-elpa-mirror
0	/tmp/disk-usage.straight/straight/repos/melpa/working
4.0K	/tmp/disk-usage.straight/straight/repos/melpa/service
12K	/tmp/disk-usage.straight/straight/repos/melpa/scripts
21M	/tmp/disk-usage.straight/straight/repos/melpa/recipes
4.0K	/tmp/disk-usage.straight/straight/repos/melpa/packages
0	/tmp/disk-usage.straight/straight/repos/melpa/packages-stable
4.0K	/tmp/disk-usage.straight/straight/repos/melpa/package-build/.github/workflows
0	/tmp/disk-usage.straight/straight/repos/melpa/package-build/.github
76K	/tmp/disk-usage.straight/straight/repos/melpa/package-build
52K	/tmp/disk-usage.straight/straight/repos/melpa/logo
8.0K	/tmp/disk-usage.straight/straight/repos/melpa/html/partials
28K	/tmp/disk-usage.straight/straight/repos/melpa/html/js
4.0K	/tmp/disk-usage.straight/straight/repos/melpa/html/css
48K	/tmp/disk-usage.straight/straight/repos/melpa/html
0	/tmp/disk-usage.straight/straight/repos/melpa/html-stable
8.0K	/tmp/disk-usage.straight/straight/repos/melpa/docker/syncer
8.0K	/tmp/disk-usage.straight/straight/repos/melpa/docker/rsyncd
20K	/tmp/disk-usage.straight/straight/repos/melpa/docker/nginx
16K	/tmp/disk-usage.straight/straight/repos/melpa/docker/logprocessor
12K	/tmp/disk-usage.straight/straight/repos/melpa/docker/builder
4.0K	/tmp/disk-usage.straight/straight/repos/melpa/docker
4.0K	/tmp/disk-usage.straight/straight/repos/melpa/.github/workflows
4.0K	/tmp/disk-usage.straight/straight/repos/melpa/.github
4.0K	/tmp/disk-usage.straight/straight/repos/melpa/.git/logs/refs/heads
4.0K	/tmp/disk-usage.straight/straight/repos/melpa/.git/logs/refs/remotes/origin
0	/tmp/disk-usage.straight/straight/repos/melpa/.git/logs/refs/remotes
0	/tmp/disk-usage.straight/straight/repos/melpa/.git/logs/refs
4.0K	/tmp/disk-usage.straight/straight/repos/melpa/.git/logs
0	/tmp/disk-usage.straight/straight/repos/melpa/.git/objects/info
844K	/tmp/disk-usage.straight/straight/repos/melpa/.git/objects/pack
0	/tmp/disk-usage.straight/straight/repos/melpa/.git/objects
4.0K	/tmp/disk-usage.straight/straight/repos/melpa/.git/refs/remotes/origin
0	/tmp/disk-usage.straight/straight/repos/melpa/.git/refs/remotes
0	/tmp/disk-usage.straight/straight/repos/melpa/.git/refs/tags
4.0K	/tmp/disk-usage.straight/straight/repos/melpa/.git/refs/heads
0	/tmp/disk-usage.straight/straight/repos/melpa/.git/refs
60K	/tmp/disk-usage.straight/straight/repos/melpa/.git/hooks
0	/tmp/disk-usage.straight/straight/repos/melpa/.git/branches
4.0K	/tmp/disk-usage.straight/straight/repos/melpa/.git/info
468K	/tmp/disk-usage.straight/straight/repos/melpa/.git
80K	/tmp/disk-usage.straight/straight/repos/melpa
20K	/tmp/disk-usage.straight/straight/repos/straight.el/watcher
0	/tmp/disk-usage.straight/straight/repos/straight.el/tests/mocks/.emacs.d/straight/repos/straight-mock-repo/subdir
8.0K	/tmp/disk-usage.straight/straight/repos/straight.el/tests/mocks/.emacs.d/straight/repos/straight-mock-repo
0	/tmp/disk-usage.straight/straight/repos/straight.el/tests/mocks/.emacs.d/straight/repos
0	/tmp/disk-usage.straight/straight/repos/straight.el/tests/mocks/.emacs.d/straight/build/straight-mock-repo
0	/tmp/disk-usage.straight/straight/repos/straight.el/tests/mocks/.emacs.d/straight/build
0	/tmp/disk-usage.straight/straight/repos/straight.el/tests/mocks/.emacs.d/straight
0	/tmp/disk-usage.straight/straight/repos/straight.el/tests/mocks/.emacs.d
0	/tmp/disk-usage.straight/straight/repos/straight.el/tests/mocks
24K	/tmp/disk-usage.straight/straight/repos/straight.el/tests
16K	/tmp/disk-usage.straight/straight/repos/straight.el/scripts
12K	/tmp/disk-usage.straight/straight/repos/straight.el/benchmark
4.0K	/tmp/disk-usage.straight/straight/repos/straight.el/.github/workflows
12K	/tmp/disk-usage.straight/straight/repos/straight.el/.github/ISSUE_TEMPLATE
4.0K	/tmp/disk-usage.straight/straight/repos/straight.el/.github
4.0K	/tmp/disk-usage.straight/straight/repos/straight.el/.git/logs/refs/heads/fix
0	/tmp/disk-usage.straight/straight/repos/straight.el/.git/logs/refs/heads
4.0K	/tmp/disk-usage.straight/straight/repos/straight.el/.git/logs/refs/remotes/origin
0	/tmp/disk-usage.straight/straight/repos/straight.el/.git/logs/refs/remotes
0	/tmp/disk-usage.straight/straight/repos/straight.el/.git/logs/refs
4.0K	/tmp/disk-usage.straight/straight/repos/straight.el/.git/logs
0	/tmp/disk-usage.straight/straight/repos/straight.el/.git/objects/info
3.0M	/tmp/disk-usage.straight/straight/repos/straight.el/.git/objects/pack
0	/tmp/disk-usage.straight/straight/repos/straight.el/.git/objects
4.0K	/tmp/disk-usage.straight/straight/repos/straight.el/.git/refs/remotes/origin
0	/tmp/disk-usage.straight/straight/repos/straight.el/.git/refs/remotes
0	/tmp/disk-usage.straight/straight/repos/straight.el/.git/refs/tags
4.0K	/tmp/disk-usage.straight/straight/repos/straight.el/.git/refs/heads/fix
0	/tmp/disk-usage.straight/straight/repos/straight.el/.git/refs/heads
0	/tmp/disk-usage.straight/straight/repos/straight.el/.git/refs
60K	/tmp/disk-usage.straight/straight/repos/straight.el/.git/hooks
0	/tmp/disk-usage.straight/straight/repos/straight.el/.git/branches
4.0K	/tmp/disk-usage.straight/straight/repos/straight.el/.git/info
20K	/tmp/disk-usage.straight/straight/repos/straight.el/.git
792K	/tmp/disk-usage.straight/straight/repos/straight.el
0	/tmp/disk-usage.straight/straight/repos
4.0K	/tmp/disk-usage.straight/straight
0	/tmp/disk-usage.straight
98M	total"
Packages:
"straight"                n/a                  fix/nongnu-depth e3469e3 2022-07-29
"org-elpa"                n/a                  n/a
"melpa"                   n/a                  master b33e8f7 2022-07-28
"gnu-elpa-mirror"         n/a                  master 55b898c 2022-07-26
"nongnu-elpa"             n/a                  main baad781 2022-07-24
"el-get"                  melpa                master bf3dba4 2022-06-02
"emacsmirror-mirror"      n/a                  master bc49945 2022-07-27

Don't have time to make a proper patch of this right now, but that may help.

[progfolio]

Haven't made a patch for this because I don't think it's as simple as just defaulting to a shallow clone. We have to consider the case where a user may have a lockfile which needs to access the recipe repositories history. I'll have to think on a clean way to implement this and may not be able to get to it any time soon. If anyone has any ideas, I am willing to help review any pull requests for such a feature.

[raxod502]

I think https://github.com/radian-software/straight.el/issues/149 is the future for low disk space consumption. Unfortunately, I do not believe it is possible to use shallow clones when the user has a lockfile, because you can't shallow-clone a specific ("unadvertised") commit from many Git repositories, you can only supply the name of a branch.

[raxod502]

Agreed 300-400MB disk space is silly and the long setup time bothers me as well.

[raxod502]

One mitigation we could do is default to shallow clones for the recipe repos, and just fallback to a full clone when we need a specific commit.