Nabla
Nabla copied to clipboard
Devsh-Graphics-Programming
The Nabla repository is a little too large to casually include in projects (due to unused git objects)
Description
Hi, I did a fresh clone of Nabla and it required downloading 1.88 GB:
Cloning into 'Nabla'...
remote: Enumerating objects: 114400, done.
remote: Counting objects: 100% (1568/1568), done.
remote: Compressing objects: 100% (645/645), done.
remote: Total 114400 (delta 988), reused 1317 (delta 826), pack-reused 112832
Receiving objects: 100% (114400/114400), 1.88 GiB | 18.20 MiB/s, done.
Resolving deltas: 100% (85301/85301), done.
I'd love to be able to include it in my project for some of the glsl built-ins, but this is a little too large for me to casually include it as a git submodule if I just want to use one or two functions.
I ran a script from https://stackoverflow.com/a/46085465 to get the sizes of objects in the repository and it came up with example_tests:
eb171b419446 64MiB examples_tests/13.Silverlining/bin/.pbtmpc3PzpB
f25a8fce24f4 64MiB examples_tests/13.Silverlining/bin/.pbtmphgywp3
b73caa80802b 68MiB examples_tests/media/noises/spp_benchmark_4k_512.exr
416977c7e0d2 76MiB examples_tests/media/sponza.zip
84e57b3c513d 77MiB examples_tests/media/sponza.zip
7cd3b5db497e 77MiB examples_tests/media/sponza.zip
66e69dc8cd36 77MiB examples_tests/media/sponza.zip
6b0b55959fcd 77MiB examples_tests/media/sponza.zip
776c942fe2c2 78MiB examples_tests/41.VisibilityBuffer/bin/visibilitybuffer_rwdi.iobj
963cad7c3095 88MiB examples_tests/media/sponza.zip
I ran git filter-repo --invert-paths --path examples_tests/ --force to see what things would look like once that directory was removed from git history and the size of .git went down to 500MB. Running the script again results in site_media being the next culprit:
17d0dd1da35c 19MiB site_media/readme/gifs/autoexposure/exposure.gif
c5978c572a40 20MiB site_media/readme/gifs/denoisingalbedoandnormals/denoising 3.gif
d21db36eb090 22MiB site_media/readme/gifs/denoisingalbedoandnormals/denoising 4.gif
3e1e88317ee2 22MiB site_media/readme/gifs/raytracingflythrough/raytracing 4.gif
b180c2ef6602 23MiB site_media/readme/gifs/denoisingalbedoandnormals/denoising 2.gif
b78d5c002d14 24MiB site_media/readme/gifs/raytracingflythrough/raytracing 5.gif
87f630d9d65d 24MiB site_media/readme/gifs/denoisingalbedoandnormals/denoising 1.gif
7615cd16a19a 24MiB site_media/readme/gifs/denoisingalbedoandnormals/denoising 5.gif
6b3e61d6a6f9 24MiB site_media/readme/gifs/raytracingflythrough/raytracing 2.gif
6b1e3fa94e47 57MiB site_media/readme/gifs/raytracingflythrough/raytracing 6.gif
After running git filter-repo --invert-paths --path site_media --force the size went down again to a very acceptable 50MB.
Solution proposal
If you're comfortable with removing these two directories from git history then I say go for it. Obviously that's necessarily an easy decision to make though. examples_tests is now a submodule but (I might be mistaken) it seems like it was part of this repository in the past.
Unfortunately we realize the problem exists and we will filter this history at some point, thanks for pointing it out. I've tried to cut the size first here https://github.com/Devsh-Graphics-Programming/Nabla/pull/600. Shall we sort this out @devshgraphicsprogramming?
@expenses current nice solution would be to do shallow clone, you will reduce the size to 6.19 MiB
$ git clone --depth=1 [email protected]:Devsh-Graphics-Programming/Nabla.git .
Cloning into '.'...
remote: Enumerating objects: 2663, done.
remote: Counting objects: 100% (2663/2663), done.
remote: Compressing objects: 100% (2275/2275), done.
remote: Total 2663 (delta 386), reused 1789 (delta 310), pack-reused 0
Receiving objects: 100% (2663/2663), 6.19 MiB | 6.11 MiB/s, done.
Resolving deltas: 100% (386/386), done.
If Nabla is your submodule you can shallow init and update it too
git submodule update --init --depth=1 <module_path_to_nabla>
If you build Nabla I recommend to let CMake do submodule updates for you, to request shallow updates we have NBL_CI_GIT_SUBMODULES_SHALLOW option, you can also exclude whole examples_tests (and parents) from the submodule request init & update by turning off NBL_BUILD_EXAMPLES option, to toggle the request you can use NBL_UPDATE_GIT_SUBMODULE option
but (I might be mistaken) it seems like it was part of this repository in the past.
indeed, got filtered with git-filter-repo :D
I'd love to be able to include it in my project for some of the glsl built-ins, but this is a little too large for me to casually include it as a git submodule if I just want to use one or two functions.
@expenses you're probably better off justcopying the GLSL over into your project, because we're about to delete it in favour of HLSL.
If you're comfortable with removing these two directories from git history then I say go for it. Obviously that's necessarily an easy decision to make though. examples_tests is now a submodule but (I might be mistaken) it seems like it was part of this repository in the past.
The ditt branch is pretty far behind, and we need to keep it alive for some time (until a new Vulkan Path Tracer replaces the OpenGL+OpenCL+CUDA abomination we made).
Once that's done I'm somewhat open to "truncating" the history.
However I'll probably move out nbl/builtin/hlsl to a submodule first.
@expenses current nice solution would be to do shallow clone, you will reduce the size to 6.19 MiB
Ah, that's a great solution, I totally forgot about shallow clones!
@expenses you're probably better off justcopying the GLSL over into your project, because we're about to delete it in favour of HLSL.
Yep fair enough, I want to switch (back) to HLSL but I'm stuck with GLSL until I can use 8-bit ints, 64-bit atomics and BDA nicely.
@expenses current nice solution would be to do shallow clone, you will reduce the size to 6.19 MiB
Ah, that's a great solution, I totally forgot about shallow clones!
@expenses you're probably better off justcopying the GLSL over into your project, because we're about to delete it in favour of HLSL.
Yep fair enough, I want to switch (back) to HLSL but I'm stuck with GLSL until I can use 8-bit ints, 64-bit atomics and BDA nicely.
8bit is a problem, 64bit work just fine, BDA incoming
ok there's a lot of options in our CMake now to control shallow vs deep submodule checkout also we don't recursively clone all submodules anyway