Only clone necessary part of git repository dependency (`--depth`)

[Rotzbua]

• [x] Feature Request.
• [x] PlatformIO Core.

Configuration

Operating system: Win 10, Visual Studio Code

PlatformIO Version (platformio --version): PlatformIO Core, version 6.1.5

Description of problem

If a dependency is a tag on a git repository, the complete git repository is cloned. Depending on the repository the traffic and amount of files can be huge. This causes a long transfer and processing time which decreases the user experience.

Git repositories are used for development built and testing.

Example platformio.ini:

[env:heltec_wifi_kit_32]
platform = espressif32@^5.2.0
platform_packages =
  ; use a upstream tag
  framework-arduinoespressif32 @ https://github.com/espressif/arduino-esp32#5b61930bb732d70f8aaa650047b9752365c9698c
framework = arduino
board = heltec_wifi_kit_32

Actual Results

Traffic: 1.60 GiB

Expected Results

Traffic: 187.19 MiB -> 10 times less

Solution hint

If possible use the --depth=1 argument from git to only clone the necessary files. Ref: https://git-scm.com/docs/git-clone

[ivankravets]

If possible use the --depth=1

We use it, see https://github.com/platformio/platformio-core/blob/develop/platformio/package/vcsclient.py#L195

If you specify a custom commit or tag, we need to clone the whole repo and later switch to the target.

Is it possible to clone "commit" or "tag" with --depth=1?

[Jason2866]

Afaik you can not checkout a specific commit without doing a full git clone. https://stackoverflow.com/questions/31278902/how-to-shallow-clone-a-specific-commit-with-depth-1

[Rotzbua]

Right. Shallow cloning a commit is not possible. It is just possible with fetch.

According to my example it would be:

git init
git remote add origin https://github.com/espressif/arduino-esp32
git fetch --depth=1 origin 5b61930bb732d70f8aaa650047b9752365c9698c
git checkout FETCH_HEAD

Thanks for pointing to the line of code. My quick working solution is:

    def export(self):
        is_commit = self.is_commit_id(self.tag)
        args = ["clone", "--recursive"]
        if not self.tag or not is_commit:
            args += ["--depth", "1"]
            if self.tag:
                args += ["--branch", self.tag]
        args += [self.remote_url, self.src_dir]
        if not is_commit:
            assert self.run_cmd(args, cwd=os.getcwd())
        if is_commit:
            assert self.run_cmd(["init", self.src_dir], cwd=os.getcwd())
            assert self.run_cmd(["remote", "add", "origin", self.remote_url])
            assert self.run_cmd(["fetch", "--depth=1", "origin", self.tag])
            
            assert self.run_cmd(["reset", "--hard", self.tag])
            return self.run_cmd(
                ["submodule", "update", "--init", "--recursive", "--force"]
            )
        return True

An alternative is to download a zip. Github offers this over a generic URL:

https://github.com/espressif/arduino-esp32/archive/commitORref.zip

The example config then would look like:

[env:heltec_wifi_kit_32]
platform = espressif32@^5.2.0
platform_packages =
  ; use a upstream tag
  framework-arduinoespressif32 @ https://github.com/espressif/arduino-esp32/archive/5b61930bb732d70f8aaa650047b9752365c9698c.zip
framework = arduino
board = heltec_wifi_kit_32

[Garfius]

There are many people who only make one-way use of [git clone], especially platform_packages and platform. We would greatly appreciate a [Shallow Cloning] option.

Many thanks for the good work.

https://www.howtogeek.com/devops/how-to-use-git-shallow-clone-for-faster-repository-cloning/