git: keep the refs space locally when fetching non-SHA revisions

[tejlmand]

Example where fetch of non SHA revisions is kept locally. Test not updated.

Inspired by: https://github.com/NordicPlayground/fw-nrfconnect-nrf/pull/1437

To allow later usage of the fetched refs space then git fetch will always keep a local ref space even fetching non-SHA refs spaces, such as branches and PRs.

Signed-off-by: Torsten Rasmussen [email protected]

[tejlmand]

Oh, tests passes, guess we should improve the unit test cases ;)

[marc-hb]

as we are already keeping remote stuff under refs/west/* we might as well keep whatever west fetches in there, because then we have it available for later use, and thus may reduce number of additional git fetches and hence improved performance. Note, this is NOT about mirroring the remote refs space, simply just storing the information we anyway have fetched, so that we might avoid fetching the same again.

It's unfortunate that Git doesn't allow us to fetch all refs without actually storing them anywhere, just getting the blobs and objects.

But then they would go away on the next garbage collection, no?

Beyond not being able to collect the refs when they are no longer needed, it just feels weird. [...] at least personally I'm bothered by all the west/xyz output I'm seeing showing up in git log --oneline output, for example.

Connecting all the dots above, I just remembered that git already has a built-in feature to cache objects and automatically expire and garbage collect them after some configurable time: it's the reflog. So here's the pseudo code of a maybe not so crazy idea:

git fetch  => refs/west/*

# too many uncollectable refs/west/*
# collapse them all in the reflog a single branch

for ref in  refs/west/*
  git update-ref refs/west/west_global_cache $ref
  git update-ref -d $ref
done

[mbolivar]

Connecting all the dots above, I just remembered that git already has a built-in feature to cache objects and automatically expire and garbage collect them after some configurable time: it's the reflog

Good point!

Since manifest-rev is reflogged already, do we need this then?

[tejlmand]

Connecting all the dots above, I just remembered that git already has a built-in feature to cache objects and automatically expire and garbage collect them after some configurable time: it's the reflog

Good point!

Since manifest-rev is reflogged already, do we need this then?

@marc-hb Good point on updating the reflog, and thus cleaning the refs/west will also help te ensure the SHAs are available for a certain time, while not have all those refs/west references popping up in git log. As I reply in https://github.com/zephyrproject-rtos/west/pull/331#discussion_r351196763 , there needs to be done some cleanup in refs/west, and this appears to be a nice way of doing that.

[tejlmand]

@marc-hb @mbolivar This PR has now been updated to clear all refs under refs/west after a fetch. This creates loose objects for SHAs not referenced elsewhere, and those objects are garbage collected after 2 weeks (default).

If revision is non-SHA, it is accessible in: refs/west/. As example, a manifest pointing having: revision: <SHA> will result in:

$ git log
dc9e1fb (HEAD, ncs/master, manifest-rev) mpsl: Enable zero latency IRQs for MPSL
4f70d0a nfc: documentation: removed obsolete statements
814a529 doc: remove wrong modem limitations in bsdlib doc

and when using a non-SHA, such as pull/<no>/head

$ git log
9ec8d9e (HEAD, refs/west/pull/120/head, manifest-rev) crypto: Updated nrf_oberon to version 3.0.3
dc9e1fb (ncs/master) mpsl: Enable zero latency IRQs for MPSL
4f70d0a nfc: documentation: removed obsolete statements
814a529 doc: remove wrong modem limitations in bsdlib doc

The benefit of having refs/west/pull/120/head available is that other west (extension-)commands can use the knowledge of manifest revision and west ref space to determine if manifest-rev is pointing to the non-SHA revision in manifest file.

Example, manifest:

   revision: pull/120/head

if manifest-rev == refs/west/{project.revision}, the repo has been updated. if manifest-rev != refs/west/{project.revision}, manifest file has changed since last west update was executed.

[mbolivar-nordic]

@tejlmand any plans to get back to this?

[tejlmand]

Every time I look at my list of open PRs, then I feel so sad for this one that it haven't got the attention it deserves. It's still on my list of things I would like to do.

[mbolivar-nordic]

OK, I will leave it open then.

[mbolivar-nordic]

@tejlmand please reopen if you have an update.