jgit timestamp provider do not work with --depth=1

[laeubi]

If one only checkout the head with --depth=1 the jgit timestamps are wrong. We should investigate if it is possible to use the remote history in that case as it is vast-full in most cases (verification builds) to check out the whole history.

[HannesWell]

Maybe git log can be used together with the remote: https://git-scm.com/docs/git-log

Using the argument -n 1 should only return the latest commit that touches a given file/folder.

But in general I don't expect the checkout depth to have a significant influence on the overall runtime of the build. At least for the p2 build the runtime of the actions/checkout was only increased from 5 to 8sec. Of course for larger repos the difference would be greater, but larger repos probably also build longer. So I expect the fraction of the overall runtime will not change significantly.

[laeubi]

At least for the p2 build the runtime of the actions/checkout was only increased from 5 to 8sec.

I think github is really good at caching its own infra. But for example the swt binaries is 750 MB where the actual HEAD is only 9 MB ... Also its more that its hard to catch that issue, I would also be okay with failing the build as a first step if such a state is detected.

[HannesWell]

At least for the p2 build the runtime of the actions/checkout was only increased from 5 to 8sec.

I think github is really good at caching its own infra. But for example the swt binaries is 750 MB where the actual HEAD is only 9 MB ...

In such extreme cases it could be useful.

Also its more that its hard to catch that issue, I would also be okay with failing the build as a first step if such a state is detected.

That would indeed be very useful in general! I wonder how this could be detected. Does git log or however the timestamp is obtain then return a special value if the file/folder is not found in the available history?

[laeubi]

I have not checked this, but if I checkout a single commit then i get a "you are in a detached head state" message from git, don't know if depth=1 is such a detached state, I'm more a git user than understanding all these special features ;-)

[mickaelistria]

"detached HEAD" is a valid case, and almost all CI builds are in detached head. Various workflows make that even on the development workstation one can be in detached head. And Tycho can still build. Tycho is confused when history is missing. With "--depth=1", there is only 1 commit in history. I don't know whether there are some flags or other stuff that Tycho can use to detect history is missing; but for sure "detached head" isn't an indicator here.

[laeubi]

I don't know whether there are some flags or other stuff that Tycho can use to detect history is missing; but for sure "detached head" isn't an indicator here.

I don't wanted to indicate that it is exactly that, I just wanted to say that git seems to know about the current (special) state so maybe there is a similar indicator for the history depth at least git has an "unshallow" command, so there must be some kind of way to detect this:

https://stackoverflow.com/questions/58704510/how-to-get-whole-git-history-afterwards-git-clone-depth-1

here it i indicated that there is a git rev-parse --is-shallow-repository but I don't knwo if Jgit support it

https://stackoverflow.com/questions/37531605/how-to-test-if-git-repository-is-shallow

[tomaswolf]

1. With a shallow clone, the commit that last touched a particular file (or file in a directory) may not be present, so timestamps may be wrong.
2. JGit has no support yet for shallow cloning. There's a pending change that implements some part of it. Unshallow is not yet part of that change. Note that a clone may still have shallow commits even after git fetch --unshallow if the upstream repo is shallow itself.
3. If a repository has shallow commits, there is a .git/shallow file listing the commits where the history is cut off.
4. Querying the upstream log to figure out timestamps when the local clone is shallow is probably going to be slow, and may not work if the upstream repo itself is shallow.

[laeubi]

@tomaswolf thanks for the insigh, tycho itself do not clone the repo but "Github Action" do it, so if the repository itself is shallow I assume it does not make a difference if I use --depth=1 or "full" depth?

3. f a repository has shallow commits, there is a .git/shallow

So effectivly one could check if that file exits and emmit a warning? Or could the file be present even if I do git fetch --unshallow and we need to parse its contents? Doe JGit maybe already support getting this "shallow" info form a Repository?

[tomaswolf]

tycho itself do not clone the repo but "Github Action" do it

You could specify fetch-depth: 0 to make the Github checkout action do a full clone.

, so if the repository itself is shallow I assume it does not make a difference if I use --depth=1 or "full" depth?

Yes, if the upstream repo already is shallow, the clone will be shallow, too.

3. f a repository has shallow commits, there is a .git/shallow

So effectivly one could check if that file exits and emmit a warning?

I don't know what the correct way to deal with a shallow repo would be for tycho-buildtimestamp-jgit.

Or could the file be present even if I do git fetch --unshallow and we need to parse its contents?

As mentioned above: if the upstream repo already has shallow commits, the clone may have shallow commits even if a full clone was done.

Doe JGit maybe already support getting this "shallow" info form a Repository?

No; not yet.

[laeubi]

I don't know what the correct way to deal with a shallow repo would be for tycho-buildtimestamp-jgit.

My current soloution would be to do check if the repo is shallow and then print a warning about that recommending to do a "full clone", as you said if the repo is already shallow it wont help, but at least the user is aware that timestamps are wrong then.