pharo-project
Pharo CI stopped working - libgit errors
I am not exactly sure what is the source of the problem, but all Pharo CI/CD on Linux and Mac stopped working in last 24 hours. Win builds seem unaffected.
I have seen different errors. Mostly IceGenericError: error reading from the zlib stream, but few times IceGenericError: bad packet length.
MetacelloNotification: Loaded -> BaselineOfUMLProfiles-CompatibleUserName.1658909608 --- https://github.com/openponk/uml-profiles.git[master] --- https://github.com/openponk/uml-profiles.git[master]
I got an error while cloning: There was an authentication error while trying to execute the operation: .
This happens usually because you didn't provide a valid set of credentials.
You may fix this problem in different ways:
1. adding your keys to ssh-agent, executing ssh-add ~/.ssh/id_rsa in your command line.
2. adding your keys in settings (open settings browser search for "Use custom SSH keys" and
add your public and private keys).
IceGenericError: error reading from the zlib stream
IceLibgitErrorVisitor>>visitGenericError:
IceLibgitErrorVisitor>>visitERROR:
LGit_GIT_ERROR>>acceptError:
[ :error |
location exists ifTrue: [ location ensureDeleteAll ].
error acceptError: (IceLibgitErrorVisitor onContext: self) ] in IceGitClone>>execute in Block: [ :error |...
FullBlockClosure(BlockClosure)>>cull:
Context>>evaluateSignal:
Context>>handleSignal:
LGit_GIT_ERROR(Exception)>>signal
LGit_GIT_ERROR class(LGitCallReturnHandler class)>>signalWith:
LGitReturnCodeEnum>>handleLGitReturnCode
LGitRepository(LGitExternalObject)>>withReturnHandlerDo:
LGitRepository>>clone:options:to:
LGitRepository>>clone:options:
[location ensureCreateDirectory.
repo := LGitRepository on: location.
cloneOptions := repo cloneOptionsStructureClass withCredentialsProvider: (IceCredentialsProvider defaultForRemoteUrl: url).
"Keeping references, because if not the GC take them."
checkoutOptions := cloneOptions checkoutOptions.
callbacks := cloneOptions fetchOptions callbacks.
callbacks transferProgress: IceGitTransferProgress new.
checkoutOptions checkoutStrategy: LGitCheckoutStrategyEnum git_checkout_force.
checkoutOptions progressCallback: IceGitCheckoutProgress new.
repo clone: url options: cloneOptions.
(LGitRemote of: repo named: 'origin')
lookup;
setUrl: url.
] in IceGitClone>>execute in Block: [location ensureCreateDirectory....
FullBlockClosure(BlockClosure)>>on:do:
IceGitClone>>execute
IceRepositoryCreator>>cloneRepository
[
self validate.
self isCloning
ifTrue: [ self cloneRepository ]
ifFalse: [ self addLocalRepository ] ] in IceRepositoryCreator>>createRepository in Block: [...
FullBlockClosure(BlockClosure)>>on:do:
IceRepositoryCreator>>createRepository
[ repository := builder createRepository ] in MCGitHubRepository(MCGitBasedNetworkRepository)>>createIcebergRepositoryFor: in Block: [ repository := builder createRepository ]
FullBlockClosure(BlockClosure)>>on:do:
MCGitHubRepository(MCGitBasedNetworkRepository)>>createIcebergRepositoryFor:
[ ^ self createIcebergRepositoryFor: urlToUse ] in MCGitHubRepository(MCGitBasedNetworkRepository)>>createIcebergRepositoryWithFallbackFor:url: in Block: [ ^ self createIcebergRepositoryFor: urlToUse ]
FullBlockClosure(BlockClosure)>>on:do:
MCGitHubRepository(MCGitBasedNetworkRepository)>>createIcebergRepositoryWithFallbackFor:url:
[ | remote |
remote := IceGitRemote url: remoteUrl.
self createIcebergRepositoryWithFallbackFor: remote url: remoteUrl ] in MCGitHubRepository(MCGitBasedNetworkRepository)>>getOrCreateIcebergRepository in Block: [ | remote |...
OrderedCollection(Collection)>>detect:ifFound:ifNone:
OrderedCollection(Collection)>>detect:ifNone:
MCGitHubRepository(MCGitBasedNetworkRepository)>>getOrCreateIcebergRepository
3. using HTTPS instead SSH (Just use an url in the form https://etc.git)./ I will try to clone the HTTPS variant.
Error with status code 1:
653 travis_wait /home/runner/.smalltalkCI/helpers.sh
Source of the log: https://github.com/OpenPonk/class-editor/runs/7543017828?check_suite_focus=true
Probably related to https://github.com/pharo-vcs/iceberg/issues/1600 and https://github.com/hpi-swa/smalltalkCI/issues/562
I am bitten by this in an other project (pillar-markup/Microdown).
I believe it is related to https://github.com/pharo-project/pharo/issues/11222.
Is this happening for GitHub actions only? or others too?
@guillep I use only GitHub actions, unfortunately, but @badetitou said in https://github.com/hpi-swa/smalltalkCI/issues/562#issuecomment-1198182062 that GitLab works fine for him
Note that some GitHub actions work randomly (maybe if the baseline load only a few dependencies) It is the case for
- https://github.com/badetitou/Carrefour
- https://github.com/moosetechnology/Fast-Java
GitHub
Binding between Famix and FAST. Contribute to badetitou/Carrefour development by creating an account on GitHub.
Contribute to moosetechnology/FAST-JAVA development by creating an account on GitHub.
Most of my projects use Moose and all these fail consistently at the time it usually fetches the Moose, but that is probably because the Moose is large. My small projects with few simple dependencies (and no Moose) seem to mostly work, but some of them not (and these fail always the same time as well).
I get it not when building, but this piece of code in a fresh image (before I apply my github credentials)
IceGitHubAPI new in: [ :api | 50 timesRepeat: [ api get: 'repos/pillar-markup/Microdown'] ]
However, I am not sure it is related.
After credentials I can do around 5000 api calls per hour, which so far has been enough. It is in the iceberg wiki.
But how to make sure the credentials are set in the CI I have no idea. Also, I have no idea why the CI is suddenly broken.
After doing this:
#(
'https://github.com/SeasideSt/Seaside-Legacy30'
'https://github.com/pharo-project/pharo-zeroconf.git'
'https://github.com/pharo-project/pharo.git'
'https://github.com/pharo-project/pharo-launcher.git'
'https://github.com/pharo-project/pharo-vm.git'
'https://github.com/pharo-project/pharo-site.git'
'https://github.com/pharo-project/pharo-changelogs.git'
'https://github.com/pharo-project/pharo-core.git'
) do: [ :url |
Transcript show: url; cr.
[
IceRepositoryCreator new
url: url;
location: FileLocator temp / ('IceRepositoryCreator Test ' , Time microsecondClockValue asString);
createRepository
] on: Error do: [ :error |
Transcript show: error asString; cr ]
] separatedBy: [ Transcript cr ]
My Transcript shows:
https://github.com/SeasideSt/Seaside-Legacy30
IceGenericError: error reading from the zlib stream
https://github.com/pharo-project/pharo-zeroconf.git
https://github.com/pharo-project/pharo.git
IceGenericError: error reading from the zlib stream
https://github.com/pharo-project/pharo-launcher.git
https://github.com/pharo-project/pharo-vm.git
IceGenericError: bad packet length
https://github.com/pharo-project/pharo-site.git
IceGenericError: bad packet length
https://github.com/pharo-project/pharo-changelogs.git
https://github.com/pharo-project/pharo-core.git
IceGenericError: error reading from the zlib stream
So: 3 OK, 2 times ‘bad packet length’ and 3 times ‘error reading from the zlib stream’.
I try setting up the token as described in the Iceberg wiki and proposed by @kasperosterbye in https://github.com/moosetechnology/Moose/commit/30d5dc67beaa3d28103e845ebba2e29e60d93af1 It does not work
Yes I have killed this token in my personal account right after the test
GitHub action virtual environment has changed the 20220724. Maybe it created the problem for us (updating of one lib) https://github.com/actions/virtual-environments/blob/main/images/linux/Ubuntu2004-Readme.md
GitHub Actions virtual environments. Contribute to actions/virtual-environments development by creating an account on GitHub.
And also, we have this issue now
MetacelloNameNotDefinedError: project group, or package named: 'FileTree' not found when used in requires: or includes: field of package: 'Metacello-FileTree' for version: baseline of BaselineOfMetacello.
I suspect the problem has to do with the following point in the changes listed for libgit2 v1.0.1:
- A bug where the smart HTTP transport could not read large data packets has been fixed. Previously, fetching from servers like Gerrit, that sent large data packets, would error.
The following gives v1.0.0 as the version of libgit2 used in Pharo:
LGitLibrary uniqueInstance version "=> #(1 0 0)"
Using that version of libgit2, the C program given below reproduces the error:
$ ./test
libgit2 version: 1.0.0
cloning to: /tmp/libgit2_clone_test.uhfXQw
git_clone returned: -1
error message: error reading from the zlib stream
Upgrading to v1.0.1 fixes the error:
$ ./test
libgit2 version: 1.0.1
cloning to: /tmp/libgit2_clone_test.TBNpFL
git_clone returned: 0
I used MacPorts to install the different versions of libgit2, making use of the instructions on the page “How to install an older version of a port”. The ‘Portfile’ for v1.0.1 can be found in: https://github.com/macports/macports-ports/commit/abe3087564d83e7e. A diff for v1.0.0 is given below.
test.c:
#include <unistd.h>
#include <stdio.h>
#include <git2/global.h>
#include <git2/clone.h>
#include <git2/errors.h>
int main() {
int major, minor, rev;
git_libgit2_init();
if (git_libgit2_version(&major, &minor, &rev) != 0) {
printf("git_libgit2_version failed\n");
return 1;
}
printf("libgit2 version: %i.%i.%i\n", major, minor, rev);
char dir[] = "/tmp/libgit2_clone_test.XXXXXX";
if (mkdtemp(dir) == NULL) {
printf("mkdtemp failed\n");
return 1;
}
printf ("cloning to: %s\n", dir);
git_repository *repo;
int value = git_clone(&repo, "https://github.com/SeasideSt/Seaside-Legacy30", dir, NULL);
printf("git_clone returned: %i\n", value);
const git_error *error = git_error_last();
if (error != NULL)
printf("error message: %s\n", error->message);
return (value == 0) ? 0 : 1;
}
Makefile:
test: test.c
cc -I/opt/local/include -L/opt/local/lib test.c -o test -lgit2
Portfile diff:
diff --git a/devel/libgit2/Portfile b/devel/libgit2/Portfile
index 7c7d6c532e0..e8120559c40 100644
--- a/devel/libgit2/Portfile
+++ b/devel/libgit2/Portfile
@@ -9 +9 @@ PortGroup legacysupport 1.0
-github.setup libgit2 libgit2 1.0.1 v
+github.setup libgit2 libgit2 1.0.0 v
@@ -27,3 +27,3 @@ homepage https://libgit2.org/
-checksums rmd160 3f9ced0e0dff170a8156e4aa8fcb0abce66c8f60 \
- sha256 5dae7cb32b6977cd95ed849d24f3692f0b7e9eb9b0ee9ffaa14baebb9cac76e1 \
- size 5304918
+checksums rmd160 f46ca0500f159e058d854ed6aeb8d4418420b419 \
+ sha256 1d3135077f7b0401c1172e41f561cadd06fd159e75aa24d710de1bd3a24b1440 \
+ size 5302852
I can confirm that version 1.0.0 is used in the CI https://github.com/moosetechnology/Moose/runs/7626118950?check_suite_focus=true#step:4:65
MOOSE - Platform for software and data analysis. . Contribute to moosetechnology/Moose development by creating an account on GitHub.
Was not the libgit 1.0.0 there the whole time? I wonder why is it problem now. Maybe GitHub changed something about data transfers (since GitLab does not seem affected)?
The description of the change in libgit2 I referred to says “fetching from servers like Gerrit, that sent large data packets, would error.” I assume GitHub just didn’t send large data packets before but now does.
Sadly, the error seems to be a bit random. I've run again the failing builds and sometimes fail but other times don't. And for jobs having a build matrix, they failed for some configurations but not for others, even when the only difference is the packages loaded.
I was able to reproduce the problem without using GitHub actions at all, but by doing a docker build in my notebook. The first 2 times it failed with IceGenericError: error reading from the zlib stream and IceGenericError: bad packet length, but the third time worked.
As mentioned in https://github.com/pharo-project/pharo/issues/11481#issuecomment-1201163105, libgit2 needs an update in the Pharo vm... where does this need to be reported to get it done?
Here but the team is on vacation :( I will see if we can do something with christophe monday. And yes we will have to make sure that not everybody is on vacation at the same time.
It is a bit crazy that such changes are not backward compatible. :( and impacts everybody.
@Ducasse thanks for the reply Stef. I suspected that France is in holiday mode right now ;-). Just wanted to make sure it reaches the right people.
It does :) we got impacted too. I cannot build some of my projects and not even load the code :(
Is this happening for GitHub actions only? or others too?
I have some GitLab project that fails with the same error now
hello back, sadly, I have not yet a solution here... I am on holidays and my times to dedicate to this issue (or any) are (very)limited. In any case what I dug here is:
- nothing changed from our side, VMs are still the same, and code is still the same.
- hence, something had to change from the other side, which seems to be the case :)
- libgit2 library may have become incorrect for the context, and we are using for the zeroconf version a pre-built library that we put on the path (see here: https://github.com/guillep/libgit_build).
- I do not have the elements with me here to build a new one myself.
Ultimately, the right way to do this would be to not use zeroconf for the CI (since is very hard to ensure it will always work) and rely on the OBS versions (https://build.opensuse.org/package/show/devel:languages:pharo:latest/pharo9) which should provide the right working version. I understand that would be maybe too radical, but eventually that would be the path... So, I would like to suggest two tests I can't do right now to see if we can untangle this:
- remove libgit2 from the downloaded zeroconf vm as an extra step before running the tests from CI to let pharo rely on the system libgit2 (ensuring it is present).
- figure out how to use the OBS version.
Of course first one is a lot easier, but Ideally we should have a dockerized github action that encapsulates all this (version 2) and makes the running of tests better (maybe @gcotelli can help here? in Argentina is winter, no holidays there! :P)
@estebanlm, about:
- libgit2 library may have become incorrect for the context […]
See my comment from August 1st: my ‘test.c’ reproduces the error with libgit2 v1.0.0, but not with libgit2 v1.0.1. The relevant change between those versions is probably the fix for “a bug where the smart HTTP transport could not read large data packets” as stated in the changes listed for libgit2 v1.0.1.
yes, I know that... but that does not change the fact that 1.0.0 was working and now it doesn't so they changed something. and that does not changes the fact that I can't build a new zeroconf version with the right library at the moment ;)
On Aug 10 2022, at 3:58 pm, Kris @.***> wrote:
@estebanlm (https://github.com/estebanlm), about:
libgit2 library may have become incorrect for the context […] See my comment from August 1st (https://github.com/pharo-project/pharo/issues/11481#issuecomment-1201163105): my ‘test.c’ reproduces the error with libgit2 v1.0.0, but not with libgit2 v1.0.1. The relevant change between those versions is probably the fix for “a bug where the smart HTTP transport could not read large data packets” as stated in the changes listed for libgit2 v1.0.1 (https://github.com/libgit2/libgit2/releases/tag/v1.0.1).
— Reply to this email directly, view it on GitHub (https://github.com/pharo-project/pharo/issues/11481#issuecomment-1210713248), or unsubscribe (https://github.com/notifications/unsubscribe-auth/AAD5MXRZPSQGYABJFHMJ4G3VYOYP5ANCNFSM542EGMHQ). You are receiving this because you were mentioned.
@estebanlm OK no problem, I couldn’t quite tell from your comment whether you had read that so just wanted to make sure. I assume that what changed is that GitHub started sending large data packets too, and perhaps GitLab now as well (per @badetitou’s earlier comment).