apache
[MNG-8145] GOAWAY received
Filipe Roque opened MNG-8145 and commented
When using a repository mirror with an nginx reverse proxy, HTTP/2 connection may fail due to reaching keepalive_requests:
[ERROR] Failed to execute goal org.apache.felix:maven-bundle-plugin:5.1.9:manifest (bundle-manifest) on project commons-io: Execution bundle-manifest of goal org.apache.felix:maven-bundle-plugin:5.1.9:manifest failed: Plugin org.apache.felix:maven-bundle-plugin:5.1.9 or one of its dependencies could not be resolved: Failed to collect dependencies at org.apache.felix:maven-bundle-plugin:jar:5.1.9 -> org.apache.maven:maven-archiver:jar:3.5.2: Failed to read artifact descriptor for org.slf4j:slf4j-api:jar:1.7.25: The following artifacts could not be resolved: org.slf4j:slf4j-api:pom:1.7.25 (absent): Could not transfer artifact org.slf4j:slf4j-api:pom:1.7.25 from/to maven-localrepository1 (https://localhost:8443/central): /127.0.0.1:42086: GOAWAY received -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the '-e' switch
[ERROR] Re-run Maven using the '-X' switch to enable verbose output
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginResolutionException
This can be reproduced with:
git clone --quiet --branch rel/commons-io-2.16.1 [email protected]:apache/commons-io.git && cd commons-io
1. builds OK without mirror and empty local repository
TEMP_M2=$(mktemp -d)
echo $TEMP_M2
cat >$TEMP_M2/empty_settings.xml <<EOF
<?xml version="1.0" encoding="UTF-8"?>
<settings xmlns="http://maven.apache.org/SETTINGS/1.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0 http://maven.apache.org/xsd/settings-1.0.0.xsd">
</settings>
EOF
/opt/maven/apache-maven-4.0.0-beta-3/bin/mvn -s $TEMP_M2/empty_settings.xml -Dmaven.repo.local=$TEMP_M2 -q clean verify -DskipTests
1. prepare a nginx mirror
mkdir -p /tmp/docker-ssl-proxy/
openssl req -subj '/CN=localhost' -x509 -newkey rsa:4096 -nodes -keyout /tmp/docker-ssl-proxy/key.pem -out /tmp/docker-ssl-proxy/cert.pem -days 365
cat >/tmp/docker-ssl-proxy/default.conf <<EOF
server {
listen 443 ssl;
http2 on;
1. lower default value of 1000 for easier demonstration
keepalive_requests 500;
ssl_certificate /etc/nginx/conf.d/cert.pem;
ssl_certificate_key /etc/nginx/conf.d/key.pem;
location /central {
proxy_pass https://repo.maven.apache.org/maven2;
}
}
EOF
docker run -d --rm \
--name my-custom-nginx-container \
-p 8443:443 \
-v /tmp/docker-ssl-proxy/:/etc/nginx/conf.d/ \
nginx:1.27.0
cp /etc/ssl/certs/adoptium/cacerts /tmp/docker-ssl-proxy/mykeystore.jks
keytool \
-keystore /tmp/docker-ssl-proxy/mykeystore.jks \
-storepass changeit \
-importcert \
-alias SITENAME \
-trustcacerts \
-file /tmp/docker-ssl-proxy/cert.pem
TEMP_M2=$(mktemp -d)
echo $TEMP_M2
cat >$TEMP_M2/mirror_settings.xml <<EOF
<?xml version="1.0" encoding="UTF-8"?>
<settings xmlns="http://maven.apache.org/SETTINGS/1.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0 http://maven.apache.org/xsd/settings-1.0.0.xsd">
<mirrors>
<mirror>
<id>maven-localrepository1</id>
<mirrorOf>central</mirrorOf>
<url>https://localhost:8443/central</url>
</mirror>
</mirrors>
</settings>
EOF
export MAVEN_OPTS='-Djavax.net.ssl.trustStore=/tmp/docker-ssl-proxy/mykeystore.jks -Djavax.net.ssl.trustStorePassword=changeit'
/opt/maven/apache-maven-4.0.0-beta-3/bin/mvn -s $TEMP_M2/mirror_settings.xml -Dmaven.repo.local=$TEMP_M2 -q clean verify -DskipTests
Does not happen with, which only use HTTP/1.1
/opt/maven/apache-maven-3.9.7/bin/mvn -s $TEMP_M2/mirror_settings.xml -Dmaven.repo.local=$TEMP_M2 -q clean verify -DskipTests
/opt/maven/apache-maven-4.0.0-beta-3/bin/mvn -s $TEMP_M2/mirror_settings.xml -Dmaven.repo.local=$TEMP_M2 -Dmaven.resolver.transport=apache -q clean verify -DskipTests
/opt/maven/apache-maven-4.0.0-beta-3/bin/mvn -s $TEMP_M2/mirror_settings.xml -Dmaven.repo.local=$TEMP_M2 -Dmaven.resolver.transport=wagon -q clean verify -DskipTests
/opt/maven/apache-maven-4.0.0-beta-3/bin/mvn -s $TEMP_M2/mirror_settings.xml -Dmaven.repo.local=$TEMP_M2 -Dmaven.resolver.transport=jdk -Daether.transport.jdk.httpVersion=HTTP_1_1 -q clean verify -DskipTests
/opt/maven/apache-maven-4.0.0-beta-3/bin/mvn -s $TEMP_M2/mirror_settings.xml -Dmaven.repo.local=$TEMP_M2 -Dmaven.resolver.transport=jdk -Daether.transport.jdk.httpVersion.maven-localrepository1=HTTP_1_1 -q clean verify -DskipTests
Affects: 4.0.0-beta-3
Issue Links:
- MRESOLVER-584 Jetty / JDK transport: HTTP2 GOAWAY improvement ("is duplicated by")
- MNG-8194 Resolver 2.0.1
1 votes, 4 watchers
Tamas Cservenak commented
Just FTR, Maven 4 uses Java HttpClient transport by default, and user can use -Daether.transport.jdk.httpVersion=HTTP_1_1 user property to force HTTP/1.1 (as by default HTTP/2 is used if remote end supports). See https://maven.apache.org/resolver-archives/resolver-2.0.0-alpha-11/configuration.html
Other options are to switch transport to "apache", or "jetty"
Michael Osipov commented
From my PoV, we are out of control on low level issues unless what Tamas Cservenak has written....
Michael Osipov commented
Tamas Cservenak. I believe we can close this one since it is not a Maven issue.
Paul Scholz commented
Note: The problem is also present in Jetty transport, see MRESOLVER-584 for more details.
Tamas Cservenak commented
Filipe Roque I just realized you DO provided reproducer env! Thanks a LOT and sorry for not spotting it sooner :(
Tamas Cservenak commented
Am moving this issue out of beta-4: MRESOLVER-584 does NOT resolve this issue, merely "improves" the UX (lesses the probability of getting GOAWAY by lowering lifespan of JDK and Jetty clients).
This is a bug in Java (https://bugs.openjdk.org/browse/JDK-8335181). Meanwhile fixed 24, and newer 17 and 21 versions.
