upgraded micronaut to 4

[munishchouhan]

This PR will upgrade micronaut to ~4.4.3~ ~4.5.x~ 4.6.x

[munishchouhan]

Getting this error:

11:35:38.638 [main] ERROR io.micronaut.runtime.Micronaut - Error starting Micronaut server: Bean definition [io.seqera.wave.Boostrap] could not be loaded: Error instantiating bean of type  [io.seqera.wave.Boostrap]

Message: Cannot invoke "java.util.Map.get(Object)" because "this.registries" is null
Path Taken: new Boostrap()
io.micronaut.context.exceptions.BeanInstantiationException: Bean definition [io.seqera.wave.Boostrap] could not be loaded: Error instantiating bean of type  [io.seqera.wave.Boostrap]

Message: Cannot invoke "java.util.Map.get(Object)" because "this.registries" is null
Path Taken: new Boostrap()
	at io.micronaut.context.DefaultBeanContext.initializeContext(DefaultBeanContext.java:1979)
	at io.micronaut.context.DefaultApplicationContext.initializeContext(DefaultApplicationContext.java:290)
	at io.micronaut.context.DefaultBeanContext.readAllBeanDefinitionClasses(DefaultBeanContext.java:3337)
	at io.micronaut.context.DefaultBeanContext.finalizeConfiguration(DefaultBeanContext.java:3690)
	at io.micronaut.context.DefaultBeanContext.start(DefaultBeanContext.java:345)
	at io.micronaut.context.DefaultApplicationContext.start(DefaultApplicationContext.java:198)
	at io.micronaut.runtime.Micronaut.start(Micronaut.java:73)
	at io.seqera.wave.Application.main(Application.groovy:41)
Caused by: io.micronaut.context.exceptions.BeanInstantiationException: Error instantiating bean of type  [io.seqera.wave.Boostrap]

Message: Cannot invoke "java.util.Map.get(Object)" because "this.registries" is null
Path Taken: new Boostrap()
	at io.micronaut.context.DefaultBeanContext.resolveByBeanFactory(DefaultBeanContext.java:2322)
	at io.micronaut.context.DefaultBeanContext.doCreateBean(DefaultBeanContext.java:2277)
	at io.micronaut.context.DefaultBeanContext.doCreateBean(DefaultBeanContext.java:2289)
	at io.micronaut.context.DefaultBeanContext.createRegistration(DefaultBeanContext.java:3056)
	at io.micronaut.context.SingletonScope.getOrCreate(SingletonScope.java:81)
	at io.micronaut.context.DefaultBeanContext.findOrCreateSingletonBeanRegistration(DefaultBeanContext.java:2958)
	at io.micronaut.context.DefaultBeanContext.initializeEagerBean(DefaultBeanContext.java:2676)
	at io.micronaut.context.DefaultBeanContext.initializeContext(DefaultBeanContext.java:1973)
	... 7 common frames omitted
Caused by: java.lang.NullPointerException: Cannot invoke "java.util.Map.get(Object)" because "this.registries" is null
	at io.seqera.wave.auth.RegistryConfig.getRegistryKeys(RegistryConfig.groovy:51)
	at io.seqera.wave.auth.RegistryCredentialsProviderImpl.getDefaultCredentials0(RegistryCredentialsProviderImpl.groovy:82)
	at io.seqera.wave.auth.RegistryCredentialsProviderImpl.getDefaultCredentials(RegistryCredentialsProviderImpl.groovy:71)
	at io.seqera.wave.Boostrap.init(Boostrap.groovy:44)
	at io.seqera.wave.$Boostrap$Definition.initialize(Unknown Source)
	at io.seqera.wave.$Boostrap$Definition.instantiate(Unknown Source)
	at io.micronaut.context.DefaultBeanContext.resolveByBeanFactory(DefaultBeanContext.java:2307)
	... 14 common frames omitted

[munishchouhan]

Issue raised with Micornaut team about unit tests not working as exxpected https://github.com/micronaut-projects/micronaut-core/issues/10051

[marcodelapierre]

@munishchouhan can you review the suggestion in https://github.com/micronaut-projects/micronaut-core/issues/10051, to see if it could help for this PR? - thank you

[munishchouhan]

@marcodelapierre it is blocked by https://github.com/seqeralabs/libseqera/pull/7

[marcodelapierre]

I see, thanks Munish.

@pditommaso on Monday let's discuss prioritising the upgrade of Nextflow to Groovy 4

[munishchouhan]

getting error, while running tests

Cannot create mock for class io.seqera.wave.service.aws.AwsEcrService . Mocking of non-interface types requires a code generation library. Please put an up-to-date version of byte-buddy or cglib-nodep on the class path.
org.spockframework.mock.CannotCreateMockException: Cannot create mock for class io.seqera.wave.service.aws.AwsEcrService . Mocking of non-interface types requires a code generation library. Please put an up-to-date version of byte-buddy or cglib-nodep on the class path.
	at org.spockframework.mock.runtime.ProxyBasedMockFactory.create(ProxyBasedMockFactory.java:47)
	at org.spockframework.mock.runtime.JavaMockFactory.createInternal(JavaMockFactory.java:58)
	at org.spockframework.mock.runtime.JavaMockFactory.create(JavaMockFactory.java:38)
	at org.spockframework.mock.runtime.CompositeMockFactory.create(CompositeMockFactory.java:44)
	at org.spockframework.lang.SpecInternals.createMock(SpecInternals.java:50)
	at org.spockframework.lang.SpecInternals.createMockImpl(SpecInternals.java:305)
	at org.spockframework.lang.SpecInternals.createMockImpl(SpecInternals.java:295)
	at org.spockframework.lang.SpecInternals.MockImpl(SpecInternals.java:104)
	at io.seqera.wave.auth.RegistryCredentialsProviderTest.should get credentials from user(RegistryCredentialsProviderTest.groovy:111)

[pditommaso]

@munishchouhan can you please have a look at this conflict? ☝️

[munishchouhan]

@munishchouhan can you please have a look at this conflict? ☝️

sure

[pditommaso]

Managed to upgrade libseqera deps, but many tests still fails. Not sure that's because I've also upgraded to MN 4.4.3 from 4.1.6

[munishchouhan]

Managed to upgrade libseqera deps, but many tests still fails. Not sure that's because I've also upgraded to MN 4.4.3 from 4.1.6

ok sure, I will fix them

[munishchouhan]

Error using wave:

% wave -i ubuntu --wave-endpoint http://localhost:9090
aa0ca0c91267.ngrok.app/wt/f948f4f98162/library/ubuntu:latest

% docker pull  aa0ca0c91267.ngrok.app/wt/f948f4f98162/library/ubuntu:latest
latest: Pulling from wt/f948f4f98162/library/ubuntu
aa21f24e1940: Verifying Checksum
error pulling image configuration: image config verification failed for digest sha256:28bc33afd4c9a541270cdae3de1f78190908ffb34081e40e636fb7fec32434e3

[munishchouhan]

Some of the tests are failing because after upgrading the body content is coming in unpooledContent and body is null

I have created with the micronaut team to know about the reason

https://github.com/micronaut-projects/micronaut-core/issues/10898

[munishchouhan]

another issue which we are facing and already reported https://github.com/micronaut-projects/micronaut-core/issues/10833

[munishchouhan]

minimal reproducer added : https://github.com/munishchouhan/NullBodyProblem

[munishchouhan]

Every test is passing now, will test in local and dev

[munishchouhan]

Tested on dev, getting error while pulling the image

% docker pull wave.dev-tower.net/wt/8aa183c9107a/wave/build/dev:bwa--11cd60ae67b658b3
bwa--11cd60ae67b658b3: Pulling from wt/8aa183c9107a/wave/build/dev
6360b3717211: Pulling fs layer
2ec3f7ad9b3c: Verifying Checksum
7716ca300600: Verifying Checksum
4f4fb700ef54: Waiting
8c61d418774c: Waiting
03dae77ff45c: Waiting
aab7f787139d: Waiting
837d55536720: Waiting
897362c12ca7: Waiting
3893cbe24e91: Waiting
d1b61e94977b: Waiting
31adf7680554: Waiting
6f86207547cb: Waiting
error pulling image configuration: image config verification failed for digest sha256:1839cd7c6b4fbb11b991c4ddc24c3fcfe116faed0b5615aecd2ed053b0b6061a

[munishchouhan]

Today, I have researched on why this error is coming because I only added @body annotation in the endpoint and nothing else; checked this issue out, but no luck https://github.com/micronaut-projects/micronaut-core/issues/9694

compared logs with no error and with error fail.log pass.log

Compared request and response, but still nothing: Request with no error:

HTTP Request: Method = GET, URI = /v2/wt/9e4b50eebc23/hrma017/dev/blobs/sha256:f7cc07297e94c80d4107cf27194a335b972e34286e53624bdfc6857e2c379f07, Headers = {Host=[3653-85-136-67-71.ngrok-free.app], User-Agent=[docker/24.0.9 go/go1.20.13 git-commit/fca702d kernel/6.5.0-15-generic os/linux arch/arm64 UpstreamClient(Docker-Client/26.1.4 \(darwin\))], Accept-Encoding=[gzip], X-Forwarded-For=[85.136.67.71], X-Forwarded-Host=[3653-85-136-67-71.ngrok-free.app], X-Forwarded-Proto=[https]}, Body = null
Forwarding blobs request 'docker.io/hrma017/dev@sha256:f7cc07297e94c80d4107cf27194a335b972e34286e53624bdfc6857e2c379f07' to 'https://production.cloudflare.docker.com/registry-v2/docker/registry/v2/blobs/sha256/f7/f7cc07297e94c80d4107cf27194a335b972e34286e53624bdfc6857e2c379f07/data?PARAMS+OMITTED'

Request with error:

HTTP Request: Method = GET, URI = /v2/wt/59bc5d3e43fd/hrma017/dev/blobs/sha256:f7cc07297e94c80d4107cf27194a335b972e34286e53624bdfc6857e2c379f07, Headers = {Host=[3653-85-136-67-71.ngrok-free.app], User-Agent=[docker/24.0.9 go/go1.20.13 git-commit/fca702d kernel/6.5.0-15-generic os/linux arch/arm64 UpstreamClient(Docker-Client/26.1.4 \(darwin\))], Accept-Encoding=[gzip], X-Forwarded-For=[85.136.67.71], X-Forwarded-Host=[3653-85-136-67-71.ngrok-free.app], X-Forwarded-Proto=[https]}, Body = null
Forwarding blobs request 'docker.io/hrma017/dev@sha256:f7cc07297e94c80d4107cf27194a335b972e34286e53624bdfc6857e2c379f07' to 'https://production.cloudflare.docker.com/registry-v2/docker/registry/v2/blobs/sha256/f7/f7cc07297e94c80d4107cf27194a335b972e34286e53624bdfc6857e2c379f07/data?PARAMS+OMITTED'

[munishchouhan]

Temporary redirect: Pull with no error: HTTP Response: Status = TEMPORARY_REDIRECT, Headers = {content-type=[application/octet-stream], date=[Wed, 12 Jun 2024 20:01:53 GMT], docker-distribution-api-version=[registry/2.0], strict-transport-security=[max-age=31536000], Content-Length=[0], Location=[https://production.cloudflare.docker.com/registry-v2/docker/registry/v2/blobs/sha256/f7/f7cc07297e94c80d4107cf27194a335b972e34286e53624bdfc6857e2c379f07/data?verify=1718225513-hD7Vgx2hrOEa2mPTf7xWjgkq8KI%3D], Connection=[close]}, Body = null

pull with error: HTTP Response: Status = TEMPORARY_REDIRECT, Headers = {content-type=[application/octet-stream], date=[Wed, 12 Jun 2024 20:08:08 GMT], docker-distribution-api-version=[registry/2.0], strict-transport-security=[max-age=31536000], Connection=[close], Location=[https://production.cloudflare.docker.com/registry-v2/docker/registry/v2/blobs/sha256/f7/f7cc07297e94c80d4107cf27194a335b972e34286e53624bdfc6857e2c379f07/data?verify=1718225888-%2BZ7%2FSJDupPjSltYDkXno1s2sNYY%3D], Content-Length=[0]}, Body = null

[munishchouhan]

One difference i found that when i am trying with version 1.8, at the time of pulling there is Manifest cache hit but that is not happening when trying with this branch

[pditommaso]

it's weird because the logs shows the container digests are the same both with current version and MN 4.

Archive.zip

[pditommaso]

The problem is that the redirect response is broken.

Using MN3

HTTP/1.1 307 Temporary Redirect
Content-Length: 0
Content-Type: application/octet-stream
Date: Thu, 13 Jun 2024 16:53:19 GMT
Docker-Distribution-Api-Version: registry/2.0
Location: https://production.cloudflare.docker.com/registry-v2/docker/registry/v2/blobs/sha256/d2/d25f557d7f31bf7acfac935859b5153da41d13c41f2b468d16f729a5b883634f/data?verify=1718300599-%2FcKeJWBLD%2BMJg1L2xa6UKRzhOws%3D
Strict-Transport-Security: max-age=31536000
Connection: close

Using MN 4

HTTP/1.1 200 OK
Content-Length: 0
Date: Thu, 13 Jun 2024 16:54:38 GMT
Connection: close

[pditommaso]

This snippet is failing

https://github.com/seqeralabs/wave/blob/9975952547cc87a6bc1bb3e2a2dec9c7264042e4/src/main/groovy/io/seqera/wave/controller/RegistryProxyController.groovy#L310-L317

HttpStatus.valueOf(resp.statusCode) should be 307, instead it retuns 200. Also the location harder is not added. It looks like that changes do not take affect

[pditommaso]

Gotcha!

This is ok

    @Get("/redirect")
    HttpResponse redirect() {
        final override = Map.of(
                'Location', 'https://foo.com',  // <-- the location can be relative to the origin host, override it to always return a fully qualified URI
                'Content-Length', '0',  // <-- make sure to set content length to zero, some services return some content even with the redirect header that's discarded by this response
                'Connection', 'close' ) // <-- make sure to return connection: close header otherwise docker hangs
        return HttpResponse
                .status(HttpStatus.valueOf(307))
                .headers(toMutableHeaders([:], override))
    }

This returns 200

    @Get("/redirect")
    CompletableFuture<MutableHttpResponse<?>> redirect() {
        final override = Map.of(
                'Location', 'https://foo.com',  // <-- the location can be relative to the origin host, override it to always return a fully qualified URI
                'Content-Length', '0',  // <-- make sure to set content length to zero, some services return some content even with the redirect header that's discarded by this response
                'Connection', 'close' ) // <-- make sure to return connection: close header otherwise docker hangs
        def ret =  HttpResponse
                .status(HttpStatus.valueOf(307))
                .headers(toMutableHeaders([:], override))

        CompletableFuture.completedFuture(ret)
    }

Can you please create a test case app, so we'll report an issue with MN folks

[munishchouhan]

Can you please create a test case app, so we'll report an issue with MN folks

ok sure

[munishchouhan]

Create the test app, but both cases returning 200 https://github.com/munishchouhan/temporaryredirectproblem

[pditommaso]

even worse!

[pditommaso]

Can you add me to that repo?

[munishchouhan]

Can you add me to that repo?

@pditommaso added, redirect is working when not adding any headers

[pditommaso]

Nope, was this the problem

[munishchouhan]

Nope, was this the problem

Thank you, I will create issue in mn repo after dinner