Error "Request Entity Too Large"

[pditommaso]

Submitting a build request using a large Conda environment, the following error is thrown:

io.micronaut.http.client.exceptions.HttpClientResponseException: Client 'http://surrealv1:8100': Request Entity Too Large
	at io.micronaut.http.client.netty.DefaultHttpClient.handleStreamHttpError(DefaultHttpClient.java:1531)
	at io.micronaut.http.client.netty.DefaultHttpClient.lambda$streamRequestThroughChannel$47(DefaultHttpClient.java:1522)
	at reactor.core.publisher.FluxFlatMap$FlatMapMain.onNext(FluxFlatMap.java:386)
	at io.micronaut.reactive.reactor.instrument.ReactorSubscriber.onNext(ReactorSubscriber.java:57)
	at reactor.core.publisher.FluxCreate$BufferAsyncSink.drain(FluxCreate.java:814)
	at reactor.core.publisher.FluxCreate$BufferAsyncSink.next(FluxCreate.java:739)
	at reactor.core.publisher.FluxCreate$SerializedFluxSink.next(FluxCreate.java:161)
	at io.micronaut.http.client.netty.DefaultHttpClient.lambda$streamRequestThroughChannel0$48(DefaultHttpClient.java:1566)
	at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590)
	at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557)
	at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492)
	at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636)
	at io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625)
	at io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105)
	at io.micronaut.http.client.netty.DefaultHttpClient$StreamStreamHttpResponseHandler.buildResponse(DefaultHttpClient.java:2343)
	at io.micronaut.http.client.netty.DefaultHttpClient$StreamStreamHttpResponseHandler.buildResponse(DefaultHttpClient.java:2331)
	at io.micronaut.http.client.netty.DefaultHttpClient$BaseHttpResponseHandler.channelReadInstrumented(DefaultHttpClient.java:2092)
	at io.micronaut.http.client.netty.DefaultHttpClient$BaseHttpResponseHandler.channelReadInstrumented(DefaultHttpClient.java:2010)
	at io.micronaut.http.client.netty.SimpleChannelInboundHandlerInstrumented.channelRead0(SimpleChannelInboundHandlerInstrumented.java:49)
	at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:102)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at io.micronaut.http.netty.reactive.HandlerPublisher.channelRead(HandlerPublisher.java:325)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at io.micronaut.http.netty.stream.HttpStreamsHandler.channelRead(HttpStreamsHandler.java:280)
	at io.micronaut.http.netty.stream.HttpStreamsClientHandler.channelRead(HttpStreamsClientHandler.java:180)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:333)
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:454)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
	at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:840)

The build raising this error is the following https://wave.seqera.io/view/builds/90d5550d6507c09c_1

[munishchouhan]

This build id is no longer available.

[munishchouhan]

I did ask the question in surrealdb discord channel, but no luck yet. I will dig more into it.

[munishchouhan]

I think it can be solved by increasing the size of micronaut.server.max-request-size, its default size is 10mb https://micronaut-projects.github.io/micronaut-core/latest/guide/configurationreference.html#io.micronaut.http.server.HttpServerConfiguration

[munishchouhan]

I will reproduce the issue and test it

[pditommaso]

Surely it's not a Micronaut *server* setting. It may be the client one

[munishchouhan]

I tried the same conda file to reproduce this error but there was no error in dev, it maybe because of the surreal updated version I will try to reproduce it with the old one

[munishchouhan]

finally able to reproduce the error, will work on fix now

[munishchouhan]

opened issue with surreal team https://github.com/surrealdb/surrealdb/issues/4452

[pditommaso]

Have you tried the setting micronaut.http.client.max-content-length ?

https://micronaut-projects.github.io/micronaut-docs-mn2/2.0.3/guide/configurationreference.html

[munishchouhan]

Have you tried the setting micronaut.http.client.max-content-length ?

https://micronaut-projects.github.io/micronaut-docs-mn2/2.0.3/guide/configurationreference.html

Not yet, thank you i will try now

[munishchouhan]

Have you tried the setting micronaut.http.client.max-content-length ?

https://micronaut-projects.github.io/micronaut-docs-mn2/2.0.3/guide/configurationreference.html

still same error

[munishchouhan]

I have tried the following options, but no luck:

1. micronaut.server.max-request-size
2. micronaut.http.services.*.max-content-length
3. micronaut.server.netty.max-initial-line-length
4. micronaut.http.client.max-content-length

[munishchouhan]

We have hit issue one year back also https://github.com/seqeralabs/wave/pull/253#issuecomment-1632462806 it is indeed surrealdb limitation we changed our strategy that time to send cves in different calls. they did advice us to use java drivers, but there was a compatibility issue. drivers works with Java 17. They did mentioned to change the version to 8. But that issue is still open https://github.com/surrealdb/surrealdb.java/issues/58

[munishchouhan]

we are using java 17 in micronaut 4 so there we can switch to surreal drivers

[pditommaso]

Which is the current max allowed length? is the limitation coming from the protocol? http vs websocket?

[munishchouhan]

current limit is 16kb https://github.com/surrealdb/surrealdb/issues/1545#issuecomment-1364651802 it can be overcome using websocket https://github.com/surrealdb/surrealdb/issues/1545#issuecomment-1368805215

[pditommaso]

Then let's add a truncation of content longer > 16k here for now

https://github.com/seqeralabs/wave/blob/51703862fac7def3e52dab3365760873b2151806/src/main/groovy/io/seqera/wave/controller/ContainerController.groovy#L307-L309

[munishchouhan]

Then let's add a truncation of content longer > 16k here for now

https://github.com/seqeralabs/wave/blob/51703862fac7def3e52dab3365760873b2151806/src/main/groovy/io/seqera/wave/controller/ContainerController.groovy#L307-L309

We only need to truncate them when saving, so it is better to truncate here https://github.com/seqeralabs/wave/blob/760f2173989208831e18ef9ea29a06ecb8868bd7/src/main/groovy/io/seqera/wave/service/persistence/impl/SurrealPersistenceService.groovy#L150

[pditommaso]

You are right

[pditommaso]

It would be better to solve this using Surreal SQL statement, instead of truncating the Conda env.

[munishchouhan]

It would be better to solve this using Surreal SQL statement, instead of truncating the Conda env.

ok sure, i will work on it

[munishchouhan]

Tried different ways to add it using SQL, the issue is paring of containerfile

{
    "code": 400,
    "details": "Request problems detected",
    "description": "There is a problem with your request. Refer to the documentation for further information.",
    "information": "There was a problem with the database: Parse error: Failed to parse query at line 6 column 29 expected an identifier\n  |\n6 | containerFile: \"FROM mambaorg/micromamba:1.5.8-lunar\n  |                ^ \n"
}

I changed it to the bytes which works, but then it will break the decoding when loading the objects from surrealdb

I will dig further to find better solutions