Vert.x stress test problem after 16xxx request sent

[winter-lau]

Questions

Vert.x stress test problem after 16xxx request sent

Version

4.5.x

Steps to reproduce

Server code: ` package demo;

import io.vertx.core.AbstractVerticle; import io.vertx.core.Vertx; import io.vertx.core.http.HttpServer; import io.vertx.core.http.HttpServerResponse;

public class SimpleHTTPServer extends AbstractVerticle {

@Override public void start() { HttpServer server = vertx.createHttpServer();

server.requestHandler(request -> { HttpServerResponse response = request.response(); response.end("Hello, World!"); });

server.listen(8080, result -> { if (result.succeeded()) { System.out.println("Server started on port 8080"); } else { System.err.println(STR."Server failed to start: {result.cause()}"); } }); }

public static void main(String[] args) { Vertx vertx = Vertx.vertx(); vertx.deployVerticle(new SimpleHTTPServer()); } } `

2. Test method:

we use ApacheBench to test, Here is the results:

1. first round:

% ab -c 100 -n 10000 http://localhost:8080/ This is ApacheBench, Version 2.3 <$Revision: 1903618 $> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient) Completed 1000 requests Completed 2000 requests Completed 3000 requests Completed 4000 requests Completed 5000 requests Completed 6000 requests Completed 7000 requests Completed 8000 requests Completed 9000 requests Completed 10000 requests Finished 10000 requests

2. second round:

% ab -c 100 -n 10000 http://localhost:8080/ This is ApacheBench, Version 2.3 <$Revision: 1903618 $> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient) Completed 1000 requests Completed 2000 requests Completed 3000 requests Completed 4000 requests Completed 5000 requests Completed 6000 requests

The test hung here.

We have test multi round, the program hung each time when 16xxx requests sent to http service

Extra

MacBook Pro with M1 Pro/16G mem/Java 21.0.2 Vert.x 4.5.x or any older 4.x version

[vietj]

what do you mean by hung ? you mean hung from the client perspective ? can you do a thread dump of the server ? did you tune your OS ?

[winter-lau]

what do you mean by hung ? you mean hung from the client perspective ? can you do a thread dump of the server ? did you tune your OS ?

the server not response, ApacheBench exit with "apr_socket_recv: Operation timed out (60)"

i don't tune OS

[vietj]

can you verify you are not running out of file descriptors or port ?

On Mon, Jul 22, 2024 at 12:23 PM Winter Lau @.***> wrote:

what do you mean by hung ? you mean hung from the client perspective ? can you do a thread dump of the server ? did you tune your OS ?

the server not response, ApacheBench exit with "apr_socket_recv: Operation timed out (60)"

i don't tune OS

— Reply to this email directly, view it on GitHub https://github.com/eclipse-vertx/vert.x/issues/5255#issuecomment-2242614547, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABXDCRYIFHF4SBFNOHBPFDZNTMSLAVCNFSM6AAAAABLH7JYSWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENBSGYYTINJUG4 . You are receiving this because you commented.Message ID: @.***>

[winter-lau]

can you verify you are not running out of file descriptors or port ? … On Mon, Jul 22, 2024 at 12:23 PM Winter Lau @.> wrote: what do you mean by hung ? you mean hung from the client perspective ? can you do a thread dump of the server ? did you tune your OS ? the server not response, ApacheBench exit with "apr_socket_recv: Operation timed out (60)" i don't tune OS — Reply to this email directly, view it on GitHub <#5255 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABXDCRYIFHF4SBFNOHBPFDZNTMSLAVCNFSM6AAAAABLH7JYSWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENBSGYYTINJUG4 . You are receiving this because you commented.Message ID: @.>

There are no error messages during the hang, and the issue automatically returns to normal after waiting a few seconds.

The validator code is very simple, you can try it and reproduce it 100%.

[vietj]

my os settings are most different than yours and it might not reproduce, that's why I need more context from you to understand what is going on before trying to reproduce

On Mon, Jul 22, 2024 at 12:41 PM Winter Lau @.***> wrote:

can you verify you are not running out of file descriptors or port ? … <#m_4294785064893347229_> On Mon, Jul 22, 2024 at 12:23 PM Winter Lau @.> wrote: what do you mean by hung ? you mean hung from the client perspective ? can you do a thread dump of the server ? did you tune your OS ? the server not response, ApacheBench exit with "apr_socket_recv: Operation timed out (60)" i don't tune OS — Reply to this email directly, view it on GitHub <#5255 (comment) https://github.com/eclipse-vertx/vert.x/issues/5255#issuecomment-2242614547>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABXDCRYIFHF4SBFNOHBPFDZNTMSLAVCNFSM6AAAAABLH7JYSWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENBSGYYTINJUG4 https://github.com/notifications/unsubscribe-auth/AABXDCRYIFHF4SBFNOHBPFDZNTMSLAVCNFSM6AAAAABLH7JYSWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENBSGYYTINJUG4 . You are receiving this because you commented.Message ID: @.>

There are no error messages during the hang, and the issue automatically returns to normal after waiting a few seconds.

The validator code is very simple, you can try it and reproduce it 100%.

— Reply to this email directly, view it on GitHub https://github.com/eclipse-vertx/vert.x/issues/5255#issuecomment-2242647171, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABXDCROBALC3LWHEQAG3VDZNTOVPAVCNFSM6AAAAABLH7JYSWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENBSGY2DOMJXGE . You are receiving this because you commented.Message ID: @.***>

[winter-lau]

What more information can I provide? I've reproduced the issue on both my and my colleague's macs, the computers are the default system and haven't been tuned with any parameters

[winter-lau]

In my windows11 laptop, the above issue doesn't happen

[surajkumar]

@winter-lau if you're running your test using the same machine then it will not work. A common mistake that new developers make is attempting to benchmark a server using a client on the same machine. Your NIC may not be able to handle it or the OS is not able to support it. The server not responding is because the server does not receive anything from the underlying system.

Additionally, no single server can handle the volumes you are trying to test. In most real world scenarios, each server usually handles around 1000 connections (or % based depending on utilization of your system). This is when you need to start scaling.

If you want accurate results, consider using seperate machines or a cloud based solution designed for testing your traffic.

I am basing this on an assumption due to the lack of information provided in your issue.