async-http-client icon indicating copy to clipboard operation
async-http-client copied to clipboard

tuning for high high concurrency when most urls are invalid

Open crazysvn opened this issue 3 months ago • 3 comments

I have like 100M urls to visit, but 70% of them are invalid. how to tune it to get high speed? right now it performance at only 300k/s, although if all the urls are valid, the speed can achieve 3M/s. I want 5M/s at least(when 70% of the urls are invalid).. when I use apache httpclient4 with multi threads, the speed can be 3M/s. I expected ACH to be more effective. what should I do to improve?

public AsyncBatchHttpClient() { DefaultAsyncHttpClientConfig.Builder configBuilder = new DefaultAsyncHttpClientConfig.Builder() .setHandshakeTimeout(1000) .setConnectTimeout(Duration.ofSeconds(1)) .setConnectionTtl(Duration.ofSeconds(15)) .setRequestTimeout(Duration.ofSeconds(10)) .setReadTimeout(Duration.ofSeconds(10)) ////.setShutdownQuietPeriod(Duration.ofSeconds(15)) .setMaxConnections(100000) .setIoThreadsCount(10000) .setPooledConnectionIdleTimeout(Duration.ofSeconds(60)) .setSoLinger(0) .setTcpNoDelay(true) .setMaxConnectionsPerHost(5) .setFollowRedirect(true) .setMaxRedirects(3) .setMaxRequestRetry(1) .setKeepAlive(false) .setUseInsecureTrustManager(true) .setDisableHttpsEndpointIdentificationAlgorithm(true);

    this.httpClient = Dsl.asyncHttpClient(configBuilder);
    
    System.out.println("MaxConnections: "+httpClient.getConfig().getMaxConnections());
}

crazysvn avatar Aug 27 '25 08:08 crazysvn