How should I use MilvusClientV2Pool when accessing multiple databases?

[pipi-olo]

I am using MilvusClientV2Pool. Since I need to access multiple databases, I create a separate ClientPool for each database, using the collection name as the key.

My problem is that when I use ClientPool, the latency becomes significantly higher compared to when I don't use it.

Here are my questions:

Q1. Am I using ClientPool incorrectly? Please let me know the proper way to access multiple Milvus databases.

Q2. Should I use client.useDatabase within a single ClientPool? When I do this, the performance becomes very slow.

Q3. Is there a way to collect metrics for MilvusClient in a Java application?

[yhmo]

MilvusClientV2Pool uses a predefined ConnectConfig to create MilvusClient objects. All MilvusClient objects are using the same ConnectConfig:

ConnectConfig connectConfig = ConnectConfig.builder()
                .uri("http://localhost:19530")
                .dbName("aaa")
                .build();
MilvusClientV2Pool pool = new MilvusClientV2Pool(poolConfig, connectConfig);

In MilvusClientV2Pool, each key represents an auto-scaled pool of MilvusClient objects. When a thread want to get a client from the pool, the pool returns an idle client to the thread. If no idle client is available, the pool will create a new MilvusClient object with the predefined ConnectConfig. Only when the thread returns the client by returnClient(), the client is marked as "idle" again.

Pool before scaled:

Pool after scaled:

[yhmo]

For each pool, the max number of MilvusClient objects are defined by the PoolConfig.maxTotalPerKey. If all the client objects are used by getClient(), and the number of MilvusClient objects has reached the maxTotalPerKey, the next call of getClient() will have to wait no more than PoolConfig.maxBlockWaitDuration. If the getClient() could not get an idle client after PoolConfig.maxBlockWaitDuration, it will throw an exception "Timeout waiting for idle object".

[yhmo]

Use these methods to observe the number of MilvusClient objects in each pool:

    public int getIdleClientNumber(String key) 
    public int getActiveClientNumber(String key)
    public int getTotalIdleClientNumber()
    public int getTotalActiveClientNumber()

An example to demo the major functions of ClientPool: https://github.com/milvus-io/milvus-sdk-java/blob/master/examples/src/main/java/io/milvus/v2/ClientPoolExample.java

[yhmo]

For multi-database usage, one ClientPool currently can only handle one database since there is only one pre-defined ConnectConfig. One pool for one database:

 ConnectConfig connectConfig_A = ConnectConfig.builder()
                .uri("http://localhost:19530")
                .dbName("aaa")
                .build();
 MilvusClientV2Pool pool_A = new MilvusClientV2Pool(poolConfig, connectConfig_A);


ConnectConfig connectConfig_B = ConnectConfig.builder()
                .uri("http://localhost:19530")
                .dbName("bbb")
                .build();
 MilvusClientV2Pool pool_B = new MilvusClientV2Pool(poolConfig, connectConfig_B);

I will consider supporting multiple databases in one ClientPool.

[yhmo]

Typically, you can use one key for all collections in one database, no need to use different keys for different collections.

MilvusClientV2Pool pool_A = new MilvusClientV2Pool(poolConfig, connectConfig);

Thread t1 = new Thread(() -> {
    MilvusClientV2 client = pool_A.getClient("for_all_collections");
    client.search(SearchReq.builder().collectionName("coll_1").build());
});
t1.start();

Thread t2 = new Thread(() -> {
    MilvusClientV2 client = pool_A.getClient("for_all_collections");
    client.search(SearchReq.builder().collectionName("coll_2").build());
});
t2.start();

[xiaofan-luan]

For each pool, the max number of MilvusClient objects are defined by the PoolConfig.maxTotalPerKey. If all the client objects are used by getClient(), and the number of MilvusClient objects has reached the maxTotalPerKey, the next call of getClient() will have to wait no more than PoolConfig.maxBlockWaitDuration. If the getClient() could not get an idle client after PoolConfig.maxBlockWaitDuration, it will throw an exception "Timeout waiting for idle object".

Why do we need to wait?

We can create a new connection, and if when we returned the client number > maxTotalPerKey we destroy the client

[yhmo]

For each pool, the max number of MilvusClient objects are defined by the PoolConfig.maxTotalPerKey. If all the client objects are used by getClient(), and the number of MilvusClient objects has reached the maxTotalPerKey, the next call of getClient() will have to wait no more than PoolConfig.maxBlockWaitDuration. If the getClient() could not get an idle client after PoolConfig.maxBlockWaitDuration, it will throw an exception "Timeout waiting for idle object".

Why do we need to wait?

We can create a new connection, and if when we returned the client number > maxTotalPerKey we destroy the client

100 threads call getClient() at the same time, there will be maxTotalPerKey + 100 clients in use

[xiaofan-luan]

what is the default value of maxTotalPerKey?

we should at least set it to 300 or more.

[pipi-olo]

Typically, you can use one key for all collections in one database, no need to use different keys for different collections.

MilvusClientV2Pool pool_A = new MilvusClientV2Pool(poolConfig, connectConfig);

Thread t1 = new Thread(() -> {
    MilvusClientV2 client = pool_A.getClient("for_all_collections");
    client.search(SearchReq.builder().collectionName("coll_1").build());
});
t1.start();

Thread t2 = new Thread(() -> {
    MilvusClientV2 client = pool_A.getClient("for_all_collections");
    client.search(SearchReq.builder().collectionName("coll_2").build());
});
t2.start();

That's right. We used milvus client pools, all using the same key. It's faster than having a different key for each collection, but it's still slow.

[pipi-olo]

what is the default value of maxTotalPerKey?

we should at least set it to 300 or more.

My goal is QPS of over 200 with a Search P99 of 100ms. In the Milvus JDK library v2.5.12, the defaults are maxIdlePerKey: 5, minIdlePerKey: 0, maxTotalPerKey: 10, and maxTotal: 50. I've set all these values to 15 with a replica count of 4. Is there a reason why maxTotal needs to be at least 300?

cf) I have removed the milvus client pool. I am using a milvus client with a replica count of 12. Through this, I have achieved the level I was aiming for.

Is the Milvus client implemented to be blocking after a search? If two search requests come in for one client, does the second search only start after the first one has completely finished?

[yhmo]

• minIdlePerKey ---------------------- the pool will reserve at least "minIdlePerKey" of idle clients for per key, even if no one calls getClient(). If you set minIdlePerKey = 1, the pool will automatically create one client for when the sub-pool of the key is created. Since the first call of getClient(key) is to create a sub-pool and ask a client, it is no difference whether set minIdlePerKey to 0 or 1. If you set minIdlePerKey=5, the first call of getClient(key) will create 5 clients and return one of them, it might take 1 second(create 5 clients, each takes 200ms to connect milvus server) before returning this client.

• maxIdlePerKey --------------------- If the number of clients of a key is smaller than this value, and someone calls getClient(), the pool will create a new client for the caller. If the number of idle clients of a key exceeds maxIdlePerKey and no one calls getClient(), the pool will destroy some clients until the number of idle clients is equal to this value, which means maxIdlePerKey clients will remain in memory, waiting for the next call of getClient().

• maxTotalPerKey -------------------- the max total number of clients per key: idle clients + in-use clients, cannot exceed this value. If the number of in-use clients is equal this value, the pool will not allow to create new client for the next call of getClient(). The next call of getClient() will wait maxBlockWaitDuration, if no client becomes idle in maxBlockWaitDuration, getClient() will throw an exception.

• maxTotal ----------------------------- the max total number of clients of all keys: all idle clients + all in-use clients

[yhmo]

[yhmo]

Search request is blocking rpc call. If you call client.search(), the search() will be blocked until it receives results from milvus server. So, the answer is yes: if two search requests come in for one client, the second search only starts after the first one has finished.

You can use multiple clients to increase QPS, the ClientPool is mainly for this purpose. Increasing the maxTotalPerKey and maxTotal values can increase QPS, since there will be more clients maintained in the pool.

If you stop search actions, some of the clients will be destroyed to reduce the number to be smaller than maxIdlePerKey. Assume you start search actions again, the pool will create more clients until the number is equal to maxTotalPerKey. Note that the maxTotal will limit the total number of clients. Let's say there are 3 keys, maxTotalPerKey=10, and you set maxTotal=20, then some of the keys will not have enough clients to hit the maxTotalPerKey. The number of each key could be 5 + 7 + 8 or 3 + 10 + 7, or even 0 + 10 + 10, etc.

@xiaofan-luan is right, set maxTotal to a large value is good practice. If you don't know how many keys will be created, it is ok to set maxTotal to 10000, just let the number of clients of each key be limited by maxTotalPerKey.

Each time a MilvusClient is created, it takes 100ms ~ 200ms to build the rpc channel between client and milvus server. So, the qps is much lower when the pool increases the number of clients from maxIdlePerKey to maxTotalPerKey. Once the number of clients reaches maxTotalPerKey, the qps is best. A larger maxIdlePerKey will get larger qps. The max qps will eventually depend on the server-side ability.

[yhmo]

@pipi-olo I just submitted a pr: https://github.com/milvus-io/milvus-sdk-java/pull/1595 In this pr, the ClientPool will support different connect configurations for different keys. And this pr increases the default maxIdlePerKey from 5 to 10, maxTotalPerKey from 10 to 30, maxTotal from 50 to 1000.

The example ClientPoolExample demonstrates the usage. It prints the number of idle clients and active(in use) clients in the console, we can see how the numbers are changed across multiple threads.

io.milvus.v2.ClientPoolExample
Database created: AA
Database created: BB
Database created: CC
Collection 'java_sdk_example_pool_v2' created in database 'AA'
Collection 'java_sdk_example_pool_v2' created in database 'BB'
Collection 'java_sdk_example_pool_v2' created in database 'CC'
======================================================================
Total 0 idle clients and 3 active clients
Key 'AA': 0 idle clients and 1 active clients
Key 'BB': 0 idle clients and 1 active clients
Key 'CC': 0 idle clients and 1 active clients
======================================================================
======================================================================
Total 0 idle clients and 4 active clients
Key 'AA': 0 idle clients and 1 active clients
Key 'BB': 0 idle clients and 1 active clients
Key 'CC': 0 idle clients and 1 active clients
======================================================================
======================================================================
Total 0 idle clients and 9 active clients
Key 'AA': 0 idle clients and 1 active clients
Key 'BB': 0 idle clients and 1 active clients
Key 'CC': 0 idle clients and 1 active clients
======================================================================
======================================================================
Total 0 idle clients and 11 active clients
Key 'AA': 0 idle clients and 1 active clients
Key 'BB': 0 idle clients and 1 active clients
Key 'CC': 0 idle clients and 1 active clients
======================================================================
======================================================================
Total 0 idle clients and 15 active clients
Key 'AA': 0 idle clients and 2 active clients
Key 'BB': 0 idle clients and 1 active clients
Key 'CC': 0 idle clients and 2 active clients
======================================================================
======================================================================
Total 0 idle clients and 17 active clients
Key 'AA': 0 idle clients and 4 active clients
Key 'BB': 0 idle clients and 4 active clients
Key 'CC': 0 idle clients and 4 active clients
======================================================================
======================================================================
Total 0 idle clients and 27 active clients
Key 'AA': 0 idle clients and 6 active clients
Key 'BB': 0 idle clients and 5 active clients
Key 'CC': 0 idle clients and 5 active clients
======================================================================
======================================================================
Total 0 idle clients and 48 active clients
Key 'AA': 0 idle clients and 6 active clients
Key 'BB': 0 idle clients and 5 active clients
Key 'CC': 0 idle clients and 5 active clients
======================================================================
======================================================================
Total 0 idle clients and 54 active clients
Key 'AA': 0 idle clients and 6 active clients
Key 'BB': 0 idle clients and 5 active clients
Key 'CC': 0 idle clients and 5 active clients
======================================================================
======================================================================
Total 0 idle clients and 59 active clients
Key 'AA': 0 idle clients and 6 active clients
Key 'BB': 0 idle clients and 5 active clients
Key 'CC': 0 idle clients and 5 active clients
======================================================================
======================================================================
Total 0 idle clients and 65 active clients
Key 'AA': 0 idle clients and 8 active clients
Key 'BB': 0 idle clients and 7 active clients
Key 'CC': 0 idle clients and 6 active clients
======================================================================
======================================================================
Total 0 idle clients and 70 active clients
Key 'AA': 0 idle clients and 23 active clients
Key 'BB': 0 idle clients and 21 active clients
Key 'CC': 0 idle clients and 21 active clients
======================================================================
======================================================================
Total 0 idle clients and 76 active clients
Key 'AA': 0 idle clients and 25 active clients
Key 'BB': 0 idle clients and 25 active clients
Key 'CC': 0 idle clients and 23 active clients
======================================================================
======================================================================
Total 0 idle clients and 83 active clients
Key 'AA': 0 idle clients and 27 active clients
Key 'BB': 0 idle clients and 27 active clients
Key 'CC': 0 idle clients and 25 active clients
======================================================================
======================================================================
Total 0 idle clients and 89 active clients
Key 'AA': 0 idle clients and 29 active clients
Key 'BB': 0 idle clients and 28 active clients
Key 'CC': 0 idle clients and 27 active clients
======================================================================
======================================================================
Total 6 idle clients and 88 active clients
Key 'AA': 2 idle clients and 28 active clients
Key 'BB': 3 idle clients and 26 active clients
Key 'CC': 1 idle clients and 28 active clients
======================================================================
======================================================================
Total 0 idle clients and 101 active clients
Key 'AA': 0 idle clients and 32 active clients
Key 'BB': 0 idle clients and 30 active clients
Key 'CC': 0 idle clients and 29 active clients
======================================================================
======================================================================
Total 0 idle clients and 106 active clients
Key 'AA': 0 idle clients and 32 active clients
Key 'BB': 0 idle clients and 31 active clients
Key 'CC': 0 idle clients and 29 active clients
======================================================================
======================================================================
Total 0 idle clients and 111 active clients
Key 'AA': 0 idle clients and 33 active clients
Key 'BB': 0 idle clients and 32 active clients
Key 'CC': 0 idle clients and 30 active clients
======================================================================
======================================================================
Total 1 idle clients and 110 active clients
Key 'AA': 1 idle clients and 35 active clients
Key 'BB': 0 idle clients and 36 active clients
Key 'CC': 0 idle clients and 34 active clients
======================================================================
======================================================================
Total 0 idle clients and 111 active clients
Key 'AA': 0 idle clients and 36 active clients
Key 'BB': 0 idle clients and 37 active clients
Key 'CC': 0 idle clients and 36 active clients
======================================================================
======================================================================
Total 0 idle clients and 120 active clients
Key 'AA': 0 idle clients and 36 active clients
Key 'BB': 0 idle clients and 37 active clients
Key 'CC': 0 idle clients and 36 active clients
======================================================================
======================================================================
Total 0 idle clients and 137 active clients
Key 'AA': 0 idle clients and 42 active clients
Key 'BB': 1 idle clients and 40 active clients
Key 'CC': 0 idle clients and 39 active clients
======================================================================
======================================================================
Total 0 idle clients and 141 active clients
Key 'AA': 0 idle clients and 42 active clients
Key 'BB': 1 idle clients and 43 active clients
Key 'CC': 1 idle clients and 41 active clients
======================================================================
======================================================================
Total 0 idle clients and 149 active clients
Key 'AA': 0 idle clients and 42 active clients
Key 'BB': 0 idle clients and 46 active clients
Key 'CC': 0 idle clients and 46 active clients
======================================================================
======================================================================
Total 0 idle clients and 155 active clients
Key 'AA': 0 idle clients and 47 active clients
Key 'BB': 0 idle clients and 48 active clients
Key 'CC': 2 idle clients and 47 active clients
======================================================================
======================================================================
Total 0 idle clients and 150 active clients
Key 'AA': 0 idle clients and 50 active clients
Key 'BB': 0 idle clients and 50 active clients
Key 'CC': 0 idle clients and 50 active clients
======================================================================
======================================================================
Total 0 idle clients and 150 active clients
Key 'AA': 0 idle clients and 50 active clients
Key 'BB': 0 idle clients and 50 active clients
Key 'CC': 0 idle clients and 50 active clients
======================================================================
======================================================================
Total 0 idle clients and 150 active clients
Key 'AA': 0 idle clients and 50 active clients
Key 'BB': 0 idle clients and 50 active clients
Key 'CC': 0 idle clients and 50 active clients
======================================================================
======================================================================
Total 0 idle clients and 150 active clients
Key 'AA': 0 idle clients and 50 active clients
Key 'BB': 0 idle clients and 50 active clients
Key 'CC': 0 idle clients and 50 active clients
======================================================================
======================================================================
Total 0 idle clients and 150 active clients
Key 'AA': 0 idle clients and 50 active clients
Key 'BB': 0 idle clients and 50 active clients
Key 'CC': 0 idle clients and 50 active clients
======================================================================
======================================================================
Total 0 idle clients and 150 active clients
Key 'AA': 0 idle clients and 50 active clients
Key 'BB': 0 idle clients and 50 active clients
Key 'CC': 0 idle clients and 50 active clients
======================================================================
======================================================================
Total 0 idle clients and 149 active clients
Key 'AA': 0 idle clients and 50 active clients
Key 'BB': 1 idle clients and 49 active clients
Key 'CC': 0 idle clients and 50 active clients
======================================================================
======================================================================
Total 2 idle clients and 148 active clients
Key 'AA': 0 idle clients and 50 active clients
Key 'BB': 2 idle clients and 48 active clients
Key 'CC': 0 idle clients and 50 active clients
======================================================================
======================================================================
Total 1 idle clients and 149 active clients
Key 'AA': 0 idle clients and 50 active clients
Key 'BB': 1 idle clients and 49 active clients
Key 'CC': 0 idle clients and 50 active clients
======================================================================
======================================================================
Total 0 idle clients and 150 active clients
Key 'AA': 0 idle clients and 50 active clients
Key 'BB': 0 idle clients and 50 active clients
Key 'CC': 0 idle clients and 50 active clients
======================================================================
======================================================================
Total 1 idle clients and 150 active clients
Key 'AA': 0 idle clients and 50 active clients
Key 'BB': 0 idle clients and 50 active clients
Key 'CC': 0 idle clients and 50 active clients
======================================================================
======================================================================
Total 0 idle clients and 149 active clients
Key 'AA': 0 idle clients and 50 active clients
Key 'BB': 0 idle clients and 50 active clients
Key 'CC': 0 idle clients and 50 active clients
======================================================================
======================================================================
Total 0 idle clients and 150 active clients
Key 'AA': 0 idle clients and 50 active clients
Key 'BB': 0 idle clients and 50 active clients
Key 'CC': 0 idle clients and 50 active clients
======================================================================
======================================================================
Total 1 idle clients and 149 active clients
Key 'AA': 0 idle clients and 50 active clients
Key 'BB': 0 idle clients and 50 active clients
Key 'CC': 1 idle clients and 49 active clients
======================================================================
======================================================================
Total 0 idle clients and 150 active clients
Key 'AA': 0 idle clients and 50 active clients
Key 'BB': 0 idle clients and 50 active clients
Key 'CC': 0 idle clients and 50 active clients
======================================================================
======================================================================
Total 1 idle clients and 149 active clients
Key 'AA': 0 idle clients and 50 active clients
Key 'BB': 0 idle clients and 50 active clients
Key 'CC': 0 idle clients and 50 active clients
======================================================================
======================================================================
Total 0 idle clients and 150 active clients
Key 'AA': 0 idle clients and 50 active clients
Key 'BB': 0 idle clients and 50 active clients
Key 'CC': 0 idle clients and 50 active clients
======================================================================
======================================================================
Total 1 idle clients and 148 active clients
Key 'AA': 0 idle clients and 50 active clients
Key 'BB': 1 idle clients and 49 active clients
Key 'CC': 0 idle clients and 50 active clients
======================================================================
======================================================================
Total 0 idle clients and 150 active clients
Key 'AA': 0 idle clients and 50 active clients
Key 'BB': 0 idle clients and 50 active clients
Key 'CC': 0 idle clients and 50 active clients
======================================================================
======================================================================
Total 0 idle clients and 150 active clients
Key 'AA': 0 idle clients and 50 active clients
Key 'BB': 0 idle clients and 50 active clients
Key 'CC': 0 idle clients and 50 active clients
======================================================================
======================================================================
Total 2 idle clients and 147 active clients
Key 'AA': 1 idle clients and 46 active clients
Key 'BB': 1 idle clients and 49 active clients
Key 'CC': 0 idle clients and 46 active clients
======================================================================
======================================================================
Total 11 idle clients and 224 active clients
Key 'AA': 0 idle clients and 33 active clients
Key 'BB': 0 idle clients and 50 active clients
Key 'CC': 0 idle clients and 35 active clients
======================================================================
======================================================================
Total 0 idle clients and 201 active clients
Key 'AA': 0 idle clients and 50 active clients
Key 'BB': 1 idle clients and 49 active clients
Key 'CC': 0 idle clients and 48 active clients
======================================================================
======================================================================
Total 0 idle clients and 150 active clients
Key 'AA': 0 idle clients and 50 active clients
Key 'BB': 0 idle clients and 50 active clients
Key 'CC': 0 idle clients and 50 active clients
======================================================================
======================================================================
Total 0 idle clients and 150 active clients
Key 'AA': 0 idle clients and 50 active clients
Key 'BB': 0 idle clients and 50 active clients
Key 'CC': 0 idle clients and 50 active clients
======================================================================
======================================================================
Total 0 idle clients and 150 active clients
Key 'AA': 0 idle clients and 50 active clients
Key 'BB': 0 idle clients and 50 active clients
Key 'CC': 0 idle clients and 50 active clients
======================================================================
======================================================================
Total 1 idle clients and 149 active clients
Key 'AA': 1 idle clients and 49 active clients
Key 'BB': 0 idle clients and 50 active clients
Key 'CC': 0 idle clients and 50 active clients
======================================================================
======================================================================
Total 0 idle clients and 150 active clients
Key 'AA': 0 idle clients and 50 active clients
Key 'BB': 0 idle clients and 50 active clients
Key 'CC': 0 idle clients and 50 active clients
======================================================================
======================================================================
Total 0 idle clients and 150 active clients
Key 'AA': 0 idle clients and 50 active clients
Key 'BB': 0 idle clients and 50 active clients
Key 'CC': 0 idle clients and 50 active clients
======================================================================
======================================================================
Total 0 idle clients and 150 active clients
Key 'AA': 0 idle clients and 50 active clients
Key 'BB': 0 idle clients and 50 active clients
Key 'CC': 0 idle clients and 50 active clients
======================================================================
======================================================================
Total 0 idle clients and 150 active clients
Key 'AA': 0 idle clients and 50 active clients
Key 'BB': 0 idle clients and 50 active clients
Key 'CC': 0 idle clients and 50 active clients
======================================================================
======================================================================
Total 1 idle clients and 149 active clients
Key 'AA': 0 idle clients and 50 active clients
Key 'BB': 1 idle clients and 49 active clients
Key 'CC': 0 idle clients and 50 active clients
======================================================================
======================================================================
Total 0 idle clients and 150 active clients
Key 'AA': 0 idle clients and 50 active clients
Key 'BB': 0 idle clients and 50 active clients
Key 'CC': 0 idle clients and 50 active clients
======================================================================
======================================================================
Total 1 idle clients and 150 active clients
Key 'AA': 0 idle clients and 50 active clients
Key 'BB': 0 idle clients and 50 active clients
Key 'CC': 0 idle clients and 50 active clients
======================================================================
======================================================================
Total 0 idle clients and 150 active clients
Key 'AA': 0 idle clients and 50 active clients
Key 'BB': 0 idle clients and 50 active clients
Key 'CC': 0 idle clients and 50 active clients
======================================================================
======================================================================
Total 0 idle clients and 150 active clients
Key 'AA': 0 idle clients and 50 active clients
Key 'BB': 0 idle clients and 50 active clients
Key 'CC': 0 idle clients and 50 active clients
======================================================================
======================================================================
Total 0 idle clients and 150 active clients
Key 'AA': 0 idle clients and 50 active clients
Key 'BB': 0 idle clients and 50 active clients
Key 'CC': 0 idle clients and 50 active clients
======================================================================
======================================================================
Total 0 idle clients and 150 active clients
Key 'AA': 0 idle clients and 50 active clients
Key 'BB': 0 idle clients and 50 active clients
Key 'CC': 0 idle clients and 50 active clients
======================================================================
======================================================================
Total 0 idle clients and 150 active clients
Key 'AA': 0 idle clients and 50 active clients
Key 'BB': 0 idle clients and 50 active clients
Key 'CC': 0 idle clients and 50 active clients
======================================================================
======================================================================
Total 0 idle clients and 150 active clients
Key 'AA': 0 idle clients and 50 active clients
Key 'BB': 0 idle clients and 50 active clients
Key 'CC': 0 idle clients and 50 active clients
======================================================================
======================================================================
Total 0 idle clients and 150 active clients
Key 'AA': 0 idle clients and 50 active clients
Key 'BB': 0 idle clients and 50 active clients
Key 'CC': 0 idle clients and 50 active clients
======================================================================
======================================================================
Total 1 idle clients and 150 active clients
Key 'AA': 0 idle clients and 50 active clients
Key 'BB': 0 idle clients and 50 active clients
Key 'CC': 0 idle clients and 50 active clients
======================================================================
======================================================================
Total 0 idle clients and 150 active clients
Key 'AA': 0 idle clients and 50 active clients
Key 'BB': 0 idle clients and 50 active clients
Key 'CC': 0 idle clients and 50 active clients
======================================================================
======================================================================
Total 0 idle clients and 150 active clients
Key 'AA': 0 idle clients and 50 active clients
Key 'BB': 0 idle clients and 50 active clients
Key 'CC': 0 idle clients and 50 active clients
======================================================================
======================================================================
Total 0 idle clients and 150 active clients
Key 'AA': 0 idle clients and 50 active clients
Key 'BB': 1 idle clients and 49 active clients
Key 'CC': 0 idle clients and 50 active clients
======================================================================
======================================================================
Total 0 idle clients and 150 active clients
Key 'AA': 0 idle clients and 50 active clients
Key 'BB': 0 idle clients and 50 active clients
Key 'CC': 0 idle clients and 50 active clients
======================================================================
======================================================================
Total 0 idle clients and 150 active clients
Key 'AA': 0 idle clients and 50 active clients
Key 'BB': 0 idle clients and 50 active clients
Key 'CC': 0 idle clients and 50 active clients
======================================================================
======================================================================
Total 1 idle clients and 150 active clients
Key 'AA': 0 idle clients and 50 active clients
Key 'BB': 0 idle clients and 50 active clients
Key 'CC': 0 idle clients and 50 active clients
======================================================================
======================================================================
Total 0 idle clients and 150 active clients
Key 'AA': 0 idle clients and 50 active clients
Key 'BB': 0 idle clients and 50 active clients
Key 'CC': 0 idle clients and 50 active clients
======================================================================
======================================================================
Total 1 idle clients and 150 active clients
Key 'AA': 0 idle clients and 50 active clients
Key 'BB': 0 idle clients and 50 active clients
Key 'CC': 0 idle clients and 50 active clients
======================================================================
======================================================================
Total 0 idle clients and 150 active clients
Key 'AA': 0 idle clients and 50 active clients
Key 'BB': 0 idle clients and 50 active clients
Key 'CC': 0 idle clients and 50 active clients
======================================================================
======================================================================
Total 0 idle clients and 150 active clients
Key 'AA': 0 idle clients and 50 active clients
Key 'BB': 0 idle clients and 50 active clients
Key 'CC': 0 idle clients and 50 active clients
======================================================================
======================================================================
Total 1 idle clients and 149 active clients
Key 'AA': 0 idle clients and 50 active clients
Key 'BB': 0 idle clients and 50 active clients
Key 'CC': 0 idle clients and 50 active clients
======================================================================
======================================================================
Total 0 idle clients and 150 active clients
Key 'AA': 0 idle clients and 50 active clients
Key 'BB': 0 idle clients and 50 active clients
Key 'CC': 0 idle clients and 50 active clients
======================================================================
======================================================================
Total 0 idle clients and 150 active clients
Key 'AA': 0 idle clients and 50 active clients
Key 'BB': 0 idle clients and 50 active clients
Key 'CC': 0 idle clients and 50 active clients
======================================================================
======================================================================
Total 0 idle clients and 150 active clients
Key 'AA': 0 idle clients and 50 active clients
Key 'BB': 0 idle clients and 50 active clients
Key 'CC': 0 idle clients and 50 active clients
======================================================================
======================================================================
Total 0 idle clients and 150 active clients
Key 'AA': 0 idle clients and 50 active clients
Key 'BB': 0 idle clients and 50 active clients
Key 'CC': 0 idle clients and 50 active clients
======================================================================
======================================================================
Total 0 idle clients and 150 active clients
Key 'AA': 0 idle clients and 50 active clients
Key 'BB': 0 idle clients and 50 active clients
Key 'CC': 0 idle clients and 50 active clients
======================================================================
======================================================================
Total 0 idle clients and 150 active clients
Key 'AA': 0 idle clients and 50 active clients
Key 'BB': 0 idle clients and 50 active clients
Key 'CC': 0 idle clients and 50 active clients
======================================================================
======================================================================
Total 0 idle clients and 150 active clients
Key 'AA': 0 idle clients and 50 active clients
Key 'BB': 0 idle clients and 50 active clients
Key 'CC': 0 idle clients and 50 active clients
======================================================================
======================================================================
Total 0 idle clients and 150 active clients
Key 'AA': 0 idle clients and 50 active clients
Key 'BB': 0 idle clients and 50 active clients
Key 'CC': 0 idle clients and 50 active clients
======================================================================
======================================================================
Total 0 idle clients and 150 active clients
Key 'AA': 0 idle clients and 50 active clients
Key 'BB': 0 idle clients and 50 active clients
Key 'CC': 0 idle clients and 50 active clients
======================================================================
======================================================================
Total 1 idle clients and 149 active clients
Key 'AA': 0 idle clients and 50 active clients
Key 'BB': 1 idle clients and 49 active clients
Key 'CC': 0 idle clients and 50 active clients
======================================================================
======================================================================
Total 0 idle clients and 150 active clients
Key 'AA': 0 idle clients and 50 active clients
Key 'BB': 0 idle clients and 50 active clients
Key 'CC': 0 idle clients and 50 active clients
======================================================================
======================================================================
Total 0 idle clients and 150 active clients
Key 'AA': 0 idle clients and 50 active clients
Key 'BB': 0 idle clients and 50 active clients
Key 'CC': 0 idle clients and 50 active clients
======================================================================
======================================================================
Total 3 idle clients and 147 active clients
Key 'AA': 0 idle clients and 50 active clients
Key 'BB': 3 idle clients and 47 active clients
Key 'CC': 0 idle clients and 50 active clients
======================================================================
======================================================================
Total 0 idle clients and 150 active clients
Key 'AA': 0 idle clients and 50 active clients
Key 'BB': 0 idle clients and 50 active clients
Key 'CC': 0 idle clients and 50 active clients
======================================================================
======================================================================
Total 0 idle clients and 150 active clients
Key 'AA': 0 idle clients and 50 active clients
Key 'BB': 0 idle clients and 50 active clients
Key 'CC': 0 idle clients and 50 active clients
======================================================================
======================================================================
Total 1 idle clients and 150 active clients
Key 'AA': 0 idle clients and 50 active clients
Key 'BB': 0 idle clients and 50 active clients
Key 'CC': 0 idle clients and 50 active clients
======================================================================
======================================================================
Total 0 idle clients and 150 active clients
Key 'AA': 0 idle clients and 50 active clients
Key 'BB': 0 idle clients and 50 active clients
Key 'CC': 0 idle clients and 50 active clients
======================================================================
======================================================================
Total 0 idle clients and 150 active clients
Key 'AA': 0 idle clients and 50 active clients
Key 'BB': 0 idle clients and 50 active clients
Key 'CC': 0 idle clients and 50 active clients
======================================================================
======================================================================
Total 0 idle clients and 150 active clients
Key 'AA': 0 idle clients and 50 active clients
Key 'BB': 0 idle clients and 50 active clients
Key 'CC': 0 idle clients and 50 active clients
======================================================================
======================================================================
Total 0 idle clients and 150 active clients
Key 'AA': 0 idle clients and 50 active clients
Key 'BB': 0 idle clients and 50 active clients
Key 'CC': 0 idle clients and 50 active clients
======================================================================
======================================================================
Total 1 idle clients and 150 active clients
Key 'AA': 0 idle clients and 50 active clients
Key 'BB': 0 idle clients and 50 active clients
Key 'CC': 0 idle clients and 50 active clients
======================================================================
Search thread Thread-129 finished
Key 'BB': 0 idle clients and 50 active clients
Search thread Thread-87 finished
Key 'BB': 0 idle clients and 50 active clients
Search thread Thread-99 finished
Key 'BB': 1 idle clients and 49 active clients
Search thread Thread-51 finished
Key 'BB': 0 idle clients and 50 active clients
Search thread Thread-39 finished
Key 'BB': 0 idle clients and 50 active clients
Search thread Thread-15 finished
Key 'BB': 1 idle clients and 49 active clients
Search thread Thread-37 finished
Key 'AA': 1 idle clients and 50 active clients
Search thread Thread-115 finished
Key 'AA': 1 idle clients and 49 active clients
Search thread Thread-7 finished
Key 'AA': 1 idle clients and 49 active clients
Search thread Thread-93 finished
Key 'BB': 1 idle clients and 49 active clients
Search thread Thread-57 finished
Key 'BB': 0 idle clients and 50 active clients
Search thread Thread-13 finished
Key 'AA': 1 idle clients and 49 active clients
Search thread Thread-9 finished
Key 'BB': 1 idle clients and 49 active clients
Search thread Thread-81 finished
Key 'BB': 0 idle clients and 50 active clients
Search thread Thread-75 finished
Key 'BB': 1 idle clients and 49 active clients
Search thread Thread-89 finished
Key 'CC': 0 idle clients and 50 active clients
Search thread Thread-19 finished
Key 'AA': 1 idle clients and 49 active clients
Search thread Thread-159 finished
Key 'BB': 0 idle clients and 50 active clients
Search thread Thread-227 finished
Key 'CC': 0 idle clients and 50 active clients
Search thread Thread-193 finished
Key 'AA': 1 idle clients and 49 active clients
Search thread Thread-31 finished
Key 'AA': 0 idle clients and 50 active clients
Search thread Thread-199 finished
Key 'AA': 0 idle clients and 50 active clients
Search thread Thread-63 finished
Key 'BB': 2 idle clients and 49 active clients
Search thread Thread-117 finished
Key 'BB': 1 idle clients and 49 active clients
Search thread Thread-165 finished
Key 'BB': 1 idle clients and 49 active clients
Search thread Thread-175 finished
Key 'AA': 1 idle clients and 49 active clients
Search thread Thread-173 finished
Key 'CC': 1 idle clients and 49 active clients
Search thread Thread-85 finished
Key 'AA': 1 idle clients and 49 active clients
Search thread Thread-69 finished
Key 'BB': 2 idle clients and 48 active clients
Search thread Thread-21 finished
Key 'BB': 1 idle clients and 49 active clients
Search thread Thread-43 finished
Key 'AA': 1 idle clients and 49 active clients
Search thread Thread-121 finished
Key 'AA': 1 idle clients and 49 active clients
Search thread Thread-65 finished
Key 'CC': 0 idle clients and 50 active clients
Search thread Thread-55 finished
Key 'AA': 0 idle clients and 50 active clients
Search thread Thread-107 finished
Key 'CC': 1 idle clients and 49 active clients
Search thread Thread-271 finished
Key 'AA': 2 idle clients and 48 active clients
Search thread Thread-145 finished
Key 'AA': 1 idle clients and 49 active clients
Search thread Thread-123 finished
Key 'BB': 1 idle clients and 49 active clients
Search thread Thread-185 finished
Key 'CC': 0 idle clients and 50 active clients
Search thread Thread-33 finished
Key 'BB': 1 idle clients and 49 active clients
Search thread Thread-97 finished
Key 'AA': 0 idle clients and 50 active clients
Search thread Thread-231 finished
Key 'BB': 1 idle clients and 49 active clients
Search thread Thread-207 finished
Key 'BB': 1 idle clients and 49 active clients
Search thread Thread-113 finished
Key 'CC': 1 idle clients and 49 active clients
Search thread Thread-47 finished
Key 'CC': 1 idle clients and 49 active clients
Search thread Thread-211 finished
Key 'AA': 1 idle clients and 49 active clients
Search thread Thread-73 finished
Key 'AA': 0 idle clients and 50 active clients
Search thread Thread-95 finished
Key 'CC': 1 idle clients and 49 active clients
Search thread Thread-365 finished
Key 'CC': 1 idle clients and 49 active clients
Search thread Thread-1 finished
Key 'AA': 1 idle clients and 49 active clients
Search thread Thread-105 finished
Key 'BB': 1 idle clients and 49 active clients
Search thread Thread-71 finished
Key 'CC': 0 idle clients and 50 active clients
Search thread Thread-181 finished
Key 'AA': 0 idle clients and 50 active clients
Search thread Thread-127 finished
Key 'AA': 0 idle clients and 50 active clients
Search thread Thread-45 finished
Key 'BB': 0 idle clients and 50 active clients
Search thread Thread-187 finished
Key 'AA': 1 idle clients and 49 active clients
Search thread Thread-137 finished
Key 'CC': 1 idle clients and 49 active clients
Search thread Thread-109 finished
Key 'AA': 0 idle clients and 50 active clients
Search thread Thread-169 finished
Key 'AA': 1 idle clients and 49 active clients
Search thread Thread-91 finished
Key 'AA': 1 idle clients and 49 active clients
Search thread Thread-139 finished
Key 'AA': 1 idle clients and 49 active clients
Search thread Thread-241 finished
Key 'AA': 0 idle clients and 50 active clients
Search thread Thread-131 finished
Key 'CC': 0 idle clients and 50 active clients
Search thread Thread-3 finished
Key 'BB': 1 idle clients and 49 active clients
Search thread Thread-143 finished
Key 'CC': 1 idle clients and 49 active clients
Search thread Thread-149 finished
Key 'CC': 0 idle clients and 50 active clients
Search thread Thread-61 finished
Key 'AA': 1 idle clients and 49 active clients
Search thread Thread-25 finished
Key 'AA': 1 idle clients and 49 active clients
Search thread Thread-225 finished
Key 'BB': 1 idle clients and 49 active clients
Search thread Thread-477 finished
Key 'BB': 1 idle clients and 49 active clients
Search thread Thread-283 finished
Key 'AA': 0 idle clients and 50 active clients
Search thread Thread-27 finished
Key 'BB': 1 idle clients and 49 active clients
Insert thread Thread-50 finished
Key 'BB': 1 idle clients and 49 active clients
Search thread Thread-157 finished
Key 'AA': 1 idle clients and 49 active clients
Search thread Thread-151 finished
Key 'AA': 1 idle clients and 49 active clients
Search thread Thread-221 finished
Key 'CC': 1 idle clients and 49 active clients
Search thread Thread-189 finished
Key 'BB': 1 idle clients and 49 active clients
Search thread Thread-79 finished
Key 'AA': 1 idle clients and 49 active clients
Search thread Thread-153 finished
Key 'BB': 1 idle clients and 49 active clients
Search thread Thread-161 finished
Key 'CC': 1 idle clients and 49 active clients
Search thread Thread-473 finished
Key 'CC': 0 idle clients and 50 active clients
Search thread Thread-133 finished
Key 'AA': 1 idle clients and 49 active clients
Search thread Thread-111 finished
Key 'BB': 0 idle clients and 50 active clients
Search thread Thread-101 finished
Key 'CC': 0 idle clients and 50 active clients
Search thread Thread-201 finished
Key 'BB': 1 idle clients and 49 active clients
Search thread Thread-119 finished
Key 'CC': 1 idle clients and 49 active clients
Search thread Thread-499 finished
Key 'AA': 1 idle clients and 49 active clients
Search thread Thread-419 finished
Key 'CC': 0 idle clients and 50 active clients
Insert thread Thread-26 finished
Key 'BB': 2 idle clients and 48 active clients
Search thread Thread-205 finished
Key 'AA': 0 idle clients and 50 active clients
Search thread Thread-351 finished
Key 'BB': 2 idle clients and 48 active clients
Search thread Thread-135 finished
Key 'BB': 1 idle clients and 49 active clients
Search thread Thread-167 finished
Key 'CC': 1 idle clients and 49 active clients
Search thread Thread-237 finished
Key 'BB': 2 idle clients and 48 active clients
Search thread Thread-275 finished
Key 'CC': 0 idle clients and 50 active clients
Search thread Thread-303 finished
Key 'BB': 0 idle clients and 50 active clients
Search thread Thread-171 finished
Key 'BB': 1 idle clients and 49 active clients
Search thread Thread-437 finished
Key 'CC': 1 idle clients and 49 active clients
Search thread Thread-141 finished
Key 'BB': 1 idle clients and 49 active clients
Search thread Thread-337 finished
Key 'AA': 1 idle clients and 49 active clients
Search thread Thread-233 finished
Key 'CC': 1 idle clients and 49 active clients
Search thread Thread-229 finished
Key 'AA': 0 idle clients and 50 active clients
Search thread Thread-269 finished
Key 'CC': 2 idle clients and 48 active clients
Insert thread Thread-42 finished
Key 'AA': 1 idle clients and 49 active clients
Search thread Thread-335 finished
Key 'CC': 1 idle clients and 49 active clients
Search thread Thread-541 finished
Key 'AA': 1 idle clients and 49 active clients
Search thread Thread-301 finished
Key 'AA': 1 idle clients and 49 active clients
Insert thread Thread-292 finished
Key 'CC': 1 idle clients and 49 active clients
Search thread Thread-343 finished
Key 'AA': 1 idle clients and 49 active clients
Insert thread Thread-68 finished
Key 'BB': 1 idle clients and 49 active clients
Insert thread Thread-134 finished
Key 'BB': 0 idle clients and 50 active clients
Insert thread Thread-174 finished
Key 'AA': 1 idle clients and 49 active clients
Search thread Thread-103 finished
Key 'AA': 0 idle clients and 50 active clients
Insert thread Thread-20 finished
Key 'BB': 0 idle clients and 50 active clients
Search thread Thread-431 finished
Key 'CC': 0 idle clients and 50 active clients
Search thread Thread-125 finished
Key 'CC': 0 idle clients and 50 active clients
Search thread Thread-333 finished
Key 'BB': 1 idle clients and 49 active clients
Search thread Thread-155 finished
Key 'CC': 0 idle clients and 50 active clients
Search thread Thread-179 finished
Key 'CC': 1 idle clients and 49 active clients
Search thread Thread-77 finished
Key 'CC': 1 idle clients and 49 active clients
Search thread Thread-287 finished
Key 'CC': 1 idle clients and 49 active clients
Insert thread Thread-100 finished
Key 'CC': 2 idle clients and 48 active clients
Search thread Thread-49 finished
Key 'AA': 0 idle clients and 50 active clients
Search thread Thread-191 finished
Key 'CC': 1 idle clients and 49 active clients
Search thread Thread-147 finished
Key 'BB': 1 idle clients and 49 active clients
Search thread Thread-399 finished
Key 'BB': 0 idle clients and 50 active clients
Search thread Thread-23 finished
Key 'CC': 1 idle clients and 49 active clients
Search thread Thread-363 finished
Key 'BB': 1 idle clients and 49 active clients
Search thread Thread-251 finished
Key 'CC': 0 idle clients and 50 active clients
Search thread Thread-11 finished
Key 'CC': 0 idle clients and 50 active clients
Search thread Thread-249 finished
Key 'BB': 1 idle clients and 49 active clients
Search thread Thread-293 finished
Key 'CC': 1 idle clients and 49 active clients
Insert thread Thread-156 finished
Key 'AA': 1 idle clients and 49 active clients
Insert thread Thread-16 finished
Key 'CC': 2 idle clients and 48 active clients
Search thread Thread-83 finished
Key 'CC': 0 idle clients and 50 active clients
Search thread Thread-163 finished
Key 'AA': 1 idle clients and 49 active clients
Insert thread Thread-206 finished
Key 'BB': 0 idle clients and 50 active clients
Insert thread Thread-474 finished
Key 'AA': 1 idle clients and 49 active clients
Insert thread Thread-44 finished
Key 'BB': 1 idle clients and 49 active clients
Insert thread Thread-74 finished
Key 'BB': 1 idle clients and 49 active clients
Search thread Thread-255 finished
Key 'BB': 1 idle clients and 49 active clients
Search thread Thread-297 finished
Key 'BB': 0 idle clients and 50 active clients
Insert thread Thread-396 finished
Key 'AA': 1 idle clients and 49 active clients
Search thread Thread-277 finished
Key 'AA': 2 idle clients and 48 active clients
Search thread Thread-217 finished
Key 'AA': 0 idle clients and 50 active clients
Search thread Thread-281 finished
Key 'CC': 1 idle clients and 49 active clients
Search thread Thread-353 finished
Key 'CC': 1 idle clients and 49 active clients
Search thread Thread-481 finished
Key 'AA': 1 idle clients and 49 active clients
Search thread Thread-239 finished
Key 'CC': 1 idle clients and 49 active clients
Search thread Thread-453 finished
Key 'BB': 1 idle clients and 49 active clients
Search thread Thread-263 finished
Key 'CC': 1 idle clients and 49 active clients
Insert thread Thread-10 finished
Key 'CC': 1 idle clients and 49 active clients
Insert thread Thread-60 finished
Search thread Thread-305 finished
Key 'CC': 1 idle clients and 48 active clients
Key 'AA': 0 idle clients and 50 active clients
Insert thread Thread-106 finished
Key 'CC': 0 idle clients and 50 active clients
Insert thread Thread-358 finished
Key 'CC': 1 idle clients and 49 active clients
Search thread Thread-291 finished
Key 'BB': 0 idle clients and 50 active clients
Search thread Thread-267 finished
Key 'BB': 0 idle clients and 50 active clients
Search thread Thread-315 finished
Key 'BB': 1 idle clients and 49 active clients
Insert thread Thread-64 finished
Key 'CC': 1 idle clients and 49 active clients
Search thread Thread-247 finished
Key 'AA': 1 idle clients and 49 active clients
Search thread Thread-5 finished
Key 'CC': 0 idle clients and 50 active clients
Search thread Thread-577 finished
Key 'AA': 0 idle clients and 50 active clients
Insert thread Thread-186 finished
Key 'AA': 0 idle clients and 50 active clients
Insert thread Thread-270 finished
Key 'AA': 1 idle clients and 49 active clients
Insert thread Thread-12 finished
Key 'AA': 1 idle clients and 50 active clients
Insert thread Thread-54 finished
Key 'AA': 1 idle clients and 49 active clients
Insert thread Thread-332 finished
Key 'BB': 1 idle clients and 49 active clients
Search thread Thread-209 finished
Key 'CC': 1 idle clients and 49 active clients
Search thread Thread-355 finished
Key 'AA': 1 idle clients and 49 active clients
Search thread Thread-519 finished
Key 'BB': 0 idle clients and 50 active clients
Insert thread Thread-260 finished
Key 'BB': 1 idle clients and 49 active clients
Search thread Thread-483 finished
Key 'BB': 1 idle clients and 49 active clients
Insert thread Thread-28 finished
Key 'CC': 0 idle clients and 50 active clients
Insert thread Thread-314 finished
Key 'BB': 0 idle clients and 50 active clients
Insert thread Thread-132 finished
Key 'AA': 0 idle clients and 50 active clients
Search thread Thread-273 finished
Key 'BB': 1 idle clients and 49 active clients
Search thread Thread-379 finished
Key 'AA': 0 idle clients and 50 active clients
Insert thread Thread-72 finished
Key 'AA': 3 idle clients and 47 active clients
Insert thread Thread-300 finished
Key 'AA': 0 idle clients and 50 active clients
Insert thread Thread-36 finished
Key 'AA': 1 idle clients and 49 active clients
Insert thread Thread-558 finished
Key 'AA': 1 idle clients and 49 active clients
Search thread Thread-417 finished
Key 'BB': 1 idle clients and 49 active clients
Insert thread Thread-462 finished
Key 'AA': 1 idle clients and 49 active clients
Search thread Thread-515 finished
Key 'CC': 1 idle clients and 50 active clients
Search thread Thread-309 finished
Key 'BB': 1 idle clients and 49 active clients
Search thread Thread-369 finished
Key 'BB': 1 idle clients and 49 active clients
Insert thread Thread-22 finished
Key 'CC': 1 idle clients and 49 active clients
Search thread Thread-585 finished
Key 'BB': 1 idle clients and 49 active clients
Search thread Thread-415 finished
Key 'AA': 1 idle clients and 49 active clients
Search thread Thread-439 finished
Key 'AA': 1 idle clients and 49 active clients
Insert thread Thread-328 finished
Key 'CC': 1 idle clients and 50 active clients
Insert thread Thread-164 finished
Key 'BB': 1 idle clients and 49 active clients
Insert thread Thread-152 finished
Key 'BB': 1 idle clients and 49 active clients
Search thread Thread-413 finished
Key 'CC': 1 idle clients and 49 active clients
Insert thread Thread-112 finished
Key 'CC': 1 idle clients and 49 active clients
Insert thread Thread-56 finished
Key 'BB': 2 idle clients and 49 active clients
Search thread Thread-373 finished
Key 'AA': 1 idle clients and 49 active clients
Insert thread Thread-256 finished
Key 'CC': 1 idle clients and 49 active clients
Insert thread Thread-6 finished
Key 'AA': 1 idle clients and 49 active clients
Insert thread Thread-18 finished
Key 'AA': 1 idle clients and 49 active clients
Insert thread Thread-218 finished
Key 'BB': 1 idle clients and 49 active clients
Search thread Thread-195 finished
Key 'BB': 1 idle clients and 49 active clients
Search thread Thread-521 finished
Key 'CC': 1 idle clients and 49 active clients
Insert thread Thread-86 finished
Key 'BB': 1 idle clients and 49 active clients
Insert thread Thread-204 finished
Key 'AA': 2 idle clients and 48 active clients
Insert thread Thread-48 finished
Key 'AA': 1 idle clients and 49 active clients
Insert thread Thread-176 finished
Key 'BB': 1 idle clients and 49 active clients
Search thread Thread-527 finished
Key 'CC': 0 idle clients and 50 active clients
Insert thread Thread-136 finished
Key 'CC': 1 idle clients and 49 active clients
Search thread Thread-285 finished
Key 'BB': 2 idle clients and 48 active clients
Insert thread Thread-248 finished
Key 'BB': 1 idle clients and 49 active clients
Insert thread Thread-214 finished
Key 'CC': 1 idle clients and 49 active clients
Search thread Thread-67 finished
Key 'AA': 1 idle clients and 49 active clients
Search thread Thread-449 finished
Key 'CC': 1 idle clients and 49 active clients
Search thread Thread-377 finished
Key 'CC': 1 idle clients and 49 active clients
Search thread Thread-321 finished
Key 'BB': 1 idle clients and 49 active clients
Insert thread Thread-46 finished
Key 'CC': 0 idle clients and 50 active clients
Insert thread Thread-188 finished
Key 'BB': 1 idle clients and 49 active clients
Insert thread Thread-458 finished
Key 'BB': 2 idle clients and 48 active clients
Insert thread Thread-98 finished
Key 'BB': 2 idle clients and 48 active clients
Insert thread Thread-184 finished
Key 'CC': 1 idle clients and 49 active clients
Insert thread Thread-70 finished
Key 'CC': 2 idle clients and 48 active clients
Insert thread Thread-52 finished
Key 'CC': 1 idle clients and 49 active clients
Search thread Thread-53 finished
Key 'CC': 0 idle clients and 50 active clients
Insert thread Thread-62 finished
Key 'BB': 1 idle clients and 49 active clients
Search thread Thread-551 finished
Key 'CC': 1 idle clients and 49 active clients
Search thread Thread-495 finished
Key 'BB': 0 idle clients and 50 active clients
Insert thread Thread-38 finished
Key 'BB': 1 idle clients and 49 active clients
Search thread Thread-563 finished
Key 'CC': 1 idle clients and 49 active clients
Search thread Thread-579 finished
Key 'BB': 1 idle clients and 49 active clients
Search thread Thread-529 finished
Key 'AA': 1 idle clients and 49 active clients
Insert thread Thread-120 finished
Key 'AA': 1 idle clients and 49 active clients
Insert thread Thread-66 finished
Key 'AA': 0 idle clients and 50 active clients
Insert thread Thread-92 finished
Key 'BB': 2 idle clients and 48 active clients
Insert thread Thread-90 finished
Key 'AA': 1 idle clients and 49 active clients
Search thread Thread-307 finished
Key 'AA': 0 idle clients and 50 active clients
Insert thread Thread-58 finished
Key 'CC': 0 idle clients and 50 active clients
Search thread Thread-463 finished
Key 'AA': 0 idle clients and 50 active clients
Search thread Thread-259 finished
Key 'AA': 0 idle clients and 50 active clients
Search thread Thread-279 finished
Key 'BB': 1 idle clients and 49 active clients
Search thread Thread-197 finished
Key 'CC': 1 idle clients and 49 active clients
Search thread Thread-257 finished
Key 'CC': 1 idle clients and 49 active clients
Insert thread Thread-142 finished
Key 'CC': 1 idle clients and 49 active clients
Insert thread Thread-532 finished
Key 'CC': 1 idle clients and 49 active clients
Search thread Thread-339 finished
Key 'BB': 1 idle clients and 49 active clients
Insert thread Thread-196 finished
Key 'CC': 1 idle clients and 49 active clients
Insert thread Thread-116 finished
Key 'BB': 1 idle clients and 49 active clients
Search thread Thread-395 finished
Key 'CC': 1 idle clients and 49 active clients
Insert thread Thread-504 finished
Key 'AA': 1 idle clients and 49 active clients
Insert thread Thread-114 finished
Key 'AA': 0 idle clients and 50 active clients
Insert thread Thread-484 finished
Key 'CC': 0 idle clients and 50 active clients
Insert thread Thread-14 finished
Key 'BB': 0 idle clients and 50 active clients
Search thread Thread-317 finished
Key 'CC': 0 idle clients and 50 active clients
Search thread Thread-361 finished
Key 'AA': 1 idle clients and 49 active clients
Search thread Thread-561 finished
Key 'BB': 1 idle clients and 49 active clients
Insert thread Thread-198 finished
Key 'AA': 1 idle clients and 49 active clients
Insert thread Thread-326 finished
Key 'BB': 1 idle clients and 49 active clients
Insert thread Thread-294 finished
Key 'AA': 1 idle clients and 49 active clients
Search thread Thread-535 finished
Key 'AA': 1 idle clients and 49 active clients
Search thread Thread-389 finished
Key 'CC': 1 idle clients and 49 active clients
Search thread Thread-375 finished
Key 'BB': 1 idle clients and 49 active clients
Search thread Thread-385 finished
Key 'AA': 1 idle clients and 49 active clients
Search thread Thread-367 finished
Key 'AA': 1 idle clients and 49 active clients
Search thread Thread-381 finished
Key 'BB': 2 idle clients and 48 active clients
Search thread Thread-443 finished
Key 'CC': 0 idle clients and 50 active clients
Insert thread Thread-296 finished
Key 'BB': 0 idle clients and 50 active clients
Insert thread Thread-148 finished
Key 'CC': 0 idle clients and 50 active clients
Insert thread Thread-40 finished
Key 'CC': 0 idle clients and 50 active clients
Search thread Thread-497 finished
Key 'CC': 1 idle clients and 49 active clients
Insert thread Thread-222 finished
Key 'AA': 1 idle clients and 49 active clients
Search thread Thread-409 finished
Key 'AA': 0 idle clients and 50 active clients
Search thread Thread-371 finished
Key 'CC': 1 idle clients and 49 active clients
Search thread Thread-573 finished
Key 'BB': 1 idle clients and 49 active clients
Search thread Thread-17 finished
Key 'CC': 1 idle clients and 49 active clients
Search thread Thread-403 finished
Key 'AA': 1 idle clients and 49 active clients
Search thread Thread-387 finished
Key 'BB': 0 idle clients and 50 active clients
Insert thread Thread-432 finished
Key 'AA': 1 idle clients and 49 active clients
Insert thread Thread-540 finished
Key 'AA': 1 idle clients and 49 active clients
Insert thread Thread-226 finished
Key 'CC': 1 idle clients and 49 active clients
Insert thread Thread-232 finished
Key 'CC': 0 idle clients and 50 active clients
Insert thread Thread-288 finished
Key 'AA': 1 idle clients and 49 active clients
Insert thread Thread-4 finished
Key 'CC': 0 idle clients and 50 active clients
Insert thread Thread-268 finished
Key 'CC': 2 idle clients and 48 active clients
Search thread Thread-509 finished
Key 'CC': 1 idle clients and 49 active clients
Search thread Thread-539 finished
Key 'CC': 0 idle clients and 50 active clients
Search thread Thread-475 finished
Key 'AA': 2 idle clients and 48 active clients
Search thread Thread-29 finished
Key 'CC': 1 idle clients and 49 active clients
Search thread Thread-359 finished
Key 'CC': 1 idle clients and 49 active clients
Search thread Thread-489 finished
Key 'BB': 1 idle clients and 49 active clients
Insert thread Thread-8 finished
Key 'BB': 0 idle clients and 50 active clients
Insert thread Thread-392 finished
Key 'BB': 0 idle clients and 50 active clients
Insert thread Thread-336 finished
Key 'AA': 1 idle clients and 49 active clients
Insert thread Thread-104 finished
Key 'BB': 1 idle clients and 49 active clients
Insert thread Thread-236 finished
Key 'BB': 1 idle clients and 49 active clients
Insert thread Thread-140 finished
Key 'BB': 1 idle clients and 49 active clients
Insert thread Thread-146 finished
Key 'BB': 1 idle clients and 49 active clients
Insert thread Thread-402 finished
Key 'AA': 1 idle clients and 49 active clients
Search thread Thread-537 finished
Key 'BB': 1 idle clients and 49 active clients
Search thread Thread-427 finished
Key 'AA': 1 idle clients and 49 active clients
Search thread Thread-265 finished
Key 'AA': 2 idle clients and 48 active clients
Search thread Thread-393 finished
Key 'BB': 1 idle clients and 49 active clients
Insert thread Thread-276 finished
Key 'AA': 1 idle clients and 49 active clients
Search thread Thread-517 finished
Key 'AA': 1 idle clients and 49 active clients
Search thread Thread-441 finished
Key 'BB': 1 idle clients and 49 active clients
Search thread Thread-331 finished
Key 'AA': 1 idle clients and 49 active clients
Search thread Thread-345 finished
Key 'BB': 1 idle clients and 49 active clients
Insert thread Thread-220 finished
Key 'CC': 1 idle clients and 49 active clients
Insert thread Thread-368 finished
Key 'BB': 1 idle clients and 49 active clients
Insert thread Thread-110 finished
Key 'BB': 1 idle clients and 49 active clients
Insert thread Thread-32 finished
Key 'BB': 2 idle clients and 48 active clients
Insert thread Thread-272 finished
Key 'BB': 2 idle clients and 48 active clients
Search thread Thread-547 finished
Key 'AA': 1 idle clients and 49 active clients
Search thread Thread-511 finished
Key 'AA': 1 idle clients and 49 active clients
Search thread Thread-405 finished
Key 'BB': 1 idle clients and 49 active clients
Insert thread Thread-302 finished
Key 'BB': 0 idle clients and 50 active clients
Insert thread Thread-434 finished
Key 'BB': 1 idle clients and 49 active clients
Insert thread Thread-78 finished
Key 'AA': 0 idle clients and 50 active clients
Search thread Thread-583 finished
Key 'AA': 1 idle clients and 49 active clients
Search thread Thread-219 finished
Key 'BB': 0 idle clients and 50 active clients
Search thread Thread-245 finished
Key 'CC': 1 idle clients and 49 active clients
Insert thread Thread-510 finished
Key 'AA': 2 idle clients and 48 active clients
Search thread Thread-447 finished
Key 'BB': 0 idle clients and 50 active clients
Insert thread Thread-360 finished
Key 'AA': 0 idle clients and 50 active clients
Search thread Thread-325 finished
Key 'AA': 1 idle clients and 49 active clients
Insert thread Thread-0 finished
Key 'AA': 1 idle clients and 49 active clients
Insert thread Thread-126 finished
Key 'AA': 1 idle clients and 49 active clients
Search thread Thread-597 finished
Key 'BB': 1 idle clients and 49 active clients
Insert thread Thread-308 finished
Key 'BB': 1 idle clients and 49 active clients
Insert thread Thread-210 finished
Key 'AA': 1 idle clients and 49 active clients
Insert thread Thread-284 finished
Key 'BB': 0 idle clients and 50 active clients
Search thread Thread-503 finished
Key 'CC': 1 idle clients and 49 active clients
Search thread Thread-357 finished
Key 'BB': 1 idle clients and 49 active clients
Insert thread Thread-490 finished
Key 'CC': 1 idle clients and 49 active clients
Insert thread Thread-250 finished
Key 'CC': 0 idle clients and 50 active clients
Insert thread Thread-160 finished
Key 'CC': 0 idle clients and 50 active clients
Insert thread Thread-130 finished
Key 'CC': 1 idle clients and 49 active clients
Insert thread Thread-34 finished
Key 'CC': 1 idle clients and 49 active clients
Insert thread Thread-596 finished
Key 'BB': 0 idle clients and 50 active clients
Search thread Thread-599 finished
Key 'CC': 0 idle clients and 50 active clients
Search thread Thread-553 finished
Key 'AA': 1 idle clients and 49 active clients
Search thread Thread-421 finished
Key 'AA': 0 idle clients and 50 active clients
Search thread Thread-471 finished
Key 'BB': 1 idle clients and 49 active clients
Search thread Thread-567 finished
Key 'BB': 0 idle clients and 50 active clients
Search thread Thread-213 finished
Key 'BB': 0 idle clients and 50 active clients
Insert thread Thread-516 finished
Key 'AA': 1 idle clients and 49 active clients
Insert thread Thread-242 finished
Key 'BB': 1 idle clients and 49 active clients
Insert thread Thread-468 finished
Key 'AA': 1 idle clients and 49 active clients
Insert thread Thread-30 finished
Key 'AA': 2 idle clients and 48 active clients
Insert thread Thread-138 finished
Key 'AA': 0 idle clients and 50 active clients
Insert thread Thread-400 finished
Key 'CC': 1 idle clients and 49 active clients
Search thread Thread-383 finished
Key 'CC': 1 idle clients and 49 active clients
Search thread Thread-571 finished
Key 'AA': 1 idle clients and 49 active clients
Insert thread Thread-572 finished
Key 'BB': 1 idle clients and 49 active clients
Insert thread Thread-180 finished
Key 'AA': 0 idle clients and 50 active clients
Insert thread Thread-234 finished
Key 'AA': 1 idle clients and 49 active clients
Insert thread Thread-534 finished
Key 'AA': 0 idle clients and 50 active clients
Search thread Thread-543 finished
Key 'BB': 1 idle clients and 49 active clients
Insert thread Thread-200 finished
Key 'BB': 3 idle clients and 47 active clients
Search thread Thread-505 finished
Key 'AA': 1 idle clients and 49 active clients
Search thread Thread-493 finished
Key 'AA': 0 idle clients and 50 active clients
Search thread Thread-429 finished
Key 'BB': 1 idle clients and 49 active clients
Search thread Thread-469 finished
Key 'AA': 1 idle clients and 49 active clients
Search thread Thread-559 finished
Key 'AA': 1 idle clients and 49 active clients
Search thread Thread-253 finished
Key 'AA': 1 idle clients and 49 active clients
Insert thread Thread-2 finished
Key 'BB': 1 idle clients and 49 active clients
Insert thread Thread-266 finished
Key 'BB': 1 idle clients and 49 active clients
Insert thread Thread-418 finished
Key 'CC': 1 idle clients and 49 active clients
Insert thread Thread-264 finished
Key 'AA': 1 idle clients and 49 active clients
Search thread Thread-341 finished
Key 'CC': 2 idle clients and 48 active clients
Search thread Thread-569 finished
Key 'CC': 1 idle clients and 49 active clients
Search thread Thread-35 finished
Key 'CC': 1 idle clients and 49 active clients
Search thread Thread-223 finished
Key 'AA': 1 idle clients and 49 active clients
Search thread Thread-549 finished
Key 'BB': 1 idle clients and 49 active clients
Search thread Thread-425 finished
Key 'CC': 1 idle clients and 49 active clients
Search thread Thread-243 finished
Key 'BB': 1 idle clients and 49 active clients
Search thread Thread-595 finished
Key 'AA': 1 idle clients and 49 active clients
Search thread Thread-323 finished
Key 'CC': 1 idle clients and 49 active clients
Search thread Thread-397 finished
Key 'AA': 0 idle clients and 50 active clients
Insert thread Thread-158 finished
Key 'BB': 0 idle clients and 50 active clients
Insert thread Thread-422 finished
Key 'BB': 2 idle clients and 48 active clients
Insert thread Thread-80 finished
Key 'BB': 1 idle clients and 49 active clients
Insert thread Thread-436 finished
Key 'CC': 1 idle clients and 49 active clients
Search thread Thread-183 finished
Key 'BB': 1 idle clients and 49 active clients
Search thread Thread-575 finished
Key 'CC': 1 idle clients and 49 active clients
Search thread Thread-347 finished
Key 'CC': 1 idle clients and 49 active clients
Search thread Thread-235 finished
Key 'AA': 0 idle clients and 50 active clients
Search thread Thread-591 finished
Key 'BB': 1 idle clients and 49 active clients
Search thread Thread-581 finished
Key 'CC': 0 idle clients and 50 active clients
Search thread Thread-545 finished
Key 'CC': 0 idle clients and 50 active clients
Insert thread Thread-150 finished
Key 'AA': 1 idle clients and 49 active clients
Insert thread Thread-564 finished
Key 'AA': 2 idle clients and 48 active clients
Search thread Thread-467 finished
Key 'CC': 1 idle clients and 49 active clients
Search thread Thread-319 finished
Key 'AA': 1 idle clients and 49 active clients
Search thread Thread-329 finished
Key 'CC': 1 idle clients and 49 active clients
Search thread Thread-203 finished
Key 'CC': 0 idle clients and 50 active clients
Search thread Thread-423 finished
Key 'BB': 0 idle clients and 50 active clients
Search thread Thread-313 finished
Key 'AA': 1 idle clients and 49 active clients
Search thread Thread-513 finished
Key 'BB': 0 idle clients and 49 active clients
Insert thread Thread-384 finished
Key 'AA': 1 idle clients and 49 active clients
Insert thread Thread-124 finished
Key 'CC': 1 idle clients and 49 active clients
Insert thread Thread-538 finished
Key 'CC': 1 idle clients and 49 active clients
Insert thread Thread-542 finished
Key 'BB': 1 idle clients and 49 active clients
Insert thread Thread-494 finished
Key 'BB': 1 idle clients and 49 active clients
Insert thread Thread-102 finished
Key 'AA': 0 idle clients and 50 active clients
Insert thread Thread-24 finished
Key 'AA': 1 idle clients and 49 active clients
Insert thread Thread-258 finished
Key 'AA': 2 idle clients and 48 active clients
Insert thread Thread-182 finished
Key 'BB': 1 idle clients and 49 active clients
Search thread Thread-465 finished
Key 'BB': 1 idle clients and 49 active clients
Search thread Thread-525 finished
Key 'BB': 1 idle clients and 49 active clients
Search thread Thread-589 finished
Key 'AA': 1 idle clients and 49 active clients
Search thread Thread-501 finished
Key 'BB': 0 idle clients and 50 active clients
Search thread Thread-261 finished
Key 'BB': 1 idle clients and 49 active clients
Search thread Thread-555 finished
Key 'BB': 1 idle clients and 49 active clients
Search thread Thread-507 finished
Key 'BB': 1 idle clients and 49 active clients
Search thread Thread-327 finished
Key 'BB': 1 idle clients and 49 active clients
Insert thread Thread-118 finished
Key 'CC': 1 idle clients and 49 active clients
Insert thread Thread-96 finished
Key 'AA': 1 idle clients and 49 active clients
Insert thread Thread-192 finished
Key 'AA': 1 idle clients and 49 active clients
Insert thread Thread-278 finished
Key 'BB': 1 idle clients and 49 active clients
Insert thread Thread-216 finished
Key 'AA': 1 idle clients and 49 active clients
Insert thread Thread-324 finished
Key 'AA': 1 idle clients and 49 active clients
Insert thread Thread-82 finished
Key 'CC': 2 idle clients and 48 active clients
Insert thread Thread-172 finished
Key 'CC': 0 idle clients and 50 active clients
Insert thread Thread-202 finished
Key 'CC': 1 idle clients and 49 active clients
Search thread Thread-177 finished
Key 'BB': 1 idle clients and 49 active clients
Search thread Thread-531 finished
Key 'BB': 2 idle clients and 48 active clients
Search thread Thread-433 finished
Key 'AA': 1 idle clients and 49 active clients
Search thread Thread-289 finished
Key 'AA': 0 idle clients and 50 active clients
Search thread Thread-411 finished
Key 'BB': 3 idle clients and 47 active clients
Search thread Thread-295 finished
Key 'AA': 1 idle clients and 49 active clients
Search thread Thread-445 finished
Key 'AA': 1 idle clients and 49 active clients
Search thread Thread-451 finished
Key 'AA': 2 idle clients and 48 active clients
Search thread Thread-523 finished
Key 'AA': 3 idle clients and 47 active clients
Search thread Thread-59 finished
Key 'CC': 1 idle clients and 49 active clients
Search thread Thread-349 finished
Key 'AA': 4 idle clients and 46 active clients
Search thread Thread-391 finished
Key 'AA': 5 idle clients and 45 active clients
Insert thread Thread-84 finished
Key 'AA': 6 idle clients and 44 active clients
Search thread Thread-487 finished
Key 'AA': 7 idle clients and 43 active clients
Search thread Thread-565 finished
Key 'AA': 8 idle clients and 42 active clients
Insert thread Thread-154 finished
Key 'CC': 1 idle clients and 49 active clients
Insert thread Thread-144 finished
Key 'AA': 9 idle clients and 41 active clients
Insert thread Thread-446 finished
Key 'BB': 4 idle clients and 46 active clients
Insert thread Thread-128 finished
Key 'BB': 5 idle clients and 45 active clients
Insert thread Thread-166 finished
Key 'CC': 1 idle clients and 49 active clients
Search thread Thread-407 finished
Key 'CC': 0 idle clients and 50 active clients
Insert thread Thread-318 finished
Key 'AA': 10 idle clients and 40 active clients
Insert thread Thread-416 finished
Key 'BB': 6 idle clients and 44 active clients
Insert thread Thread-178 finished
Key 'CC': 1 idle clients and 49 active clients
Insert thread Thread-552 finished
Key 'AA': 9 idle clients and 37 active clients
Insert thread Thread-228 finished
Key 'AA': 10 idle clients and 32 active clients
Insert thread Thread-592 finished
Key 'CC': 1 idle clients and 49 active clients
Insert thread Thread-168 finished
Key 'AA': 9 idle clients and 33 active clients
Search thread Thread-459 finished
Key 'BB': 7 idle clients and 43 active clients
Insert thread Thread-340 finished
Key 'CC': 1 idle clients and 49 active clients
Insert thread Thread-304 finished
Key 'CC': 2 idle clients and 48 active clients
Insert thread Thread-562 finished
Key 'CC': 3 idle clients and 47 active clients
Insert thread Thread-190 finished
Key 'CC': 0 idle clients and 50 active clients
Insert thread Thread-88 finished
Key 'CC': 1 idle clients and 49 active clients
Insert thread Thread-378 finished
Key 'AA': 6 idle clients and 36 active clients
Insert thread Thread-320 finished
Key 'BB': 8 idle clients and 42 active clients
Insert thread Thread-194 finished
Key 'BB': 9 idle clients and 41 active clients
Insert thread Thread-364 finished
Key 'CC': 1 idle clients and 49 active clients
Search thread Thread-41 finished
Key 'CC': 0 idle clients and 50 active clients
Search thread Thread-557 finished
Key 'CC': 1 idle clients and 49 active clients
Search thread Thread-311 finished
Key 'CC': 1 idle clients and 49 active clients
Search thread Thread-435 finished
Key 'BB': 9 idle clients and 40 active clients
Insert thread Thread-262 finished
Key 'CC': 1 idle clients and 49 active clients
Insert thread Thread-454 finished
Key 'CC': 1 idle clients and 49 active clients
Insert thread Thread-566 finished
Key 'BB': 10 idle clients and 39 active clients
Insert thread Thread-598 finished
Key 'CC': 1 idle clients and 49 active clients
Insert thread Thread-362 finished
Key 'BB': 6 idle clients and 38 active clients
Insert thread Thread-312 finished
Key 'AA': 7 idle clients and 35 active clients
Insert thread Thread-528 finished
Key 'AA': 8 idle clients and 34 active clients
Insert thread Thread-286 finished
Key 'CC': 1 idle clients and 49 active clients
Insert thread Thread-580 finished
Key 'CC': 1 idle clients and 49 active clients
Insert thread Thread-354 finished
Key 'AA': 9 idle clients and 33 active clients
Insert thread Thread-282 finished
Key 'AA': 10 idle clients and 31 active clients
Search thread Thread-479 finished
Key 'CC': 1 idle clients and 49 active clients
Search thread Thread-593 finished
Key 'CC': 1 idle clients and 49 active clients
Search thread Thread-587 finished
Key 'CC': 1 idle clients and 49 active clients
Insert thread Thread-472 finished
Key 'CC': 2 idle clients and 48 active clients
Search thread Thread-533 finished
Key 'CC': 3 idle clients and 47 active clients
Search thread Thread-491 finished
Key 'CC': 4 idle clients and 46 active clients
Insert thread Thread-334 finished
Key 'CC': 5 idle clients and 44 active clients
Insert thread Thread-350 finished
Key 'BB': 7 idle clients and 37 active clients
Insert thread Thread-380 finished
Key 'BB': 8 idle clients and 36 active clients
Insert thread Thread-376 finished
Key 'CC': 6 idle clients and 44 active clients
Insert thread Thread-342 finished
Insert thread Thread-570 finished
Key 'AA': 10 idle clients and 29 active clients
Key 'AA': 10 idle clients and 29 active clients
Insert thread Thread-162 finished
Key 'AA': 9 idle clients and 28 active clients
Search thread Thread-299 finished
Key 'CC': 7 idle clients and 43 active clients
Search thread Thread-461 finished
Key 'CC': 8 idle clients and 42 active clients
Search thread Thread-401 finished
Key 'CC': 9 idle clients and 41 active clients
Search thread Thread-215 finished
Key 'CC': 10 idle clients and 40 active clients
Insert thread Thread-306 finished
Key 'AA': 9 idle clients and 28 active clients
Insert thread Thread-108 finished
Key 'AA': 10 idle clients and 27 active clients
Insert thread Thread-388 finished
Key 'CC': 10 idle clients and 39 active clients
Insert thread Thread-554 finished
Key 'BB': 9 idle clients and 35 active clients
Insert thread Thread-428 finished
Key 'BB': 10 idle clients and 33 active clients
Insert thread Thread-212 finished
Key 'BB': 9 idle clients and 33 active clients
Insert thread Thread-386 finished
Key 'BB': 9 idle clients and 32 active clients
Search thread Thread-457 finished
Key 'AA': 9 idle clients and 26 active clients
Insert thread Thread-398 finished
Key 'BB': 10 idle clients and 31 active clients
Insert thread Thread-496 finished
Key 'CC': 8 idle clients and 38 active clients
Insert thread Thread-348 finished
Key 'AA': 10 idle clients and 25 active clients
Insert thread Thread-224 finished
Key 'BB': 10 idle clients and 30 active clients
Insert thread Thread-408 finished
Key 'AA': 10 idle clients and 24 active clients
Insert thread Thread-476 finished
Key 'BB': 9 idle clients and 28 active clients
Insert thread Thread-322 finished
Key 'CC': 9 idle clients and 37 active clients
Insert thread Thread-316 finished
Key 'CC': 10 idle clients and 36 active clients
Insert thread Thread-506 finished
Key 'BB': 9 idle clients and 28 active clients
Insert thread Thread-280 finished
Key 'CC': 9 idle clients and 35 active clients
Insert thread Thread-76 finished
Key 'CC': 10 idle clients and 34 active clients
Search thread Thread-485 finished
Key 'CC': 6 idle clients and 33 active clients
Insert thread Thread-584 finished
Key 'BB': 10 idle clients and 26 active clients
Insert thread Thread-586 finished
Key 'CC': 7 idle clients and 32 active clients
Insert thread Thread-456 finished
Key 'AA': 9 idle clients and 23 active clients
Insert thread Thread-94 finished
Key 'CC': 8 idle clients and 31 active clients
Insert thread Thread-500 finished
Key 'BB': 2 idle clients and 26 active clients
Insert thread Thread-290 finished
Key 'BB': 3 idle clients and 25 active clients
Insert thread Thread-442 finished
Key 'CC': 9 idle clients and 30 active clients
Insert thread Thread-440 finished
Key 'BB': 4 idle clients and 24 active clients
Insert thread Thread-244 finished
Key 'CC': 10 idle clients and 29 active clients
Insert thread Thread-578 finished
Key 'BB': 5 idle clients and 23 active clients
Insert thread Thread-544 finished
Key 'CC': 10 idle clients and 26 active clients
Insert thread Thread-330 finished
Key 'AA': 9 idle clients and 22 active clients
Insert thread Thread-406 finished
Key 'CC': 4 idle clients and 27 active clients
Search thread Thread-455 finished
Key 'CC': 5 idle clients and 26 active clients
Insert thread Thread-382 finished
Key 'CC': 6 idle clients and 25 active clients
Insert thread Thread-414 finished
Key 'AA': 9 idle clients and 21 active clients
Insert thread Thread-482 finished
Key 'BB': 6 idle clients and 22 active clients
Insert thread Thread-344 finished
Key 'BB': 7 idle clients and 21 active clients
Insert thread Thread-246 finished
Key 'AA': 9 idle clients and 20 active clients
Insert thread Thread-464 finished
Key 'BB': 8 idle clients and 20 active clients
Insert thread Thread-230 finished
Key 'BB': 9 idle clients and 19 active clients
Insert thread Thread-438 finished
Key 'AA': 9 idle clients and 19 active clients
Insert thread Thread-252 finished
Key 'AA': 10 idle clients and 18 active clients
Insert thread Thread-352 finished
Key 'CC': 7 idle clients and 24 active clients
Insert thread Thread-366 finished
Key 'AA': 9 idle clients and 17 active clients
Insert thread Thread-298 finished
Key 'CC': 8 idle clients and 23 active clients
Insert thread Thread-548 finished
Key 'BB': 10 idle clients and 18 active clients
Insert thread Thread-404 finished
Key 'BB': 10 idle clients and 17 active clients
Insert thread Thread-488 finished
Key 'BB': 10 idle clients and 16 active clients
Insert thread Thread-502 finished
Key 'CC': 9 idle clients and 22 active clients
Insert thread Thread-274 finished
Key 'CC': 10 idle clients and 16 active clients
Insert thread Thread-560 finished
Key 'BB': 10 idle clients and 15 active clients
Insert thread Thread-522 finished
Key 'AA': 9 idle clients and 16 active clients
Insert thread Thread-514 finished
Key 'CC': 6 idle clients and 20 active clients
Insert thread Thread-410 finished
Key 'BB': 9 idle clients and 14 active clients
Insert thread Thread-568 finished
Insert thread Thread-536 finished
Key 'BB': 10 idle clients and 13 active clients
Key 'CC': 7 idle clients and 19 active clients
Insert thread Thread-170 finished
Key 'BB': 10 idle clients and 10 active clients
Insert thread Thread-310 finished
Key 'CC': 8 idle clients and 18 active clients
Insert thread Thread-420 finished
Key 'AA': 10 idle clients and 15 active clients
Insert thread Thread-470 finished
Key 'BB': 9 idle clients and 11 active clients
Insert thread Thread-426 finished
Key 'AA': 9 idle clients and 12 active clients
Insert thread Thread-524 finished
Key 'BB': 10 idle clients and 10 active clients
Insert thread Thread-520 finished
Key 'CC': 9 idle clients and 17 active clients
Insert thread Thread-430 finished
Key 'CC': 10 idle clients and 15 active clients
Insert thread Thread-254 finished
Key 'BB': 9 idle clients and 9 active clients
Insert thread Thread-374 finished
Key 'BB': 10 idle clients and 8 active clients
Insert thread Thread-122 finished
Key 'BB': 10 idle clients and 6 active clients
Insert thread Thread-424 finished
Key 'CC': 5 idle clients and 15 active clients
Insert thread Thread-460 finished
Key 'CC': 6 idle clients and 14 active clients
Insert thread Thread-448 finished
Key 'CC': 7 idle clients and 13 active clients
Insert thread Thread-530 finished
Key 'BB': 10 idle clients and 6 active clients
Insert thread Thread-508 finished
Key 'CC': 8 idle clients and 12 active clients
Insert thread Thread-590 finished
Key 'BB': 9 idle clients and 5 active clients
Insert thread Thread-346 finished
Key 'CC': 9 idle clients and 11 active clients
Insert thread Thread-208 finished
Key 'CC': 10 idle clients and 10 active clients
Insert thread Thread-492 finished
Key 'AA': 8 idle clients and 13 active clients
Insert thread Thread-444 finished
Key 'AA': 9 idle clients and 12 active clients
Insert thread Thread-466 finished
Key 'CC': 7 idle clients and 9 active clients
Insert thread Thread-574 finished
Key 'CC': 8 idle clients and 8 active clients
Insert thread Thread-556 finished
Key 'CC': 9 idle clients and 7 active clients
Insert thread Thread-394 finished
Key 'CC': 10 idle clients and 6 active clients
Insert thread Thread-480 finished
Key 'AA': 10 idle clients and 11 active clients
Insert thread Thread-546 finished
Key 'AA': 10 idle clients and 10 active clients
Insert thread Thread-450 finished
Key 'AA': 10 idle clients and 9 active clients
Insert thread Thread-518 finished
Key 'BB': 10 idle clients and 4 active clients
Insert thread Thread-550 finished
Key 'CC': 9 idle clients and 5 active clients
Insert thread Thread-338 finished
Key 'BB': 9 idle clients and 3 active clients
Insert thread Thread-412 finished
Key 'CC': 10 idle clients and 4 active clients
Insert thread Thread-370 finished
Key 'CC': 9 idle clients and 3 active clients
Insert thread Thread-372 finished
Key 'AA': 4 idle clients and 8 active clients
Insert thread Thread-512 finished
Key 'BB': 10 idle clients and 2 active clients
Insert thread Thread-582 finished
Key 'AA': 5 idle clients and 7 active clients
Insert thread Thread-498 finished
Key 'AA': 6 idle clients and 6 active clients
Insert thread Thread-452 finished
Key 'BB': 9 idle clients and 1 active clients
Insert thread Thread-588 finished
Key 'AA': 7 idle clients and 5 active clients
Insert thread Thread-526 finished
Key 'CC': 10 idle clients and 1 active clients
Insert thread Thread-238 finished
Key 'CC': 9 idle clients and 0 active clients
Insert thread Thread-240 finished
Key 'AA': 8 idle clients and 4 active clients
Insert thread Thread-486 finished
Key 'AA': 9 idle clients and 3 active clients
Insert thread Thread-594 finished
Key 'AA': 10 idle clients and 2 active clients
Insert thread Thread-390 finished
Key 'AA': 10 idle clients and 1 active clients
Insert thread Thread-478 finished
Key 'CC': 10 idle clients and 0 active clients
Insert thread Thread-576 finished
Key 'AA': 10 idle clients and 0 active clients
Insert thread Thread-356 finished
Key 'BB': 10 idle clients and 0 active clients
======================================================================
Total 30 idle clients and 0 active clients
Key 'AA': 10 idle clients and 0 active clients
Key 'BB': 10 idle clients and 0 active clients
Key 'CC': 10 idle clients and 0 active clients
======================================================================
10000 rows persisted in collection 'java_sdk_example_pool_v2' of database 'AA'
10000 rows persisted in collection 'java_sdk_example_pool_v2' of database 'BB'
10000 rows persisted in collection 'java_sdk_example_pool_v2' of database 'CC'
Collection 'java_sdk_example_pool_v2' dropped in database 'AA'
Collection 'java_sdk_example_pool_v2' dropped in database 'BB'
Collection 'java_sdk_example_pool_v2' dropped in database 'CC'
Database dropped: AA
Database dropped: BB
Database dropped: CC
30000 insert requests and 30000 search requests finished in 22.946 seconds
======================================================================
Total 30 idle clients and 0 active clients
Key 'AA': 10 idle clients and 0 active clients
Key 'BB': 10 idle clients and 0 active clients
Key 'CC': 10 idle clients and 0 active clients
======================================================================
======================================================================
Total 0 idle clients and 0 active clients
Key 'AA': 0 idle clients and 0 active clients
Key 'BB': 0 idle clients and 0 active clients
Key 'CC': 0 idle clients and 0 active clients
======================================================================

Process finished with exit code 0

[xiaofan-luan]

maxTotalPerKey

I think maxTotalPerKey too small (10) is the main problem, we would expect it to be 50.

[yhmo]

This pr https://github.com/milvus-io/milvus-sdk-java/pull/1604 changed the maxTotalPerKey default value to 50.