locust
locust copied to clipboard
locustio
Performance difference between v2.41.5 and 2.13.0
Prerequisites
- [x] I am using the latest version of Locust
- [x] I am reporting a bug, not asking a question
Description
Hi,
We tried load testing on our services with v2.41.5, using the following command:
locust -f <...> --host=<...> --users 3000 --spawn-rate 3000 --run-time 60s --headless --processes 4
And then we tried the same load test with v2.13.0 with the following command: Terminal 1
locust --master -f <...> --expect-workers=3 --headless --users 3000 --spawn-rate 3000 --run-time 60
Terminal 2
for ((i=1;i<=3;i++));
do
locust --worker --master-host=127.0.0.1 -f <...> &
done
We see that that there is a huge difference in reported performance: v2.41.5 shows p95 that is significantly slower (higher number) compared to using v2.13.0.
Is this a known issue, or are we using the new version incorrectly somehow?
Command line
locust -f <...> --host=<...> --users 3000 --spawn-rate 3000 --run-time 60s --headless --processes 4
Locustfile contents
from locust import HttpUser, task, constant_throughput
class TopicClassificationUser(HttpUser):
"""
Simple Locust user class for load testing the topic classification API endpoint.
"""
# Wait 1 second between requests (for 1 RPS per user)
wait_time = constant_throughput(1)
host = "http://....:8000" # purposely hidden IP
@task
def test_topic_classification(self):
"""
Single test task for topic classification API.
"""
# Prepare the payload
payload = {
"inputs": [
{
"name": "INPUT0",
"shape": [1, 1],
"datatype": "BYTES",
"data": ["what are movies"],
"parameters": {"content_type": "str"},
}
]
}
# Headers
headers = {"Content-Type": "application/json"}
# Make the request
with self.client.post(
"/v2/models/topic_classification/infer",
json=payload,
headers=headers,
catch_response=True,
) as response:
if response.status_code == 200:
try:
result = response.json()
# Check response structure
if self.validate_response(result):
response.success()
else:
response.failure("Response format validation failed")
except Exception as e:
response.failure(f"Invalid JSON response: {str(e)}")
else:
response.failure(f"HTTP {response.status_code}: {response.text}")
def validate_response(self, response_data):
"""
Validate that the response matches the expected format.
"""
try:
# Check required top-level fields
if "model_name" not in response_data:
print("Missing model_name field")
return False
if "model_version" not in response_data:
print("Missing model_version field")
return False
if "outputs" not in response_data:
print("Missing outputs field")
return False
# Check model_name and model_version values
if response_data["model_name"] != "topic_classification":
print(
f"Expected model_name 'topic_classification', got '{response_data['model_name']}'"
)
return False
if response_data["model_version"] != "1":
print(
f"Expected model_version '1', got '{response_data['model_version']}'"
)
return False
# Check outputs structure
outputs = response_data["outputs"]
if not isinstance(outputs, list) or len(outputs) == 0:
print("Outputs should be a non-empty list")
return False
output = outputs[0]
required_output_fields = ["name", "datatype", "shape", "data"]
for field in required_output_fields:
if field not in output:
print(f"Missing output field: {field}")
return False
# Check output field values
if output["name"] != "OUTPUT0":
print(f"Expected output name 'OUTPUT0', got '{output['name']}'")
return False
if output["datatype"] != "BYTES":
print(f"Expected datatype 'BYTES', got '{output['datatype']}'")
return False
if output["shape"] != [1]:
print(f"Expected shape [1], got {output['shape']}")
return False
# Check data structure
data = output["data"]
if not isinstance(data, list) or len(data) == 0:
print("Data should be a non-empty list")
return False
# Check that data contains JSON string with labels
data_item = data[0]
if not isinstance(data_item, str):
print("Data item should be a string")
return False
# Try to parse the JSON string in data
import json
try:
parsed_data = json.loads(data_item)
if not isinstance(parsed_data, list) or len(parsed_data) == 0:
print("Parsed data should be a non-empty list")
return False
item = parsed_data[0]
if "labels" not in item:
print("Missing 'labels' field in parsed data")
return False
labels = item["labels"]
if not isinstance(labels, list):
print("Labels should be a list")
return False
# Check for expected labels (Entertainment, Films)
if "Entertainment" not in labels or "Films" not in labels:
print(
f"Expected labels to contain 'Entertainment' and 'Films', got {labels}"
)
return False
except json.JSONDecodeError as e:
print(f"Failed to parse JSON in data: {e}")
return False
print(f"Response validation successful: {response_data}")
return True
except Exception as e:
print(f"Response validation error: {e}")
return False
Python version
3.12 when using v2.41.5 of locust, and python 3.10.12 when using v2.13.0 of locust
Locust version
v2.41.5
Operating system
Canonical, Ubuntu, 22.04 LTS Minimal, amd64 jammy minimal image built on 2025-10-01
There is a significant difference in SSLContext handling between Python 3.12 and 3.10, breaking performance, especially when launching users rapidly.
python-requests introduced a workaround for this issue in 2.32.0, but it seems to have been reverted in a recent release (2.32.5). Can you try requests 2.32.4?
I tried with requests package version 2.32.4. Whilst its better than before, the results are still quite different than the older version of locust:
Results with locust 2.41.5 & requests 2.32.4
- Command used:
locust -f topic_model.py --users 3000 --spawn-rate 3000 --run-time 60s --headless --csv=results --processes 7
Result:
Type Name # reqs # fails | Avg Min Max Med | req/s failures/s
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
POST /v2/models/topic_classification/infer 180000 0(0.00%) | 367 110 748 360 | 3009.65 0.00
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
Aggregated 180000 0(0.00%) | 367 110 748 360 | 3009.65 0.00
Response time percentiles (approximated)
Type Name 50% 66% 75% 80% 90% 95% 98% 99% 99.9% 99.99% 100% # reqs
--------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------
POST /v2/models/topic_classification/infer 360 430 470 490 530 550 570 610 720 740 750 180000
--------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------
Aggregated 360 430 470 490 530 550 570 610 720 740 750 180000
Results with locust 2.13.0 & requests 2.32.5
- Terminal 1:
locust --master -f topic_model.py --expect-workers=6 --headless --users 3000 --spawn-rate 3000 --run-time 60 - Terminal 2:
for ((i=1;i<=6;i++));
do
locust --worker --master-host=127.0.0.1 -f topic_model.py &
done
Results:
Type Name # reqs # fails | Avg Min Max Med | req/s failures/s
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
POST /v2/models/topic_classification/infer 179572 0(0.00%) | 90 18 718 77 | 2987.35 0.00
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
Aggregated 179572 0(0.00%) | 90 18 718 77 | 2987.35 0.00
Response time percentiles (approximated)
Type Name 50% 66% 75% 80% 90% 95% 98% 99% 99.9% 99.99% 100% # reqs
--------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------
POST /v2/models/topic_classification/infer 77 93 100 110 140 160 210 520 670 710 720 179572
--------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------
Aggregated 77 93 100 110 140 160 210 520 670 710 720 179572
As you can see, with the older version of locust, p95 is 160ms, whereas with the newer version, p95 is 550ms
The locust script, the service being tested are identical.
Hi,
Thanks for your analysis.
The SSL context issue, more likely related to the Python version, not Locust version.
You ramp up is kind of high and can get you into trouble, both with the SSL context (which I think is the main issue), but also with OS level things. Anything above 100/s per worker is kinda extreme (there should be a warning in your logs about that - can you share the rest of your logs too?).
How much worse was it before you downgraded requests? We might want to not use the most recent version until we can construct a workaround. Can you maybe test using FastHttpUser too?
The SSL context issue, more likely related to the Python version, not Locust version.
Which Python version do you recommend to use then?
Anything above 100/s per worker is kinda extreme (there should be a warning in your logs about that - can you share the rest of your logs too?).
Here are the logs from the worker script (running v2.13.0 version of locust):
Worker logs:
root@ai-load-testing-spot-vm-1:~/beta_k8s_us_east5_cluster# [2025-10-16 05:46:35,207] ai-load-testing-spot-vm-1/WARNING/locust.main: System open file limit '1024' is below minimum setting '10000'.
It's not high enough for load testing, and the OS didn't allow locust to increase it by itself.
See https://github.com/locustio/locust/wiki/Installation#increasing-maximum-number-of-open-files-limit for more info.
[2025-10-16 05:46:35,208] ai-load-testing-spot-vm-1/WARNING/locust.main: System open file limit '1024' is below minimum setting '10000'.
It's not high enough for load testing, and the OS didn't allow locust to increase it by itself.
See https://github.com/locustio/locust/wiki/Installation#increasing-maximum-number-of-open-files-limit for more info.
[2025-10-16 05:46:35,210] ai-load-testing-spot-vm-1/WARNING/locust.main: System open file limit '1024' is below minimum setting '10000'.
It's not high enough for load testing, and the OS didn't allow locust to increase it by itself.
See https://github.com/locustio/locust/wiki/Installation#increasing-maximum-number-of-open-files-limit for more info.
[2025-10-16 05:46:35,210] ai-load-testing-spot-vm-1/WARNING/locust.main: System open file limit '1024' is below minimum setting '10000'.
It's not high enough for load testing, and the OS didn't allow locust to increase it by itself.
See https://github.com/locustio/locust/wiki/Installation#increasing-maximum-number-of-open-files-limit for more info.
[2025-10-16 05:46:35,211] ai-load-testing-spot-vm-1/INFO/locust.main: Starting Locust 2.13.0
[2025-10-16 05:46:35,211] ai-load-testing-spot-vm-1/INFO/locust.main: Starting Locust 2.13.0
[2025-10-16 05:46:35,212] ai-load-testing-spot-vm-1/INFO/locust.main: Starting Locust 2.13.0
[2025-10-16 05:46:35,212] ai-load-testing-spot-vm-1/WARNING/locust.main: System open file limit '1024' is below minimum setting '10000'.
It's not high enough for load testing, and the OS didn't allow locust to increase it by itself.
See https://github.com/locustio/locust/wiki/Installation#increasing-maximum-number-of-open-files-limit for more info.
[2025-10-16 05:46:35,213] ai-load-testing-spot-vm-1/WARNING/locust.main: System open file limit '1024' is below minimum setting '10000'.
It's not high enough for load testing, and the OS didn't allow locust to increase it by itself.
See https://github.com/locustio/locust/wiki/Installation#increasing-maximum-number-of-open-files-limit for more info.
[2025-10-16 05:46:35,213] ai-load-testing-spot-vm-1/INFO/locust.main: Starting Locust 2.13.0
[2025-10-16 05:46:35,214] ai-load-testing-spot-vm-1/INFO/locust.main: Starting Locust 2.13.0
[2025-10-16 05:46:35,215] ai-load-testing-spot-vm-1/INFO/locust.main: Starting Locust 2.13.0
Response validation successful: {'model_name': 'topic_classification', 'model_version': '1', 'outputs': [{'name': 'OUTPUT0', 'datatype': 'BYTES', 'shape': [1], 'data': ['[{"labels": ["Entertainment", "Films"]}]']}]}
Response validation successful: {'model_name': 'topic_classification', 'model_version': '1', 'outputs': [{'name': 'OUTPUT0', 'datatype': 'BYTES', 'shape': [1], 'data': ['[{"labels": ["Entertainment", "Films"]}]']}]}
Response validation successful: {'model_name': 'topic_classification', 'model_version': '1', 'outputs': [{'name': 'OUTPUT0', 'datatype': 'BYTES', 'shape': [1], 'data': ['[{"labels": ["Entertainment", "Films"]}]']}]}
...
- The rest of the logs from are all just
Response validation successful: {'model_name': 'topic_classification', 'model_version': '1', 'outputs': [{'name': 'OUTPUT0', 'datatype': 'BYTES', 'shape': [1], 'data': ['[{"labels": ["Entertainment", "Films"]}]']}]}
Master logs:
root@ai-load-testing-spot-vm-1:~/beta_k8s_us_east5_cluster# ./start_benchmarking_service.sh ./profanity_service/topic_model.py 3000 3000 60 6
[2025-10-16 05:46:31,235] ai-load-testing-spot-vm-1/INFO/root: Waiting for workers to be ready, 0 of 6 connected
Type Name # reqs # fails | Avg Min Max Med | req/s failures/s
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
Aggregated 0 0(0.00%) | 0 0 0 0 | 0.00 0.00
[2025-10-16 05:46:32,237] ai-load-testing-spot-vm-1/INFO/root: Waiting for workers to be ready, 0 of 6 connected
Type Name # reqs # fails | Avg Min Max Med | req/s failures/s
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
Aggregated 0 0(0.00%) | 0 0 0 0 | 0.00 0.00
[2025-10-16 05:46:33,238] ai-load-testing-spot-vm-1/INFO/root: Waiting for workers to be ready, 0 of 6 connected
[2025-10-16 05:46:34,239] ai-load-testing-spot-vm-1/INFO/root: Waiting for workers to be ready, 0 of 6 connected
[2025-10-16 05:46:35,210] ai-load-testing-spot-vm-1/INFO/locust.runners: Worker ai-load-testing-spot-vm-1_93b61a3a34cb4d31a837065a21f740c4 (index 0) reported as ready. 1 workers connected.
[2025-10-16 05:46:35,210] ai-load-testing-spot-vm-1/INFO/locust.runners: Worker ai-load-testing-spot-vm-1_abdd684258e049b589f83fc3899132d8 (index 1) reported as ready. 2 workers connected.
[2025-10-16 05:46:35,212] ai-load-testing-spot-vm-1/INFO/locust.runners: Worker ai-load-testing-spot-vm-1_b0b9536e262c4275ac930c44b7a5871b (index 2) reported as ready. 3 workers connected.
[2025-10-16 05:46:35,212] ai-load-testing-spot-vm-1/INFO/locust.runners: Worker ai-load-testing-spot-vm-1_335d1c0ead7643cda8451a41f3442d18 (index 3) reported as ready. 4 workers connected.
[2025-10-16 05:46:35,214] ai-load-testing-spot-vm-1/INFO/locust.runners: Worker ai-load-testing-spot-vm-1_538ffac4ebd94d1390eef647215b33b6 (index 4) reported as ready. 5 workers connected.
[2025-10-16 05:46:35,214] ai-load-testing-spot-vm-1/INFO/locust.runners: Worker ai-load-testing-spot-vm-1_9ff3582cf421445b9e31b7c18bd10972 (index 5) reported as ready. 6 workers connected.
Type Name # reqs # fails | Avg Min Max Med | req/s failures/s
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
Aggregated 0 0(0.00%) | 0 0 0 0 | 0.00 0.00
[2025-10-16 05:46:35,240] ai-load-testing-spot-vm-1/INFO/locust.main: Run time limit set to 60 seconds
[2025-10-16 05:46:35,240] ai-load-testing-spot-vm-1/INFO/locust.main: Starting Locust 2.13.0
[2025-10-16 05:46:35,241] ai-load-testing-spot-vm-1/INFO/locust.runners: Sending spawn jobs of 3000 users at 3000.00 spawn rate to 6 ready workers
[2025-10-16 05:46:35,241] ai-load-testing-spot-vm-1/WARNING/locust.runners: Your selected spawn rate is very high (>100/worker), and this is known to sometimes cause issues. Do you really need to ramp up that fast?
[2025-10-16 05:46:35,348] ai-load-testing-spot-vm-1/INFO/locust.runners: All users spawned: {"TopicClassificationUser": 3000} (3000 total users)
Type Name # reqs # fails | Avg Min Max Med | req/s failures/s
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
Aggregated 0 0(0.00%) | 0 0 0 0 | 0.00 0.00
Type Name # reqs # fails | Avg Min Max Med | req/s failures/s
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
POST /v2/models/topic_classification/infer 8030 0(0.00%) | 273 29 792 120 | 1481.00 0.00
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
Aggregated 8030 0(0.00%) | 273 29 792 120 | 1481.00 0.00
Type Name # reqs # fails | Avg Min Max Med | req/s failures/s
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
POST /v2/models/topic_classification/infer 15578 0(0.00%) | 189 26 792 100 | 2442.75 0.00
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
Aggregated 15578 0(0.00%) | 189 26 792 100 | 2442.75 0.00
Type Name # reqs # fails | Avg Min Max Med | req/s failures/s
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
POST /v2/models/topic_classification/infer 17105 0(0.00%) | 179 26 792 100 | 2518.25 0.00
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
Aggregated 17105 0(0.00%) | 179 26 792 100 | 2518.25 0.00
Type Name # reqs # fails | Avg Min Max Med | req/s failures/s
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
POST /v2/models/topic_classification/infer 26210 0(0.00%) | 152 26 792 97 | 2709.71 0.00
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
Aggregated 26210 0(0.00%) | 152 26 792 97 | 2709.71 0.00
Type Name # reqs # fails | Avg Min Max Med | req/s failures/s
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
POST /v2/models/topic_classification/infer 29225 0(0.00%) | 144 26 792 93 | 2677.60 0.00
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
Aggregated 29225 0(0.00%) | 144 26 792 93 | 2677.60 0.00
Type Name # reqs # fails | Avg Min Max Med | req/s failures/s
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
POST /v2/models/topic_classification/infer 35305 0(0.00%) | 135 26 792 91 | 2804.80 0.00
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
Aggregated 35305 0(0.00%) | 135 26 792 91 | 2804.80 0.00
Type Name # reqs # fails | Avg Min Max Med | req/s failures/s
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
POST /v2/models/topic_classification/infer 44266 0(0.00%) | 127 26 792 91 | 3005.60 0.00
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
Aggregated 44266 0(0.00%) | 127 26 792 91 | 3005.60 0.00
Type Name # reqs # fails | Avg Min Max Med | req/s failures/s
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
POST /v2/models/topic_classification/infer 45796 0(0.00%) | 125 26 792 90 | 2850.70 0.00
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
Aggregated 45796 0(0.00%) | 125 26 792 90 | 2850.70 0.00
Type Name # reqs # fails | Avg Min Max Med | req/s failures/s
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
POST /v2/models/topic_classification/infer 53179 0(0.00%) | 121 26 792 89 | 3002.10 0.00
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
Aggregated 53179 0(0.00%) | 121 26 792 89 | 3002.10 0.00
Type Name # reqs # fails | Avg Min Max Med | req/s failures/s
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
POST /v2/models/topic_classification/infer 62273 0(0.00%) | 117 26 792 89 | 2995.00 0.00
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
Aggregated 62273 0(0.00%) | 117 26 792 89 | 2995.00 0.00
Type Name # reqs # fails | Avg Min Max Med | req/s failures/s
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
POST /v2/models/topic_classification/infer 62273 0(0.00%) | 117 26 792 89 | 2995.00 0.00
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
Aggregated 62273 0(0.00%) | 117 26 792 89 | 2995.00 0.00
Type Name # reqs # fails | Avg Min Max Med | req/s failures/s
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
POST /v2/models/topic_classification/infer 71160 0(0.00%) | 116 26 792 90 | 2991.60 0.00
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
Aggregated 71160 0(0.00%) | 116 26 792 90 | 2991.50 0.00
Type Name # reqs # fails | Avg Min Max Med | req/s failures/s
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
POST /v2/models/topic_classification/infer 80264 0(0.00%) | 113 26 792 90 | 2988.60 0.00
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
Aggregated 80264 0(0.00%) | 113 26 792 90 | 2988.60 0.00
Type Name # reqs # fails | Avg Min Max Med | req/s failures/s
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
POST /v2/models/topic_classification/infer 80264 0(0.00%) | 113 26 792 90 | 2988.60 0.00
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
Aggregated 80264 0(0.00%) | 113 26 792 90 | 2988.60 0.00
Type Name # reqs # fails | Avg Min Max Med | req/s failures/s
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
POST /v2/models/topic_classification/infer 89303 0(0.00%) | 111 26 792 89 | 2992.50 0.00
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
Aggregated 89303 0(0.00%) | 111 26 792 89 | 2992.50 0.00
Type Name # reqs # fails | Avg Min Max Med | req/s failures/s
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
POST /v2/models/topic_classification/infer 98444 0(0.00%) | 110 26 792 89 | 3011.60 0.00
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
Aggregated 98444 0(0.00%) | 110 26 792 89 | 3011.60 0.00
Type Name # reqs # fails | Avg Min Max Med | req/s failures/s
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
POST /v2/models/topic_classification/infer 98444 0(0.00%) | 110 26 792 89 | 3011.60 0.00
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
Aggregated 98444 0(0.00%) | 110 26 792 89 | 3011.60 0.00
Type Name # reqs # fails | Avg Min Max Med | req/s failures/s
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
POST /v2/models/topic_classification/infer 107342 0(0.00%) | 109 26 792 89 | 3000.20 0.00
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
Aggregated 107342 0(0.00%) | 109 26 792 89 | 3000.20 0.00
Type Name # reqs # fails | Avg Min Max Med | req/s failures/s
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
POST /v2/models/topic_classification/infer 116350 0(0.00%) | 107 26 792 88 | 2998.40 0.00
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
Aggregated 116350 0(0.00%) | 107 26 792 88 | 2998.40 0.00
Type Name # reqs # fails | Avg Min Max Med | req/s failures/s
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
POST /v2/models/topic_classification/infer 116350 0(0.00%) | 107 26 792 88 | 2998.40 0.00
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
Aggregated 116350 0(0.00%) | 107 26 792 88 | 2998.40 0.00
Type Name # reqs # fails | Avg Min Max Med | req/s failures/s
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
POST /v2/models/topic_classification/infer 125555 0(0.00%) | 107 26 792 89 | 2990.80 0.00
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
Aggregated 125555 0(0.00%) | 107 26 792 89 | 2990.80 0.00
Type Name # reqs # fails | Avg Min Max Med | req/s failures/s
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
POST /v2/models/topic_classification/infer 134523 0(0.00%) | 106 26 792 90 | 3001.20 0.00
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
Aggregated 134523 0(0.00%) | 106 26 792 90 | 3001.20 0.00
Type Name # reqs # fails | Avg Min Max Med | req/s failures/s
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
POST /v2/models/topic_classification/infer 134523 0(0.00%) | 106 26 792 90 | 3001.20 0.00
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
Aggregated 134523 0(0.00%) | 106 26 792 90 | 3001.20 0.00
Type Name # reqs # fails | Avg Min Max Med | req/s failures/s
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
POST /v2/models/topic_classification/infer 143504 0(0.00%) | 106 26 792 90 | 2994.00 0.00
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
Aggregated 143504 0(0.00%) | 106 26 792 90 | 2994.00 0.00
Type Name # reqs # fails | Avg Min Max Med | req/s failures/s
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
POST /v2/models/topic_classification/infer 152642 0(0.00%) | 106 26 792 90 | 3003.70 0.00
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
Aggregated 152642 0(0.00%) | 106 26 792 90 | 3003.70 0.00
Type Name # reqs # fails | Avg Min Max Med | req/s failures/s
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
POST /v2/models/topic_classification/infer 152642 0(0.00%) | 106 26 792 90 | 3003.70 0.00
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
Aggregated 152642 0(0.00%) | 106 26 792 90 | 3003.70 0.00
Type Name # reqs # fails | Avg Min Max Med | req/s failures/s
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
POST /v2/models/topic_classification/infer 161526 0(0.00%) | 105 26 792 90 | 2993.70 0.00
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
Aggregated 161526 0(0.00%) | 105 26 792 90 | 2993.80 0.00
Type Name # reqs # fails | Avg Min Max Med | req/s failures/s
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
POST /v2/models/topic_classification/infer 170700 0(0.00%) | 105 26 792 90 | 2994.30 0.00
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
Aggregated 170700 0(0.00%) | 105 26 792 90 | 2994.30 0.00
[2025-10-16 05:47:35,240] ai-load-testing-spot-vm-1/INFO/locust.main: --run-time limit reached, shutting down
Type Name # reqs # fails | Avg Min Max Med | req/s failures/s
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
POST /v2/models/topic_classification/infer 170700 0(0.00%) | 105 26 792 90 | 2994.30 0.00
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
Aggregated 170700 0(0.00%) | 105 26 792 90 | 2994.30 0.00
[2025-10-16 05:47:35,290] ai-load-testing-spot-vm-1/INFO/locust.runners: Worker 'ai-load-testing-spot-vm-1_335d1c0ead7643cda8451a41f3442d18' (index 3) quit. 0 workers ready.
[2025-10-16 05:47:35,291] ai-load-testing-spot-vm-1/INFO/locust.runners: Worker 'ai-load-testing-spot-vm-1_9ff3582cf421445b9e31b7c18bd10972' (index 5) quit. 0 workers ready.
[2025-10-16 05:47:35,325] ai-load-testing-spot-vm-1/INFO/locust.runners: Worker 'ai-load-testing-spot-vm-1_538ffac4ebd94d1390eef647215b33b6' (index 4) quit. 0 workers ready.
[2025-10-16 05:47:35,332] ai-load-testing-spot-vm-1/INFO/locust.runners: Worker 'ai-load-testing-spot-vm-1_93b61a3a34cb4d31a837065a21f740c4' (index 0) quit. 0 workers ready.
[2025-10-16 05:47:35,335] ai-load-testing-spot-vm-1/INFO/locust.runners: Worker 'ai-load-testing-spot-vm-1_b0b9536e262c4275ac930c44b7a5871b' (index 2) quit. 0 workers ready.
[2025-10-16 05:47:35,378] ai-load-testing-spot-vm-1/INFO/locust.runners: Worker 'ai-load-testing-spot-vm-1_abdd684258e049b589f83fc3899132d8' (index 1) quit. 0 workers ready.
[2025-10-16 05:47:35,379] ai-load-testing-spot-vm-1/INFO/locust.runners: The last worker quit, stopping test.
[2025-10-16 05:47:36,242] ai-load-testing-spot-vm-1/INFO/locust.main: Shutting down (exit code 0)
Type Name # reqs # fails | Avg Min Max Med | req/s failures/s
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
POST /v2/models/topic_classification/infer 179367 0(0.00%) | 105 26 792 91 | 2986.42 0.00
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
Aggregated 179367 0(0.00%) | 105 26 792 91 | 2986.42 0.00
Response time percentiles (approximated)
Type Name 50% 66% 75% 80% 90% 95% 98% 99% 99.9% 99.99% 100% # reqs
--------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------
POST /v2/models/topic_classification/infer 91 110 120 130 160 190 250 560 750 790 790 179367
--------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------
Aggregated 91 110 120 130 160 190 250 560 750 790 790 179367
Here are the logs for v2.41.5 of locust:
(python3.12_env) root@ai-load-testing-spot-vm-1:~/beta_k8s_us_east5_cluster/profanity_service# locust -f topic_model.py --users 3000 --spawn-rate 3000 --run-time 60s --headless --csv=results --processes 7
[2025-10-16 06:00:00,146] ai-load-testing-spot-vm-1/INFO/locust.main: Starting Locust 2.41.5
[2025-10-16 06:00:00,182] ai-load-testing-spot-vm-1/INFO/locust.runners: ai-load-testing-spot-vm-1_13ebc7131e1348d58e620435ec7d7efb (index 0) reported as ready. 1 workers connected.
[2025-10-16 06:00:00,183] ai-load-testing-spot-vm-1/INFO/locust.runners: ai-load-testing-spot-vm-1_159ce85216e84231aff21604db7f5137 (index 1) reported as ready. 2 workers connected.
[2025-10-16 06:00:00,183] ai-load-testing-spot-vm-1/INFO/locust.runners: ai-load-testing-spot-vm-1_e40036979e0949ee93d1a119aa5f1e28 (index 2) reported as ready. 3 workers connected.
[2025-10-16 06:00:00,183] ai-load-testing-spot-vm-1/INFO/locust.runners: ai-load-testing-spot-vm-1_24442440e2c94ada9238fa8e30007c03 (index 3) reported as ready. 4 workers connected.
[2025-10-16 06:00:00,184] ai-load-testing-spot-vm-1/INFO/locust.runners: ai-load-testing-spot-vm-1_886eb538c5e544bb95dcd954c808de82 (index 4) reported as ready. 5 workers connected.
[2025-10-16 06:00:00,184] ai-load-testing-spot-vm-1/INFO/locust.runners: ai-load-testing-spot-vm-1_33ce861996074d60a040c1e493fdb60b (index 5) reported as ready. 6 workers connected.
Type Name # reqs # fails | Avg Min Max Med | req/s failures/s
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
Aggregated 0 0(0.00%) | 0 0 0 0 | 0.00 0.00
[2025-10-16 06:00:00,364] ai-load-testing-spot-vm-1/INFO/locust.runners: ai-load-testing-spot-vm-1_2319c0a8a1774cbf8ed12a84d0f95f9d (index 6) reported as ready. 7 workers connected.
[2025-10-16 06:00:01,183] ai-load-testing-spot-vm-1/INFO/locust.main: Run time limit set to 60 seconds
[2025-10-16 06:00:01,183] ai-load-testing-spot-vm-1/INFO/locust.runners: Sending spawn jobs of 3000 users at 3000.00 spawn rate to 7 ready workers
[2025-10-16 06:00:01,183] ai-load-testing-spot-vm-1/WARNING/locust.runners: Your selected spawn rate is very high (>100/worker), and this is known to sometimes cause issues. Do you really need to ramp up that fast?
[2025-10-16 06:00:01,332] ai-load-testing-spot-vm-1/INFO/locust.runners: All users spawned: {"TopicClassificationUser": 3000} (3000 total users)
Response validation successful: {'model_name': 'topic_classification', 'model_version': '1', 'outputs': [{'name': 'OUTPUT0', 'datatype': 'BYTES', 'shape': [1], 'data': ['[{"labels": ["Entertainment", "Films"]}]']}]}
Response validation successful: {'model_name': 'topic_classification', 'model_version': '1', 'outputs': [{'name': 'OUTPUT0', 'datatype': 'BYTES', 'shape': [1], 'data': ['[{"labels": ["Entertainment", "Films"]}]']}]}
Response validation successful: {'model_name': 'topic_classification', 'model_version': '1', 'outputs': [{'name': 'OUTPUT0', 'datatype': 'BYTES', 'shape': [1], 'data': ['[{"labels": ["Entertainment", "Films"]}]']}]}
....
Response validation successful: {'model_name': 'topic_classification', 'model_version': '1', 'outputs': [{'name': 'OUTPUT0', 'datatype': 'BYTES', 'shape': [1], 'data': ['[{"labels": ["Entertainment", "Films"]}]']}]}
Response validation successful: {'model_name': 'topic_classification', 'model_version': '1', 'outputs': [{'name': 'OUTPUT0', 'datatype': 'BYTES', 'shape': [1], 'data': ['[{"labels": ["Entertainment", "Films"]}]']}]}
[2025-10-16 06:01:01,183] ai-load-testing-spot-vm-1/INFO/locust.main: --run-time limit reached, shutting down
[2025-10-16 06:01:01,184] ai-load-testing-spot-vm-1/INFO/locust.runners: Got quit message from master, shutting down...
[2025-10-16 06:01:01,184] ai-load-testing-spot-vm-1/INFO/locust.runners: Got quit message from master, shutting down...
[2025-10-16 06:01:01,184] ai-load-testing-spot-vm-1/INFO/locust.runners: Got quit message from master, shutting down...
[2025-10-16 06:01:01,184] ai-load-testing-spot-vm-1/INFO/locust.runners: Got quit message from master, shutting down...
[2025-10-16 06:01:01,184] ai-load-testing-spot-vm-1/INFO/locust.runners: Got quit message from master, shutting down...
[2025-10-16 06:01:01,184] ai-load-testing-spot-vm-1/INFO/locust.runners: Got quit message from master, shutting down...
[2025-10-16 06:01:01,184] ai-load-testing-spot-vm-1/INFO/locust.runners: Got quit message from master, shutting down...
[2025-10-16 06:01:01,239] ai-load-testing-spot-vm-1/INFO/locust.main: Shutting down (exit code 0)
[2025-10-16 06:01:01,239] ai-load-testing-spot-vm-1/INFO/locust.runners: Worker 'ai-load-testing-spot-vm-1_24442440e2c94ada9238fa8e30007c03' (index 3) quit. 0 workers ready.
[2025-10-16 06:01:01,242] ai-load-testing-spot-vm-1/INFO/locust.main: Shutting down (exit code 0)
[2025-10-16 06:01:01,242] ai-load-testing-spot-vm-1/INFO/locust.runners: Worker 'ai-load-testing-spot-vm-1_13ebc7131e1348d58e620435ec7d7efb' (index 0) quit. 0 workers ready.
[2025-10-16 06:01:01,242] ai-load-testing-spot-vm-1/INFO/locust.main: Shutting down (exit code 0)
[2025-10-16 06:01:01,242] ai-load-testing-spot-vm-1/INFO/locust.runners: Worker 'ai-load-testing-spot-vm-1_2319c0a8a1774cbf8ed12a84d0f95f9d' (index 6) quit. 0 workers ready.
[2025-10-16 06:01:01,266] ai-load-testing-spot-vm-1/INFO/locust.main: Shutting down (exit code 0)
[2025-10-16 06:01:01,267] ai-load-testing-spot-vm-1/INFO/locust.runners: Worker 'ai-load-testing-spot-vm-1_33ce861996074d60a040c1e493fdb60b' (index 5) quit. 0 workers ready.
[2025-10-16 06:01:01,267] ai-load-testing-spot-vm-1/INFO/locust.main: Shutting down (exit code 0)
[2025-10-16 06:01:01,267] ai-load-testing-spot-vm-1/INFO/locust.runners: Worker 'ai-load-testing-spot-vm-1_886eb538c5e544bb95dcd954c808de82' (index 4) quit. 0 workers ready.
[2025-10-16 06:01:01,280] ai-load-testing-spot-vm-1/INFO/locust.main: Shutting down (exit code 0)
[2025-10-16 06:01:01,280] ai-load-testing-spot-vm-1/INFO/locust.main: Shutting down (exit code 0)
[2025-10-16 06:01:01,280] ai-load-testing-spot-vm-1/INFO/locust.runners: Worker 'ai-load-testing-spot-vm-1_e40036979e0949ee93d1a119aa5f1e28' (index 2) quit. 0 workers ready.
[2025-10-16 06:01:01,280] ai-load-testing-spot-vm-1/INFO/locust.runners: Worker 'ai-load-testing-spot-vm-1_159ce85216e84231aff21604db7f5137' (index 1) quit. 0 workers ready.
[2025-10-16 06:01:01,280] ai-load-testing-spot-vm-1/INFO/locust.runners: The last worker quit, stopping test.
[2025-10-16 06:01:02,186] ai-load-testing-spot-vm-1/INFO/locust.main: Shutting down (exit code 0)
Type Name # reqs # fails | Avg Min Max Med | req/s failures/s
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
POST /v2/models/topic_classification/infer 180000 0(0.00%) | 372 108 702 370 | 3008.50 0.00
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
Aggregated 180000 0(0.00%) | 372 108 702 370 | 3008.50 0.00
Response time percentiles (approximated)
Type Name 50% 66% 75% 80% 90% 95% 98% 99% 99.9% 99.99% 100% # reqs
--------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------
POST /v2/models/topic_classification/infer 370 440 480 500 530 550 570 580 630 690 700 180000
--------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------
Aggregated 370 440 480 500 530 550 570 580 630 690 700 180000
How much worse was it before you downgraded requests?
Type Name # reqs # fails | Avg Min Max Med | req/s failures/s
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
POST /v2/models/topic_classification/infer 180000 0(0.00%) | 371 84 771 370 | 3008.79 0.00
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
Aggregated 180000 0(0.00%) | 371 84 771 370 | 3008.79 0.00
Response time percentiles (approximated)
Type Name 50% 66% 75% 80% 90% 95% 98% 99% 99.9% 99.99% 100% # reqs
--------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------
POST /v2/models/topic_classification/infer 370 440 480 500 530 550 580 590 630 750 770 180000
--------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------
Aggregated 370 440 480 500 530 550 580 590 630 750 770 180000
- These are results when using locust v2.41.5, and requests v2.32.5. These are similar to when using requests 2.32.4. It turns out that eariler I had seen latencies of p95 of 50 seconds when these versions cause I was running the test on VSCode terminal.. as soon as I moved to an independent terminal, the results changed to the ones pasted above.. however, its still much worse than when using locust v2.13.0.. so there is still some issue.
Can you maybe test using FastHttpUser too?
Type Name # reqs # fails | Avg Min Max Med | req/s failures/s
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
POST /v2/models/topic_classification/infer 180000 0(0.00%) | 279 57 630 280 | 3013.56 0.00
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
Aggregated 180000 0(0.00%) | 279 57 630 280 | 3013.56 0.00
Response time percentiles (approximated)
Type Name 50% 66% 75% 80% 90% 95% 98% 99% 99.9% 99.99% 100% # reqs
--------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------
POST /v2/models/topic_classification/infer 280 360 400 420 470 490 510 520 600 620 630 180000
--------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------
Aggregated 280 360 400 420 470 490 510 520 600 620 630 180000
- This was with v2.41.5 of locust, requests 2.32.4. The results are very similar to when just using HttpUser.
One clarification, I tried locust version 2.13.0, with requests 2.32.3 and python version 3.9.24 + FastHttpUser, and that worked very well! I could do 1000 rps on a single worker, and the p95 latency was lesser than 100ms.. which is what we expected from the service being tested!! So for now, we will stick to that configuration.
Uhm... This is an important error message, please fix this issue before doing anything else.
[2025-10-16 05:46:35,207] ai-load-testing-spot-vm-1/WARNING/locust.main: System open file limit '1024' is below minimum setting '10000'.
It's not high enough for load testing, and the OS didn't allow locust to increase it by itself.
See https://github.com/locustio/locust/wiki/Installation#increasing-maximum-number-of-open-files-limit for more info.
This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 20 days.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}