"ERROR: Not enough resources: free memory less than 512MB: 184MB" with large ammos

[citizen-stig]

I often get mentioned error with a large ammo file. I have more than 4G memory available on host. I get error after about 10-15 seconds after start. Error doesn't appear regularly

Example of ammo https://gist.github.com/citizen-stig/d9582e22a31ee0462cb5

My load.ini:

[phantom]
instances = 60000
address = 10.11.1.100:8080
rps_schedule = const (2, 4004) 
writelog = all 
threads = 10
[aggregator]
time_periods = 1 2 3 4 5 6 7 8 9 10 20 30 40 50 60 70 80 90 100 150 200 250 300 350 400 450 500 600 700 750 800 900 1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 

Is it possible to fix this?

[nettorta]

1. Can you attach tank.log ?
2. You can disable plugin

[tank] plugin_rcheck =

and/or specify options for this plugin: http://yandextank.readthedocs.org/en/latest/configuration.html#resource-check

[citizen-stig]

1. tank.log -https://gist.github.com/citizen-stig/393d429b913bc6cbea65
2. I've opened htop for checking resources and it actually consumes almost all resources. The main question, why this happens. If I turn off this plugin, host will fail with no memory error.

[direvius]

@citizen-stig try specifying less instances, for example:

instances=1000

[citizen-stig]

@direvius, so technically, yandex tank cannot handle high load with a big ammo? I mean if I really need a lot of instances, I will get this error if ammo file is pretty big?

[direvius]

@citizen-stig you have your timeout set for 10s and load of 2 requests per second. The maximum number of instances that can be used in such circumstances is 10 * 2 = 20. Why do you need 60000?

Also, it is not Yandex.Tank who can't handle high load with big ammo. It is actually your server. We had some tests with 100К instances and 40GB of ammo. But we also have more then 100 GB of RAM. It is strange to expect that you get a lot of instances for free. Yes, they are lightweight, but not zero weight.

[citizen-stig]

@direvius, 2 requests per second set for easier reproducing. I've created this bug, because it reproduced not constantly, and when yandex.tank can start without this error, resource usage is very low, even on load much bigger than 2 seconds. I mean, that problem can be in process of generating of .stpd files.

[direvius]

@citizen-stig well, according to tank.log the stpd generation is finished long before you get that error. Would you please collect the information about memory usage of each of tank's processes and post it here? (including phantom)

[citizen-stig]

I'll try to provide this information soon.