Phase duration exceeded. Cannot launch all the users

[deanstef]

trafficstars

Hi! I am experiencing Phase duration exceeded, but not all users were launched (334 users, 5.6% of phase) error. I am trying to launch a number of users sending HTTP POSTs to a server, querying different ports. This is a snap of my tsung xml:

<!-- Client side setup -->
  <clients>
    <client host="localhost" use_controller_vm="true" maxusers="30000"/>
  </clients>

  <!-- Server side setup -->
  <servers>
    <server host="10.22.36.107" port="8545" type="tcp"></server>
    <server host="10.22.36.107" port="8547" type="tcp"></server>
    <server host="10.22.36.107" port="8549" type="tcp"></server>
    <server host="10.22.36.107" port="8551" type="tcp"></server>
  </servers>

  <load>
    <arrivalphase phase="1" duration="1" unit="minute">
        <users arrivalrate="100" unit="second"></users>
    </arrivalphase>
  </load>

From controller log seems that due to phase duration exceed not all users are launched. What causes phase duration exceed?

[tisba]

Okay, let's continue here :)

Have you made sure that you are not overloading the controller instance? Are there any other errors or messages in the log files?

[deanstef]

Yes better here sorry :)

I am not sure if I am overloading the controller. How can I check it? I don't have other errors neither in the controller log (just some alarm_handler messages) nor in the tsung.log file.

[tisba]

You can use tsung to monitor itself, but you can also consult your "normal" system monitoring (or worst case just do a top).

[deanstef]

Ok. I am new to tsung, do I need to monitor the client where I launch my workload (i.e. localhost)? If yes, this is my server load:

[tisba]

You have to monitor everything basically. You might want to monitor the system under test (your target) as well as the systems generating the load (including the Tsung controller).

[tisba]

hmm, interesting. That doesn't really look that overloaded. What version of tsung are you running?

Also you could try to extend the test period and/or limit the arrival rate. The rate is relatively high, especially if you are running. But if the defined sessions are not that complex it should be doable.

[deanstef]

I am sorry, in my snap there was a typo. The arrival rate for such OS monitoring is arrivalrate="100". However I am experiencing same issues with higher rates (like 1000) which I definitely need to test.

I am running the latest version of tsung. I have 4 sessions like that:

<session name="http_n3-n4" probability="25" type="ts_http">
        <!-- transaction from node2 to node0 -->
        <request> 
                <dyn_variable name="my_response3" re="[\s\S]*?(.*jsonrpc.*)" />
                <http url="http://10.22.36.107:8549/" method="POST" version="1.1" content_type='application/json' contents='--JSON CONTENT--'></http>
        </request>
        <setdynvars sourcetype="eval"
            code='fun({Pid,DynVars})->
                {ok, Res}=ts_dynvars:lookup(my_response3,DynVars),
                Timestamp=os:system_time(millisecond),
                io:format("~w, ~p~n", [Timestamp, Res]) end.'>
            <var name="resp" />
        </setdynvars>
    </session>

I will try now with extended test period.

[deanstef]

I tried increasing load duration and phase duration. In both tests same error Phase duration exceeded.

[tisba]

The error is expected, as you are actively terminating the test by limiting the duration. The question is, if you managed to launch more clients

[deanstef]

No, and I am afraid this is my problem. I am also using one client (localhost) as you can see above from my configuration. Indeed, whit arrivalrate>=1000 I am also experiencing error_connect_eaddrinuse and error_abort_max_conn_retries.

I think I am missing something in my clients config.

[tisba]

eaddrinuse is an indicator that you are running out of ephemeral ports.

Keep in mind, that every user you launch, will create a new connection, makes one request and finishes. Maybe that is not what you want to achieve. You can e.g. make multiple requests per client (which will re-use the connection if the target supports it).

[deanstef]

I just want to stress my server with a number of requests and observe the throughput. Whats the best way for that? Do I need to use multiple clients even if I have just one machine running tsung?

[tisba]

Without knowing more details that is not straight forward to answer.

The first question is if you want to stress your server including the connection establishment. If this is the case, it might be better to have more servers where traffic is originating from. If you "just" want to make HTTP requests and do not care about re-using the connection, you can launch let's say 10 users, and let everyone make 100 requests. You could use <for> for this combined with <think>.

[deanstef]

I need to send a number of HTTP posts to my server. Thus, I need to print out on a file part of each response from the server. However, I might want to test also loads with connection establishment. I have two questions regarding your solutions:

1. Is not clear to me how can I simulate more machines to generate traffic (virtual IPs?)
2. Should I use <for> inside each session? Furthermore, why do I need <think>? It will delay the requests and that's not what I want.

[tisba]

1. one option would be described here: http://tsung.erlang-projects.org/user_manual/conf-client-server.html#advanced-setup - the other is basically get more (virtual) servers :)
2. You can use <think> with <for> to get a approximated request rate per session

[deanstef]

1. Ok. I am planning to use one host so I will try the first option. Thank you :)
2. Is there any example for that?

[tisba]

Please see the documentation on how to use these options.

You don't need the wait time. It's sometimes useful to get a approximate rate. If you make one request and wait for one second, you make 1 request per second per active user. If you have 1000 active clients, you make 1000 requests per second.

Try to build something like:

for { request; think }

[deanstef]

Hi @tisba , I managed to send transactions at an approximate rate using <for> and <think> as you suggested. Actually I have k users sending n requests per second in an infinite loop. This works well with low number of users. For big number of users (e.g. 500+) tsung do not behave as expected, indeed starts to send less requests than the k/s (with k users).

Following you can find the measurements for 300 and 500 users. As you can observe in the second test tsung does not send 500 req/s as expected. Do you have any suggestion?

Here the result with 300 users:

and here with 500 users:

[tisba]

I do rarely use Tsung in controler-only mode. But I find it a bit hard to believe that we are hitting a limit here. In distributed mode I can assure you that it scales very well to -much- higher numbers.

That being said, if you cannot share more details, like full configuration, log files etc. it's very hard to figure out what is going on.

[deanstef]

Thank you @tisba . Attached you can find the full configuration and the logs for the test with 500 users.

tsung-parity_transTest-loopNOERL.txt

[email protected]

tsung.log

[deanstef]

Hey @tisba is there a way in tsung to log the Requests that are made? The report seems based on the throughput, so the responses received by the server. What if the server ignores some request? My impression is that my tsung script works well, so I just want to figure out the input rate I am generating with tsung.

[MoorthiRaj]

Hey @tisba @deanstef, I am new to tsung tool and lagging in understanding the report. Can you help me with the same.

Also I have one more request, I have achieved 100k concurrent users chat in distributed load testing. Here the arrivalrate is 100/s. If I update the arrivalrate to 200 users are not getting created more than 85k+ and test run does not stopped.

Can you tell me any solution for this as 'Mean' value for tr_authenticate, tr_Chat, tr_close are high.

Note: I have attached the script file and report screenshot here.

Thank you.

tsung-100k-users-chat-script.txt

[MoorthiRaj]

For this query - "Can you tell me any solution for this as 'Mean' value for tr_authenticate, tr_Chat, tr_close are high." I found the solution. I have modified the tsung script and adjusted the average response time i.e. Mean for all the transactions.