WATCHDOG violation when calling get_stats() in python automation API in STL - regression stability

[syaakov]

Setup details:

Setups details	Description
Server:	UCSC-C240-M4SX
CPU:	1 x Intel(R) Xeon(R) CPU E5-2667 v3 @ 3.20GHz
Ports count	2 x 10Gbps @ VMXNET3 Ethernet Controller
RAM:	4440 MB
NICs:	E1000
OS:	Fedora 21

Error message:

In test_pause_resume_update_dynamic_profile.
Error:
Traceback (most recent call last):
File "/tmp/trex-scripts/automation/regression/stateless_tests/stl_client_test.py", line 881, in test_pause_resume_update_dynamic_profile
self.pause_resume_update_streams_iteration(delay = 5, expected_pps = 25000)
File "/tmp/trex-scripts/automation/regression/stateless_tests/stl_client_test.py", line 387, in pause_resume_update_streams_iteration
tx_pps = self.c.get_stats(ports = [0])[0]['opackets'] / delay
File "/tmp/trex-scripts/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py", line 51, in wrap2
ret = f(*args, **kwargs)
File "/tmp/trex-scripts/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py", line 1455, in get_stats
output = self._get_stats_common(ports, sync_now)
File "/tmp/trex-scripts/automation/trex_control_plane/interactive/trex/common/trex_client.py", line 2827, in _get_stats_common
raise TRexError(rc)
trex.common.trex_exceptions.TRexError: *** [RPC] - Failed to send message to server general exception

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/tmp/trex-scripts/automation/regression/stateless_tests/stl_client_test.py", line 901, in test_pause_resume_update_dynamic_profile
assert False , '{0}'.format(e)
AssertionError: *** [RPC] - Failed to send message to server general exception

During handling of the above exception, another exception occurred

Traceback:

Traceback (most recent call last):
File "/usr/lib64/python3.6/unittest/case.py", line 60, in testPartExecutor
yield
File "/usr/lib64/python3.6/unittest/case.py", line 622, in run
testMethod()
File "/tmp/trex-scripts/automation/regression/stateless_tests/stl_client_test.py", line 904, in test_pause_resume_update_dynamic_profile
self.cleanup()
File "/tmp/trex-scripts/automation/regression/stateless_tests/stl_client_test.py", line 215, in cleanup
self.c.remove_all_captures()
File "/tmp/trex-scripts/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py", line 48, in wrap2
raise TRexError("'{0}' - is not valid while disconnected".format(func_name))
trex.common.trex_exceptions.TRexError: 'remove_all_captures' - is not valid while disconnected

Standard output:

Stdout:

>>>>>>>>>>>>>>> Daemon log <<<<<<<<<<<<<<<
11-21 09:10:49 trex_server INFO Processing get_trex_log() command.
11-21 09:11:07 trex_server INFO Processing get_trex_daemon_log() command.

>>>>>>>>>>>>>>>> Trex log <<<<<<<<<<<<<<<<
Starting Scapy server..... Scapy server is started
The ports are bound/configured.
Starting TRex v2.92 please wait ...
set driver name net_vmxnet3
driver capability : TSO
set dpdk queues mode to ONE_QUE
Number of ports found: 2
zmq publisher at: tcp://*:4500
wait 5 sec .....
port : 0
------------
link : link : Link Up - speed 10000 Mbps - full-duplex
promiscuous : 0
port : 1
------------
link : link : Link Up - speed 10000 Mbps - full-duplex
promiscuous : 0
number of ports : 2
max cores for 2 ports : 1
tx queues per port : 3
-------------------------------
RX core uses TX queue number 65535 on all ports
core, c-port, c-queue, s-port, s-queue, lat-queue
------------------------------------------
1 0 0 1 0 0
-------------------------------
Cleanup of old namespaces related to Linux-based stack
Cleanup Done


*** WATCHDOG violation detected on task 'ZMQ sync request-response' which have failed to response to the signal ***