Large output from event scripts may cripple RabbitMQ or ems

[KLuka]

Hi, we were stress testing OpenBaton and we found that when life-cycle event scripts produce large outputs (>2MB) it is possible that some issues will appear. I guess that large outputs may disrupt the RabbitMQ service and strange things start to happen.

To reproduce this issues one can use iperf example with bellow script added to INSTANTIATE event. It also helps to instantiate many NSDs at the same time (more than three for example).

#!/bin/sh
find /

Unable to start new NSR

After stress test users cannot launch additional NSRs because OpenBaton isn't able to find VNFM. Manager is still listed in web GUI, but when NSD is launched we get this error:

2016-07-22 14:15:20.817 ERROR 2080 --- [http-nio-8080-exec-3] o.o.n.a.e.GlobalExceptionHandler         : Exception was thrown -> Return message: VNFManager with endpoint: generic is not registered or not enable or not active.

org.openbaton.exceptions.NotFoundException: VNFManager with endpoint: generic is not registered or not enable or not active.

EMS is getting killed

In some cases the ems was killed because it lost the connection to RabbitMQ after the script was executed. After that NSR stayed in NULL state. Log from ems when it is killed (/var/log/openbaton/ems-receiver.log):

...
WARNING:pika.adapters.base_connection:Socket closed when connection was open
WARNING:pika.connection:Disconnected from RabbitMQ at 172.18.150.32:5672 (501): FRAME_ERROR - type 3, first 16 octets = <<".... 61%  114M 3">>: {invalid_frame_end_marker, 46}
CRITICAL:pika.adapters.blocking_connection:Connection close detected; result=BlockingConnection__OnClosedArgs(connection=<pika.adapters.select_connection.SelectConnection object at 0xa731d0>, reason_code=501, reason_text='FRAME_ERROR - type 3, first 16 octets = <<".... 61%  114M 3">>: {invalid_frame_end_marker, 46}')

[lorenzotomasini]

This is important, we could look into rabbit features for splitting the message