Connection aborted. RemoteDisconnected('Remote end closed connection without response')

[LFED-FP]

Work environment

Questions	Answers
Type of issue	Bug
OS version (server)	Ubuntu I believe
OS version (client)	MacOS
PHP version	7.2
MISP version	2.4.133
PyMisp version	2.4.133

In Dockerfile

RUN sed -i "s/max_execution_time = 30/max_execution_time = 300/" /etc/php/7.2/apache2/php.ini ; \
    sed -i "s/memory_limit = 128M/memory_limit = 8192M/" /etc/php/7.2/apache2/php.ini ; \
    sed -i "s/upload_max_filesize = 2M/upload_max_filesize = 50M/" /etc/php/7.2/apache2/php.ini ; \
    sed -i "s/post_max_size = 8M/post_max_size = 50M/" /etc/php/7.2/apache2/php.ini

In MISP ui

max_execution_time (300)… OK (recommended: 300)
memory_limit (8192M)… OK (recommended: 2048M)
upload_max_filesize (50M)… OK (recommended: 50M)
post_max_size (50M)… OK (recommended: 50M)

Expected behavior

I have purpose written code which locates events with specific tags, creates new tags, tags all found events with new tags, republishes those events, & then deletes the old tags. Essentially I am updating tags on events & removing old tags.

Actual behavior

Some tags have over 9,000+ events associated with them. When I encounter such tags, the code goes about its normal execution. As the code attempts to search for all the events for a particular tag with previously statement total, I get this error message {"asctime": "2021-01-19 21:46:07,358", "timestamp": 1611092767.394494, "name": "misp-feeds", "filename": "search_and_destroy.py", "funcName": "_search", "lineno": 228, "level": "ERROR", "levelno": 40, "message": "Searching for events failed. kwargs: {'controller': 'events', 'tags': 'malware:GandCrab', 'limit': 500, 'date_to': 1611092482, 'published': True, 'page': 4} Reason: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response',))", "severity": "ERROR"}

I tried "streaming" events by limiting how many events are returned by MISP in a while loop, cycling through until there are no events returned old_events = client._search(controller='events', tags=old_tag_name, limit=500, date_to=now, published=True) However, I get the same error. I had a tag with 9,015 event. My code processed exactly 1,500 events and then refuses to continue even after I stop the script and run it again.

Steps to reproduce the behavior

I looked through other issues related to this and found a similar issue https://github.com/MISP/MISP/issues/6805. Im not exactly sure how to reproduce this.

Logs, screenshots, configuration dump, ...

[Rafiot]

Just making sure, you're incrementing the page parameter in every loop. right?

Something like:

i = 1
while True:
    old_events = client._search(controller='events', tags=old_tag_name, limit=500, page=i, date_to=now, published=True)
    if len(old_events) < 500:
        break
    i += 1

If yes, the issue you opened on the MISP project (https://github.com/MISP/MISP/issues/6887) is where you will get an answer.

[LFED-FP]

Yes I am incrementing the page variable :-)

[Rafiot]

Woops, sorry, I forgot a follow up on this issue. Do you still have the problem?