connectors icon indicating copy to clipboard operation
connectors copied to clipboard

Published 20 hours ago • verified publisher icon OpenCTI-Platform

[Malpedia] No progress bar and crash

Open Ken-Abruzzi opened this issue 3 years ago • 16 comments

Hello! @SamuelHassine

Description

The connector ran for the first time after I deploy it but no progress bar is shown in the web page. image And I assure that the connector actually ran via reading the log file image And I can also prove it by find out the entities created by author "malpedia", the results show that these entities are created when I first ran the connector. image After that, the connector has never run for more than 10 days.

Environment

  1. OS (where OpenCTI server runs): Ubuntu Server LTS 20.04
  2. OpenCTI version: OpenCTI 4.5.5
  3. OpenCTI client: frontend

Reproducible Steps

Steps to create the smallest reproducible scenario:

  1. I installed the connector using docker-compose.
  2. I leave the MALPEDIA_AUTH_KEY empty. image

Expected Output

  1. Every time the connector start a new work progress, there should be a progress bar in the web page, such as below: image
  2. The progress should start every 172800 seconds as I set up in the docker-compose.yml file.

Actual Output

  1. No progress bar is shown even if the connector ran successfully once.
  2. The connector never runs again.

Additional information

At the end of the first progress, the log shows some errors: image The docker status is running: image

Thanks for your help!

Ken-Abruzzi avatar Jun 28 '21 04:06 Ken-Abruzzi

@rhaist Hello! Thanks for your attention. I click the "reset the connector state", nothing happens. image I change "MALPEDIA_INTERVAL_SEC" from 179200 to 216000 in the docker-compose.yml file and then deploy the container again. The connector begins to work again. I change the docker-compose.yml file only to update the container. Actually, the "MALPEDIA_INTERVAL_SEC" field does not matter. image You can see that the connector imports new data today (June 29). The last time the connector worked is on June 15. image Still no progress bars show on the web page. image

Ken-Abruzzi avatar Jun 30 '21 01:06 Ken-Abruzzi

@rhaist Hello! I find that the read and write operations show effective numbers when the malpedia connector runs. image

Ken-Abruzzi avatar Jun 30 '21 03:06 Ken-Abruzzi

Hi @Ken-Abruzz, Thanks for the detailed analysis. Much appreciated.

The Malpedia connector is from a time where the work API didn't exist yet in OpenCTI, thus no progress in the UI currently.

I am waiting for the STIX output to be finished by the Malpedia team before I work on the connector again and fix this.

I'll update this issue as soon as there's news.

rhaist avatar Jun 30 '21 05:06 rhaist

Hello @rhaist . The connector doesn't start to send bundles at the frequency as the yaml file tells it. Resetting the connector doesn't work, neither does "docker restart docker_connector-urlhaus_1". I have to reboot my machine to wake up the connector.

Ken-Abruzzi avatar Aug 27 '21 03:08 Ken-Abruzzi

Hey @Ken-Abruzzi

Based on your other github issue, it seems that this issue is connector independent. Try either doing a docker restart or to entirely remove the docker container docker stop, docker rm and docker-compose up -d again. Does this help?

Is the Malpedia connector or the urlhaus connector the problem in your current issue?

Regards,

nor3th avatar Aug 29 '21 14:08 nor3th

@nor3th Hello! I'm sorry to make a mistake in the above comment about regarding "urlhaus". But both urlhaus and malpedia share the same problem.

Ken-Abruzzi avatar Sep 01 '21 00:09 Ken-Abruzzi

Does simply doing a docker restart or stop as described above also help?

nor3th avatar Sep 01 '21 10:09 nor3th

@rhaist any chance to have this connector updated to generate a bundle?

SamuelHassine avatar Sep 02 '21 07:09 SamuelHassine

I still don't have an ETA from Malpedia. Try to solve this ASAP. Sorry for the inconvenience.

rhaist avatar Sep 02 '21 08:09 rhaist

The problem with "no progress bar" has been explained in #479 . And now I update the connector to version 5.0.0. The error recorded in the log is not "Failed to establish a new connection [ERRNO 111]: connection refused". The new error is "Failed to establish a new connection: [Errno -3] Try again". This error is mentioned in #467 . And it is promised to be solved in a new version. image But the timer stops when there is 157890 seconds left, and hence no new work starts. image

Ken-Abruzzi avatar Sep 18 '21 03:09 Ken-Abruzzi

When I reboot my computer, the connector runs again. But it is stopped by a new error: image

Ken-Abruzzi avatar Sep 22 '21 08:09 Ken-Abruzzi

@rhaist any update for this connector?

SamuelHassine avatar Feb 01 '22 00:02 SamuelHassine

Malpedia wanted to do an update this week. Will update the connector right afterwards. Should not take long.

rhaist avatar Feb 01 '22 07:02 rhaist

@rhaist Any news on the connector?

SamuelHassine avatar Mar 22 '22 06:03 SamuelHassine

@Jipegien @nino-filigran @Megafredo @helene-nguyen we need a full refactor of this connector to generate STIX bundles / works instead of querying API / send stix bundles without works.

SamuelHassine avatar Jan 16 '24 00:01 SamuelHassine

@helene-nguyen @Megafredo to do after other rework on enrichement connectors, instead of VMRay one. thanks!

Jipegien avatar Jan 16 '24 08:01 Jipegien

@Megafredo @Jipegien, just FYI, since confidence level modifications this connector is not working anymore which makes urgent to upgrade it to STIX bundle generation.

Short story long: as the connector is trying to directly update things in the platform it crashes everytime it has an insufficent confidence level to do so...

SamuelHassine avatar Mar 25 '24 00:03 SamuelHassine

Thanks for the information @SamuelHassine, it is currently in the roadmap, but should I change the priority of my task and focus on malpedia ?

Megafredo avatar Mar 25 '24 06:03 Megafredo