MISP error handling causes malformed state

[mahmut-the-guy]

1. Start MISP connector with blank state.
2. Once connector starts, expect it to set state with page_number (and increment it).
3. After some time, during restSearch queries, client fails to fetch data due to server responding with "429 rate limit exceeded" (or any other server error)
4. State is set to current date and all the data between MISP_IMPORT_FROM_DATE and current data ignored in the next run, since state now contains the current date.

Expected behavior

State of timestamp (import from date) should only be updated if all the data is successfully fetched. Otherwise it should remain at the last data checkpoint with relevant page number.

[nino-filigran]

@helene-nguyen or @Megafredo could you please help assess (and fixing this bug, if it's an actual one)? Thanks a lot!

[helene-nguyen]

@nino-filigran @mahmut-the-guy I'm on it :)

[helene-nguyen]

@mahmut-the-guy sorry the late answer, receiving a status 429 is not an error, it is the other server "kindly" asking you to please stop spamming requests. Obviously, your rate of requests has been too high and the server is not willing to accept this.

You might want to check if the rate limit is set on MISP:

Concerning the state, only the connector last run is taking into account and if you have the error raised, the last_event_timestamp will not change because no event is detected.

I will close this issue but it can be re-opened if needed :)