restore-files.py crashes when missing find_element returns None

[kmcmahon1959]

Description

When using restore-files to incrementally update a second instance of OpenCTI the process crashes whenever a referenced element is not in the data being restored (normally because it's already in the system). The return from find_element in the resolve_missing function (line 79 currently) is not checked for "None" (as is done in the restore_files function at line 121).

Environment

1. OS (where OpenCTI server runs): Ubuntu 20.4
2. OpenCTI version: 5.3.11
3. OpenCTI client: python
4. Other environment details:

Reproducible Steps

Steps to create the smallest reproducible scenario:

1. Run backup-files on system 1
2. copy output to system 2
3. Run restore-files on system 2

Expected Output

Ingest of all elements.

Actual Output

INFO:root:Restore run directory @ 20220314T200100Z Traceback (most recent call last): File "/opt/opencti-highside-sync/connectors-master/external-import/restore-files/src/restore-files.py", line 188, in RestoreFilesInstance.start() File "/opt/opencti-highside-sync/connectors-master/external-import/restore-files/src/restore-files.py", line 181, in start self.restore_files() File "/opt/opencti-highside-sync/connectors-master/external-import/restore-files/src/restore-files.py", line 133, in restore_files self.resolve_missing(dir_date, ids, missing_element, acc) File "/opt/opencti-highside-sync/connectors-master/external-import/restore-files/src/restore-files.py", line 87, in resolve_missing self.resolve_missing(dir_date, element_ids, missing_element, acc) File "/opt/opencti-highside-sync/connectors-master/external-import/restore-files/src/restore-files.py", line 80, in resolve_missing refs = ref_extractors([data]) File "/opt/opencti-highside-sync/connectors-master/external-import/restore-files/src/restore-files.py", line 17, in ref_extractors for key in data.keys(): AttributeError: 'NoneType' object has no attribute 'keys' Killed

[richard-julien]

Backup restore is not designed to do incremental synchronization.

To do that you can use the builtin synchronizer.

[kmcmahon1959]

Ok, now I'm using restore-files.py for just restoring a full backup. And I got this same error again. Note: it was after a restart of restore-files, but all of the input data was in place. It just needs: "if missing_element is not None:" after line 79 in resolve_missing (just like in restore_files).

[richard-julien]

Any chance to send me your backup directory to reproduce locally? (We can of course sign an NDA or anything required)