Update `observed_data.number_observed` using `pycti`

[eliataylor]

Description

Our connect needs to increment the number_observed property. From the active connector, I am getting some number_observed greater than 1, but it's not clear the exact context of those updates. Using pycti I've tried many combinations of fields but never get number_observed to increment.

Environment

1. OS: Osx
2. OpenCTI version: 5.3.10
3. OpenCTI client: pycti

Reproducible Steps

1. run

        context = ["aee931ee-f2e4-45a6-bae4-7b3ee0a6e49a"]        
        observed = self.helper.api.observed_data.read(filters=[{"key": "objectContains", "values": context}])
        if observed is not None:
            fields = {}
            fields["stix_id"] = observed["standard_id"]
            fields["id"] = observed["id"]
            fields['number_observed'] = observed["number_observed"] + 1
            # fields["objects"] = observed["objectsIds"]
            fields["objects"] = observed["objects"]
            fields["last_observed"] = observed["last_observed"]
            fields["first_observed"] = observed["first_observed"]
            fields['update'] = True
            self.helper.api.observed_data.create(**fields)

Expected Output

The Observed Data's number_observed property increments by 1 each run.

Actual Output

No change.

Additional information

I found this related Issue - https://github.com/OpenCTI-Platform/opencti/issues/1444 - but the fix is already applied to 5.3.10

Screenshots (optional)

[SamuelHassine]

Hello @eliataylor,

Can you please try to put a confidence level to your observed data greater or equel to 15 because this is the default one in the platform?

If your confidence level is greater or equal than the existing one (so you can put always the same), the entity will be updated. Update=true is now used only in some merging circumstances.

Kind regards, Samuel

[eliataylor]

@SamuelHassine , I wish i could say that worked. I tried all variations of

            fields = {}
            fields["stix_id"] = observed["standard_id"]
            fields["id"] = observed["id"]
            fields['number_observed'] = observed["number_observed"] + 1
            # fields["createdBy"] = observed["createdBy"]
            # fields["objects"] = observed["objectsIds"]
            fields["objects"] = observed["objects"]
            fields["confidence"] = 100
            fields["last_observed"] = observed["last_observed"]
            fields["first_observed"] = observed["first_observed"]
            # fields["objectMarking"] = observed["objectMarkingIds"]
            # fields["objectLabel"] = observed["objectLabelIds"]
            # fields['update'] = True
            # self.helper.api.observed_data.update_field(**fields)
            self.helper.api.observed_data.create(**fields)

[SamuelHassine]

Ok @eliataylor, you're right, we have a special portion of the API code that prevent this update if the first_observed / last_observed are the same. We will fix this asap. In the mean time:

If you change (a little) the first_observed or the last_observed, you will enter in this condition:

 const timePatch = handleRelationTimeUpdate(updatePatch, element, 'first_observed', 'last_observed');
    // Upsert the count only if a time patch is applied.
    if (isNotEmptyField(timePatch)) {
      const basePatch = { number_observed: element.number_observed + updatePatch.number_observed };
      const patch = { ...basePatch, ...timePatch };
      const patched = await patchAttributeRaw(element, patch, { upsert: true });
      impactedInputs.push(...patched.impactedInputs);
      patchInputs.push(...patched.updatedInputs);
    }

So this means the number_observed will be updated with "existing number_observed" + "passed number_observed". In your case, if you would like to increment, just pass "1".

Kind regards, Samuel

[eliataylor]

@SamuelHassine , unfortunately it's still not budging even when incrementing last_observed by 1 second each run.

            fields = {}
            fields["stix_id"] = observed["standard_id"]
            fields["id"] = observed["id"]
            fields['number_observed'] = 1
            # fields['number_observed'] = observed["number_observed"] + 1
            fields["objects"] = observed["objects"]
            fields["confidence"] = 100
            fields["first_observed"] = observed["first_observed"]
            date = datetime.strptime(observed["last_observed"], self.dateFormat)
            date += timedelta(seconds=1)
            fields["last_observed"] = datetime.strftime(date, self.dateFormat)
            fields['update'] = True
            self.helper.api.observed_data.create(**fields)

However, this idea does match up with the fact my runtime Connector is setting greater than 1. Of course, I would like to ingest out of chronological order so those dates may not change despite the number_observed.

[SamuelHassine]

We have another customer where this is working, can you display the "fields" at each run and verify the last_observed is actually incremented?

[eliataylor]

on 5.3.10?

[SamuelHassine]

Yes this should work since a while :)

[eliataylor]

@SamuelHassine , I've verified through breakpoints neither last_observed nor number_observed are updated, despite sending the updated fields object.

Both runs show:

The only difference in my queried object and update fields is '2022-08-05T20:16:34.000000Z' vs '2022-08-05T20:16:34.000Z'

in my code above I'm using self.dateFormat = "%Y-%m-%dT%H:%M:%S.%fZ"

[eliataylor]

@SamuelHassine , i now see i'm getting this error:

{'name': 'Variable "$input" got invalid value { id: "3936c937-8315-49e3-9532-efbed95c8d07", entity_type: "Url", parent_types: ["Basic-Object", "Stix-Object", "Stix-Core-Object", "Stix-Cyber-Observable"], standard_id: "url--e9d97d97-6e83-501a-bed5-91786286d4f9", spec_version: "2.1", created_at: "2022-09-13T17:55:58.610Z", updated_at: "2022-09-13T17:55:58.635Z", createdById: null } at "input.objects[0]"; String cannot represent a non string value: { id: "3936c937-8315-49e3-9532-efbed95c8d07", entity_type: "Url", parent_types: ["Basic-Object", "Stix-Object", "Stix-Core-Object", "Stix-Cyber-Observable"], standard_id: "url--e9d97d97-6e83-501a-bed5-91786286d4f9", spec_version: "2.1", created_at: "2022-09-13T17:55:58.610Z", updated_at: "2022-09-13T17:55:58.635Z", createdById: null }', 'message': 'Variable "$input" got invalid value { id: "3936c937-8315-49e3-9532-efbed95c8d07", entity_type: "Url", parent_types: ["Basic-Object", "Stix-Object", "Stix-Core-Object", "Stix-Cyber-Observable"], standard_id: "url--e9d97d97-6e83-501a-bed5-91786286d4f9", spec_version: "2.1", created_at: "2022-09-13T17:55:58.610Z", updated_at: "2022-09-13T17:55:58.635Z", createdById: null } at "input.objects[0]"; String cannot represent a non string value: { id: "3936c937-8315-49e3-9532-efbed95c8d07", entity_type: "Url", parent_types: ["Basic-Object", "Stix-Object", "Stix-Core-Object", "Stix-Cyber-Observable"], standard_id: "url--e9d97d97-6e83-501a-bed5-91786286d4f9", spec_version: "2.1", created_at: "2022-09-13T17:55:58.610Z", updated_at: "2022-09-13T17:55:58.635Z", createdById: null }'}
ERROR:root:Variable "$input" got invalid value { id: "3936c937-8315-49e3-9532-efbed95c8d07", entity_type: "Url", parent_types: ["Basic-Object", "Stix-Object", "Stix-Core-Object", "Stix-Cyber-Observable"], standard_id: "url--e9d97d97-6e83-501a-bed5-91786286d4f9", spec_version: "2.1", created_at: "2022-09-13T17:55:58.610Z", updated_at: "2022-09-13T17:55:58.635Z", createdById: null } at "input.objects[0]"; String cannot represent a non string value: { id: "3936c937-8315-49e3-9532-efbed95c8d07", entity_type: "Url", parent_types: ["Basic-Object", "Stix-Object", "Stix-Core-Object", "Stix-Cyber-Observable"], standard_id: "url--e9d97d97-6e83-501a-bed5-91786286d4f9", spec_version: "2.1", created_at: "2022-09-13T17:55:58.610Z", updated_at: "2022-09-13T17:55:58.635Z", createdById: null }

obviously observed_data.create(**fields) shouldn't be touching the contained objects when only updating the Observed Data fields.

[eliataylor]

@SamuelHassine , resolved. in the end. confidence and update didn't matter. I do need to increment the last_observed by a millisecond and only pass number_observed=1 to increment it's value. My issue otherwise was that objects needs to just be an array of Ids, not the whole stix objects.

            fields = {}
            fields["stix_id"] = observed["standard_id"]
            fields['number_observed'] = 1
            # fields['number_observed'] = observed["number_observed"] + 1 #this seems more appropriate.
            fields["objects"] = observed["objectsIds"]
            fields["first_observed"] = observed["first_observed"]
            date = datetime.strptime(observed["last_observed"], self.dateFormat)
            date += timedelta(milliseconds=1)
            fields["last_observed"] = datetime.strftime(date, self.dateFormat)
            # fields["confidence"] = observed["confidence"]
            # fields['update'] = True 
            self.helper.api.observed_data.create(**fields)

Not sure if you want to close this given the necessary patches on these upsert conditions.