Need filtering capability to limit number of records

[agrawald]

Scenario

While using taxii2-connector as docker container with openCTI, we are getting following issue, as we are trying to load a very huge collection "collection-id".

ERROR

{"timestamp": "2024-10-02T03:40:57.618330Z", "level": "ERROR", "name": "TAXII2 Connector -", "message": "413 Client Error: Payload Too Large for url: https://some-taxii2-server/taxii2/rootapi/collections/collection-id/objects/?added_after=2024-09-27T23%3A40%3A55.592945Z", "exc_info": "Traceback (most recent call last):\n File \"/opt/opencti-taxii2/taxii2.py\", line 214, in run\n self.poll_entire_root(root)\n File \"/opt/opencti-taxii2/taxii2.py\", line 273, in poll_entire_root\n self.poll(coll)\n File \"/opt/opencti-taxii2/taxii2.py\", line 345, in poll\n response = collection.get_objects(**filters)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.11/site-packages/taxii2client/v21/__init__.py\", line 376, in get_objects\n return self._conn.get(self.objects_url, headers={\"Accept\": accept}, params=query_params)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.11/site-packages/taxii2client/common.py\", line 310, in get\n raise e\n File \"/usr/local/lib/python3.11/site-packages/taxii2client/common.py\", line 300, in get\n resp.raise_for_status()\n File \"/usr/local/lib/python3.11/site-packages/requests/models.py\", line 1024, in raise_for_status\n raise HTTPError(http_error_msg, response=self)\nrequests.exceptions.HTTPError: 413 Client Error: Payload Too Large for url: https://some-taxii2-server/taxii2/rootapi/collections/collection-id/objects/?added_after=2024-09-27T23%3A40%3A55.592945Z"}

Expected

We are expecting to have an option to limit number of records, using limit parameter which can be injected as environment variable

Code Reference

https://github.com/OpenCTI-Platform/connectors/blob/b9eb01cdf0d97b568986643408c092f1467282e7/external-import/taxii2/src/taxii2.py#L341

[annoyingapt]

Is this the taxi post stream connector? Sorry, I know you linked to the taxii connector, I've seen that before when adding objects, hence why I asked. Does setting a limit fix the issue for you?

[agrawald]

I believe, yes. This is for external-import for taxii2 connector. I have provided the code reference.

[annoyingapt]

I could have a look at this later. Do you have the capability to test the python before it gets pushed to a container? To make sure it resolves your issue?

[agrawald]

Sorry I cannot test that for you.

[annoyingapt]

Ok sweet, this is done, also found a bug with more being set to None on the first request (moved above the request). More is now being returned as limit is being used. Before I raise a PR I'm going see if there is anything else I can add. You can check it out here: https://github.com/annoyingapt/connectors/tree/taxii2_improvements