OpenCTI-Platform
OpenCTI TAXII Feed - 413 Content Too Large
Description
We are trying to connect OpenTAXII collection, which is very large, to OpenCTI, using TAXII feeds. While processing the collection, we are getting following error
{"category":"APP","context":"Taxii ingestion execution","errors":[{"attributes":{"genre":"TECHNICAL","http_status":500},"message":"Request failed with status code 413","name":"UNKNOWN_ERROR","stack":"GraphQLError: Request failed with status code 413 at error (/opt/opencti/build/src/config/errors.js:7:10) at UnknownError (/opt/opencti/build/src/config/errors.js:81:47) at Object._logWithError (/opt/opencti/build/src/config/conf.js:238:17) at Object.error (/opt/opencti/build/src/config/conf.js:247:48) at /opt/opencti/build/src/manager/ingestionManager.ts:402:18 at processTicksAndRejections (node:internal/process/task_queues:95:5) at async Promise.all (index 0) at async Promise.all (index 1) at ingestionHandler (/opt/opencti/build/src/manager/ingestionManager.ts:508:5) at /opt/opencti/build/src/manager/ingestionManager.ts:529:9 at iit.#runHandlerAndScheduleTimeout (/opt/opencti/build/node_modules/set-interval-async/dist/set-interval-async-timer.cjs:36:13) at Timeout._onTimeout (/opt/opencti/build/node_modules/set-interval-async/dist/set-interval-async-timer.cjs:29:13)"},{"message":"Request failed with status code 413","name":"AxiosError","stack":"AxiosError: Request failed with status code 413 at settle (/opt/opencti/build/node_modules/axios/lib/core/settle.js:19:12) at IncomingMessage.handleStreamEnd (/opt/opencti/build/node_modules/axios/lib/adapters/http.js:599:11) at IncomingMessage.emit (node:events:531:35) at endReadableNT (node:internal/streams/readable:1696:12) at processTicksAndRejections (node:internal/process/task_queues:82:21) at yKt.request (/opt/opencti/build/node_modules/axios/lib/core/Axios.js:45:41) at processTicksAndRejections (node:internal/process/task_queues:95:5) at taxiiHttpGet (/opt/opencti/build/src/manager/ingestionManager.ts:314:29) at taxiiV21DataHandler (/opt/opencti/build/src/manager/ingestionManager.ts:375:24) at async Promise.all (index 0) at async Promise.all (index 1) at ingestionHandler (/opt/opencti/build/src/manager/ingestionManager.ts:508:5) at /opt/opencti/build/src/manager/ingestionManager.ts:529:9 at iit.#runHandlerAndScheduleTimeout (/opt/opencti/build/node_modules/set-interval-async/dist/set-interval-async-timer.cjs:36:13) at Timeout._onTimeout (/opt/opencti/build/node_modules/set-interval-async/dist/set-interval-async-timer.cjs:29:13)"}],"level":"error","message":"Request failed with status code 413","name":"BLOCK_FEED_CONTEXT","source":"backend","timestamp":"2024-10-18T03:41:31.368Z","version":"6.3.6"}
Environment
- OS (where OpenCTI server runs): AWS RHEL EC2
- OpenCTI version: OpenCTI 6.3.6
- OpenCTI client: NA
- Other environment details: NA
Reproducible Steps
Steps to create the smallest reproducible scenario:
- Configure a TAXII Feeds for OpenTAXII which has a large collection
Expected Output
Large Collection from OpenTAXII should make use of paginations to fetch thereby, importing all the objects.
Actual Output
Errors out as OpenCTI is trying to fetch everything in one REST API call.
Additional information
NA
Screenshots (optional)
NA
@agrawald do you maybe a feed that we can use to be able to reproduce?
Apologies @nino-filigran I am not allowed to share the feed as part of my corporate agreement. However, I can tell you that the feed has more than 45000 STIX records. I will still check with my team.
Apologies @nino-filigran I am not allowed to share the feed as part of my corporate agreement. However, I can tell you that the feed has more than 45000 STIX records. I will still check with my team.
Apologies, will not be able to help you with the test data. However, if you do decide to implement pagination for TAXII feeds, I can download and test it out for you on a branch if you would prefer.
Thanks @agrawald, I'm still trying to figure out a Taxii with this amount of data for now!
@nino-filigran what is stopping you to implement pagination for the TAXII feeds.
@agrawald nothing form a technical standpoint as far as I know, but simply the current workload of the team. We have quite some bugs opened already, which have a higher priority at the moment, in addition of the delivery of features.
Hello @agrawald ! We've pushed a change that will be available on the next minor version: a new environment variable INGESTION_MANAGER__TAXII_FEED__LIMIT_PER_REQUEST will be avaible to configure your TAXII ingestion pagination.
Wow! Thank you so much
-- Thanks Dheeraj (DJ) Agrawal
On Sat, 1 Feb 2025 at 3:52 AM, Jeremy Cloarec @.***> wrote:
Hello @agrawald https://github.com/agrawald ! We've pushed a change that will be available on the next minor version: a new environment variable INGESTION_MANAGER__TAXII_FEED__LIMIT_PER_REQUEST will be avaible to configure your TAXII ingestion pagination.
— Reply to this email directly, view it on GitHub https://github.com/OpenCTI-Platform/opencti/issues/8697#issuecomment-2627799472, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABESDFB4BRSQFX7QFVYDC2T2NOS6VAVCNFSM6AAAAABQFCARZKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMMRXG44TSNBXGI . You are receiving this because you were mentioned.Message ID: @.***>