Issues with Storing and Accessing Large Data (>12-13 Bytes) via Public Gateway

[Ali-Usama]

Description

I'm integrating rust-ipfs into a Substrate blockchain to enable decentralized storage capabilities for our nodes. The integration involves using offchain workers to interact with an IPFS node, managed by rust-ipfs, for storing and retrieving data. While testing this setup, I've encountered an issue where I'm unable to access data larger than approximately 12 to 13 bytes through a public IPFS gateway. Smaller data sizes work as expected and are accessible without issues.

Steps to Reproduce

• Initialize the IPFS node using rust-ipfs with this configuration.
• Store data on IPFS using the Substrate offchain worker, which interacts with the rust-ipfs instance.
• Attempt to access the stored data through a public IPFS gateway (e.g., https://ipfs.io/ipfs/<CID>).

Expected Behavior

Data of any size, when stored on IPFS using rust-ipfs through our Substrate blockchain integration, should be retrievable via public IPFS gateways.

Actual Behavior

When attempting to access data larger than 12 to 13 bytes through a public gateway, the request fails (504: Gateway Timeout Error). Smaller data sizes are retrievable without any issues.

Additional Information

Rust-IPFS version: forked rust-ipfs Substrate version: polkadot-v0.9.43

I suspect this might be related to how rust-ipfs handles data chunking or broadcasting of CID announcements to the IPFS network, particularly for larger data sizes. However, I am not entirely sure if the issue lies within the configuration of the rust-ipfs node, the data storage process, or the retrieval/query mechanism.

Request for Assistance

Could you provide insights or recommendations on how to address this issue? Specifically, I am looking for:

• Confirmation if this is a known issue with rust-ipfs or if it might be related to my integration approach.
• Any configuration changes or optimizations that could help in successfully storing and accessing larger data sizes via public IPFS gateways.
• Best practices for debugging and resolving such issues when integrating rust-ipfs with Substrate blockchains.

Thank you for your support and looking forward to your guidance on resolving this challenge.