ipfs-search

ipfs-search copied to clipboard

https://docs.ipfs.io/how-to/best-practices-for-nft-data/#metadata

dokterbob

The more timeouts, the more parallel fetches we want to do (up until a certain maximum, of course).

dokterbob

so we can fine-tune the kind of requests that IPFS gets of various types - e.g. we can handle a lot of Stat() calls, but much less Ls() calls and...

dokterbob

Everything else is a server error and is very unlikely to spontaneously succeed the next time. Later: re-enable all retries and max retry limit through nsq.

dokterbob

https://godoc.org/github.com/ipfs/go-ipfs-http-api Offers streaming, context closing and full IPFS API compatibility (eventually).

dokterbob

``` Apr 14 18:17:04 boole ipfs-crawler[32158]: [119B blob data] Apr 14 18:17:04 boole ipfs-crawler[32158]: [126B blob data] Apr 14 18:17:04 boole ipfs-crawler[32158]: [139B blob data] Apr 14 18:17:04 boole ipfs-crawler[32158]:...

dokterbob

Separate for IPFS/Tika and Elasticsearch.

dokterbob

dokterbob

Allows us to have named workers, cleans up code, etc.

dokterbob

Example: https://gateway.ipfs.io/ipfs/bafybeihoqfxb3jvqxviom4mqmu3ntabqdfqse7f5u26z32qxifjhksknyi/094-gossipsub-v1.1-headerimage+grid.jpeg results in https://gateway.ipfs.io/ipfs/bafybeihoqfxb3jvqxviom4mqmu3ntabqdfqse7f5u26z32qxifjhksknyi/094-gossipsub-v1.1-headerimage%20grid.jpeg being requested through ipfs-tika, resulting in 404.

dokterbob