ipfs-search

ipfs-search copied to clipboard

dokterbob

Crawler crashed with no elastic index

3

Hi, recently I did a pull on the repo and when run with docker-compose up the crawler crashed with the following error `ipfs-crawler_1 | Creating AMQP channels.` `ipfs-crawler_1 | Starting...

littlespeechless

We're getting 'Too Many Requests' and should deal with them intelligently, e.g. querying a different node or simply waiting for a bit.

dokterbob

REUSE compliance

1

https://reuse.software/ Ref: * https://reuse.software/ * https://github.com/ipfs-search/dweb-search-frontend/pull/210

dokterbob

dokterbob

1. Implement `BulkGetter` similar to `BulkIndexer`. 2. Use `_id` query on multi-index search API over GET api. https://www.elastic.co/guide/en/elasticsearch/reference/6.8/query-dsl-ids-query.html 3. Allow [doc values](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-fields.html#docvalue-fields) instead of source filtering. This should increase performance...

dokterbob

Due to large amount of no-ops in backend, it seems not to be the case.

dokterbob

How to access this site over IPFS?

11

Can this site be accessed through only IPFS? And peers running server actions (NodeJS) and Database in a p2p way?

danimesq

Creation of a flake for nixos

2

After some thoughts, here are the following considerations regarding making a flake for ipfs-search - Consider a flake like a docker file and a docker compose file. However in terms...

happysalada

Publish metadata through HAMT on IPFS

3

For direct acccess. Possibly strategies: a) Use built-in experimental IPFS HAMT, which works transparently. b) Use [IPLD-HAMT](https://ipld.io/specs/advanced-data-layouts/hamt/spec/), which is in progress. Ref: #183

dokterbob