bitmagnet-io
store data in Elasticsearch
I have a little idea that if the data is stored in Elasticsearch, the query speed is quite fast. It may be better than storing it in PostgreSQL.
It had been asked a few times before. It would need a huge rewrite of the code, and probably not happening anytime soon. https://github.com/bitmagnet-io/bitmagnet/issues/298#issuecomment-2394099444
Spoiler alert: I'm working on it right now, but it'll be an optional plugin, not in the core.
Spoiler alert: I'm working on it right now, but it'll be an optional plugin, not in the core.
Thank you very much. This project has helped me a lot. Before, I only scraped other people's websites. Now that I have database permissions, I can do a lot more. I can use regular expressions to get the videos I want. Thank you very much.
Spoiler alert: I'm working on it right now, but it'll be an optional plugin, not in the core.
[root@iZsj9yq7s0pbZ ~]# curl http://127.0.0.1:9200/_cat/count/torrent_files 1760407081 01:58:01 15893968 [root@iZsj9yq7s0pbZ ~]# curl http://127.0.0.1:9200/_cat/count/torrents 1760407091 01:58:11 7197838 a085ddfe2ad1 docker.elastic.co/elasticsearch/elasticsearch:8.11.0 "/bin/tini -- /usr/l…" 2 weeks ago Up 2 weeks 0.0.0.0:9200->9200/tcp, [::]:9200->9200/tcp, 0.0.0.0:9300->9300/tcp, [::]:9300->9300/tcp elasticsearch 8.4GB (virtual 9.81GB)
The data volume of torrent_files is 15893968, and the data volume of torrents is 7197838, which only takes up 9.81GB. Maybe Elasticsearch is a good choice. At least we don’t need the torrent_contents table.
Spoiler alert: I'm working on it right now, but it'll be an optional plugin, not in the core.
How is this being implemented? is there docs on this plugin system? I am also in the middle of writing a meilisearch syncer. any information would be helpful.
Before ES gets added, because its a bit of a PITA to self host, has anyone tried using PostgreSQL text search or a separate sqlite db just for text search before going down the ES rabit hole to see if ether of dose are fast enough. My guess is that ether or one of dose will be enough for self hosted that in most case will have a handful of user. just saying
has anyone tried using PostgreSQL text search
This is exactly what bitmagnet currently does.
