discuit icon indicating copy to clipboard operation
discuit copied to clipboard

Feature: integrate meilisearch for searching

Open Codycody31 opened this issue 10 months ago • 3 comments

Adds support for the use of meilisearch for quick indexing and searching of discuit communities.

Required

  • [x] Move code for updating a communities index used before the API returns info to separate function
  • [x] Rather than placing search code in the core move to internal/search so we can have a package for it and not have to worry about names being the same
  • [x] Integration into the UI for searching (only if it is enabled)
  • [x] When indexing, break up the community name on each capital letter (to fix searching for Dev not matching DiscuitDev, etc)
  • [x] Allow searching on the communities page but default it when blank to the standard API

If Possible

  • [x] Index posts and users
  • [x] Sorting based on members, post creation time, user's account age, etc

General

After this is merged, someone who is better at UI stuff than me should probably make the searching be nice lol

Known Issues

  • When searching for a community while creating a new post, the picture is not used due to it not being included in the data set to meilisearch

Related Issues:

  • closes: #3
  • closes: #15
  • closes: #28

Codycody31 avatar Apr 09 '24 18:04 Codycody31

A new issue has been found! batching of data, as currently, it would attempt to dump the entire posts table to the index, however, the posts table has over 27K records sooooo. I'll need to batch the data, good thing is I already have code for it, just need to convert it over.

Codycody31 avatar Apr 22 '24 17:04 Codycody31

@previnder, the indexing logic along with everything else is finished. Now when pulling data to index it takes it in batches of 5000 records (I'll change it if you want it to be higher or lower, probably higher around 10,000 or 15,000). When pushing the data over to meilisearch we chunk it into batches of 50MiB (or less if we don't have that much data). By default, the searching ability is disabled and still makes use of the Google search to allow for the hoster to fully configure everything without having to worry about meilisearch till they are ready.

Codycody31 avatar Apr 26 '24 13:04 Codycody31

Merged latest, and now configured to also work with environment variables

Codycody31 avatar Jun 16 '24 14:06 Codycody31

Along with the conflicting files, I've found some other bugs when implementing this PR in a fork, I'll take a look at fixing the issues either today, or this weekend.

Codycody31 avatar Oct 10 '24 15:10 Codycody31

Note, going to refactor a bit of code to be a bit more search engine agnositic, and just refer to the package as search, rather than meilisearch as that can be decided when we store it in context.

Codycody31 avatar Oct 10 '24 15:10 Codycody31

@previnder I cut out the searching in the UI for communities, not sure how we want this to be done. If you want I can also remove the searching from the main search bar in the UI as it's not complete yet and only supports searching communities in the UI.

Codycody31 avatar Oct 14 '24 15:10 Codycody31