mastodon
mastodon copied to clipboard
mastodon
Search Modernization while preserving #hashtag paradigm
Pitch
Search is lacking in many ways and is inconsistent across web and mobile apps. If an instance has elastic search, search should be able to handle more complexity and do so with user experience in mind as to what they're trying to achieve by searching - discovery - either finding exactly what they're looking for or being guided to discover what they don't know.
Boolean Search
For example, with elastic search enabled - Boolean searches should be possible to search for "#hiking and #pnw" to find posts tagged with #hiking and #pnw. The search experience should also assume that any word is a hashtag and search for them. For example, searching for "#hiking #pnw" does a full text search for those words (it appears it is an any, not an AND) but it won't show the click through hashtags of #hking or #pnw because it considers it a full-text-search. (screenshots below)
existing issue for bool: https://github.com/mastodon/mastodon/issues/21778
Synonyms (related items)
In addition to Boolean search and treating hashtags as full text as well as hashtag search, we should extend the search output to show what most people presume search is. We should offer "related searches" such as by synonym. So, if someone searches for #hike the first positions should be exact matched, but under a topic such as "Related searches" we should show #hiker #hiking #trails - This will help guide people to discovery of related keywords they may not be thinking of. This would allow search without robots or algorithms limiting this or hiding it behind complexity that isn't needed. Guide people to discovery.
Tokenization of query
Tokenization of query parsing would be nice so if people ask questions, the query could be tokenized to show results such as "where to go hiking" could show results for #hiking and if they said "where to go hiking in California" it could show hashtags for #hiking #california. This is more advanced, but shows how we could extend search to do what people search for - without having to violate #hashtags
Existing Problems with Hashtag search:
Why doesn't it show full text search by default?
Remove the hashtag and search and now we're seeing what i'd expect if i search.
Another account, where you see I have full text search with a saved post.
Motivation
The idea is to provide what people want from search, without changing the behavior to full text search being based on hashtags but provide that "human friendly search" as the default position.
All a robot does for full text search is assume tokenization based on its algorithm/dictionary/stemming/and people massaging it. I believe we can stick with hashtags, use them as the means to make things discoverable (opt in) and make keyword search a first rate citizen while offering assistance/rails in front to help with discoverability.
Perhaps even posts tagged with a hashtag and in public, should be considered searchable and our idea should be that a public post with a hashtag is consent to be searched.
Right now, it's discoverable via the hashtag when so much more could be achieved if it was shown in full text search via the hashtag.
If a link is approved or a publisher is approved and those links are trending, they should be searchable as well. Searching within # explore should allow people to discover trending hashtags, trending news and trending accounts to follow.
Extending search to allow people to opt in their profile page to be searchable via #hashtags would be highly welcomed for user discovery as well
When i search for hashtag, the best thing that could happen is
Show people with #hashtag in their bio (not implemented) Show hashtags that contain the hashtag (current process) Show posts with hashtag
The mobile app is broken in a similar way where the pivot for searching for hashtag doesn't show posts. The bug is linked above.
" a public post with a hashtag is consent to be searched"
A reasonable suggestion, albeit if such an extended search would be implemented, posts sent before the implementation date should be left out of its index, even if they had hashtags. Otherwise users should informed to remove their more delicate posts, which would be too much to ask.
Some version of this warning has been in Mastodon UI since January 2, 2018: "This post won't be listed under any hashtag as it is not public. Only public posts can be searched by hashtag."
Here's the earliest commit I found: https://github.com/mastodon/mastodon/commit/b6af88192ff48372c5f6ed1321f21d99aaffcd3f
I think it's safe to say that from the earliest days of Mastodon "public post with a hashtag is consent to be searched" was not only implied, but explicitly stated.
After a lot of digging around, the easiest thing may be to just go to the /hashtag results when query = hashtag and have a different page for pivoting for actual "hashtag search" as i don't think it makes much sense to send regular users to search hashtags when they really mean to find posts containing hashtags.
i still think elastic search may be beneficial for hashtagged searches so people can do Boolean search operators such as "and" "or" "+" "-" so on and so forth... not sure how amicable the community is towards that since it would tokenize non hashtag words too. But maybe the hashtag and public post is the consent needed... no idea.