New HexDocs home page

[josevalim]

trafficstars

Today's home page is fully done in the client, we should probably mirror that here.

Overall, the main feature we want to have is full-text search across (at least) one or more packages. You can add packages by typing # which will trigger an autocomplete, such as #phoenix_live_view. You can then press : to autocomplete a given version, otherwise the latest is picked from the hex.pm API. If the search only has the package information, such as #ecto or #ecto:1.0.0, and no search term, we go to the particular package (+ version).

Anything else we could do when it comes to search? /cc @ericmj @ruslandoga @wojtekmach

I will work on a mockup to the new homepage. The mix hex.search and ExDoc can come later.

[josevalim]

Once we submit the search, we should go to a page like this: https://claude.site/artifacts/3324d578-85b1-411c-accd-c002f37f301b

I am currently in touch with Livebook's designer so he can mockup the new home page plus search. @paulo-valim made himself available to implement the designs as HTML + Tailwind. We already have endpoints in Hex for:

1. Package autocompletion
2. Docs search (coming with #47)
3. And another for creating short URLs

We will just need someone to implement the front-end bits. I will post all future updates here.

[josevalim]

@ruslandoga I am wondering: should we also index the most recent versions of packages as phoenix-latest, so it is easy for us to always query the latest? In the same way that hexdocs already tracks that hexdocs.pm/phoenix means latest?

[josevalim]

Another potential feature we want to borrow from ExDoc is using title:, doc: and type: to scope search. Here are all features available in ExDoc search:

• Multiple words (such as foo bar) are searched as OR
• Use * anywhere (such as fo*) as wildcard
• Use + before a word (such as +foo) to make its presence required
• Use - before a word (such as -foo) to make its absence required
• Use : to search on a particular field (such as field:word). The available fields are title, doc and type
• Use WORD^NUMBER (such as foo^2) to boost the given word
• Use WORD~NUMBER (such as foo~2) to do a search with edit distance on word

[josevalim]

@ruslandoga @ericmj @wojtekmach @supersimple I just want to share with you the new designs for the home page:

And the package search functionality:

We will work on converting those to HTML, then I will let you know once that's ready :)

[supersimple]

Looks good. I know at least one person that will be happy ;)

[lpil]

Hello everyone! @ghivert may be able to help out here with the frontend with Gleam+Lustre.

P.S. Love the design there, the homepage especially looks really nice.

[ghivert]

Hi everyone! @lpil told me about a potential rewrite with Lustre in Gleam. I’d be happy to help on the subject and implement the frontend part!

If I understand everything correctly, the backend part is already written, so the main task is only to implement the mockup as a real web app? Is there some requirements like SSR, or anything else?

In case you have a Figma/Sketch/whatever format for the mockups, maybe it could directly translate into a Lustre + Tailwind?

[josevalim]

Oh! Being to able to keep it within the BEAM community would be fantastic, thanks for chiming in @lpil and @ghivert!

If I understand everything correctly, the backend part is already written, so the main task is only to implement the mockup as a real web app? Is there some requirements like SSR, or anything else?

Yes, the backend is existing Hex APIs and the newly added Search API. Our goal is also to serve a static page, so no SSR. We probably want the homepage (the first screen) to be indexable though but I guess that should be doable. And I was told most engines index using a JS engine anyway?

In case you have a Figma/Sketch/whatever format for the mockups, maybe it could directly translate into a Lustre + Tailwind?

We do have one. We also have someone who can do the translation from Figma mockup to HTML + Tailwind. Or would you rather tackle it too?

This is very exciting!

[ghivert]

I'm really happy to be able to help!

Yes, the backend is existing Hex APIs and the newly added Search API. Our goal is also to serve a static page, so no SSR. We probably want the homepage (the first screen) to be indexable though but I guess that should be doable. And I was told most engines index using a JS engine anyway?

What we could do is performing some SSG in that case: no full-fledges SSR, but the HTML statically served contains the initial render. So it could be nicely indexed by search engines, and client-side runtimes will then take care of rendering the content after that (with hydration). I think it makes sense at least for the homepage to be pre-rendered, to improve search engine indexing. Also, Google is now indexing web sites with a JS engine enabled. I'm not sure Bing is doing it too however.

I think the SSR question is also related to the support of the Open Graph Protocol for Hex. Without SSR, it's not possible to have proper Open Graph tags. With OG tags, we could provide nice displays on some applications, like Discord or Slack when linking HexDocs URL. I'm not sure it's really useful, and it will be more work.

I think it's possible to start with a first implementation entirely client-side, and potentially migrate server-side later if needed (if things like OG tags are required).

We do have one. We also have someone who can do the translation from Figma mockup to HTML + Tailwind. Or would you rather tackle it too?

To be fully transparent on that point, Tailwind is not my favorite tech. If you have someone who's expert with (or at least used to), I think it can be a way to gain some time, but they're too much busy, I can tackle it too. 🙂

Talking about time, do you have some deadline for a working implementation?

[josevalim]

To be fully transparent on that point, Tailwind is not my favorite tech. If you have someone who's expert with (or at least used to), I think it can be a way to gain some time, but they're too much busy, I can tackle it too. 🙂

We are open to alternatives, if you have any mind. :) I suggested Tailwind because it is most likely others will be familiar with it. @paulo-valim made himself available to translate to Tailwind but that will probably only happen within a month. We don't have a deadline per se, given it is all open source.

[josevalim]

Correction, @paulo-valim has already started working on HTML + Tailwind, so we should have something as base sooner than later. :)

[ghivert]

I'm on the pure CSS side of the force! I consider CSS simpler than any framework, but as any take, it's highly debatable 😉

If @paulo-valim started working on it, I'd suggest to wait for him to finish, and then implement the HTML in the Lustre code. That will be less work UI-related and more time to properly implement the flows!

Thanks for your answers about the deadlines. I can start taking a look this week-end. And experimenting with the search API if it's available?

[josevalim]

I can start taking a look this week-end. And experimenting with the search API if it's available?

Perfect. We will work on getting the API descriptions for you before the weekend (hopefully tomorrow!).

[ghivert]

I can start taking a look this week-end. And experimenting with the search API if it's available?

Perfect. We will work on getting the API descriptions for you before the weekend (hopefully tomorrow!).

Thanks ! No worry though, if you need some time, I can still postpone experimentation (I thought API descriptions were already written). Do not crush for me! 🙂

[wojtekmach]

Hey everyone, some updates:

1. Currently hexdocs.pm homepage search autocomplete hits hexdocs.pm/sitemap.xml which is inefficient as it contains irrelevant data for what we require for new search. We're adding an endpoint https://hexdocs.pm/package_names.csv which contains sorted list of package names, e.g.:

package1
package2

etc.

2. We're working on exposing Typesense which will be available at ~search.hex.pm~ search.hexdocs.pm.

3. We can now create short links for hexdocs.pm:

$ mix hex.user key generate
Username: wojtekmach
Account password: ***
Generating key...
***
$ curl -H "authorization: $HEX_API_KEY" https://hex.pm/api/short_url --json '{"url":"https://hexdocs.pm/#test"}'
{"url":"https://hex.pm/l/EmTm6"}

This will be relevant when based on list of selected packages we'll generate a "short ink" for https://hexdocs.pm/#{search}.

I'll post here once 1. and 2. are done. Hopefully this is enough information to iterate on the frontend that will consume those endpoints, let me know otherwise!

[josevalim]

Oh, the short links API requires a user token? So let's remove this functionality from the proof of concept for now. So overall:

1. A listing of all packages will come from a CSV file, with packages sorted one per line separated by \n

2. Once you find a package, you can get its versions from https://hex.pm/api/packages/PACKAGE_NAME

3. The search endpoint will be exposed as https://github.com/hexpm/hexdocs/issues/47 - for now you can use https://search.staging.hexdocs.pm/?q= and it points to a TypeSense collection

4. We won't have the share feature for now. However, we probably don't need to? If whenever we add a package+version to the sidebar, we update the URL, we are good to go. We can do ?p[phoenix]=1.5.6&p[gleam]=1.3.4&q=hello+world and so on

Please let us know if you have any questions @ghivert!

[ghivert]

Hi! I'm really sorry for being so long to answer, I had some personal & professional issues to handle urgently. 😳 But I'm back on track.

I started playing with the different API. I implemented a quick PoC with Lustre, that you can find here.

That first draft is made of two pages:

1. The first one would be the home page, able to provide an autocomplete on packages, and can query the different package versions if needed.
2. The second page is able to get the results from the TypeSense API, by searching for a term. It is also able to search for specific packages in the search (it's tested and working).

Because a demo is always nice, here's two short videos to illustrate the possible actions:

Details

https://github.com/user-attachments/assets/878ecc3f-0da1-460e-ae99-598eb2e4308b https://github.com/user-attachments/assets/805e04f8-ac9e-4f8d-99f3-510f07f1efff

---

Because I was unsure of the user flows we desired for HexDocs homepage, I prefered to stay simple and implement the different parts of the logic to be able to reuse them later with the different screens and flow. Because paulo-valim is making the HTML, I also stuck with simple plain HTML. Of course, we can expect better screens in the end. 😄 I just wanted to advance fast on the subject to check if that's what you're looking for.

Nowadays the code is living in a separate repo for the PoC, but I suppose it should also integrate HexDocs repo directly? In that case I think it's wiser to let you tell me where to integrate and how, but I can also take a look and find an appropriate location. 😄

---

Everything here is a bit rough and needs more polishing. Here's how I understand the user flows from the two screenshots above:

1. User connects on the home page, search for a package, autocomplete shows up, and clicking on one redirects to corresponding HexDocs page.
2. User connects on the search page, inputs packages if needed, then type a term, and we trigger a search request, trying to find the best corresponding matches. User can also type some special characters in the search bar to activated advanced search (# to specify where to search, etc.)

Is it OK? Is there anything I'm missing?

[ruslandoga]

👋 @ghivert

Just a note: https://github.com/ghivert/hexdocs-search seems to be private :) Or maybe it's something else but it returns 404 for me right now.

[ghivert]

Oh ! Thanks for warning ! I changed the visibility !

[josevalim]

Hi @ghivert! First off, no need to apologize, we are all working on our free time, so any contributions and updates are really welcome! :heart: The demos are nice and it is an exciting preview of what is to come!

I will talk a bit about the flows but that's what I have on my mind and I am very open to suggestions.

Home page

The home page should work as a search engine. You join the website, the input is focused, and you start typing. If you type #, we will assume you want to autocomplete a package. If you add a : after the package name, we will show its versions. For example:

1. If I type "gleam", nothing shows up.
2. If I type "gleam #", then package autocompletion appears
3. If I type "gleam #lustre:", then version autocompletion for that package appears

Once you type something, you can submit the search by clicking the button or pressing enter. Doing so takes you to search page.

Search page

The search page converts the query from the previous page into something a bit more structured. Any package that you added to your input goes to the sidebar. If no version is specified, we automatically pick the latest. New packages can be added at any time, which triggers a new search. If you do a search with no packages, we should probably show a warning asking them to add packages to filter the results.

Structure

I believe the application is a SPA. As you perform queries and add packages, they should automatically change the URL, so we can easily share results with someone else.

WDYT?

Just one note, I am a Gleam noob, what is the best way to run the project? Perhaps we can add the steps in the README? I tried what is there and it didn't work. Thanks and have a great weekend!

[josevalim]

Hi @ghivert, sorry for the delay, we finally have the front page ready: https://github.com/paulo-valim/hex - we will focus on the results page next. Thank you @paulo-valim!

[ghivert]

Hi! Thanks for the message! No worry, I took a bit of time to clean everything on my own too! I completely forgot to ping here while modifying the repo! (My bad, I'm a bit bad at communicating on GitHub sometimes, notifications tend to grow and they get lost. 😳)

Count on me to implement the designs! I started to implement the flows as you stated, I think they're great and they make sense. I started to think on how to implement them properly, and I improved a bit the code structure in the repo. I also wrote a bit of documentation to help get you started, with more detailed information on installing Gleam and getting the code running.

I have a busy schedule those weeks, so tell me if it's too slow or anything, I'll see how to go a bit faster!

[paulo-valim]

♥️

[paulo-valim]

Hi everyone, I just pushed the HTML for the settings page.

If you have any questions, feel free to reach out!

Also, when everything is ready, just ping me so I can review the designs and see if they need any adjustments.

Thanks =)

[josevalim]

Hi @ghivert, just a quick check to see if there is anything else we can help with. If for some reason you can't move this forward for now, it is ok, just let us know :) Thanks!

[ghivert]

Hi ! Thanks for the message! I'm working as I can with the lot of work we currently have at my company (life of a startup), but I already started to integrate the Tailwind designs, and the different parts of the search, with a working autocomplete with both packages & versions. I also integrated the management of URL queries to be able to share a search on the web, taking the form of http://[address]/search?q=[query]&packages=[list of packages].

Here is an up-to-date demo of the code, currently pushed on the repo (still the same, HexDocs Search). Launching the demo should be easier than ever, with a simple gleam run -m lustre/dev start, once Gleam is installed (you can get it installed easily with a package manager, like brew install gleam, or with asdf or mise). Tailwind is directly downloaded for you, and once the web server is live, you can see it running at localhost:1234. I'll make sure to host a temporary version on an accessible web page somewhere tomorrow (like a CloudFlare or Netlify page).

https://github.com/user-attachments/assets/6727224e-27ad-4e4d-bded-ab6968faafbb

I'm not 100% sure I'm using the TypeSense endpoint correctly, and displaying the information as they should. I'm simply launching a request GET https://search.hexdocs.pm/?q=for&query_by=title,doc&page=1. I feel like responses are not as clean as the mock-up, but that could be expected? You could see how it looks in the demo currently. I wonder also if we should handle pagination, because I can't see it in the screens.

---

Just a quick word to say I'm sorry if I'm slowing your process of the new release for HexDocs. I know you already said I did not have to apologize for anything, but I uses HexDocs every day in my work now, and I'd really love to not put you and the team in a bad situation because I'm too slow!

[josevalim]

The video looks great! Thanks for sharing! :heart: I will try to check it out today or during this week. For now I have one question: what is missing? From the video is looks quite complete/done? :)

[ghivert]

Here's a link to a deployed version, as mentionned. Packages getting is not working, because the endpoint does not work. However, everything can be used otherwise!

For the missing things, I think an autocomplete is missing on the "Package name" on the search page, as well as an autocomplete for the version. Some buttons are not working, mainly because I don't know where to point exactly. The HTML content can be wrong at some places, because I did not know where to point at, or because the package link is missing in the TypeSense response.

I think it's mainly a question of a bit more time to get everything as desired, and another review of the HTML now that it's usable. 😄 But I also think the code can easily be improved over time. Do you have some plans on what to do next?

[josevalim]

PS: I know I am very delayed on this. I am sorry :) It is near the top of my queue.

[josevalim]

Hi @ghivert, sorry for the delay once more. It looks great and I finally have answers!

1. We are working on having https://hexdocs.pm/package_names.csv with all package names but it is not ready yet

2. However, you can use https://hex.pm/api/packages/PACKAGE_NAME to get all package versions :)

3. Btw, regarding the "latest", we won't have "latest" in the package entry itself... instead, if no version is given, we will look up the latest version using the API request above and replace the latest by the actual number

4. When doing the actual search requests, you want to filter the query to only the selected "package-version" pairs. This can be done by appending this to the URL: filter_by=package:=req-0.5.12||package:=jose-2.1.0||package:=...

5. Finally, the link to go to the package is going to be this (sorry, Elixir code):

[name, vsn] = String.split(document.package, "-", parts: 2)
"https://hexdocs.pm/#{name}/#{vsn}/#{document.ref}"

6. The preview is the same URL as above, except you open it up in an iframe passing ?preview=true&theme=THEME, where THEME is one of "light", "dark" or "system".

Btw, would you be happy if we sent some PRs as well for other changes unrelated to the above?

Thank you!!!!