Warning or error for orphan docs

[amimas]

🚀 Feature

When the site is built, show a warning or an error about orphaned docs page. An orphaned doc would be the one that has no links from any other docs or in the sidebar navigation. As a result, that doc may never be discovered.

Have you read the Contributing Guidelines on issues?

Yes

Motivation

This is one of the ways to improve seo for a technical docs site. All docs pages should be navigateable through internal links in the site. In this case, it can be either through links from another doc or through sidebar navigation.

Sphinx based site has an option to be able to detect this kind of orphaned docs. It'd be great if we could catch similar issues in Docusaurus.

https://docs.readthedocs.io/en/stable/guides/technical-docs-seo-guide.html

Pitch

I think the easiest way to implement is to check if each docs listed in one of the sidebar.

Whether each docs have at least one link from another doc, might be tricky to implement because every docs have to parsed and keep track of links. It's probably also safe to assume that every docs should be in the sidebar. So, maybe only checking sidebars for orphaned docs is enough.

This should be configurable so that it'll be either an error or warning. Making it an error will be helpful for CI process.

A similar warning for orphaned assets can be helpful. There's no point in having an image file included in the site if it's not used anywhere in the docs. Perhaps orphaned docs asset should be a separate feature request.

[slorber]

Hi,

This is something I actually noticed and wanted to work on some day :)

I think we should also allow a frontmatter or config like secret: true, so that if it's not linked from anywhere, it does not warn/error. Sometimes people want a page with no link and no sitemap on purpose, until they are ready to publish.

[amimas]

I think we should also allow a frontmatter or config like secret: true, so that if it's not linked from anywhere, it does not warn/error. Sometimes people want a page with no link and no sitemap on purpose, until they are ready to publish.

I'm not so sure about that. It may give some people an incorrect impression of the content of that page being "secured".

Even if there are no links to that "secret" page from other docs, it does not make that page secured in anyway. Search engine crawlers might still locate it or someone can still find that page (manually or accidentally). Someone from the inside might even post a link to it if they're upset for any reason.

The only way contents should be secured is if there are authentication in place; this brings up a completely separate topic. Or, the site should not be published publicly; i.e. publish internally that is only accessible through VPN.

Bottom line is, "secrets" should be handled veeeery carefully 😄

[slorber]

@amimas this is an option that would be documented. The user should understand that the link is still accessible, just without any link.

I use this feature extensively on my own blog, based on a gatsby theme: https://github.com/narative/gatsby-theme-novela#post

This is useful because it permits me to share an actual link for blog post reviews before final publish, without requiring to create PR deployments.

Some sites only use Github pages, not all are on Netlify, yet they may want a doc review from community folks before including it in the sidebar. Having an actual link to share is more convenient than asking them to review the MD on Github (particularly if it's interactive MDX)

[amimas]

Isn't this already doable right now? Even if the warning/error message about orphaned docs feature is implemented, in this case you'd just set that option to be a warning. Not sure what the value of the metadata in front matter. In your case, it needs to be an orphaned page. Maybe I'm getting tripped over the term "secret" as metadata.

[slorber]

We can see later if having an error/warning is a problem. I'm ok to check for unreferenced docs. I probably wouldn't mind if the build show a few warnings in case I have private docs anyway, so we'll figure later if that is annoying for some users.

[vthirupathi]

Hi, is this currently being worked on? If not, I'd like to take a stab at it if its still a desired feature @slorber

[slorber]

Thanks, you can work on it

Le jeu. 12 nov. 2020 à 21:14, vthirupathi [email protected] a écrit :

Hi, is this currently being worked on? If not, I'd like to take a stab at it if its still a desired feature

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/facebook/docusaurus/issues/2875#issuecomment-726315586, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAFW6PWYJ4JWNMXHJ7RXIQ3SPQ6ZPANCNFSM4NSHGYBQ .

[vikhramt007]

Hi, me and @cindygu4 have constructed a plan for this. We're having trouble figuring out which file in the source would be responsible for displaying the warning?

Also, in website/sidebars.js, how is docs-introduction resolved. As in, how does it know this refers to docs.md?

[slorber]

Hi @vikhramt007

I don't know what is the best place to handle this in the code.

Sidebars use doc ids, that are generated from the FS location or by frontmatter.

It's probably better to discuss first what you plan to build exactly, and its API, before rushing on implementing it, or you'll likely end up coding something we couldn't merge due to a bad design decision

[vikhramt007]

Hi @vikhramt007

I don't know what is the best place to handle this in the code.

We were just unsure of if the warning is displayed on the website's home page or when the user clicks on the docs page?

Sidebars use doc ids, that are generated from the FS location or by frontmatter.

Thanks, this really clears that up!

It's probably better to discuss first what you plan to build exactly, and its API, before rushing on implementing it, or you'll likely end up coding something we couldn't merge due to a bad design decision

Our current plan is as follows:

• In sidebar.js create a set of doc ids from the items key of each dictionary with type: 'category', as well as any standalone strings.
• look through the .md or .mdx files for each doc id and add any linked docs not yet added to the set, and repeat this process until no new docs are added
• using this complete set of non-orphan docs, create a list of all docs not in this set (orphan docs)
• Display all these orphan docs either in the home page or docs page

Would this be a valid approach? Thanks for the input

[slorber]

Hi,

The type DocMetadata already contains the sidebarName attribute if the doc is associated with any sidebar, so you don't need to work on finding docs that are not in any sidebar, it's already done for you.

The concept of "orphan doc" is not "does not appear in any sidebar", but rather "has no incoming link". If a doc is not in any sidebar, but the homepage links to it, it should not be considered as orphan.

Display all these orphan docs either in the home page or docs page

This feature is about warning/error, not displaying those docs in the UI. You can reuse the reporting code used by onBrokenLinks when you detect an orphan doc, and offer a configurable option for this logging (some users may want an orphan doc on purpose for some reasons, and should be able to disable such warning)

---

I wonder if this should not be handled more generically. @amimas it make sense to also report orphan pages no? Or do we explicitly want to restrict this to docs?

There's already a method here:

export async function handleBrokenLinks({
  allCollectedLinks,
  onBrokenLinks,
  routes,
  baseUrl,
  outDir,
}: {
  allCollectedLinks: Record<string, string[]>;
  onBrokenLinks: ReportingSeverity;
  routes: RouteConfig[];
  baseUrl: string;
  outDir: string;
}) 

In which we have all the page paths and their outgoing links, it should be doable to detect which pages have no incoming links based on these datas.

[Josh-Cena]

This would be very easy to implement given all links collected (and also the solution that IMO makes the most sense). The only problem is the number of false positives. Should we allow users to declare a list of routes to be ignored? If we do it in the core, we can't rely on any metadata and have to hard-code the ignored routes.

Alternatively, because #5698 will add metadata to routes, we can use the routes' metadata to filter out those with a special secret metadata

[ctreatma]

It looks like the PR attached to this issue was targeting v2; is this or a similar feature available in v3?