p5.js-website icon indicating copy to clipboard operation
p5.js-website copied to clipboard
verified publisher icon processing

Programmatically found broken internal links

Open ff6347 opened this issue 1 year ago • 16 comments

Most appropriate sections of the p5.js website?

Other (specify if possible)

What is your operating system?

Mac OS

Web browser and version

not related to browsers

Actual Behavior

Broken links generate a 404

Expected Behavior

Links should serve the linked content

Steps to reproduce

Since I saw some reports of broken links in the website I tried a programmatic approach using broken-link-checker.

How to:

Edit: To make it easier to generate an overview I created a script that outputs GitHub flavored markdown. https://github.com/ff6347/find-broken-links

npm install broken-link-checker
npx blc -roe https://p5js.org > report-blc-prod-live.txt

# The flags do the following
# --exclude-external, -e  Will not check external links.
# --ordered, -o           Maintain the order of links as they appear in their HTML document.
# --recursive, -r         Recursively scan ("crawl") the HTML document(s).

The output needs some cleaning using regular expressions afterwards.

Remove all reports that are okay:

^├───OK───.*?$\n

Then remove all reports of urls that have no broken links:

^Getting links from:.*?$\nFinished! \d{1,1000} links found. \d{1,1000} excluded. 0 broken.

Fix some reports where there is no new line between the report and the next.

look for:

broken\.\n(\w)

Replace with:

broken.\n\n$1

These three should do most of the grunt work. There are some false positive reports about images that I removed manually at the end.

Here is what I've found:

Low Hanging Fruits

There are some links in footer and reference that create 301 redirects which can be avoided through small tweaks.

These are the lines that need a trailing slash:

Generally these are neglect-able, but a HEAD request against them returns a 301 HTTP response. This could be fixed since it is a low hanging fruit.

301 redirects

There are many links in the reference generated form the JSDoc comments in the source of p5.js that are lacking the trailing slash and generate a 301 HTTP redirect. These need some manual labor in the source code. This is a bit tedious work. It should not be a problem keeping this as is since the site-host + Astro take care of the redirects.

Broken Links

There are some broken links also generated from the JSDoc that need fixing and I found broken links in the tutorials. They are hard-coded into the .mdx files and need to be fixed one by one.

Next Steps

IMO the next steps should be

  • creating separate issues for these since fixing it just based on this list is error prone.
  • check external links as well when the internal ones are fixed.

Below you can find the report

report created: Fri, 13 Sep 2024 11:06:07 GMT

  • [ ] https://p5js.org/contribute/friendly_error_system/

    • [ ] _friendlyFileLoadError() | https://p5js.org/contribute/friendly_error_system/fes_contribution_guide.md#_friendlyerror
    • [ ] _validateParameters() | https://p5js.org/contribute/friendly_error_system/fes_contribution_guide.md#_validateparameters
    • [ ] Dev Notes | https://p5js.org/contribute/friendly_error_system/fes_contribution_guide.md#-development-notes

  • [x] https://p5js.org/contribute/release_process/

    • [x] “New p5.js release” | https://p5js.org/contribute/.github/workflows/release.yml

  • [ ] https://p5js.org/contribute/steward_guidelines/

    • [ ] p5.js’ design principles | https://p5js.org/contribute/contributor_guidelines.md#software-design-principles
    • [ ] design principles | https://p5js.org/contribute/contributor_guidelines.md#software-design-principles
    • [ ] contributor’s guidelines | https://p5js.org/contribute/contributor_guidelines.md#working-on-p5js-codebase
    • [ ] ../tasks/build/eslint-samples.js | https://p5js.org/contribute/tasks/build/eslint-samples.js
    • [ ] ../tasks/build/browserify.js | https://p5js.org/contribute/tasks/build/browserify.js
    • [ ] ../tasks/test/mocha-chrome.js | https://p5js.org/contribute/tasks/test/mocha-chrome.js

  • [ ] https://p5js.org/contribute/webgl_mode_architecture/

    • [ ] null | https://p5js.org/contribute/webgl_mode_architecture/images/line-diagram.svg
    • [ ] null | https://p5js.org/contribute/webgl_mode_architecture/images/joins.svg
    • [ ] null | https://p5js.org/contribute/webgl_mode_architecture/images/caps.svg
    • [ ] null | https://p5js.org/contribute/webgl_mode_architecture/images/flags.svg

  • [x] https://p5js.org/examples/imported-media-video-canvas/

    • [x] Video | https://p5js.org/examples/dom-video.html

  • [x] https://p5js.org/examples/imported-media-video-capture/

    • [x] Video | https://p5js.org/examples/dom-video.html
    • [x] Video Canvas | https://p5js.org/examples/dom-video-canvas.html

  • [x] https://p5js.org/examples/imported-media-video/

    • [x] Video Canvas | https://p5js.org/examples/dom-video-canvas.html

  • [ ] https://p5js.org/people/

    • [ ] Ryan Slade | https://p5js.org/people/rdslade.github.io

  • [ ] https://p5js.org/reference/console/log/

    • [ ] console.log | https://p5js.org/reference/p5/console/log
    • [ ] string | https://p5js.org/reference/p5/string

  • [ ] https://p5js.org/reference/JSON/stringify/

    • [ ] string | https://p5js.org/reference/p5/string

  • [ ] https://p5js.org/reference/p5.sound/p5.Oscillator/

    • [ ] p5.SinOsc | https://p5js.org/reference/p5.SinOsc
    • [ ] p5.TriOsc | https://p5js.org/reference/p5.TriOsc
    • [ ] p5.SqrOsc | https://p5js.org/reference/p5.SqrOsc
    • [ ] p5.SawOsc | https://p5js.org/reference/p5.SawOsc

  • [x] https://p5js.org/reference/p5.Table/addColumn/

    • [x] addColumn() | https://p5js.org/reference/p5/addColumn

  • [x] https://p5js.org/reference/p5.Table/addRow/

    • [x] addRow() | https://p5js.org/reference/p5/addRow

  • [x] https://p5js.org/reference/p5.Table/removeColumn/

    • [x] removeColumn() | https://p5js.org/reference/p5/removeColumn

  • [x] https://p5js.org/reference/p5.Table/rows/

    • [x] getRows() | https://p5js.org/reference/p5/getRows

  • [x] https://p5js.org/reference/p5/blue/

    • [x] colorMode() | https://p5js.org/reference/colorMode

  • [x] https://p5js.org/reference/p5/brightness/

    • [x] colorMode() | https://p5js.org/reference/colorMode

  • [x] https://p5js.org/reference/p5/createFilterShader/

    • [x] Introduction to Shaders | https://p5js.org/learn/getting-started-in-webgl-shaders.html

  • [x] https://p5js.org/reference/p5/describe/

    • [x] Writing accessible canvas descriptions | https://p5js.org/learn/accessible-labels.html

  • [x] https://p5js.org/reference/p5/describeElement/

    • [x] Writing accessible canvas descriptions | https://p5js.org/learn/accessible-labels.html

  • [x] https://p5js.org/reference/p5/endShape/

    • [x] writing a custom shader | https://p5js.org/learn/getting-started-in-webgl-shaders.html

  • [x] https://p5js.org/reference/p5/green/

    • [x] colorMode() | https://p5js.org/reference/colorMode

  • [x] https://p5js.org/reference/p5/gridOutput/

    • [x] Writing accessible canvas descriptions | https://p5js.org/learn/accessible-labels.html

  • [ ] https://p5js.org/reference/p5/gt/

    • [ ] > | https://p5js.org/reference/p5/%3E

  • [ ] https://p5js.org/reference/p5/gte/

    • [ ] >= | https://p5js.org/reference/p5/%3E=

  • [x] https://p5js.org/reference/p5/hue/

    • [x] colorMode() | https://p5js.org/reference/colorMode

  • [ ] https://p5js.org/reference/p5/image/

    • [ ] p5.Texture | https://p5js.org/reference/p5/p5.Texture
    • [ ] p5.FramebufferTexture | https://p5js.org/reference/p5/p5.FramebufferTexture

  • [x] https://p5js.org/reference/p5/lerpColor/

    • [x] lerp | https://p5js.org/reference/lerp
    • [x] colorMode() | https://p5js.org/reference/colorMode

  • [x] https://p5js.org/reference/p5/lightness/

    • [x] colorMode() | https://p5js.org/reference/colorMode

  • [ ] https://p5js.org/reference/p5/lt/

    • [ ] << a=""> evaluates to true if the left value is less than the right value. | https://p5js.org/reference/p5/%3C

  • [ ] https://p5js.org/reference/p5/lte/

    • [ ] <=< a=""> evaluates to true if the left value is less than or equal to the right value. | https://p5js.org/reference/p5/%3C=

  • [ ] https://p5js.org/reference/p5/model/

    • [ ] loadGeometry() | https://p5js.org/reference/p5/loadGeometry

  • [x] https://p5js.org/reference/p5/p5.Table/

    • [x] getRows() | https://p5js.org/reference/p5/getRows
    • [x] addRow() | https://p5js.org/reference/p5/addRow
    • [x] addColumn() | https://p5js.org/reference/p5/addColumn
    • [x] removeColumn() | https://p5js.org/reference/p5/removeColumn

  • [x] https://p5js.org/reference/p5/red/

    • [x] colorMode() | https://p5js.org/reference/colorMode

  • [x] https://p5js.org/reference/p5/saturation/

    • [x] colorMode() | https://p5js.org/reference/colorMode

  • [x] https://p5js.org/reference/p5/textOutput/

    • [x] Writing accessible canvas descriptions | https://p5js.org/learn/accessible-labels.html

  • [x] https://p5js.org/reference/p5/WEBGL/

    • [x] the learn page about coordinates and transformations | https://p5js.org/learn/getting-started-in-webgl-coords-and-transform.html
    • [x] the learn page about custom geometry | https://p5js.org/learn/getting-started-in-webgl-custom-geometry.html
    • [x] the learn page for styling and appearance | https://p5js.org/learn/getting-started-in-webgl-appearance.html
    • [x] the learn page section about cameras | https://p5js.org/learn/getting-started-in-webgl-appearance.html#camera
    • [x] introduction to shaders | https://p5js.org/learn/getting-started-in-webgl-shaders.html
    • [x] all the interactive WEBGL tutorials | https://p5js.org/learn/#:~:text=Getting%20Started%20in%20WebGL

  • [x] https://p5js.org/tutorials/color-gradients/

    • [x] createButton() | https://p5js.org/reference/p5/createbutton/
    • [x] Color Radial Gradient Example | https://p5js.org/examples/color-radial-gradient.html

  • [x] https://p5js.org/tutorials/conditionals-and-interactivity/

    • [x] Variables and Motion Tutorial | https://p5js.org/tutorials/variables-and-motion/
    • [x] logical operators | https://p5js.org/examples/control-logical-operators.html

  • [x] https://p5js.org/tutorials/custom-geometry/

    • [x] Styling and Appearance | https://p5js.org/tutorials/styling-and-appearance/

  • [x] https://p5js.org/tutorials/custom-shapes-and-smooth-curves/

    • [x] Melody App | https://p5js.org/tutorials/melody-app-with-nodejs/

  • [x] https://p5js.org/tutorials/data-structure-garden/

    • [x] Organizing Code with Functions | https://p5js.org/tutorials-organizing-code-with-functions/
    • [x] Melody App | https://p5js.org/tutorials/melody-app/

  • [x] https://p5js.org/tutorials/get-started/

    • [x] string | https://p5js.org/reference/p5/string/
    • [x] Strings | https://p5js.org/reference/p5/string/
    • [x] string | https://p5js.org/reference/p5/string/
    • [x] strings | https://p5js.org/reference/p5/string/
    • [x] string | https://p5js.org/reference/p5/string/

  • [x] https://p5js.org/tutorials/getting-started-with-nodejs/

    • [x] Melody App | https://p5js.org/tutorials/melody-app-with-nodejs/
    • [x] p5.Oscillator | https://p5js.org/reference/p5.Oscillator

  • [x] https://p5js.org/tutorials/how-to-optimize-your-sketches/

    • [x] p5.Vector | https://p5js.org/reference/p5.Vector
    • [x] p5.Vector | https://p5js.org/reference/p5.Vector

  • [x] https://p5js.org/tutorials/layered-rendering-with-framebuffers/

    • [x] the createFramebuffer() documentation | https://p5js.org/reference/en/p5/createFramebuffer/
    • [x] an example of Framebuffer depth blur | https://p5js.org/examples/3d-blur-using-framebuffer-depth.html
    • [x] createFramebuffer() | https://p5js.org/reference/en/p5/createFramebuffer/

  • [x] https://p5js.org/tutorials/lights-camera-materials/

    • [x] orbitControl() | https://p5js.org/reference/en/p5/orbitControl/
    • [x] ambientLight() | https://p5js.org/reference/en/p5/ambientLight/
    • [x] directionalLight() | https://p5js.org/reference/en/p5/directionalLight/
    • [x] pointLight() | https://p5js.org/reference/en/p5/pointLight/
    • [x] spotLight() | https://p5js.org/reference/en/p5/spotLight/
    • [x] imageLight() | https://p5js.org/reference/en/p5/imageLight/
    • [x] noLight() | https://p5js.org/reference/en/p5/noLight/
    • [x] fill() | https://p5js.org/reference/en/p5/fill/
    • [x] ambientMaterial() | https://p5js.org/reference/en/p5/ambientMaterial/
    • [x] specularMaterial() | https://p5js.org/reference/en/p5/specularMaterial/
    • [x] shininess() | https://p5js.org/reference/en/p5/shininess/
    • [x] emissiveMaterial() | https://p5js.org/reference/en/p5/emissiveMaterial/

  • [x] https://p5js.org/tutorials/repeating-with-loops/

    • [x] Variables and Motion | https://p5js.org/tutorials/variables-and-motion/

  • [x] https://p5js.org/tutorials/responding-to-inputs/

    • [x] Conditions and Interactivity | https://p5js.org/tutorials/conditions-and-interactivity/

  • [x] https://p5js.org/tutorials/setting-up-your-environment/

    • [x] 2D Primitive shapes | https://p5js.org/referencegroup-Shape/
    • [x] using the p5.js Web Editor with a screen reader. | https://p5js.org/tutorials/using-web-editor-with-screen-reader/
    • [x] How to Use the p5.js Web Editor with a Screen Reader | https://p5js.org/tutorials/using-web-editor-with-screen-reader/
    • [x] color | https://p5js.org/referencegroup-Color/

  • [x] https://p5js.org/tutorials/simple-melody-app/

    • [x] p5.Oscillator | https://p5js.org/reference/p5.Oscillator
    • [x] p5.Oscillator | https://p5js.org/reference/p5.Oscillator
    • [x] p5.Oscillator | https://p5js.org/reference/p5.Oscillator
    • [x] p5.Oscillator | https://p5js.org/reference/p5.Oscillator
    • [x] p5.Oscillator | https://p5js.org/reference/p5.Oscillator
    • [x] p5.Oscillator | https://p5js.org/reference/p5.Oscillator
    • [x] p5.Oscillator | https://p5js.org/reference/p5.Oscillator
    • [x] p5.Oscillator | https://p5js.org/reference/p5.Oscillator
    • [x] p5.Oscillator | https://p5js.org/reference/p5.Oscillator
    • [x] p5.Oscillator | https://p5js.org/reference/p5.Oscillator
    • [x] p5.Oscillator | https://p5js.org/reference/p5.Oscillator
    • [x] p5.Oscillator | https://p5js.org/reference/p5.Oscillator
    • [x] p5.Oscillator | https://p5js.org/reference/p5.Oscillator
    • [x] p5.Oscillator | https://p5js.org/reference/p5.Oscillator
    • [x] p5.Element | https://p5js.org/reference/p5.Element
    • [x] DOM objects | https://p5js.org/referencegroup-DOM/

  • [x] https://p5js.org/tutorials/speak-with-your-hands/

    • [x] Recursive Tree | https://p5js.org/examples/07_Repetition/05_Recursive_Tree/
    • [x] Smoke Particles | https://p5js.org/examples/15_Math_And_Physics/03_Smoke_Particle_System/
    • [x] Interactivity in p5.js | https://p5js.org/tutorial/conditionals-and-interactivity/

  • [x] https://p5js.org/tutorials/variables-and-change/

    • [x] frameCount | https://p5js.org/reference/p5/framecount/
    • [x] frameCount | https://p5js.org/reference/p5/framecount/

Would you like to work on the issue?

yes

ff6347 avatar Sep 08 '24 11:09 ff6347

Thanks @ff6347, this looks great! We have multiple open issues about broken links that we are gradually fixing by batches, since the reasons for the broken links vary.

To provide some context:

  • p5.js website Contribute pages are generated directly from p5.js repo contributor_docs folder, so broken links will be fixed there.
  • p5.js website Reference pages are generated directly from p5.js repo src folder, so we’ll address broken links in that folder.
  • p5.js website Tutorials broken links can be fixed directly on p5.js website repo tutorial folder.
  • Any broken links related to p5.sound are currently on hold, as we’re planning to release a new version of the p5.sound library within the next month.

Here’s my suggestion: Let's use this issue as the main issue to consolidate all the broken link issues. We can update the issue with the latest report when some links are fixed, for examples, some broken links in the report generated on 2024-09-08 are already fixed. For anyone interested in contributing, feel free to start by fixing broken links in the Contribute or Tutorial pages in batches, then we can move to Reference page and others. Please go ahead and open PRs as you work through them. Thank you!

Qianqianye avatar Sep 13 '24 07:09 Qianqianye

hi @Qianqianye thanks for the clarification where to look and start. 👌🏾

I updated the issue description to have check boxes rather then plaintext to track the progress. I also created a script to regenerate the report. https://github.com/ff6347/find-broken-links

I'll ping you as soon as I have the first PR.

ff6347 avatar Sep 13 '24 11:09 ff6347

Hi! I'm new to contributing and I wanted to let you know that I'm interested in fixing the broken links. I will open PR gradually.

thegodworm avatar Oct 19 '24 12:10 thegodworm

So for the contributor docs, I don't get it. reading the doc ; p5js-websites pulls from p5.js then uses astro to build the static website right? " https://p5js.org/contribute/contributor_guidelines.md#software-design-principles" doesn't work but

https://p5js.org/contribute/contributor_guidelines/#software-design-principles works, it seems it's all due to .md not being present, thus refering to the correct page. I don't know how to correct this ( p5.js or p5jswebsite repo?) or the conversion script is missconfigured?

thegodworm avatar Oct 21 '24 14:10 thegodworm

@thegodworm Astro transforms all Markdown .md files to a folder with the name of that file (minus the extension) and an index.html in it.

ff6347 avatar Oct 23 '24 09:10 ff6347

after inspections, the links are broken because they are not built correctly (via script?)

<a href="./pathToContent.md#anchor"> point To X</a> -> error 404.

works when corrected like this:

<a href="../pathToContent#anchor">** point To X</a>

adding ../ relative path correctly and removing the md.

I would ask what would be the best course of actions to take?

writing a script automatically addressing the issues? manually?

or modifying how the script for astro handle site building?

thegodworm avatar Oct 25 '24 11:10 thegodworm

@thegodworm the content is intentionally specifying a .md file so that they work when viewed on GitHub, so I think doing it as part of a script makes sense here. There's already some initial parsing and converting done here that can be modified: https://github.com/processing/p5.js-website/blob/a872651a35b08201d84392e3e65e5e8ed85b8d4c/src/scripts/builders/contribute.ts#L90-L95

davepagurek avatar Oct 25 '24 20:10 davepagurek

@davepagurek yes , I saw that, I will see then what I can do :) . I prefer to ask first because I want to be sure as it's my first collaboration 😃

thegodworm avatar Oct 25 '24 20:10 thegodworm

No problem! Happy to have you helping out 🙂

davepagurek avatar Oct 25 '24 21:10 davepagurek

Just a quick chime in while checking my teaching docs to see if/what broke and found a pretty big one: https://p5js.org/learn/ » https://p5js.org/tutorials/ ?

One idea would be to implement legacy dirs that forward, or better yet a custom 404 page that parses the URL window.location.href and compares to a list of new homes for those links? It could include a form input (if a true 404 without redirect) that creates an issue with expected URL and new URL if they research the solution.

ffd8 avatar Nov 19 '24 23:11 ffd8

I was thinking the same thing. The site needs a custom 404 page.

I'm not 100 % sure where this is deployed. If would be on cloudflare or netlify there is the possibility to configure redirect.

But I think it's GitHub pages with Cloudflare in front of it.

As far as I know, there is not the possibility to do this on GitHub pages. So a 404 page with some common links and a search might be necessary.

ff6347 avatar Nov 20 '24 05:11 ff6347

This is already being mentioned in other issues, ie #676 – but just ran into it as well for the bottom links to objects, ie:

https://p5js.org/reference/p5/createVector, at the very bottom suggests to learn more at https://p5js.org/reference/p5/createVector/#/p5.Vector, however it should be https://p5js.org/reference/p5/p5.Vector There's a link at the top of the page that correctly points to p5.vector.

Same is true for createVideo and I'm sure a handful of others.

ffd8 avatar Jan 17 '25 00:01 ffd8

Hi @davepagurek, I'd like to work on fixing the broken links on the friendly_error_system page.

as @ff6347 mentioned, these reference-style links are still broken:

_friendlyFileLoadError() → https://p5js.org/contribute/friendly_error_system/fes_contribution_guide.md#_friendlyerror _validateParameters() → https://p5js.org/contribute/friendly_error_system/fes_contribution_guide.md#_validateparameters Dev Notes → https://p5js.org/contribute/friendly_error_system/fes_contribution_guide.md#-development-notes

I believe the issue is that inline links in the contributor page markdown are being converted correctly by the rewriteRelativeMdLinks function.

https://github.com/processing/p5.js-website/blob/da0dfb5834e9446a24d5dfa67945a473c16a9d7e/src/scripts/utils.ts#L291-L306

However, the reference-style links (for example, lines 132–141 in friendly_error_system.md) on the friendly_error_system page are not being transformed into the proper relative URLs, which results in 404 broken links.

My thought was to add additional logic in the rewriteRelativeMdLinks function to handle reference-style links as well. Would it be okay if I open a PR for this? (This would be my first PR, so I’m not sure if I should just go ahead and open it, or discuss it in the issue first. Thanks for your understanding!)

junseok44 avatar Sep 03 '25 11:09 junseok44

Also, I noticed a typo in the same friendly_error_system page.

https://github.com/processing/p5.js/blob/f20262d18cf914b72899f1b1a64cdce1233da11b/contributor_docs/friendly_error_system.md?plain=1#L140

— the reference link _fesErrorMonitor() currently points to _fesErrorMontitor. That looks like it just needs a simple typo fix, so I’ll open a separate issue for it in the p5.js repo. Thanks in advance!

junseok44 avatar Sep 03 '25 11:09 junseok44