devdocs icon indicating copy to clipboard operation
devdocs copied to clipboard

Update Godot docs to include v4.2 and fix older version scraping

Open jessevanherk opened this issue 1 year ago • 5 comments

This adds docs for Godot v4.2, and does some fixups for scraping older godot releases from the upstream website.

Godot 4.2 was released on Nov 2023: https://godotengine.org/article/godot-4-2-arrives-in-style/

Since v3.5, godot upstream docs have used some new HTML layouts and file structures. Since older versions still need to be scrapable, this introduces another set of versioned filters for godot.

This change also makes small changes to the filters to handle the current website markup for the previous godot doc versions.

The @gdscript and @globalscope entries can't currently be browsed, because of an encoding mismatch between the frontend and backend; I've identified a possible fix in #2200 .

checklist

If you're updating existing documentation to its latest version, please ensure that you have:

  • [x] Updated the versions and releases in the scraper file
  • [x] Ensured the license is up-to-date
  • [x] Ensured the icons and the SOURCE file in public/icons/your_scraper_name/ are up-to-date if the documentation has a custom icon
  • [x] Ensured self.links contains up-to-date urls if self.links is defined
  • [x] Tested the changes locally to ensure:
    • The scraper still works without errors
    • The scraped documentation still looks consistent with the rest of DevDocs
    • The categorization of entries is still good

QA notes

I generated all of the godot doc versions that the scraper supports, and diffed them against the existing packaged versions to ensure as close of a match as possible.

Command to generate a local copy of the godot 4.2 docs:

bundle exec thor docs:generate [email protected]

Spot checks show that the main differences are:

  • the attribution statement, which I updated based on the attribution listed on all of the godot public docs going back to 2.1.
  • class_@gdscript and class_@globalscope filenames are escaped with %40 instead of @. I have no idea how the previous packager got those files to be @ symbols, I suspect the godot website used to encode those differently.
  • minor text edits that have been committed to upstream docs in the meantime
  • minor markup changes in the upstream HTML over time (mainly on the 3.x series)

jessevanherk avatar Apr 22 '24 05:04 jessevanherk

Hi. Thanks for doing this!

I just tried your changes and getting a 403 when trying to download Godot 4.2 docs.

root@990bc51c4450:/devdocs# bundle exec thor docs:download [email protected]
(1/1) Godot 4.2 FAILED (OpenURI::HTTPError: 403 Forbidden)
Done

I tried visiting https://docs.godotengine.org/en/4.2 on a browser and it doesn't look like I've been blacklisted or anything.

Regards

gordon-boom avatar May 04 '24 11:05 gordon-boom

root@990bc51c4450:/devdocs# bundle exec thor docs:download [email protected]

@gordon-boom I haven't done anything around uploading a packaged version to devdocs yet, so I wouldn't expect that to work at this point. If there's a step I need to do as part of this PR to get that working, let me know what it is and I'd be happy to do it!

The command I've been using thus far for testing is: bundle exec thor docs:generate [email protected]

jessevanherk avatar May 08 '24 14:05 jessevanherk

root@990bc51c4450:/devdocs# bundle exec thor docs:download [email protected]

@gordon-boom I haven't done anything around uploading a packaged version to devdocs yet, so I wouldn't expect that to work at this point. If there's a step I need to do as part of this PR to get that working, let me know what it is and I'd be happy to do it!

The command I've been using thus far for testing is: bundle exec thor docs:generate [email protected]

Ah silly me. I guess I didn't read the difference between download and generate. I will try that next time.

gordon-boom avatar May 08 '24 16:05 gordon-boom

Just tried it out. Looks good! Let's hope it gets merged.

gordon-boom avatar May 09 '24 16:05 gordon-boom

This works for me.

mattgleeson avatar Jul 11 '24 16:07 mattgleeson

a7c9db2fe8143675ebcd1d54a93b3c3881c40eea

Maungmaunglwin096062286 avatar Sep 14 '24 21:09 Maungmaunglwin096062286