refactor: load MagicMirror modules from JSON index instead from wiki

[KristjanESPERANTO]

I hope you are open to this. This PR migrates from scraping the MagicMirror wiki to consuming the official JSON modules index at https://modules.magicmirror.builders/data/modules.json.

Changes

• mmpm/constants/urls.py: Update MAGICMIRROR_MODULES_URL to point to the new JSON endpoint
• mmpm/magicmirror/database.py: Replace HTML scraping logic with JSON parsing in __download_packages__()
• mmpm/magicmirror/package.py: Replace from_raw_data() with from_json() and map new field names (name → title, url → repository, maintainer → author)
• tests/magicmirror/test_database.py: Mock HTTP requests and add test coverage for error scenarios (invalid JSON structure, network errors)
• pyproject.toml: Remove beautifulsoup4 dependency

Motivation

The MagicMirror maintainers (I'm one of them) plan to deprecate the wiki list (There is no fixed schedule yet). This change removes the dependency on wiki scraping and aligns with the official modules index.

All other projects which rely on the module list (like MMM-Remote-Control, MMM-ModInstall and MMM-Config) are already using the new API. So mmpm is the last one (at least of the projects we know).

[Bee-Mar]

On the surface this looks good to me. I must have misunderstood that endpoint was the official API. This should certainly increase stability. Later this evening or tomorrow I'll be able to look through the code more and do some local testing but otherwise I'm on board.

[KristjanESPERANTO]

This should certainly increase stability.

Yes, this is one of the main intentions :slightly_smiling_face:

Besides, through the automation pipeline, the new API also offers some more information (such as GitHub stars or keywords/tags) that consumers could use additionally.

otherwise I'm on board

Glad to hear that! :smiley:

I must have misunderstood that endpoint was the official API.

For a better understanding, I've wanted to visualize it for a long time. Now I've finally gotten around to it. This is how I imagine it:

Current flow

flowchart LR
  wiki[(module wiki list<br><i>- unreliable -</i>)]
  pipeline{{automation pipeline}}
  api[(API<br>modules.json)]
  remote[MMM-Remote-Control]
  modinstall[MMM-ModInstall]
  config[MMM-Config]
  mmpm[mmpm]
  moduleWebsite[website<br>modules.magicmirror.builders]

  wiki --> pipeline --> api
  api --> remote
  api --> modinstall
  api --> config
  api --> moduleWebsite
  wiki --> mmpm

Target flow

flowchart LR
  ui[(Form-based front end<br>for adding, editing, and<br>deleting modules<br><i>- not yet conceptualized -</i>)]
  pipeline{{automation pipeline}}
  api[(API<br>modules.json)]
  remote[MMM-Remote-Control]
  modinstall[MMM-ModInstall]
  config[MMM-Config]
  mmpm[mmpm]
  moduleWebsite[website<br>modules.magicmirror.builders]

  ui --> pipeline --> api
  api --> remote
  api --> modinstall
  api --> config
  api --> mmpm
  api --> moduleWebsite

The target concept replaces the wiki list with a form-based frontend for adding, editing, and deleting modules (still in the conceptual phase) while downstream consumers continue using the unchanged API endpoint independent of the start point of the flow.

[KristjanESPERANTO]

Hm... sadly the diagrams are not displayed on the GitHub app. But in a browser they get rendered.

[Bee-Mar]

This weekend is actually more likely for me to look through this there will likely need to be adjustments in the user interface models as well with the object keys being adjusted

[KristjanESPERANTO]

there will likely need to be adjustments in the user interface models as well with the object keys being adjusted

I don't think so. The new parser maps to the same object keys like the old parser - so your internal interface stays the same :slightly_smiling_face:

[Bee-Mar]

@KristjanESPERANTO you'd think I'd able recall that 😂 it's been so long since I've made any substantial changes to the codebase. I still need to actually double check everything, work's been extremely busy as of late

[Bee-Mar]

@KristjanESPERANTO things look good, but could you also bump the minor version to 4.3.0? The pyproject.toml, version.py and the change log would need updated with that version number

[KristjanESPERANTO]

could you also bump the minor version to 4.3.0?

Just done :slightly_smiling_face: