wikiteam Split dumpgenerator.py into modules

Split dumpgenerator.py into modules

Open emijrp opened this issue 10 years ago • 10 comments

Hello. As said in other issues (#142) and discussions, the size of dumpgenerator.py is becoming too big. The solution is split into several modules. As we want to expand to other wiki engines, the structure must reflect this, probably creating one module per engine.

I have open this issue to discuss that future structure, functions and module naming, etc. I propose:

dumpgenerator.py: main script (may be the moment for a better name?) |_dg_mediawiki.py: module with functions for mediawiki dumps ...|_mwGetPageTitles(): function for retrieve page titles ...|_mwGetImageNames(): function for retrieve image names |_dg_dokuwiki.py: module with functions for dokuwiki dumps ...|_dwGetPageTitles(): ... ...|_dwGetImageNames(): ... |_dg_wikifarms.py: module with spiders to retrieve lists of wikis from wikifarms ...|_wfGetListOfWikis(): generates a list of wikis for a wikifarm |_dg_tools.py: module for other useful stuff ...|_getWikiEngine(): returns wiki engine for a URL

As complexity increases, I think we need to offer a PyPi package and a .exe for Windows users.

Jul 04 '14 19:07 emijrp

Maybe this would be better as part of the rewrite branch?

Jul 04 '14 20:07 balr0g

Agreed, we should implement the module-based system into the rewrite branch instead and remove all the other files not related to dumpgenerator.py itself to other repositories in the GitHub organization.

Jul 26 '14 12:07 Hydriz

Documentation for Python modules and packages https://docs.python.org/2/tutorial/modules.html#packages

wikiteam/

init.py mediawiki.py dokuwiki.py wikifarms.py tools.py

Sep 22 '14 12:09 emijrp

I have given this issue High-Priority. We need to split into modules, create a package and the pip install option. Who can help in this?

Sep 23 '14 11:09 emijrp

I have given this issue High-Priority. We need to split into modules, create a package and the pip install option. Who can help in this?

It would be easier to help if there was a more clear plan. Are we supposed to add this to the rewrite branch? It's about 60 commits behind.

Sep 24 '14 14:09 PiRSquared17

I'd love to get at least the dumper to the PyPI so that pip could be used to install this script.

How do we stand on that? Should I just start rewriting that? Now that we have Travis that might be quite good thing to do.

BTW: the rewrite branch is now 1 commit ahead, 87 commits behind master.

Oct 07 '14 15:10 mrshu

I just discovered this project and I was happily using the software until I discovered that dokuwiki support is not there and development doesn't seem very active.

Is there any interest on revitalizing the project? Where the team lurks?