devdocs
devdocs copied to clipboard
freeCodeCamp
Python: scrap full documentation (instead of library reference only)
Currently, DevDocs only scraps Python library reference rooted at https://docs.python.org/2/library/index.html.
It would be great to scrap the full documentation rooted at https://docs.python.org/2/.
I agree, the same for Python 3.
For example, the global keywork is only documented in the Language reference at https://docs.python.org/3/reference/simple_stmts.html#the-global-statement
I guess at least the /reference/ should be added.
Would changing
self.root_path = 'library/index.html'
options[:only_patterns] = [/\Alibrary\//]
by
options[:only_patterns] = [/\Alibrary\//, /\Areference\//]
or
self.root_path = 'index.html'
options[:only_patterns] = [/\Aindex\.html/, /\Alibrary\//, /\Areference\//]
in https://github.com/freeCodeCamp/devdocs/blob/master/lib/docs/scrapers/python.rb be enough ?
This also means that the Python C API documentation isn't included, because it's under https://docs.python.org/3/c-api/
Same here. Every other documentation software (Dash / Zeal / Velocity) features the full documentation, but DevDocs is lagging behind.
@simon04, thanks a lot for the update! Do you mind if point an improvement point?
I tried searching for except and for try, because I wanted to see the documentation of the syntax of that statement. I couldn't find it in devdocs. Then I looked at The Python Language Reference, and from there I could find the link to The try statement. That means the documentation is scrapped and available, but hard to find. Observe:
- Compound statements page is available and can be found from the devdocs search.
- But any headers/sections under that page are not found through devdocs.
- For comparison, the datetime — Basic date and time types page is available, and we can easily find anything inside it, like datetime.date.today.
I have only checked the Language Reference section. I'm not sure if the same applies to the other parts of the documentation.
Can you please take a look at this? Or should I open this as a new bug?
Hi @denilsonsa, thank you for your suggestions. I've included h2 entries for the reference, tutorial, using pages. I've updated the docs to Python 3.10.7
Nice but entering try in the search box when C++, Rust, and Go docs are enabled leaves the Python 8.4 The try statement below the relevance threshold. It would be nice if actual keywords were treated as index terms, too.
Steps: Enter try.
Expected: an entry with a Python icon, named try, opening 8.4 The try statement.
Steps: Enter except.
Expected: an entry with a Python icon, named except, opening 8.4 The try statement.
Bonus points if entering else displays entries for 8.1 The if statement, 8.2 The while statement, 8.3 The for statement, 8.4 The try statement, 8.9.2 The async for sttement, but I don’t know if there’s enough markup in the original pages to pull this off.
@yurikhan , @simon04 that could potentially be solved by incorporating the upstream index: https://docs.python.org/3.10/genindex.html
I mean, from that index I can find a couple of pages/sections/anchors related to the try keyword, and eight for else. If the terms from that index can be incorporated into devdocs' own indexing feature, that could make the relevant docs easier to find. (i.e. Terms from the upstream index get added to a hidden synonym list inside each respective entry; that means "The if statement" would still show up in the search results when searching for "else" or for "else keyword".)
No idea how much extra work that would require, but that's something that has the potential of benefiting any other sphinx-based documentation.
Hi @yurikhan, @denilsonsa, thank you for your suggestions. I tried to address your suggestions in the latest python docs update. Please see the commit messages above for details. Thanks!
It's amazing! I can find stuff very quickly! I tried try, with, but also symbols like ? or (? or -.
else and except don't yield very good results; and if (conditional expression) can't be found (when searching for if). But, frankly, it's already good enough and we appreciate all the valuable work @simon04 have done!
Interestingly, the upstream Python docs are missing the proper index entries for finding the := walrus operator. If you search the 3.10 index, you can't find any of "Assignment expressions", "walrus" or ":=". Thus, this isn't a bug in devdocs, but in upstream docs. (And I've notified them by e-mail; they are tracking it at https://github.com/python/cpython/issues/97913 and https://github.com/python/cpython/issues/97914)
Hi, I forgot to report on the latest improvement (commit above from 2022-10-05): now, all entries from the genindex linking to /stmts/ are added. Previously, only non-text statements (such as **) were added. Now also for, if, elif, else, ... are added.