best practices for GitHub pages?

[cs01]

GitHub pages are a common way to host documentation for sites. They require one of the following to host in source code:

• master branch /docs folder
• master branch
• gh-pages branch

Right now I'm using master branch /docs for pygdbmi, and I have what I would assume is a very common project structure, yet my workflow has a little hack in it. I generate my docs with the following make recipe:

docs:
	pdoc --html  --force --output-dir docs pygdbmi
	mv docs/pygdbmi/* docs
	rmdir docs/pygdbmi

Is there a way to generate docs directly to the --output-dir? By namespacing it with the package name, it breaks it for GitHub pages.

[kernc]

What's wrong with invoking mv as above?

[kernc]

pdoc CLI usage allows for more than one module/package specified:

pdoc --html mypackage1 mypackage2 ...

1. Are functions returning Union[T, List[T]] really such a good idea?

[kernc]

Generating all modules/packages to a single landing page at index.html (that then point to package_1/index.html, package_2/index.html)

Then you'd have another doorway page (like the --http one) exhibiting a single link to your project documentation. Would you like that (i.e. leave it in docs dir as is, without moving)?

Disallowing multiple projects to have docs generated at once

No. Specifying multiple projects is the mechanism by which you make pdoc aware of, and thereby link to, symbols in disjunct packages.

---

The problem with 1. is that subsequent runs are not deterministically idempotent.

$ pdoc --html -o docs project1

# Creates a different filesystem hierarchy.
# Doesn't reliably warn for overwriting project1.
# Leaves docs/index.html to remain stale and misleading.
$ pdoc --html -o docs project1 project2

This idea also won't work because it would break existing CI / deployments.

[kernc]

I'm not convinced that running mv, explicitly, hopefully scripted, represents a hassle.

Let's maybe leave this open for a while and see how many votes it accrues.

[florimondmanca]

Hey! I'm interested in pdoc, but my use case is that of generating pages and integrating those within another generated documentation, so I won't be deploying the output of pdoc to GitHub Pages directly.

All that said, I totally agree that at least documenting a typical workflow for GitHub pages would be a nice extra.

[KyleKing]

FWIW, I have a DoIt script in Python that does some filesystem manipulation to ouput documentation to gh-pages. Would love feedback if there is a better to do this

Process:

1. Run pdoc
2. copy pdoc output to dash_charts-gh-pages/
3. Commit dash_charts-gh-pages/

Directory Structure:

• parent_dir/
  • dash_charts-master/
  • dash_charts-gh-pages/

Code:

See task_document and task_commit_docs from https://github.com/KyleKing/dash_charts/blob/master/dodo.py (note: also moves a directory with example Python code temporarily into module so they appear in the documentation, but isn't necessary)

Result: https://kyleking.me/dash_charts/

[postlund]

I have based the documentation for my project on the Cayman theme, but modified it quite a bit to fit my needs. The API reference (which is still very much work in progress) is generated by pdoc and checked in statically. To integrate it with Cayman, I modified the template to include some front matter. This also allows me to change permalink to get rid of the .html extension, so that it integrates nicely. Also had to do some other minor tweaks, e.g. fix CSS.

Generating the output is done with a small python hack I did, which also contains a "verify" action that checks if the reference is up-to-date with code and fails otherwise. The "verify" step is run by tox (and in turn GitHub Actions), so it's not possible to merge anything if the documentation is "old".

Some links for inspiration:

• The documentation can be seen at at pyatv.dev/api
• Templates for pdoc is here
• Script to generate API documentation is here (integration with tox here)

Not in any way a perfect solution, but works pretty good for my needs. Maybe it serves as inspiration for someone else. 🍰

[rirze]

I have a simple redirect at docs/index.html that contains:

<meta http-equiv="refresh" content="0; url=./{{project-name}}/" />

where one should obviously replace {{project-name}} with their project name. This way I can generate my documentation directly with using the -o docs flag and simply commit the changes.

---

A more performant line suggested by kernc:

<link rel="preload prerender" as="document" href="./{{project-name}}/">

[yohplala]

Hi there, just to say I face the same problem. I guess I will be using @rirze 's proposal.

[epogrebnyak]

Still maybe a new switch for direct output dir can work? Like pdoc --html -O docs pkg? I think it is an elegant way to use Github Pages capabilty. By requiring mv and rmdir we are loosing the user base and simplicity of use, which is pdoc great strenght.

[dkapitan]

@rirze's solution serves my needs. Here is my current Python package workflow with cookiecutter, pdoc3 and GitHub.

Setup package with cookiecutter

mkdir myproject
cd myproject 
cookiecutter gh:audreyr/cookiecutter-pypackage
pip install -e myproject

Setup documentation with pdoc3

• Create documentation in /docs folder

  doc --html -o ./docs myproject
  

• Add redirect in ./docs/index.html

  meta http-equiv="refresh" content="0; url=./{{my project}}/index.html" />
  

Setup GitHub repo

• Create a new, empty GitHub repository and follow instructions to do your first commit
• Setup GitHub Pages using the from master branch in /docs folder option in your repository settings

[epogrebnyak]

Why not save the ease of use for skipping

Add redirect in ./docs/index.html

 <meta http-equiv="refresh" content="0; url=./{{my project}}/index.html" />

entirely?

Many users may have tried sphinx, the workflow there is very clean:

sphinx-build source_dir docs
cd docs
touch .nojekyll 

In pdoc there is historically a hidden and a bit unexpected behaviour that output dir is not a direct link, but parent directory.

What I'm suggesting is too look at the angle of ease of use and predictability of behaviour. My suggestion in adding a new switch that eliminates hidden behavior, the existing switch is left as is, because it is useful for generating several packages' documentation outside of Github Pages usecase.

The intended workflow can be:

pdoc --html -O docs source_dir 
cd docs
touch .nojekyll 

[epogrebnyak]

Update: for pdoc we do not even need .nojekyll, so entire documentation Github Pages will be oneliner (and with sphinx it is not):

pdoc --html -O docs source_dir 

Looks like appealing feature to me, advantageous to "there is a best practice advice with several steps involved".

[rirze]

Because the idea is that you can have multiple modules within a project. It would be great if you could have a simple -O flag that does this, but I would ask you to really think about how to implement with the concerns that @kernc raised in this thread.

How do you handle multiple modules? How do you handle projects with a single module? What happens if you switch from single module to multiple modules?

If you have answers to these, then, please let us know and even maybe contribute a patch. Until then, I think more manual specification using the redirect is better because there is less under-hood-logic for where the entry point for your docs should be.

Plus, setting the redirect is at best a one-time step, at worst, an uncommon edit. It does not affect workflow, only initial setup and possibly major refactors.

[epogrebnyak]

Not sure I understand a use case case with multiple modules for a project, can you point to examples?

One piece of documentation is for one package, right?

[epogrebnyak]

What I'm tring to document is:

my_project 
 - docs
 - src
    - my_project 
       - __init__.py
       - a.py
       - b.py 

pdoc will create documentation in docs/my_project, no way to specify a direct output folder. For single package I want the documentation just go to docs. If your usecase is pdoc a.py b.py you use a -o switch, if you documnt a single package or a signle folder, you use an -O switch. Or is this too much confusion?

[rirze]

I'm no expert at Python packaging, but I'll try to explain what I can:

A project can consist of multiple packages. The bare minimum that qualifies a package in a project is that a package resides in a top level directory in the project/source root with a __init__.py.

This is supported in Python packaging, notably when specifying packages=find_packages() in setup.py. Python packaging supports multiple modules being installed from a single project.

I cannot confirm this, but I think it is standard practice to not do this, as it lead to confusion and so most authors isolate projects to one module. I could not find any popular packages that bundle multiple modules together. Using your example, a valid Python project can look like this:

my_project 
 - docs
 - src
    - my_project 
       - __init__.py
       - a.py
       - b.py 
    - some_other_project
	   - __init__.py
	   - c.py

I understand your use case only uses one package (as do all my packages as well), but pdoc3 supports documentation for multiple packages in the same project. You can make the point that it shouldn't or that this behavior could be disregarded for the -O flag but that's deserves a different discussion.

Hopefully this clarifies the topic.

[kernc]

Not sure I understand a use case case with multiple modules for a project, can you point to examples?

One may want to specify multiple modules for various reasons and use cases:

$ pdoc $(find -name \*.py -and -not -path \*exclude\* | filter however)

Note, HTML documents from a single command-line run are all interlinked.

Pdoc can also be used as a browsable cross-referenced on-the-fly external docs generator:

% pdoc3 -o. html urllib re
% tree
.
├── html
│   ├── entities.md
│   ├── index.md
│   └── parser.md
├── re.md
└── urllib
    ├── error.md
    ├── index.md
    ├── parse.md
    ├── request.md
    ├── response.md
    └── robotparser.md

2 directories, 10 files

[kernc]

pdoc will create documentation in docs/my_project, no way to specify a direct output folder.

You can create and commit the meta-refresh file as @rirze recommends. You need to commit it only once. If you worry about redirect delay/performance, might add in there another:

<link rel="preload prerender" as="document" href="./{{project-name}}/">

If you worry about result URL prettiness, it's just one mv-equivalent command in your docs building configuration.

---

-O switch

With literally limitless workarounds available, I honestly don't think this warrants complicating logic or a new CLI switch.

[epogrebnyak]

packages=find_packages()

Used to find subpackages, not different packages in a single project. Most python projects are single init.py, possibly with sub packages, otherwise what do you assign single package name= to in setup.py.

[epogrebnyak]

pdoc3 supports documentation for multiple packages in the same project

Which cannot exist for for pip-installable packages with setup.py.

[kernc]

Used to find subpackages, not different packages in a single project.

@epogrebnyak This is what the docs say:

packages is a list of all Python import packages that should be included in the distribution package. ...

A single distribution package can indeed package several import packages. Whether it should, I don't know.

[rirze]

Just want to chime in-- I agree with @kernc.

Subpackages mean something else-- they are sub-packages meaning they reside under a package. To the other point, it is indeed possible to ship two packages using one setup.py file using the file structure I showed above. Both of those packages would be the first name in an import statement (import my_project.a & import some_other_project.c).

If it helps, here's how poetry documents the configuration option packages for users. The documentation string should verify what we've been saying.

I feel like this has gone on a tangent. Bottom line is, the Python specification supports multiple packages being installed with one distribution file. Pdoc3 adopts that same specification. The current way pdoc3 generates documentation is generic for any number of packages in a project.

As for the future, pdoc3 could make adjustments to accommodate single-package distributions, but that would involve case-by-case logic and development and testing. It ultimately comes down to the contributors as to whether or not support the use case.

[epogrebnyak]

If it helps, here's how poetry documents the configuration option packages for users. The documentation string should verify what we've been saying.

Same docs:

If your project structure differs from the standard one supported by poetry, you can specify the packages you want to include in the final distribution.

You seem to have a strong focus on border case and not the main case. Why exactly the documentation has to be on distribution level and not on package level is unclear. It is possible to ship two packages in one distro, but who does that, why joint documentation for two packages, etc.

In sphinx interwaving different packaed is done through extension, you seem to have a strong stake in that as core feature.

It ultimately comes down to the contributors as to whether or not support the use case.

Fair enough, it is your ideas of a usecase, maybe can work on a fork to simplify the API, perhaps something useful can come out of it.

Thanks for all for detailed discussion, sorry it drove us away from Github Pages advice.

[kernc]

You seem to have a strong focus on border case and not the main case.

You seem to have a strong conviction that your case is the main case? :raised_eyebrow:

pdoc is reputed to do things exactly the opposite from Sphinx. I guess let's sit on it for a while longer.

[epogrebnyak]

You seem to have a strong conviction that your case is the main case? 🤨

Just what poetry says about non-standard project configuration, not my words.

pdoc is reputed to do things exactly the opposite from Sphinx. I guess let's sit on it for a while longer.

Good it is not accidental. Maybe more explaination it needed in the docs/man page, such as -o behavior, more people will know what to expect, can provide that for your review.

[kernc]

If any such command line switch were added, it would be named more in line of --single-toplevel. But I don't understand why we'd want to pollute everyones's namespace when better-suited, concise and powerful utilities are around.

mv docs/project/* docs
move ...  # on Windos

Now, after https://github.com/pdoc3/pdoc/pull/197, it'll certainly be clearer what's going on and where. Documentation improvements are welcome, of course.

[rirze]

Sorry, I didn't want to respond further, but your classification/understanding is misleading and I don't want to leave the state of the thread at that.

Classifying this as the "border" case is wrong. A more appropriate term would be "generic" case. It may be the case that 99% of users only package one module per distribution, but that is not the point being made or defended here. There is a reason why poetry ended up supporting multiple packages (rather than right out rejecting the feature request because this coexists with your "use case"), when in my opinion, any capable Python programmer in the project should have supported it from the start.

I'm going to unsubscribe from this topic. In summary:

• I came here to share a handy tip for accommodating GitHub Pages with pdoc3
• Had to convince someone that a core feature of the Python distribution specification exists.
• Received responses that only nitpick the answers given and didn't appreciate the facts that were being shown.
• Ended with a declaration of how pdoc3 doesn't fulfill their vision of a package documentation tool.

The way you're arguing is not beneficial nor helpful. Just... reflect on that.

[ghost]

You could simply setup a redirect

[ethanabrooks]

I have a simple redirect at docs/index.html that contains:

<meta http-equiv="refresh" content="0; url=./{{project-name}}/" />

where one should obviously replace {{project-name}} with their project name. This way I can generate my documentation directly with using the -o docs flag and simply commit the changes.

Just for n00bs like me, this looks like a file called index.html with the following content:

<!DOCTYPE html>
<html>

<head>
    <meta http-equiv="refresh" content="0; url=./{{project-name}}/" />
</head>

</html>