gno gno doc command + rpc + gnoweb view

The underlying idea is to provide roughly the same functionality as the go doc command, as well potentially that of godoc (ie. http documentation viewer/generator).

So the purpose and scope is that to be a central command and reference points to all things related to technical documentation. It should not contain tutorials and guides, but it should contain precise and up-to-date information on all tooling, commands, and specifications related to the Gnolang language.

The way I see it there are a few different types of documentation we should provide:

Standard library :heavy_check_mark:

Examples: gnodev doc errors, gnodev doc std

While most of these can be provided in an identical fashion to how Go doc parses documentation on symbols and packages, there should also be documentation for "native" types which are mapped to go types using stdlibs/stdlibs.go. The way this should be done is by declaring the prototype without the documentation, like Go for does for the symbols implemented in assembly. Providing them directly as prototypes in code also helps to develop future tooling (e.g. code autocompletion, which can be based only on the AST without exceptions for builtin code).

There should also be a documentation-only builtin package, similar to Go's, providing documentation about builtin functions and types.

gnodev tries to determine the directory of stdlibs in the same way as gnodev test currently does, and provides the --root-dir function to that effect. Potentially in the future we could provide some functionality similar to GOROOT, although this is more in the scope of #460.

Local packages and realms :heavy_check_mark:

Examples: gnodev doc ., gnodev doc ./examples/gno.land/p/demo/avl

This should work roughly as the above, parsing the AST with the comments to provide documentation about the package.

References to other packages are parsed and understood in a local context if subpackages of the current (determined through gno.mod). Otherwise, they are understood as remote packages and the --remote flag is used as below (though for CLI usage the remote will not be used).

:question: Should we add further metadata like the one that gnoview displayed, implementing #477?

Remote packages and realms

Examples: gnodev doc --remote 'staging.gno.land:36657' gno.land/p/demo/avl

This should query the remote server for documentation about a package or a realm. So the functionality is similar to that returned by the ABCI query vm/qfuncs - though for this use case there should probably be a specific query which also works on packages and is designed specifically for serving on the documentation (so has raw package documentation and documentation for all the exported symbols, ie. used as a documentation for programming realms instead of for making calls using maketx). Though for the exported functions of a realm, it could provide the same functionality that ?help is currently providing (ie. showing the commands to interact with the contract).

Manually written manual pages

Examples: gnodev man, gnodev man gnokey, gnodev man gnomod, gnodev man abci vm/qeval.

Like UNIX man, but Markdown based. This would pull markdown files as topics from the docs directory in the root, or it could also be built into gnodev using go:embed (possibly guarded by a build tag). So these provide manually written documentation about topics which are related to programming in Gno and using the command line tools of the project. Again, these are not how-tos, but specific documentation about what each flag does.

Command documentation (gnokey, gnoland, gnofaucet...)
ABCI query documentation (vm/qeval, vm/qfuncs, ...)
References (like these: interpreter internals, gnomod, language spec, ...)

HTTP viewer and generator

Considerable when everything else is done and implemented. But many of these references could also be useful to publish online, or simply for users who prefer a graphical interface to view all documentation locally when developing. So the gno.land website could be extended to provide all of the documentation provided in the command gnodev doc and gnodev man, in a similar fashion to what godoc does. This would mean

expanding ?help to provide also documentation for other symbols which are not smart contracts
providing a /man sub-path where the manual pages can be found
providing a way to browse the stdlib source and documentation
showing ?help also on packages, only to the extent of providing developer documentation (not smart contracts)

Options/functionality overview

--remote, --root-dir, --home like in other commands.
--all like go doc (when asking for the documentation of a package, expand all symbols)
--src like go doc (show the full source code for the symbol)
--unexported like go doc (show/document unexported symbols, unavailable for remote querying).

For gnodev man, only --root-dir applies

Feedback is very welcome, also on the design of the command line and flags.

Related: #28, #379, #408, #477

Feb 17 '23 12:02 thehowl

While most of these can be provided in an identical fashion to how Go doc parses documentation on symbols and packages.

We will probably have more metadata at the (see #498), and we should start parsing comments to extend the understanding of packages by our VM and dev tools.

There should also be a documentation-only built-in package, similar to Go's, providing documentation about built-in functions and types.

Yes.

gnodev tries to determine the directory of stdlibs in the same way as gnodev test currently does, and provides the --root-dir function to that effect. Potentially in the future, we could potentially provide some functionality similar to GOROOT, although this is more in the scope of https://github.com/gnolang/gno/issues/460.

Yep, also related to gno.mod (#479).

❓ Should we add further metadata like the one that gnoview displayed, implementing https://github.com/gnolang/gno/issues/477?

Later yes; short term, no.

The current focus for gnodev should be to support developers working locally. We have a first basic internet/net dependency feature with gnodev mod download. Step 1 is to focus on the source code to have something similar to go toolkit.

Step 2 will create and integrate tools to play with the production state, i.e., discovering and inspecting realm data, synchronizing it locally, etc.

Examples: gnodev doc --remote 'staging.gno.land:36657' gno.land/p/demo/avl

Yep, the idea is to make this transparent with the following:

per-remote caching folder (in $GNOROOT)
lazy gno mod download when something is missing
then, make all the commands work locally

HTTP viewer and generator

I love all the enumerated ideas 👍 .

Question: do you plan to work on this as a whole, or can you split this into smaller parts?

Thank you

Feb 17 '23 13:02 moul

Smaller parts, roughly matching the 5 paragraphs I would think. So a first POC on the standard library, and then build up from there. Also to keep the PRs small and focused :)

Feb 17 '23 14:02 thehowl

We need to enhance our approach to documentation. Here's how:

Go beyond the basics. Instead of just "documentation is for users," (https://docs.gno.land/concepts/effective-gno/#documentation-is-for-users) let's make our API self-explanatory. Include method signatures, a README.md, developer-suggested examples, and maybe even unit test names, we can create new standards to express documentation from code or .gno files in general. This should help users and developers understand and use the code directly from the source, without needing extra tutorials.
Integrate documentation with our tools. Make the API docs accessible in GnoWeb. This way, when you run GnoWeb locally, you get all the documentation for installed and developed realms, without needing extra tools or internet access.
Keep documentation in sync. Embed the docs/ folder from the repo into GnoWeb. This ensures you always have the most up-to-date documentation that matches your current git commit, even offline.

In short:

We aim to improve discoverability, transparency, and trust for users and developers.
We want a unified development experience. Develop locally, see the results, and update documentation and code in real-time before pushing to GitHub or on-chain. No more guessing the impact of your changes.

Mar 20 '24 17:03 moul

@gnolang/berty hey. I was talking with Manfred about possibly delegating out to Berty:

creating RPC endpoints on gnoland to get gnodoc for symbols.
possibly then integrating this in gnoweb.

He mentioned @D4ryl00 as a good candidate. Do you think you folks have capacity to do this?

Just to kick-start a discussion. This issue is a good meta-issue to others, but let's discuss scope ahead of working on it.

Nov 28 '24 16:11 thehowl

@gnolang/berty

apologies for the slow turnaround. here's some more context and a quick list of requirements:

current state

gno doc command line works, largely untouched over the past year
some missing features
no way to inspect documentation for packages, realms and docs on-chain
gnoweb changes incoming -> https://github.com/gnolang/gno/pull/3195
- we have a new and better codebase in which we can integrate these changes; but let's touch base with Alexis and Guillhem on implementations so we know what design to have.

where we want to get

gno.land RPC able to show documentation about packages and symbols
gnoweb able to show the documentation for everything that can be shown as source
- using the RPC from above
gno doc able to query on-the-fly about packages it doesn't have downloaded, through the RPC
- optional, but POC on the fact that there should be multiple clients of this "doc RPC": i envision adena to use it, and gno studio connect to eventually become somewhat redundant.

related work

chain domain: https://github.com/gnolang/gno/pull/2911
- tl;dr: next testnets will have their paths prefixed with testN.gno.land rather than being gno.land in general. as a consequence:
remote package download: https://github.com/gnolang/gno/pull/3123
- this is currently implemented as gno mod download, though i requested norman to eventually move this to gno get. it derives the RPC path from the "chain domain" by querying rpc.<domain>:443, this is a hack for now (eventually we want to use r/sys/zones) but can be something we also use for remote querying
package loader centralization: https://github.com/gnolang/gno/pull/2932
- this should eventually be implemented by the doc package as well, rather than rolling out its own system to create

what there is to be done

let's:

clean-up the pkg/doc codebase as necessary, and making it useful to not only work on the filesystem but also on the chain; ie. supporting MemPackage or a subset of the Store to fetch packages both on-chain and on the FS.
create RPC endpoints to get documentation; I expect you to be able to be smart and have relative liberty here to come up with specifications as you see fit, but this is roughly what I imagine
- query vm/qdoc, takes a package path and an optional symbol (can be a similar syntax to vm/qeval for the input, so gno.land/p/demo/avl.Tree.Get).
  - returns an amino-encoded data structure which contains the data of the documentation, either at package-level or for the symbol.
  - note: this shouldn't try to do smart matching as the current gno doc on the filesystem; let's expect a full pkgpath rather than allowing just avl. similarly, it shouldn't discriminate between exported/unexported (let this be done by the clients, so assume -u by default) and be case-sensitive when matching symbols.
include this in gnoweb; as I said, let's maybe ask @alexiscolin and @gfanton to pitch in and be involved in any designs/structural decisions.
- maybe work on this alongside the RPC endpoints, but in a separate PR
- I see the current home-page of packages being automatically gno doc (so, https://gno.land/p/demo/avl); but also I see that the help page, either this PR or in another one shortly following it up, could simply become a gno doc with additionally forms to construct gnokey commands.
- vm/qfuncs could become redundant at this stage.
let's look to include this also on gno doc on the CLI. currently it already understands packages installed in $GNOHOME/pkg/, but if it gets a full package path and the package is not available locally, it should possibly be capable of just making a query to the remote and unmarshaling the data.

tell me what you think; and let's maybe book a call for the end of the week.

Dec 09 '24 16:12 thehowl