Docs!

[mikeal]

We need more docs.

The new CID and Block interface aren’t documented at all.

[rvagg]

Labelling this as exp/beginner good first issue and help wanted because it would be great for someone with fresh eyes to help tackle this. We have docs in the README, they're not terrible, but it'd be great to expand them and explain some things a bit more. An examples directory would be awesome too to show off what it can do, just some basic things like what's reflected in the README that users can run.

This library is key to our JS stack, so making sure it has great entry docs is important.

[MF416]

@rvagg - currently part of the bootcamp and was looking to take this on as a first "bug fix" project. Is anyone else working on this / is this still open?

[rvagg]

In speaking to @MF416 elsewhere, my suggestion is something like:

• Focus on examples to start with - the README is currently mostly a series of short examples
• We have https://github.com/ipld/js-examples but it's a big example that goes beyond what we need here, but might be useful in learning the basic pattern
• Make an examples/ directory, try and build small example executables based on the README's snippets with some console.log() output
• Revisit README as you go to see how each of those sections could be improved based on the example experience.

The https://github.com/ipld/js-car repo is another place where I've spent quite a bit of time trying to use examples as documentation. Those are all runnable on the commandline, and are even run as part of CI to make sure they continue to work.

Of course, when you get further experience with the patterns here, you might see obvious ways to do pure documentation work in the README as well.

Two other bootcampers are going to help get some basic docs onto ipld.io for JS and they'll be pointing to this repo as the main source of documentation for JS IPLD, so improving things here will be great timing.

[AuHau]

Ehm I agree that more docs is needed. Tbh I am super confused, lost and unsure how to navigate and place this library. I am coming here from https://github.com/multiformats/js-multicodec repo looking for similar functionality that I can't find at all.

For example, where I can get a list of all codecs? Translating from codec names to codec values? Prefixing/getting prefix? Tbh. I can't find pretty much anything from the js-multicodec functionality here.

[rvagg]

@achingbrain what do you think about keeping js-multicodec alive, in a trimmed-down form, to just ship the table with simple accessors for the data? While we certainly don't want to encourage its general use, I've found use for it when I want access to the table and don't care about its size in my dependency tree. We could address that problem with documentation. I hadn't noticed the deprecation notice on there until now.

@AuHau to your question - you should consider whether you need access to the whole table. Part of the big shift we've done lately is to decouple all of these pieces so that you only have to consume the minimal pieces you need for your application. The multicodec table is getting quite large and there's not usually a good reason to have the whole thing in your application, usually you just want information about a couple of codecs and hashers and having the whole table is far too big (and it's always growing). All our new codecs and hashers ship with both their code and name on their main exports.

[AuHau]

@rvagg that makes sense. For my use-case, I don't need the whole table so I will adapt to your new approach. But I would advise documenting this "big shift" into places where people come from to this repo (like the js-multicodec, js-cid etc.).

[rvagg]

@AuHau yep, point taken, and this is the right issue to make such a comment! Our docs need more expansion.

[achingbrain]

I think having the old module still in use makes the intended style of use less clear so probably best to avoid it?

Maybe you or @Gozala or someone else who came up with the idea could write a blog post about the thought process behind the change with lots of examples and an upgrade guide?

[Gozala]

I think having the old module still in use makes the intended style of use less clear so probably best to avoid it?

I think general functionality of tagging bytes with a code and an ability to match by tag is worth having somewhere (which would correspond to addPrefix / rmPrefix), although that place is probably not js-multicodec. I am assuming most mulitformat values use same prefixing logic, if that is not the case please let me know.

Maybe you or @Gozala or someone else who came up with the idea could write a blog post about the thought process behind the change with lots of examples and an upgrade guide?

I did advocate for passing around codecs as opposed their codes and would not mind drafting a rational for it if someone is willing to review so we can post it to a blog or at least add .md file here that other repos can point to.

[vogdb]

Hi! @rvagg is there anything else needed for documentation? I see that examples folder had been added.

Also what do you think about enabling a a doc engine like documentation.js or sphinx? It would enable embedding the source code from examples folder into README.md. Then if anything major changes in the library, it would have to be updated only in examples.

[rvagg]

This issue isn't closed because I'm not convinced we have a good entry-point here for what is a critical piece of the IPLD puzzle that developers need to wrap their heads around. The docs could probably still be improved to be more explanatory and not just desciptive.

I like code-to-doc engines, but we have a problem in that we use TypeScript annotations heavily and I'm not aware of a good tool that can extract that information properly for docs.

I wrote https://github.com/rvagg/jsdoc4readme to do code-to-doc work for me in other libraries but it breaks apart the further you lean on TypeScript types. IIRC I managed to keep it just working for https://github.com/ipld/js-car but it's tricky work since the mismatches between jsdoc standards and TypeScript are non-trivial.

If you can make something work then I'm open to seeing it, but the TypeScript annotations being unmolested are critical. If it's just a matter of syncing examples and readme then that would probably be fine.

But .. also .. please don't overload dependencies too much more, any doc engine that has dependency bloat, even for devDependencies is likely going to get a :thumbsdown:. Lean is good and makes it easier to maintain and contribute.

[AuHau]

Just from brief search on the internet I have found better-docs that seems to support JSDoc Typescript annotations: https://wojciechkrysiak.medium.com/typescript-jsdoc-better-docs-7c03b6ea04df

Might be worth checking it out.

[vogdb]

Thank you for clarifying! As a beginner I will study the repo and see if I can help with the docs. Considering the doc engine. better-docs adds 218 packages with 3 high severity vulnerabilities. typedoc adds 225 with 3 high severity vulnerabilities. I guess that they don't fit under a lean dependency.

What do you think about this alternative https://github.com/marketplace/actions/markdown-embed-code-from-file ? It allows to embed files into markdown. However it is exposed as a Github action so it would be problematic for local usage.

[rvagg]

Yeah, not ideal if it has to get to an Action to take effect.

For examples we could do this the other way around, see https://github.com/rvagg/testmark.js which is an approach we're trying to adopt across the IPLD ecosystem (see also https://github.com/warpfork/go-testmark, there's a Rust one too now I believe). The idea is that test fixtures should exist in docs, and you just extract them to execute them. We could just do the same for examples too; I can think of a few repos where I've put a lot of work into examples that are also mirrored in the README--js-car being one of them--in that case the examples are part of the test run too, so I could imagine a situation where as a setup for running tests, the examples are extracted from the README (maybe not all of them, in the js-car case most of the examples don't belong in the README) and then executed.

This might even make a good CLI for testmark, something like testmark extract-files --input README.md --output examples/, if the fixtures look like filenames then they could be extracted.

[vogdb]

I see. Tests in README is an interesting approach. It reminds me of Cucumber test tool. I hope you can find enough time to make it as it looks like a serious standalone test framework.

I've looked around a bit and made a small PR for typos #166. Sorry for disturbing you.

[vmx]

Something that is missing in the current docs I just ran into was: How do I multibase encode bytes?

The answer is:

import { base64url } from "multiformats/bases/base64"
const bytes = new Uint8Array([118, 109, 120])
const multibase = base64url.encode(bytes)
console.log(multibase)

[vmx]

Another one is: How do I wrap an hash (digest) I already have created?

The answer is:

import { create as digestToMultihash } from "multiformats/hashes/digest"
const SHA2_256 = 0x12
const digest = new Uint8Array([0x76, 0x6d, 0x78, 0x69, 0x73, 0x6d, 0x79, 0x6e, 0x61, 0x6d, 0x65, 0x74, 0x68, 0x61, 0x74, 0x66, 0x61, 0x6b, 0x65, 0x73, 0x61, 0x73, 0x68, 0x61, 0x32, 0x32, 0x35, 0x36, 0x68, 0x61, 0x73, 0x68])
const multihash = digestToMultihash(SHA2_256, digest)
console.log(multihash.bytes)

[vmx]

And one more: How do I use the included varint API?

An example is:

import { varint } from "multiformats"
const number = 0x300003
const encodingLength = varint.encodingLength(number)
const encoded = new Uint8Array(encodingLength);
varint.encodeTo(number, encoded)
console.log(encoded)
const [originalNumber, numBytesDecoded] = varint.decode(encoded)
console.log(originalNumber === number, numBytesDecoded)

[vmx]

Triggered by https://github.com/multiformats/js-multiformats/issues/262.

How do I multibase decode:

import { bases } from 'multiformats/basics'

// Single known multibase.
console.log(bases.base32.decode('bozwxq'))

// Combine multiple multibases.
const base32or64 = bases.base32.decoder.or(bases.base64url.decoder)
console.log(base32or64.decode('bozwxq'))
console.log(base32or64.decode('udm14'))

You could also create your own utility that has a lookup table as mentioned at https://github.com/multiformats/js-multiformats/issues/121#issuecomment-939771137.