Resolving IPNS immedietly with Libp2p-Fetch

[Rinse12]

We’re trying to set up IPNS and IPNS-Over-Pubsub in helia. The default behavior of IPNS-Over-Pubsub router in helia is to wait to receive the the IPNS update from its gossipsub subscription. But that takes too long, and in Kubo there’s a way for asking a connected peer directly for the latest IPNS record. The difference in terms of latency is huge, helia takes ~30s to resovle an IPNS, while kubo ~2s.

We would like the helia node to fetch the latest IPNS record immediately instead of waiting for IPNS updates in the pubsub topic. The IPNS-Over-Pubsub protocol suggests using Libp2p-Fetch for this, which is what Kubo is using. But we’re hitting a roadblock because no matter how we construct the fetch identifier or key, kubo nodes always respond with NOT_FOUND.

This is how we’re constructing fetch key at the moment. Please let me know how to correct it:

const subplebbitIpnsName = "12D3KooWJ7mvJFaWHK43MYd1Au4W4mkbY7L8dQaiMBqH5bZkSsFn";
const subplebbitIpnsAsPeerId = PeerId.parse(subplebbitIpnsName);
const fetchKey = "/ipns/" + uint8ArrayToString(subplebbitIpnsAsPeerId.toBytes(), "binary");
const res = await helia.libp2p.services.fetch.fetch(peer.id, fetchKey); // always gives undefined because kubo responds with NOT_FOUND

Moreover, is implementing libp2p-fetch for IPNS on the roadmap for helia?

References: IPNS PubSub Router

[estebanabaroa]

Something weird is happening, after dialing a kubo node with IPNS over pubsub and doing:

libp2p.services.fetch.registerLookupFunction(
  '/ipns/',
   (key) => console.log('fetch received with key:', [key])
)
await name.resolve(CID.parse('k51qzi5uqu5dkhvs75chetwlvhwyvhwj8c2qq71tx62z3fekwkms32ubp6k08g').multihash)

Logs:

  libp2p:fetch look up data with identifier /ipns/ 퓶U)ǂfzӗఄ筗{;𠴇Rv.ॱ80 +0ms
fetch received with key: [ '/ipns/\x00$\b\x01\x12 퓶U)ǂfzӗఄ筗{;𠴇R\x14v.ॱ80' ]
  libp2p:fetch sending status for /ipns/ 퓶U)ǂfzӗఄ筗{;𠴇Rv.ॱ80 not found +1ms

So I assume the key is '/ipns/\x00$\b\x01\x12 퓶U)ǂfzӗఄ筗{;𠴇R\x14v.ॱ80' but when I do:

const libp2pFetchKey = '/ipns/\x00$\b\x01\x12 퓶U)ǂfzӗఄ筗{;𠴇R\x14v.ॱ80'
const value = await libp2p.services.fetch.fetch(kuboPeerId, libp2pFetchKey)

I get the log:

  libp2p:fetch dialing /libp2p/fetch/0.0.1 to 12D3KooWL49bEXmXyp2zVc5HUCjKZjVdavK7mP97Md6qi6HidSUq +0ms
  libp2p:fetch using default timeout of 10000 ms +1ms
  libp2p:fetch fetch /ipns/ 퓶U)ǂfzӗఄ筗{;𠴇Rv.ॱ80 +36ms
  libp2p:fetch received status for /ipns/ 퓶U)ǂfzӗఄ筗{;𠴇Rv.ॱ80 not found +12ms

Using the exact key copy pasted that kubo is sending doesn't work?

Also how does 'k51qzi5uqu5dkhvs75chetwlvhwyvhwj8c2qq71tx62z3fekwkms32ubp6k08g' become '/ipns/\x00$\b\x01\x12 퓶U)ǂfzӗఄ筗{;𠴇R\x14v.ॱ80'?

[gammazero]

Triage: There should be existing code examples that shows how to correctly do this concatenation. The code above incorrectly concatenates binary data.

[achingbrain]

JavaScript strings are always UTF-16 (e.g. multibyte) so if you have arbitrary binary data that you stuff into a string, and a byte of that data looks like the first byte of a multi-byte character, reading the characters back out as bytes will result in data corruption.

The tedious thing about all this is that libp2p-fetch in Go uses unmodified datastore keys for the lookup, and in Go you can treat byte arrays as strings and strings as byte arrays. This leaked into the libp2p-fetch spec which was a mistake. Further reading: https://github.com/libp2p/specs/issues/656#issuecomment-2577228339

When the JS side serializes the string to a Uint8Array to send to Kubo, it truncates the string due to characters getting interpreted as bytes incorrectly - when read by Kubo the resulting datastore key has no match so you get the "not found" error.

Instead, turn the /ipns/ fetch key prefix into a Uint8Array and append the bytes for the PeerId (e.g. the IPNS key):

import { concat } from 'uint8arrays/concat'
import { fromString } from 'uint8arrays/from-string'
import { peerIdFromString } from '@libp2p/peer-id'

const peerId = peerIdFromString('12D3KooWJ7mvJFaWHK43MYd1Au4W4mkbY7L8dQaiMBqH5bZkSsFn')
const peerIdBytes = peerId.toMultihash().bytes

const prefix = fromString('/ipns/')
const fetchKey = concat([
  prefix,
  peerIdBytes
], prefix.byteLength + peerIdBytes.byteLength)

const res = await helia.libp2p.services.fetch.fetch(kuboPeer, fetchKey)

Also how does 'k51qzi5uqu5dkhvs75chetwlvhwyvhwj8c2qq71tx62z3fekwkms32ubp6k08g' become '/ipns/\x00$\b\x01\x12 퓶U)ǂfzӗఄ筗{;𠴇R\x14v.ॱ80'?

k51qzi5uqu5dkhvs75chetwlvhwyvhwj8c2qq71tx62z3fekwkms32ubp6k08g is a libp2p-key encoded as a v1 CID. That is, the multihash embedded in the CID is an Identity hash that contains the PeerId protobuf:

import { CID }  from 'multiformats/cid'
import { peerIdFromCID } from '@libp2p/peer-id'
 
const cid = CID.parse('k51qzi5uqu5dkhvs75chetwlvhwyvhwj8c2qq71tx62z3fekwkms32ubp6k08g')
const peerId = peerIdFromCID(cid)
const peerIdBytes = peerId.toMultihash().bytes
// or 
// const peerIdBytes = cid.multihash.bytes

const prefix = fromString('/ipns/')
const fetchKey = concat([
  prefix,
  peerIdBytes
], prefix.byteLength + peerIdBytes.byteLength)

const res = await helia.libp2p.services.fetch.fetch(kuboPeer, fetchKey)

Moreover, is implementing libp2p-fetch for IPNS on the roadmap for helia?

It would be nice to have this, please open a PR if you are still working on this and have the time.

I've opened an issue to track the missing feature.