IPNS pinning: ipfs name follow

[ghost]

cc @vyzo

USAGE
  ipfs name follow <ipns-path> - Track changes to IPNS names.

SYNOPSIS
  ipfs name follow [--pin=true] [--allow-expired=true] [--] <ipns-path>

ARGUMENTS

  <ipns-path> - IPNS path of the name that should be followed.

OPTIONS

  --pin           bool   - Always pin the name's most recent content. Default: true
  --allow-expired bool   - Allow lookup even if a name's TTL is reached. Default: true

DESCRIPTION

  The 'ipfs name follow' command can be used to track changes to an IPNS name.
  A followed name will be refreshed in the background in intervals, so that
  lookups through an 'ipfs' command or the HTTP API are always instant and up-to-date.

  When called with '--pin' (default), after refreshes the name's new content
  is pinned. This is equivalent to 'ipfs pin replace /ipfs/QmOld /ipfs/QmNew'.

  When called with '--allow-expired' (default), lookups of the name through
  the 'ipfs' command or the HTTP API always return the last known content,
  even if the TTL has been reached and the name has expired. If this is the
  case, a warning is printed along with whatever is the command's respones.

  The combination of '--pin' and '--allow-expired' allows for convenient
  offline use cases. For example, on a long flight I'd like to have IPFS docs
  with me, and I'd like them to be usable at /ipns/docs.ipfs.io as always.

  Open question: pubsub

  Open question: refresh after ttl/2 or whatever
  - /ipns/QmFoo:       ttl = ipns-ttl
  - /ipns/dnslink.net: ttl = min(dns-ttl, ipns-ttl)

  Open question: storage of followings

[Stebalien]

Is this only active when the command is active? I see two different commands here: ipfs pin add --follow and ipfs resolve --follow.

Pinning

ipfs pin add --follow [--allow-expired=true] [--lazy=false] /ipns/QmId

This command will create a new (permanent) pin that will "follow" this IPNS address while the daemon is active.

Note: --lazy exists to mirror --prefetch below (basically: make a non-strict pin that doesn't download everything up-front).

Use-cases: Mirroring, backup, offline-replication.

Follow

ipfs resolve --follow [--prefetch] [--quiet] /ipns/...

Unlike the previous command, this "follow" would only remain active while the command is active. By default, unless --quiet is passed, it would print out each new target ipfs address as it finds newer IPNS records. If --prefetch is specified, this command will prefetch the linked contents. TODO: It should probably pin the contents with some form of temporary (low-priority?) pin but we don't really have those at the moment.

Use-cases: Authenticated data streams, website "feeds".

[ghost]

@Stebalien yeah your proposal sounds good too!

[vyzo]

I am not sure we need a behaviour that is only available while the command is active - do we really care all that much about seeing background resolutions displayed in the terminal?

edit: of course it's not just human eyeballs consuming the output

[vyzo]

There is also the implicit relationship to ipns pubsub. We can have the follow command work through pubsub or it can be something that does background resolutions. The former is more efficient and adds value to ipns pubsub, the latter always works regardless of whether the other end have enabled pubsub.

[Stebalien]

edit: of course it's not just human eyeballs consuming the output

Exactly. That's more for e.g., webapps.

We can have the follow command work through pubsub or it can be something that does background resolutions. The former is more efficient and adds value to ipns pubsub, the latter always works regardless of whether the other end have enabled pubsub.

I'd do both. Always use pubsub and occasionally check the DHT for updates (more frequently if we see updates through the DHT that we don't see through pubsub).

[matthewrobertbell]

An alternative could be allowing "pinning" of IPNS names, so that they are both automatically updated, the records republished by that node and the current IPFS object which is pointed to being automatically pinned as well.

This would mean that there is no requirement to keep a command running.

[rklaehn]

@Stebalien

so the ipfs resolve --follow --prefetch would only print out a new hash for a name once everything is pinned?

the ipfs pin add --follow command is permanently running in the background, but it would be great if there was a way to track what it is currently doing (what is the latest hash for the name, what is the latest hash that is fully pinned, etc.)

[vyzo]

I think I prefer the ipfs name follow approach, as it's clear that it something that works with namesys (ie it's ipns functionality).

[Stebalien]

An alternative could be allowing "pinning" of IPNS names, so that they are both automatically updated, the records republished by that node and the current IPFS object which is pointed to being automatically pinned as well.

That's the first command I proposed.

so the ipfs resolve --follow --prefetch would only print out a new hash for a name once everything is pinned?

No. Actually, it wouldn't even pin. That command would just print out the hash and then start fetching in the background. If you wanted a pin, you'd use ipfs pin add --follow ... (you could also do both).

the ipfs pin add --follow command is permanently running in the background, but it would be great if there was a way to track what it is currently doing (what is the latest hash for the name, what is the latest hash that is fully pinned, etc.)

For the latest hash, you'd just resolve the IPNS name (which should use the cache). "Fully pinned" is a bit trickier. The second we get a new hash, we'd unpin the previous version and start pinning a new one.

We could have a --keep-history (to keep past versions). However, we probably wouldn't have a way to query past versions.

I think I prefer the ipfs name follow approach, as it's clear that it something that works with namesys (ie it's ipns functionality).

Good point. I agree.

[mitar]

I think this is duplicate of #1467?

[Stebalien]

@mitar this issue exists to discuss specific designs (although it does address the needs of #1467).

[lockedshadow]

What a wonderful name! This upcoming feature looks very interesting.

And do I understand correctly that "following" nodes will republish the last seen value of given name, even if the creator of that name (with its private key) disappears once? Will the new nodes be able to resolve a value of that name, if the "following" ones will be present in network?

[vyzo]

that's an interesting idea.

[sonatagreen]

We could have a --keep-history (to keep past versions). However, we probably wouldn't have a way to query past versions.

This part feels like the sort of thing that might be best handled with IPRS involved somehow.

[hsanjuan]

Could we have most of this as a library independent from go-ipfs? A library that, given a libp2p host and an IPNS key(s) and some config, puts new value on a channel for consumption.

It would be useful for cluster, and I think for other people.

[magik6k]

This would require extracting namesys, but can be done.

[NatoBoram]

Hi! Is there any news related to this issue?

Termux has started to publish their packages via IPFS under /ipns/k51qzi5uqu5dg9vawh923wejqffxiu9bhqlze5f508msk0h7ylpac27fdgaskx and /ipns/k51qzi5uqu5dj05z8mr958kwvrg7a0wqouj5nnoo5uqu1btnsljvpznfaav9nk. I'd like to be able to automatically follow these pins whenever they're updated.

Unfortunately, right now, it requires a fair share of scripting skills to get done.

I do have something in Bash...

#!/bin/bash

function update() {
	local name="$1"
	local ipns="$2"

	printf "Updating \e[33m$name\e[0m...\n"

	local oldIpfs="$(ipfs files stat --hash $name)"
	local newIpfs="$(ipfs name resolve $ipns)"

	ipfs pin update "$oldIpfs" "$newIpfs"
	ipfs files rm -r $name
	ipfs files cp "$newIpfs" $name
	ipfs files stat $name

	printf "\n"
}

update '/termux-stable' 'k51qzi5uqu5dg9vawh923wejqffxiu9bhqlze5f508msk0h7ylpac27fdgaskx' &
update '/termux-unstable' 'k51qzi5uqu5dj05z8mr958kwvrg7a0wqouj5nnoo5uqu1btnsljvpznfaav9nk' &
wait

echo 'Done.'

But that's considerably harder to do in PowerShell Core...

#Requires -PSEdition Core

$functions = {
	function update($name, $ipns) {
		Write-Host 'Updating ' -NoNewline
		Write-Host "$name" -ForegroundColor Yellow -NoNewline
		Write-Host '...'

		$oldIpfs = $(ipfs files stat --hash $name)
		$newIpfs = $(ipfs name resolve $ipns)

		ipfs pin update $oldIpfs $newIpfs
		ipfs files rm -r $name
		ipfs files cp $newIpfs $name
		ipfs files stat $name

		Write-Host
	}
}

Start-ThreadJob -InitializationScript $functions -ScriptBlock { update '/termux-stable' 'k51qzi5uqu5dg9vawh923wejqffxiu9bhqlze5f508msk0h7ylpac27fdgaskx' }
Start-ThreadJob -InitializationScript $functions -ScriptBlock { update '/termux-unstable' 'k51qzi5uqu5dj05z8mr958kwvrg7a0wqouj5nnoo5uqu1btnsljvpznfaav9nk' }
Get-Job | Wait-Job | Receive-Job

Write-Output 'Done.'

Another problem is that it has to be executed periodically and that also requires a fair share of skills in systemd or Task Scheduler and I think following an IPNS address should be a built-in feature.

More precisely, the workflow I'm looking for is something that can

• Add an IPNS address to a tracking list
• Optionally pin all new updates
• Optionally unpin all stale updates
• Works in the background as part of a regular IPFS daemon

[RangerMauve]

This would be very nice to have.

I'm working on an IPFS devgrant to create a protocol test suite for browsers and I need to have some test files as well as the entire suite pinned and available over IPNS.

It's possible to do some hacky stuff to make it work, but first class support for ipns when pinning would make life easier.

I also saw @lidel mention that IPNS is a lot speedier now when pubsub is enabled, which would also make it easier for gateways to be reactive to updates instead of periodically polling.

[Forza-tng]

Hi! I'd also like to mention the importance of being able to pin a /ipns/some.domain.tld/ and have it automatically download new and updated files.

My use case is being file mirror for a software project and without this functionality it is very difficult to keep the local mirror updated.

[hsanjuan]

My use case is being file mirror for a software project and without this functionality it is very difficult to keep the local mirror updated.

You can use ipfs-cluster to follow one or several pins. I.e run a ipfs-cluster-service on one side and ipfs-cluster-follow on the "clients". https://cluster.ipfs.io/documentation/collaborative/

[NatoBoram]

@hsanjuan You still need some scripting, but replace ipfs by ipfs-cluster-ctl. Plus, if you only have one node to maintain and use to follow the IPNS, then you could just use ipfs directly. Even if you had multiple nodes, ipfs-cluster-service doesn't offer much over ipfs if it can't distribute part of a pin instead of the whole thing; at that point. just run the script on all ipfs clients.

If the maintainer uses IPNS instead of IPFS Cluster, then there's no good reason to use IPFS Cluster yourself, unless you have like 4+ machines.

[lanzafame]

@NatoBoram

ipfs-cluster-service doesn't offer much over ipfs if it can't distribute part of a pin instead of the whole thing

Except it offers the feature that is being requested, which is to follow a pin or pinset. Please note that you don't need to run an actual cluster of ipfs-cluster nodes for this feature to work. It only requires a single ipfs node with a single ipfs-cluster node running beside it. Would ipfs name follow be simpler? yes. Is writing the systemd files to start an ipfs and ipfs-cluster node easier than scheduling a script? yes.

Sharding is implemented in cluster but is patiently waiting on a feature in the IPFS pinning system to have it enabled #5133.

[MayeulC]

This is quite similar to https://github.com/ipfs/kubo/issues/1467

My use-case is the following: I have a document (website) that is updated through a CI system, running on the "cloud". I could publish an updated IPNS record in the same CI, and have my local IPFS servers mirror it automatically, while the CI job is ephemeral.

TBH, I'm not sure if that would be the best way to go about it (I need to ensure the content is fully mirrored before shutting down the CI job, so I may need some communication with the IPFS servers anyway), but that seems one of the simplest.

[lordcirth]

This is quite similar to #1467

My use-case is the following: I have a document (website) that is updated through a CI system, running on the "cloud". I could publish an updated IPNS record in the same CI, and have my local IPFS servers mirror it automatically, while the CI job is ephemeral.

TBH, I'm not sure if that would be the best way to go about it (I need to ensure the content is fully mirrored before shutting down the CI job, so I may need some communication with the IPFS servers anyway), but that seems one of the simplest.

Perhaps you could poll findprovs to test that other nodes are mirroring?

[MayeulC]

True.

I actually wanted to edit this comment, as I think the real benefit here is that you don't have to share the private key with many servers, only with the initial publisher, which can be ephemeral. This could have security benefits.