Planetable
Upgrading Kubo from 0.15 to 0.28
There was an IPNS issue starting in version 0.16 that prevented us from upgrading:
https://discuss.ipfs.tech/t/ipfs-name-resolve-does-not-always-return-the-freshest-cid-for-ipns-on-kubo-0-20-0/16624
Now it seems to be fixed, and there is a new feature, ipfs key sign, which could be useful in some new applications:
https://github.com/ipfs/kubo/releases/tag/v0.25.0
So, we began the work to upgrade the bundled Kubo from version 0.15 to the latest, 0.28.
Implemented a basic auto migration process in this branch: https://github.com/Planetable/Planet/tree/ipfs-migration, should take care of upgrading existing Kubo binary from version 0.15.0 to 0.28.0 ( repo version 12 -> 15 )
We've shipped a lot of fixes since Kubo 0.15 and the latest 0.32.1 release is packed with some cool features, e.g. AutoTLS. It should also fix #338.
Did you have any problems upgrading?
We've shipped a lot of fixes since Kubo 0.15 and the latest 0.32.1 release is packed with some cool features, e.g. AutoTLS. It should also fix #338.
Did you have any problems upgrading?
Hey @2color, thanks for taking a look at our project.
When 0.29 was out, during our testing, it seemed that IPNS resolve issue was still consistently happening:
https://discuss.ipfs.tech/t/ipfs-name-resolve-does-not-always-return-the-freshest-cid-for-ipns-on-kubo-0-20-0/16624/8
So, we did not upgrade.
Today I tested again with this IPNS powered ohlife.eth.limo:
/ipns/k51qzi5uqu5dhbdxrvoeqy96c3krp2t1d9hfscjyn2mwts5f1w8f1sq99xwh3r
Very happy to report it seems the unexpected behavior is gone!
Below is what I got today:
Now, with the same 0.29 installation I used earlier, it can resolve consistently.
Previously in June, like in the screenshot below, a stale result pops from time to time. It was like a result was cached somewhere, and Kubo just used it.
@kailuo and I will do more testing with the latest 0.31 and restart our upgrading efforts. Thank you @2color!
Do you have any specific IPNS name that I can test?
Hi @2color
Today I encountered this issue again with 0.34.1, please see the screenshot attached:
The IPNS in question is:
k51qzi5uqu5di0awbvmh7vanmk726n7w0pujnmcuz8vvqi19qso22u0p0gjrz2
With 0.34.1, it returns one of the two results when resolving, and one of them is outdated.
With 0.15.0, it always returns only the most recent result:
Is there a config I can tune in 0.34.1 to fix this?
@2color
For IPNS k51qzi5uqu5di0awbvmh7vanmk726n7w0pujnmcuz8vvqi19qso22u0p0gjrz2
This is 0.34.1 (Left) and 0.15.0 (Right) side by side:
An outdated result bafybeiaghevtk6l3spoeuexku574ug6a7in3irg2sm4g7iqblyybjxrsuu appears from time to time on 0.34.1 while 0.15.0 can consistently return only bafybeienoruu3xrxhf2vldc4dpwkc4keh3hl2jxa2sssnm6uqeuwtiroxa.
Additional context:
kubo 0.34.1 is started with:
/usr/local/bin/ipfs daemon --enable-pubsub-experiment --enable-namesys-pubsub
In config, Routing.AcceleratedDHTClient is set to true.
One thing to point out is that when you use the -s option, results (value from found records) are printed as they're found. So remove the -s option to make sure you get the latest.
Do you know who's publishing this IPNS record and which version of Kubothey are using?
With the latest Kubo I am only able to resolve this name to:
➜ ~ ipfs name resolve -r -n -s k51qzi5uqu5di0awbvmh7vanmk726n7w0pujnmcuz8vvqi19qso22u0p0gjrz2
/ipfs/bafybeiduvuxvknfkset4urun43lpshwfdnqou6kwuskqwsm6pe6qi5ni5q
From the IPNS inspector:
It would be interesting to see what the sequence numbers in the IPNS records are for names that resolve to more than one record.
The IPNS record was published with Kubo 0.15.0 with these parameters:
- When resolving, if I omit
-s, then two results will take turns, sometimes the latest one, sometimes the outdated one. Then I found that-scan show me all the possibilities Kubo got. - It seems the issue happens more often when the record was just published. Then, after some time (maybe 24h), it seemed like a cache had expired somewhere, so only the expected result was returned.
With 0.15.0, it publishes the record and resolves to a single result; everything is working as expected.
Starting from 0.2x, two results take turns.
In 2021, we started building a static site generator with a built-in IPFS node.
Users can build a blog with this macOS GUI and publish the site with the built-in IPFS node. Then later they can link the IPNS to their .eth / .sol names like these:
https://planetable.eth.limo/ https://planetable.sol.build/
We were following the updates of Kubo until one day a user reported that his updates could not be reliably retrieved on a self-hosted IPFS gateway. That was how I encountered this name resolve issue.
And I found it's quite strange that the issue doesn't always happen. As of right now, I'm trying to reproduce it...
The IPNS record was published with Kubo 0.15.0 with these parameters:
* When resolving, if I omit `-s`, then two results will take turns, sometimes the latest one, sometimes the outdated one. Then I found that `-s` can show me all the possibilities Kubo got. * It seems the issue happens more often when the record was just published. Then, after some time (maybe 24h), it seemed like a cache had expired somewhere, so only the expected result was returned.
With 0.15.0, it publishes the record and resolves to a single result; everything is working as expected.
Starting from 0.2x, two results take turns.
@livid In this case, which version of Kubo is publishing the record?
The sequence number in the IPNS record should prevent this behaviour from happening deterministically. That means that as long as the IPFS node can find both records, it can easily determine which is the latest.
Based on the information you've shared, it seems that for whatever reason, the newer version of Kubo fails to resolve to the latest version of the record. But it's not clear why.
My suggestion would be to try disabling pubsub both on the publishing and resolving node and see if this problem persists. This could help us isolate the source of the problem. PubSub for IPNS was never fully polished and released as a stable feature. I wouldn't be surprised if this bug is related to that.
@2color
In our case, publishing is always 0.15.0 as it is the current version we ship with the Planet app.
And in both Publishing and Resolving, PubSub is indeed enabled. I will test disabling it. Thank you!
In that case, I would suggest first testing publishing and resolving with the latest Kubo, first with PubSub disabled, and then with PubSub enabled. That would help isolate the root cause.
@2color I like the new tool you created for diagnosing IPNS. It would be super useful if that tool could become an option --verbose for ipfs name resolve in Kubo.
@livid I agree with you.
For what it's worth, you can use the gateway exposed by Kubo to get the raw IPNS record with GET localhost:8080/ipns/id?format=ipns-record and inspect the record with the ipfs name inspect command.
You still won't be able to determine if the IPNS record was resolved with PubSub or the DHT (for this you can disable PubSub and see if and how it behaves differently) , but it should help you debug this.
Just wanted to check in if there are any updates or help you need from our side?
@2color Thank you!
I think we will stay on 0.15 and keep testing newer versions. As 0.15 has everything the app needs, while the new version could break a core feature, it's safer for us to stay on 0.15 for now.