kubo
kubo copied to clipboard
ipfs
ipfs pin remote ls skips files that are created within a second of each other
Checklist
- [X] This is a bug report, not a question. Ask questions on discuss.ipfs.io.
- [X] I have searched on the issue tracker for my bug.
- [X] I am running the latest go-ipfs version or have an issue updating.
Installation method
ipfs-desktop
Version
ipfs version --all
go-ipfs version: 0.12.0-06191dfef
Repo version: 12
System version: amd64/darwin
Golang version: go1.17.6
Config
{
"API": {
"HTTPHeaders": {}
},
"Addresses": {
"API": "/ip4/127.0.0.1/tcp/5001",
"Announce": [],
"AppendAnnounce": [],
"Gateway": "/ip4/127.0.0.1/tcp/8081",
"NoAnnounce": [],
"Swarm": [
"/ip4/0.0.0.0/tcp/4001",
"/ip6/::/tcp/4001",
"/ip4/0.0.0.0/udp/4001/quic",
"/ip6/::/udp/4001/quic"
]
},
"AutoNAT": {},
"Bootstrap": [
"/dnsaddr/bootstrap.libp2p.io/p2p/QmNnooDu7bfjPFoTZYxMNLWUQJyrVwtbZg5gBMjTezGAJN",
"/dnsaddr/bootstrap.libp2p.io/p2p/QmQCU2EcMqAqQPR2i9bChDtGNJchTbq5TbXJJ16u19uLTa",
"/dnsaddr/bootstrap.libp2p.io/p2p/QmbLHAnMoJPWSCR5Zhtx6BHJX9KiKNN6tpvbUcqanj75Nb",
"/dnsaddr/bootstrap.libp2p.io/p2p/QmcZf59bWwK5XFi76CZX8cbJ4BhTzzA3gU1ZjYZcYW3dwt",
"/ip4/104.131.131.82/tcp/4001/p2p/QmaCpDMGvV2BGHeYERUEnRQAwe3N8SzbUtfsmvsqQLuvuJ",
"/ip4/104.131.131.82/udp/4001/quic/p2p/QmaCpDMGvV2BGHeYERUEnRQAwe3N8SzbUtfsmvsqQLuvuJ"
],
"DNS": {
"Resolvers": {}
},
"Datastore": {
"BloomFilterSize": 0,
"GCPeriod": "1h",
"HashOnRead": false,
"Spec": {
"mounts": [
{
"child": {
"path": "blocks",
"shardFunc": "/repo/flatfs/shard/v1/next-to-last/2",
"sync": true,
"type": "flatfs"
},
"mountpoint": "/blocks",
"prefix": "flatfs.datastore",
"type": "measure"
},
{
"child": {
"compression": "none",
"path": "datastore",
"type": "levelds"
},
"mountpoint": "/",
"prefix": "leveldb.datastore",
"type": "measure"
}
],
"type": "mount"
},
"StorageGCWatermark": 90,
"StorageMax": "10GB"
},
"Discovery": {
"MDNS": {
"Enabled": true,
"Interval": 10
}
},
"Experimental": {
"AcceleratedDHTClient": false,
"FilestoreEnabled": false,
"GraphsyncEnabled": false,
"Libp2pStreamMounting": false,
"P2pHttpProxy": false,
"StrategicProviding": false,
"UrlstoreEnabled": false
},
"Gateway": {
"APICommands": [],
"HTTPHeaders": {
"Access-Control-Allow-Headers": [
"X-Requested-With",
"Range",
"User-Agent"
],
"Access-Control-Allow-Methods": [
"GET"
],
"Access-Control-Allow-Origin": [
"*"
]
},
"NoDNSLink": false,
"NoFetch": false,
"PathPrefixes": [],
"PublicGateways": null,
"RootRedirect": "",
"Writable": false
},
"Identity": {
"PeerID": "12D3KooWL4Kt9dVHftkmjdKet99VUAk5GfS4fxSBd9tUQwyXSXGG"
},
"Internal": {},
"Ipns": {
"RecordLifetime": "",
"RepublishPeriod": "",
"ResolveCacheSize": 128
},
"Migration": {
"DownloadSources": [],
"Keep": ""
},
"Mounts": {
"FuseAllowOther": false,
"IPFS": "/ipfs",
"IPNS": "/ipns"
},
"Peering": {
"Peers": null
},
"Pinning": {
"RemoteServices": {
"alwaysNFT": {
"API": {
"Endpoint": "http://localhost:8088/ipfsapi"
},
"Policies": {
"MFS": {
"Enable": false,
"PinName": "",
"RepinInterval": ""
}
}
},
"pinata": {
"API": {
"Endpoint": "https://api.pinata.cloud/psa"
},
"Policies": {
"MFS": {
"Enable": false,
"PinName": "",
"RepinInterval": ""
}
}
}
}
},
"Plugins": {
"Plugins": null
},
"Provider": {
"Strategy": ""
},
"Pubsub": {
"DisableSigning": false,
"Router": ""
},
"Reprovider": {
"Interval": "12h",
"Strategy": "all"
},
"Routing": {
"Type": "dht"
},
"Swarm": {
"AddrFilters": null,
"ConnMgr": {
"GracePeriod": "300s",
"HighWater": 300,
"LowWater": 50,
"Type": "basic"
},
"DisableBandwidthMetrics": false,
"DisableNatPortMap": false,
"RelayClient": {},
"RelayService": {},
"Transports": {
"Multiplexers": {},
"Network": {},
"Security": {}
}
}
}
Description
We are building an implementation of the ipfs remote pinning API and cross-checking against the go-ipfs implementation.
It looks like ipfs pin remote ls skips files when two files have creation dates within a second of each other. These are the pins in our database:
filename | datecreated
----------------+--------------------------------
file2659.dat | 2022-03-04 06:00:25.91637+00
file2658.dat | 2022-03-04 06:00:24.899773+00
file2657.dat | 2022-03-04 06:00:24.03629+00
file2656.dat | 2022-03-04 06:00:22.98233+00
file2655.dat | 2022-03-04 06:00:21.954263+00
file2654.dat | 2022-03-04 06:00:21.079727+00
file26530.dat | 2022-03-04 06:00:20.258195+00
file2653.dat | 2022-03-04 06:00:19.415329+00
file26529.dat | 2022-03-04 06:00:18.625286+00
file26528.dat | 2022-03-04 06:00:17.877294+00
file26527.dat | 2022-03-04 06:00:17.073337+00
file26526.dat | 2022-03-04 06:00:16.298727+00
file26525.dat | 2022-03-04 06:00:15.34932+00
file26524.dat | 2022-03-04 06:00:14.470775+00
file26523.dat | 2022-03-04 06:00:13.695262+00
file26522.dat | 2022-03-04 06:00:12.804065+00
file26521.dat | 2022-03-04 06:00:11.933292+00
file26520.dat | 2022-03-04 06:00:11.049674+00
file2652.dat | 2022-03-04 06:00:10.250677+00
file26519.dat | 2022-03-04 06:00:09.411705+00
file26518.dat | 2022-03-04 06:00:08.597348+00
file26517.dat | 2022-03-04 06:00:07.898356+00
file26516.dat | 2022-03-04 06:00:07.213373+00
file26515.dat | 2022-03-04 06:00:06.358728+00
file26514.dat | 2022-03-04 06:00:05.547268+00
file26513.dat | 2022-03-04 06:00:04.728247+00
file26512.dat | 2022-03-04 06:00:03.76135+00
file26511.dat | 2022-03-04 06:00:02.974295+00
file26510.dat | 2022-03-04 06:00:02.070656+00
file2651.dat | 2022-03-04 06:00:01.352209+00
file2650.dat | 2022-03-04 06:00:00.575174+00
(31 rows)
This is what ipfs pin remote ls returns:
% ipfs pin remote ls --service=alwaysNFT
QmVcWnsfQ9cjZUz8oc2419XtincZ7oX2s4uy1raViDXdBy pinned file2659.dat
QmepaMJpAfYsUAStCcqTcSpSATgK7dqd6hEKcRc5EGD9M2 pinned file2658.dat
QmTPqcFUYKoB43YnJRPzTWtxSYb2NYyznNSAXcHzTkseYc pinned file2657.dat
QmRAkFamSC2SRJdGKRHMZTfzTrxn1ePTQ4oiUnEjpFRayr pinned file2656.dat
QmSLczjF7cNXCdK1zm8TmhbyHYzYV27rHf6Y1PF864wYRb pinned file2655.dat
Qmd77xJzv9apMEjE2MHNKb7Vc6PZo8hKAyxkCT7kmtR1JJ pinned file2654.dat
QmViqpreDaJHs7PxhAVokypJg1MtR5GM1XyWwbLohyztnP pinned file26530.dat
Qma9fVpAxxyAqkZYCnRUXwhRxuYmWLDoNQzSNq7YAhGiTq pinned file2653.dat
QmUYFyz2MSKiksCteGa6Dd1DqZMwDYkbPvrhZiNvd7VjFa pinned file26529.dat
QmTm2vCKU5TwE4EUN8xUdDw2NNWPDenDhxgHxcH9ckt38h pinned file26528.dat
QmVctn2yAxmS5GgY3qAFFxo4v6sYWLFXThjm2mRzuH4RQC pinned file26526.dat
QmazXhqJQjjAsGp1eBb8EzMwvLkBoxRkmePMWwx4D6snja pinned file26525.dat
QmNWLWrh9pEQciaPpsXXwgUQwNmbtK473Pa4B7VEHehcJa pinned file26524.dat
QmXyLTx4F96V9HVfnhR3sGiYd7XQBVd1vPVPe1n7N5hNMo pinned file26523.dat
QmfQN648Wq5yT2rau9RZMKGC5j6M4XkecYPegPjNSku2WL pinned file26522.dat
QmUBfhT6VngPJUkTiWBFqxHKKyCw9v84TywTQL9CVnfjPF pinned file26521.dat
QmTxQhXzpX6CJ39mVx1PvBExDvmd3amSekTKivQFZNz2Qe pinned file26520.dat
QmUP4jEAxR3UuUsE3EJwUW83LzkcUyrMVNfH9tEEExxJAa pinned file2652.dat
QmVGxJKKJU3CCEHjLMhHRp3E7KGhBy9XHpwur9Xkgkga8M pinned file26519.dat
QmbgZnodUXtDx9HN2DU4zFLuSbQj6dzRBwSxvXakizUH6v pinned file26518.dat
QmQypAywRj1ba8p881nCdKXf9uL2FU3ViMRwDvusX2DqV5 pinned file26517.dat
QmZbCfGCiMuYB2BGZit3rmwJVnfutft4WhHDuHtmuAFpBs pinned file26516.dat
QmSVukqZoLD51gzfakUFKACpukS913vPiWn2aHbBr6zTBz pinned file26515.dat
QmPhzfM6Y5Q31C6AoTrJoxAy6t3mBnU2ZH15SE7gdJ8xSk pinned file26514.dat
QmeKqYEriFyCyc253b5n1YcaQDGifZJkU41Mu3UVbKx8jn pinned file26513.dat
QmRMYfCfwyeBGiXEVgLUVmQVaSNAowtxJfduYzuQLwTXHN pinned file26512.dat
QmcvLF5R9ceKZw4GBuCD4EpKtdjDQ16Fe6EewPCoSs2rLR pinned file26511.dat
QmZZ1gGAfL8PhjdNT8xr7QNqEAKtxSuh9MY2eDr8o77bnn pinned file26510.dat
QmdzLWBgfFZCysoDPW5MP221NfCmHcepy8RyWaXoQJ6Juf pinned file2651.dat
QmP7fdDDHaMdvcieJdJZaxaKDgaZTnVpox52ZiM47G7CMK pinned file2650.dat
Note that file file26527.dat is missing from the list. Also note that file26527.dat was created within the same second as the previous file and is the 11th file in the expected output.
Your first call to our service was:
GET /ipfsapi/pins?limit=10&status=pinned
And we returned pins with the creation dates:
{"count":31,"results":[{"requestid":"399c0a15-b677-45f1-959e-3eb75e6fcab3","status":"pinned","created":"2022-03-04T06:00:25.916Z","pin":{"cid":"QmVcWnsfQ9cjZUz8oc2419XtincZ7oX2s4uy1raViDXdBy","origins":[],"name":"file2659.dat","meta":{},"delegates":[],"info":{"projectid":"3b04ec20-8785-46e5-a38b-bc1046446935","pinCount":"1"}}},{"requestid":"048408de-d979-4d92-90b6-7af2ea953245","status":"pinned","created":"2022-03-04T06:00:24.899Z","pin":{"cid":"QmepaMJpAfYsUAStCcqTcSpSATgK7dqd6hEKcRc5EGD9M2","origins":[],"name":"file2658.dat","meta":{},"delegates":[],"info":{"projectid":"3b04ec20-8785-46e5-a38b-bc1046446935","pinCount":"1"}}},{"requestid":"5ab7b350-a136-4ae5-aeaf-61d6eff9f790","status":"pinned","created":"2022-03-04T06:00:24.036Z","pin":{"cid":"QmTPqcFUYKoB43YnJRPzTWtxSYb2NYyznNSAXcHzTkseYc","origins":[],"name":"file2657.dat","meta":{},"delegates":[],"info":{"projectid":"3b04ec20-8785-46e5-a38b-bc1046446935","pinCount":"1"}}},{"requestid":"d9c85a93-8f6f-4386-944b-a24818e425e1","status":"pinned","created":"2022-03-04T06:00:22.982Z","pin":{"cid":"QmRAkFamSC2SRJdGKRHMZTfzTrxn1ePTQ4oiUnEjpFRayr","origins":[],"name":"file2656.dat","meta":{},"delegates":[],"info":{"projectid":"3b04ec20-8785-46e5-a38b-bc1046446935","pinCount":"1"}}},{"requestid":"0fca840e-0fc9-4062-b825-189ce1fd435a","status":"pinned","created":"2022-03-04T06:00:21.954Z","pin":{"cid":"QmSLczjF7cNXCdK1zm8TmhbyHYzYV27rHf6Y1PF864wYRb","origins":[],"name":"file2655.dat","meta":{},"delegates":[],"info":{"projectid":"3b04ec20-8785-46e5-a38b-bc1046446935","pinCount":"1"}}},{"requestid":"6f7b5e93-4a68-4f63-ab11-0f517b9d6738","status":"pinned","created":"2022-03-04T06:00:21.079Z","pin":{"cid":"Qmd77xJzv9apMEjE2MHNKb7Vc6PZo8hKAyxkCT7kmtR1JJ","origins":[],"name":"file2654.dat","meta":{},"delegates":[],"info":{"projectid":"3b04ec20-8785-46e5-a38b-bc1046446935","pinCount":"1"}}},{"requestid":"4138b704-3ca6-4c84-98b5-eac8bae5003f","status":"pinned","created":"2022-03-04T06:00:20.258Z","pin":{"cid":"QmViqpreDaJHs7PxhAVokypJg1MtR5GM1XyWwbLohyztnP","origins":[],"name":"file26530.dat","meta":{},"delegates":[],"info":{"projectid":"3b04ec20-8785-46e5-a38b-bc1046446935","pinCount":"1"}}},{"requestid":"f9bcb08f-b7ce-4f5f-8a1f-66ac9d8ba65d","status":"pinned","created":"2022-03-04T06:00:19.415Z","pin":{"cid":"Qma9fVpAxxyAqkZYCnRUXwhRxuYmWLDoNQzSNq7YAhGiTq","origins":[],"name":"file2653.dat","meta":{},"delegates":[],"info":{"projectid":"3b04ec20-8785-46e5-a38b-bc1046446935","pinCount":"1"}}},{"requestid":"a5b694b7-5d94-407d-be28-877f8f9df70c","status":"pinned","created":"2022-03-04T06:00:18.625Z","pin":{"cid":"QmUYFyz2MSKiksCteGa6Dd1DqZMwDYkbPvrhZiNvd7VjFa","origins":[],"name":"file26529.dat","meta":{},"delegates":[],"info":{"projectid":"3b04ec20-8785-46e5-a38b-bc1046446935","pinCount":"1"}}},{"requestid":"097b7f9f-2adf-4190-a202-9dbd1317fffe","status":"pinned","created":"2022-03-04T06:00:17.877Z","pin":{"cid":"QmTm2vCKU5TwE4EUN8xUdDw2NNWPDenDhxgHxcH9ckt38h","origins":[],"name":"file26528.dat","meta":{},"delegates":[],"info":{"projectid":"3b04ec20-8785-46e5-a38b-bc1046446935","pinCount":"1"}}}]}
Note that the final created timestamp returned was "2022-03-04T06:00:17.877Z",
to get the next page of output, go-ipfs sent:
GET /ipfsapi/pins?before=2022-03-04T06%3A00%3A17Z&limit=10&status=pinned
Omitting the millisecond portion of the timestamp. Consequently, file26527.dat which had a creation date between 6:00:17 and 6:00:18 was not included in the output.
This error should be reproduceable with any remote pinning service with high rates of pinning.
Thank you for submitting your first issue to this repository! A maintainer will be here shortly to triage and review. In the meantime, please double-check that you have provided all the necessary information to make this process easy! Any information that can help save additional round trips is useful! We currently aim to give initial feedback within two business days. If this does not happen, feel free to leave a comment. Please keep an eye on how this issue will be labeled, as labels give an overview of priorities, assignments and additional actions requested by the maintainers:
- "Priority" labels will show how urgent this is for the team.
- "Status" labels will show if this is ready to be worked on, blocked, or in progress.
- "Need" labels will indicate if additional input or analysis is required.
Finally, remember to use https://discuss.ipfs.io if you just need general support.
![welcome[bot] avatar](https://avatars.githubusercontent.com/in/4141?v=4 "Published by a pub.dev verified publisher"){kind=small}
Notes from the triage: I'll look into this, testing with a mock but I think it's an issue on the data returned you by the pinning service.
@gharriso I would like you to try lsing with curl or whatever http tool.
And if you can show a request where all the data is there with curl but not with ipfs pin ls that does mean we have an issue in our code.
The pinning service API is simple, here is the documentation of what you need to hit:
https://ipfs.github.io/pinning-services-api-spec/#tag/pins/paths/~1pins/get
Here is a demonstration of the issue using curl and the ipfs command.
First, the command line:
ipfs pin remote service add alwaysNFT http://localhost:8088/ipfsapi/ eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpZCI6IjZjZGZmZTlmLTU0N2MtNGNlYi1iNTM4LWJkNTE2NDNmOWEyYyIsImlhdCI6MTY1MDUxMjM3OSwiZXhwIjoxNjUwNTk4Nzc5fQ.qrEMJGqnFsxQfYMG9GeqYF6PdBThMRLZMIEV_Sr3x1A
$ ipfs pin remote ls --service=alwaysNFT
QmXAFDsuf3synfYnvSVz139PwYru3AEBD6SVCgw5b5waKx pinned QmXAFDsuf3synfYnvSVz139PwYru3AEBD6SVCgw5b5waKx
QmVywznsgvrBTr49xCfR5AqgqiyfnReKUbWQySDyYpfmbs pinned JKM2402-4-0
QmNZEuVn24qiwFA988uhZ4arZtoVYejWpCmfc3VJb3kLaR pinned JKM2402-5-0
QmaRLpknr6Puy4rUoLqCqqsACQK8uLjRkkrtAS7k2e1r4v pinned JKM2402-4-1
QmeSXGFXsW8pAM5vqv5Z6sLPSUd4jMRC5CZjLv62BGpVSB pinned JKM2402-5-1
QmaKBhxJrWMfnLhAhKvDG9kPqi2vqiRRC6JdtMz5RmqpyG pinned JKM2402-3-1
QmRrqfBYWGdU2zc3M1uJZ6YXKfNVrzcLp7rr4HsozXEM6p pinned JKM2402-3-0
QmfLRiuuevg7XCVbjCcaU91fsxUg7EMuNWaFATfmP5Rpfp pinned JKM2402-2-1
Qmbi1KLYqLWBUCBaQsRCRxTdNvo6HkXzagUpFA29SXtRUM pinned JKM2402-2-0
QmTu1Pbc7g78JUXZ6SUw7BMhwGCCE44ixbQzkCXXzQ5UsT pinned JKM2402-1-1
bafkreiewfy6474jejksieq2zaaiygxlb5mkckfutme2pr3cr3yamp4sh7a pinned OPENSTORE-4.155875096305949e+76-0
QmNm6ByyvZ7q1y9Eohvhjkhm6EHRtXdmeCaiFBmpMqV8WZ pinned JKM0504-6-1
QmYEYcKMi6xpv69XGfGmwQjFkvVfMffZGGVPG7kvxxvFGK pinned JKM0504-9-1
QmYiFzA3YdU1PwHcxDnm6up6Zn9qT2UgUBdiRTzhSyontu pinned JKM0504-9-0
QmcjTtF8T5weXbY8Vv32Bjisdeouy1TVvuYpdgbk8bcZPd pinned JKM0504-8-1
QmNsByRbNt6jyqXYVqV71N2dM9Zt4RCTLXiCY1cJ579wNU pinned JKM0504-8-0
QmV9oqkZVMtQVatJEWfGdXrgu5JsDyFGBkKKixAfTSQhVP pinned JKM0504-7-1
QmRrqfBYWGdU2zc3M1uJZ6YXKfNVrzcLp7rr4HsozXEM6p pinned JKM0504-0-0
QmaKBhxJrWMfnLhAhKvDG9kPqi2vqiRRC6JdtMz5RmqpyG pinned JKM0504-0-1
QmVywznsgvrBTr49xCfR5AqgqiyfnReKUbWQySDyYpfmbs pinned JKM0504-1-0
QmdgqsRen33kKBs3DpGGmmoM2L8wezyDw2zujjKeKCWJPt pinned Finale 2009 - [Let-it-be-accompanying 2].pdf
QmXcSJu8r7gzxUwnBn4jisnXq8xQ49Sam6GudpnBafAcmQ pinned 43523954-Guitar-Tab-Songbook-Led-Zeppelin-IV.pdf
Qmf6d3eFmkGtk5PkicpQdy2K99PPTrNXT28st6byg29bGn pinned 416113721-Play-Piano-With-the-Beatles-Wise-Publications-2000.pdf
QmRBoG3mMDbXEVUGQxKDTAidJuCEvNP6BWhWEyQtCVZZFb pinned 394470207-beatles-let-it-be.pdf
QmVR6DH8mcQJFb77RCYDD82xWnJwndnBoB3PeqHGcVw4wP pinned 275527801-227507825-Beatles-for-Classical-Guitar-Hill-John-K-pdf.pdf
Qmcy6YcDdmcnCWvHqzSpKxfVoRtS2miEwc8KYTWkpouEtH pinned 201394742-Let-it-be.pdf
QmZxXW9eVCRhNgzWr3ViMvUA2G1ickwc3x4MXbs5LRNGdD pinned 155001927-Beatles-For-Classical-Guitar-Arr-Joe-Washington-PDF.pdf
Just 27 pins were returned by the command line:
ipfs pin remote ls --service=alwaysNFT |wc -l
27
Here you see curl returns all 307 pins. I've used jq to simplify the output:
bash -v curl.sh|jq '.count,.results[].pin.cid'
curl --location --request GET 'http://alwaysubuntumac.local:8088/ipfsapi/pins?limit=400' \
--header 'Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpZCI6IjZjZGZmZTlmLTU0N2MtNGNlYi1iNTM4LWJkNTE2NDNmOWEyYyIsImlhdCI6MTY1MDU4NjA1MCwiZXhwIjoxNjUwNjcyNDUwfQ.Ty4DH2k4rJPmA46RYqo_g5jxpR0OZRjokOn-BCQFSSg'
307
"QmXAFDsuf3synfYnvSVz139PwYru3AEBD6SVCgw5b5waKx"
"QmWwobN856akaKHx1XvCPgcfiH8GZoFDydNPmEehJ5oA1t"
"QmeSXGFXsW8pAM5vqv5Z6sLPSUd4jMRC5CZjLv62BGpVSB"
"QmNZEuVn24qiwFA988uhZ4arZtoVYejWpCmfc3VJb3kLaR"
"QmaRLpknr6Puy4rUoLqCqqsACQK8uLjRkkrtAS7k2e1r4v"
"QmVywznsgvrBTr49xCfR5AqgqiyfnReKUbWQySDyYpfmbs"
"QmaKBhxJrWMfnLhAhKvDG9kPqi2vqiRRC6JdtMz5RmqpyG"
"QmRrqfBYWGdU2zc3M1uJZ6YXKfNVrzcLp7rr4HsozXEM6p"
"QmfLRiuuevg7XCVbjCcaU91fsxUg7EMuNWaFATfmP5Rpfp"
"Qmbi1KLYqLWBUCBaQsRCRxTdNvo6HkXzagUpFA29SXtRUM"
"QmTu1Pbc7g78JUXZ6SUw7BMhwGCCE44ixbQzkCXXzQ5UsT"
"QmTKZEmqoiKc2mnz8hFAiTouYLB1g4WmGLVESbruZjcsiK"
"QmZNphGjKZy7oBXoEFFXGhH2UmUqzx1un59LXBUEMD2Y2d"
"QmVaNtjzm42wRC4dMDk3bt4VLY9sff4PNxsHbLyX2BMX2d"
"QmRPRR8aHT4cE7xRrjwECfD5AnHzco2eDZUkQzyzGpe3a5"
"QmPnMkZRLWTyHBVUxt2fFrHjB6bLgnGu11a5ubHow4SLs6"
"QmeGQQghJNM1Gz4YAAM4rT3peq1Sg9YxVQPkFpfqmkJxbf"
"QmUDpFZTozx17HKpMEmTqBJK25kywwF57EYSfhzAxhP7Zw"
"QmfLRiuuevg7XCVbjCcaU91fsxUg7EMuNWaFATfmP5Rpfp"
"Qmbi1KLYqLWBUCBaQsRCRxTdNvo6HkXzagUpFA29SXtRUM"
"QmTu1Pbc7g78JUXZ6SUw7BMhwGCCE44ixbQzkCXXzQ5UsT"
"QmTKZEmqoiKc2mnz8hFAiTouYLB1g4WmGLVESbruZjcsiK"
"QmPzmitVKi6tEY56mM7p2shqByQsi3ckuxuEs1epkndBtt"
"QmXMkNfoAADtE33FJ2gqhk1MQvrhf6xCjSpt56pMYCPUNR"
"QmQiJ9NwZghefXb86p2ck4r13dedmYoex4JwTfAAKuYPsd"
"QmWzJ8yprZ9KkYYJfm3QLbXitHFw3KxvhiyxGa4pSK4Wix"
"QmZPkzrue5WcPNzR7oun7VkLgReGt1vhuhXYCi6rDAdz6H"
"QmRaX2VVLHTW6jM6ZvXAwXVJK2CNpG5QTtYAmEYS151pye"
"QmXtwPTqLsnyr5cfS6GUR8SYXLkTTgjHCSMWPxUpXjRu8E"
"QmWJ5YxJHD5Q9ZNW1CCyYCoHq29XjCqkVyEUJiYfC19GQG"
"Qma2LFomtUdgXVXJ4hKrLKntbSQFHEifPviHm6x2LURQj6"
"QmdwG2P4RXezCauQdtj1oHHNTiS5rYimdmTwThgCYsGnoB"
"QmcB7uPeHC3PhURCw5MnnWxVLoGHGaVEU1iCEfNbqD4k6Y"
"QmW39ukPr1bLZFS8aU5tTmh4FfJBwQB2KZWwYE7WsUej4b"
"QmcU3HT9f38AfR3hjvYP9fkJhLN6qnTwP7kGf8d1Tpt2af"
"QmNtB3mCdyoj59S6k36TfALLbxFGJubx7RGGXpo5AnsuUk"
"QmRCZLmE8PCrcakhEg2bBF2XZY2CKPS6GCxodCjCajxG4U"
"QmTnRpYW8FAT5vqRy4Xrj2CcpC333j1DY1Pq1ACsmhwDWx"
"QmRNLk4xp7bDBWbH4673pYiabUShWC7f3fKDWnHtLJSm31"
"Qmab8ZVmdgNdDpkdhVW3e5P2g5wSTYrT2vWWFeKxHDnCtB"
"QmSB546DPdKi6zzfpgyV5z85BpfjX8CF1UeDrDxLKauCro"
"QmNmmLHubB82NhYDcgqpY78mFsowJimMcYG9JS613DX4rT"
"QmbKSWWU2J66tkaYBL1FQB31Y8FtmmoGmpuL2QWr4DCVyB"
"QmeHdP5jvqVN8JJyYGrwUn5Y6ourpJ2tLWZizhtVkPyCcg"
"QmXoe56n1ijc4Y87DCWutYgTUSomcEMXa1Ck9xe36Q5DUN"
"QmV32zeAQh8dLa23vwKfD17XZLh8k8dowqYMGRMKgZxNZx"
"QmZ4rX8njpTbeaSR7cHyA7LExJ57jW4bj5t6ZoxpvLLTER"
"Qmd8WhUJY8upka9TQh6WuEKhYDHS5cSh8WCfnHU5sbb6xj"
"QmeuKR7DuRxRxXnZ3Zms3MftGdvhf7kgy1hPb6Zf1xMTGe"
"QmNmmLHubB82NhYDcgqpY78mFsowJimMcYG9JS613DX4rT"
"QmaqPAALGixCZ8dUnyZVfPuVNjsUSCEbYXL1nRWRsNaEE8"
"QmZXLWDwcCQjJSm67w2WjrYajX9Z9FV7FKLGRDZBWLGAxL"
"QmbKSWWU2J66tkaYBL1FQB31Y8FtmmoGmpuL2QWr4DCVyB"
"QmeHdP5jvqVN8JJyYGrwUn5Y6ourpJ2tLWZizhtVkPyCcg"
"QmXoe56n1ijc4Y87DCWutYgTUSomcEMXa1Ck9xe36Q5DUN"
"QmV32zeAQh8dLa23vwKfD17XZLh8k8dowqYMGRMKgZxNZx"
"QmQUySJ6kV5tV18WZM57wvReoZFs3xsVoSHVKukD1BaZFn"
"QmafUphxBh3mVmjTLbGbF2U8D1HDK3WmKhCatswUFtvPMe"
"QmaoejUT9TJkKspzQWpQxbwycgUPHUuBcxWMm4C2FavxPP"
"QmW8pzX9AmQieviSDLAinp4rju1r4xmjMH8BVXfwNdVkjN"
"Qma77zt7Z2isMrLxbwmnkkBRVtug5zZwL7HRCLimQhZS6J"
"QmPRYQDSryqh6LiGkv19pnRAbz7TiQvG9fBYcY8A5S7qxc"
"QmNUspoGaXF9ZpHk8TF17aoqcizygMzmyQzFrtHXGBEvDH"
"QmRSrU4cNf4WtbkJSquXCtKiAN93wNgj9RpoCni3SPmvAV"
"QmYitoZnWnXQWESEKBajbk4NvAinrj9uNbPJQCrq7Y3SM3"
"Qmd8oWurdNmSU8j8mtKSdKWf9tUh764v2yA2RSkK1zs4M7"
"QmZitUhbYZFa7EZbNagygFSeWeKcRRFGCzgVPz2hutW1Zk"
"QmQUySJ6kV5tV18WZM57wvReoZFs3xsVoSHVKukD1BaZFn"
"QmafUphxBh3mVmjTLbGbF2U8D1HDK3WmKhCatswUFtvPMe"
"QmdDuhQCmouCBAqNUcAse12PSxgCxh2JJ4Ti45h8ZHAyHR"
"QmYWghxrcUzUes88fxjcKv2J9Ad5A53VViwKQrn7wdesRE"
"Qma2Ec7QSPVT57Rf2MsXNmGwcBjvypPhPSWChHV9XKWwL8"
"QmdDuhQCmouCBAqNUcAse12PSxgCxh2JJ4Ti45h8ZHAyHR"
"QmZimyAbktFDZ9eBSbsX1EX2w9q2N3TSHD33uuaZY2Q3Ui"
"QmYWghxrcUzUes88fxjcKv2J9Ad5A53VViwKQrn7wdesRE"
"QmZ4rX8njpTbeaSR7cHyA7LExJ57jW4bj5t6ZoxpvLLTER"
"Qmd8oWurdNmSU8j8mtKSdKWf9tUh764v2yA2RSkK1zs4M7"
"QmZitUhbYZFa7EZbNagygFSeWeKcRRFGCzgVPz2hutW1Zk"
"QmdAR4PyNttbeUkvBvu5u9Z8XRzTxY13nM1eFTq2XKKEFL"
"QmYitoZnWnXQWESEKBajbk4NvAinrj9uNbPJQCrq7Y3SM3"
"Qma2Ec7QSPVT57Rf2MsXNmGwcBjvypPhPSWChHV9XKWwL8"
"QmZimyAbktFDZ9eBSbsX1EX2w9q2N3TSHD33uuaZY2Q3Ui"
"QmRSrU4cNf4WtbkJSquXCtKiAN93wNgj9RpoCni3SPmvAV"
"QmNUspoGaXF9ZpHk8TF17aoqcizygMzmyQzFrtHXGBEvDH"
"QmPRYQDSryqh6LiGkv19pnRAbz7TiQvG9fBYcY8A5S7qxc"
"Qma77zt7Z2isMrLxbwmnkkBRVtug5zZwL7HRCLimQhZS6J"
"Qmd8WhUJY8upka9TQh6WuEKhYDHS5cSh8WCfnHU5sbb6xj"
"QmeuKR7DuRxRxXnZ3Zms3MftGdvhf7kgy1hPb6Zf1xMTGe"
"QmW8pzX9AmQieviSDLAinp4rju1r4xmjMH8BVXfwNdVkjN"
"QmaoejUT9TJkKspzQWpQxbwycgUPHUuBcxWMm4C2FavxPP"
"QmZXLWDwcCQjJSm67w2WjrYajX9Z9FV7FKLGRDZBWLGAxL"
"QmaqPAALGixCZ8dUnyZVfPuVNjsUSCEbYXL1nRWRsNaEE8"
"QmdAR4PyNttbeUkvBvu5u9Z8XRzTxY13nM1eFTq2XKKEFL"
"QmbT4viHnbAVYosZEynq4pZygTReiHW9DmzxM4SaXvqg3s"
"QmbY5bdeZQF2k6shiBtFwDdBWPMKBaBBKa7C2KrHGctz8v"
"QmWf2UgJjn9xcZMk1xPiCvTgo5kMRmwp2fmW5YRbST55cv"
"QmX3Ds5t675yDH27y1t8w7Zt8ggMNWwFrEK8g9xHE7cUsC"
"Qmd1prgtiqPmQ4aFemnD5CNatwwoRH1bXvDtSRvXMykpWR"
"QmRrqfBYWGdU2zc3M1uJZ6YXKfNVrzcLp7rr4HsozXEM6p"
"QmRq4qQN3ST54cxR2HoiW3UhGacVR1vXqUjJYG5pKeGHqF"
"QmSY85wDmGZbCEpkxo79Sb61YQjPZMFuNZxwbq6Uc6Ax6D"
"Qmab8ZVmdgNdDpkdhVW3e5P2g5wSTYrT2vWWFeKxHDnCtB"
"QmVzaNuVyrJH8PuXHtHqg9wtHhgCQvFA5abkt5MTrE7he7"
"QmRNLk4xp7bDBWbH4673pYiabUShWC7f3fKDWnHtLJSm31"
"QmNZEuVn24qiwFA988uhZ4arZtoVYejWpCmfc3VJb3kLaR"
"QmeSXGFXsW8pAM5vqv5Z6sLPSUd4jMRC5CZjLv62BGpVSB"
"QmTnAexEvnUqwpBUcGy56GixFiUBbePkx8gLA6ua7wFmdU"
"QmTnRpYW8FAT5vqRy4Xrj2CcpC333j1DY1Pq1ACsmhwDWx"
"QmRCZLmE8PCrcakhEg2bBF2XZY2CKPS6GCxodCjCajxG4U"
"QmeQTuuXUkbAWDnhQo5pR9eXK6ZgqqYv2XhK1cC4Tt7uQa"
"QmNtB3mCdyoj59S6k36TfALLbxFGJubx7RGGXpo5AnsuUk"
"QmSDw6zaaNR4dkjM4oqHKD66QCEjUQZjCsgg7djUrP8bZK"
"QmXMkNfoAADtE33FJ2gqhk1MQvrhf6xCjSpt56pMYCPUNR"
"QmcU3HT9f38AfR3hjvYP9fkJhLN6qnTwP7kGf8d1Tpt2af"
"QmPGg8hGjw9xPe9eN3m5HMWoY5CQUivDgStD15by3j8D95"
"QmW39ukPr1bLZFS8aU5tTmh4FfJBwQB2KZWwYE7WsUej4b"
"QmNNpzZExguETMgP9nKrnJKtc7vbQzr4B2LHiqWaRmkoN8"
"QmcB7uPeHC3PhURCw5MnnWxVLoGHGaVEU1iCEfNbqD4k6Y"
"QmVaNtjzm42wRC4dMDk3bt4VLY9sff4PNxsHbLyX2BMX2d"
"QmQqLQySi1Azj2tR5y124JScK1sbveYgiSJSj9XdU7541d"
"Qmc4fFBUuQhrVxFvWZFtBCGAVXM9KYBVH6g7biJLB7efL9"
"Qma2LFomtUdgXVXJ4hKrLKntbSQFHEifPviHm6x2LURQj6"
"QmR3CDKkxq7bn4eJ3w9rcGLRLWr9AdmE2YgadbiFk4xruk"
"QmWJ5YxJHD5Q9ZNW1CCyYCoHq29XjCqkVyEUJiYfC19GQG"
"QmXtwPTqLsnyr5cfS6GUR8SYXLkTTgjHCSMWPxUpXjRu8E"
"QmaRLpknr6Puy4rUoLqCqqsACQK8uLjRkkrtAS7k2e1r4v"
"QmRaX2VVLHTW6jM6ZvXAwXVJK2CNpG5QTtYAmEYS151pye"
"QmVywznsgvrBTr49xCfR5AqgqiyfnReKUbWQySDyYpfmbs"
"QmaKBhxJrWMfnLhAhKvDG9kPqi2vqiRRC6JdtMz5RmqpyG"
"QmZPkzrue5WcPNzR7oun7VkLgReGt1vhuhXYCi6rDAdz6H"
"QmWzJ8yprZ9KkYYJfm3QLbXitHFw3KxvhiyxGa4pSK4Wix"
"QmdwG2P4RXezCauQdtj1oHHNTiS5rYimdmTwThgCYsGnoB"
"QmYEYcKMi6xpv69XGfGmwQjFkvVfMffZGGVPG7kvxxvFGK"
"QmQiJ9NwZghefXb86p2ck4r13dedmYoex4JwTfAAKuYPsd"
"QmYiFzA3YdU1PwHcxDnm6up6Zn9qT2UgUBdiRTzhSyontu"
"QmTzAb39aJ5AGBGyQuZcM6bgMc1u3EfQVGCaYMCAm8LdHN"
"QmPzmitVKi6tEY56mM7p2shqByQsi3ckuxuEs1epkndBtt"
"QmcjTtF8T5weXbY8Vv32Bjisdeouy1TVvuYpdgbk8bcZPd"
"QmWwobN856akaKHx1XvCPgcfiH8GZoFDydNPmEehJ5oA1t"
"QmNsByRbNt6jyqXYVqV71N2dM9Zt4RCTLXiCY1cJ579wNU"
"QmZNphGjKZy7oBXoEFFXGhH2UmUqzx1un59LXBUEMD2Y2d"
"QmV9oqkZVMtQVatJEWfGdXrgu5JsDyFGBkKKixAfTSQhVP"
"QmaCaoyfKmnXTH7JtKAn86fEZaf61bKFdiRBJSRdJYS5Rx"
"QmTKZEmqoiKc2mnz8hFAiTouYLB1g4WmGLVESbruZjcsiK"
"QmTu1Pbc7g78JUXZ6SUw7BMhwGCCE44ixbQzkCXXzQ5UsT"
"QmRwXqdLKyV2oH4CLtSEQH8Sco1HEig5Hd7ihj9Q1NpPVd"
"QmY7AA2aGPxg36uxVbvmhPzTF96w1rXnje6VJ8E6XbjNo2"
"QmQwFuNe9o1EF9wvUNThcEWBKW8RpWowP5P7Pujd43inSQ"
"Qmbi1KLYqLWBUCBaQsRCRxTdNvo6HkXzagUpFA29SXtRUM"
"QmNm6ByyvZ7q1y9Eohvhjkhm6EHRtXdmeCaiFBmpMqV8WZ"
"QmRQZrEKW6jmhnZc5osoatuVtmxN1njQz1SEsWHatwf259"
"QmfLRiuuevg7XCVbjCcaU91fsxUg7EMuNWaFATfmP5Rpfp"
"QmdKcjqRBPnB5c2xApFaxG9WnUUHq3J3o6ByuULCpj7EBD"
"QmSB546DPdKi6zzfpgyV5z85BpfjX8CF1UeDrDxLKauCro"
"QmUDpFZTozx17HKpMEmTqBJK25kywwF57EYSfhzAxhP7Zw"
"QmdTZeREgNEd1PoQp6ja64V9yGXFyTpGfJM6H8qa2fCZTR"
"QmeGQQghJNM1Gz4YAAM4rT3peq1Sg9YxVQPkFpfqmkJxbf"
"QmPnMkZRLWTyHBVUxt2fFrHjB6bLgnGu11a5ubHow4SLs6"
"QmWsDQwfpuH7qBRAfn7m9AQVtPcVAYxBt6kKsnKJSvJ1bk"
"QmWr4LG3NjpGmAtzaqQSarjj5yUwKWkcdx6wUjrnDR33qA"
"QmRPRR8aHT4cE7xRrjwECfD5AnHzco2eDZUkQzyzGpe3a5"
"bafkreiewfy6474jejksieq2zaaiygxlb5mkckfutme2pr3cr3yamp4sh7a"
"QmNZEuVn24qiwFA988uhZ4arZtoVYejWpCmfc3VJb3kLaR"
"QmQqLQySi1Azj2tR5y124JScK1sbveYgiSJSj9XdU7541d"
"QmNNpzZExguETMgP9nKrnJKtc7vbQzr4B2LHiqWaRmkoN8"
"QmPGg8hGjw9xPe9eN3m5HMWoY5CQUivDgStD15by3j8D95"
"QmSDw6zaaNR4dkjM4oqHKD66QCEjUQZjCsgg7djUrP8bZK"
"QmaCaoyfKmnXTH7JtKAn86fEZaf61bKFdiRBJSRdJYS5Rx"
"QmRwXqdLKyV2oH4CLtSEQH8Sco1HEig5Hd7ihj9Q1NpPVd"
"QmeQTuuXUkbAWDnhQo5pR9eXK6ZgqqYv2XhK1cC4Tt7uQa"
"QmTnAexEvnUqwpBUcGy56GixFiUBbePkx8gLA6ua7wFmdU"
"QmVzaNuVyrJH8PuXHtHqg9wtHhgCQvFA5abkt5MTrE7he7"
"QmSY85wDmGZbCEpkxo79Sb61YQjPZMFuNZxwbq6Uc6Ax6D"
"QmR3CDKkxq7bn4eJ3w9rcGLRLWr9AdmE2YgadbiFk4xruk"
"QmaRLpknr6Puy4rUoLqCqqsACQK8uLjRkkrtAS7k2e1r4v"
"QmVywznsgvrBTr49xCfR5AqgqiyfnReKUbWQySDyYpfmbs"
"QmaKBhxJrWMfnLhAhKvDG9kPqi2vqiRRC6JdtMz5RmqpyG"
"QmRrqfBYWGdU2zc3M1uJZ6YXKfNVrzcLp7rr4HsozXEM6p"
"Qmd1prgtiqPmQ4aFemnD5CNatwwoRH1bXvDtSRvXMykpWR"
"QmbT4viHnbAVYosZEynq4pZygTReiHW9DmzxM4SaXvqg3s"
"QmbY5bdeZQF2k6shiBtFwDdBWPMKBaBBKa7C2KrHGctz8v"
"QmWf2UgJjn9xcZMk1xPiCvTgo5kMRmwp2fmW5YRbST55cv"
"QmX3Ds5t675yDH27y1t8w7Zt8ggMNWwFrEK8g9xHE7cUsC"
"Qmc4fFBUuQhrVxFvWZFtBCGAVXM9KYBVH6g7biJLB7efL9"
"QmRq4qQN3ST54cxR2HoiW3UhGacVR1vXqUjJYG5pKeGHqF"
"Qmab8ZVmdgNdDpkdhVW3e5P2g5wSTYrT2vWWFeKxHDnCtB"
"QmRNLk4xp7bDBWbH4673pYiabUShWC7f3fKDWnHtLJSm31"
"QmeSXGFXsW8pAM5vqv5Z6sLPSUd4jMRC5CZjLv62BGpVSB"
"QmTnRpYW8FAT5vqRy4Xrj2CcpC333j1DY1Pq1ACsmhwDWx"
"QmRCZLmE8PCrcakhEg2bBF2XZY2CKPS6GCxodCjCajxG4U"
"QmNtB3mCdyoj59S6k36TfALLbxFGJubx7RGGXpo5AnsuUk"
"QmcU3HT9f38AfR3hjvYP9fkJhLN6qnTwP7kGf8d1Tpt2af"
"QmcB7uPeHC3PhURCw5MnnWxVLoGHGaVEU1iCEfNbqD4k6Y"
"Qma2LFomtUdgXVXJ4hKrLKntbSQFHEifPviHm6x2LURQj6"
"QmVaNtjzm42wRC4dMDk3bt4VLY9sff4PNxsHbLyX2BMX2d"
"QmYEYcKMi6xpv69XGfGmwQjFkvVfMffZGGVPG7kvxxvFGK"
"QmYiFzA3YdU1PwHcxDnm6up6Zn9qT2UgUBdiRTzhSyontu"
"QmcjTtF8T5weXbY8Vv32Bjisdeouy1TVvuYpdgbk8bcZPd"
"QmNsByRbNt6jyqXYVqV71N2dM9Zt4RCTLXiCY1cJ579wNU"
"QmV9oqkZVMtQVatJEWfGdXrgu5JsDyFGBkKKixAfTSQhVP"
"QmWr4LG3NjpGmAtzaqQSarjj5yUwKWkcdx6wUjrnDR33qA"
"QmNm6ByyvZ7q1y9Eohvhjkhm6EHRtXdmeCaiFBmpMqV8WZ"
"QmRQZrEKW6jmhnZc5osoatuVtmxN1njQz1SEsWHatwf259"
"QmdKcjqRBPnB5c2xApFaxG9WnUUHq3J3o6ByuULCpj7EBD"
"QmdTZeREgNEd1PoQp6ja64V9yGXFyTpGfJM6H8qa2fCZTR"
"QmWsDQwfpuH7qBRAfn7m9AQVtPcVAYxBt6kKsnKJSvJ1bk"
"QmdwG2P4RXezCauQdtj1oHHNTiS5rYimdmTwThgCYsGnoB"
"QmRPRR8aHT4cE7xRrjwECfD5AnHzco2eDZUkQzyzGpe3a5"
"QmPnMkZRLWTyHBVUxt2fFrHjB6bLgnGu11a5ubHow4SLs6"
"QmeGQQghJNM1Gz4YAAM4rT3peq1Sg9YxVQPkFpfqmkJxbf"
"QmUDpFZTozx17HKpMEmTqBJK25kywwF57EYSfhzAxhP7Zw"
"QmSB546DPdKi6zzfpgyV5z85BpfjX8CF1UeDrDxLKauCro"
"QmfLRiuuevg7XCVbjCcaU91fsxUg7EMuNWaFATfmP5Rpfp"
"Qmbi1KLYqLWBUCBaQsRCRxTdNvo6HkXzagUpFA29SXtRUM"
"QmQwFuNe9o1EF9wvUNThcEWBKW8RpWowP5P7Pujd43inSQ"
"QmY7AA2aGPxg36uxVbvmhPzTF96w1rXnje6VJ8E6XbjNo2"
"QmTu1Pbc7g78JUXZ6SUw7BMhwGCCE44ixbQzkCXXzQ5UsT"
"QmTKZEmqoiKc2mnz8hFAiTouYLB1g4WmGLVESbruZjcsiK"
"QmZNphGjKZy7oBXoEFFXGhH2UmUqzx1un59LXBUEMD2Y2d"
"QmWwobN856akaKHx1XvCPgcfiH8GZoFDydNPmEehJ5oA1t"
"QmPzmitVKi6tEY56mM7p2shqByQsi3ckuxuEs1epkndBtt"
"QmXMkNfoAADtE33FJ2gqhk1MQvrhf6xCjSpt56pMYCPUNR"
"QmQiJ9NwZghefXb86p2ck4r13dedmYoex4JwTfAAKuYPsd"
"QmWzJ8yprZ9KkYYJfm3QLbXitHFw3KxvhiyxGa4pSK4Wix"
"QmZPkzrue5WcPNzR7oun7VkLgReGt1vhuhXYCi6rDAdz6H"
"QmRaX2VVLHTW6jM6ZvXAwXVJK2CNpG5QTtYAmEYS151pye"
"QmXtwPTqLsnyr5cfS6GUR8SYXLkTTgjHCSMWPxUpXjRu8E"
"QmWJ5YxJHD5Q9ZNW1CCyYCoHq29XjCqkVyEUJiYfC19GQG"
"QmTzAb39aJ5AGBGyQuZcM6bgMc1u3EfQVGCaYMCAm8LdHN"
"QmW39ukPr1bLZFS8aU5tTmh4FfJBwQB2KZWwYE7WsUej4b"
"QmZyC6yNCThLdcByebw1vaJQGNKDmNkwKVRx2raNwXTQ2o"
"QmUGKFZwTcjqxAxLfosjP66vtsN2LHkVFygMYdwqrNbyGe"
"QmXzQD3Rsk8MWiZ4gCMBoYJLfLemLY6EovWU3AkviQMiEZ"
"QmRiyToHi4o9epmAAMPncHzCVwdrdh6ge4z6WjNXVLwQ39"
"QmTdEhvmMz5s4U2JKPnbJm4uxQNCsJMCTe3YrU5YASzbj8"
"QmXLNtVUeMK8RK89eWZZNKZephpXP5njG6R6RWYHn5Jt9i"
"QmZ8a71WZ1c7r4rJGo7Hydn9SvF5qaANX2Z17swHWX4Txv"
"QmeJncV9WsEp7qkKsapxVRqYvfhQTxPsve1A6rbjuffhZx"
"QmQo8qPaqtTaRQoGNbKCqgLfGuMD77fXko7W9QAez35HvK"
"QmSF59JzjXNSomMts77xMmeH56JRJaipNH5RkCfjZ3BiX6"
"QmdKBtpj6CT22ZwkoKWDsFwYwtLTN7RgBbpL22PoMH3YSm"
"QmcfC94PrCsR6psBFzNeQcyuoRYCGwrWczcNNMQswG7ip1"
"QmWvXeY1cj9GGFMBf2y6HHJz6cyaZcafuBKkgivwCnCLQx"
"QmZpVDKFGB6oaSPAwC1A7b4vWrADGzdzVvxRSP5wnSvAen"
"QmaTxf81fS81aWiHzosFPgnvL2YxdgQbXm21cyq6uGdsuo"
"QmYFrTxNRwc47d2DutL7nKr1omDZVKA8ixCSM25EUcWP52"
"QmRxjhgdyrrjyLjwYe8ynsWaYgjX4buidjiA8EFWww6qkz"
"QmWcf7nkQr5j22QnGzavGi8WVay3mVakHqDjE6QVFdpFNW"
"QmRtuiW55PuVso8L6DUCD2QKR7zfDuFUpwhuAnvE2CjTkC"
"QmSYDZMXGNE6nFZBjfgGEp7THkvhkR8Fjhh211HYp5s9Fk"
"QmT1673fd4VdDEhbD1jcQteVC6j2dBboBmUxT9fE3NST1X"
"QmUyUb2BLMEqccM9vF75HVaPRZbtzExgHXNwqRx5sk33wM"
"QmQxeHPWQJv82QRCpt7s3AsTSEg4KJGve8kh4ppvnEUe43"
"QmdLRioEbkDpL7T82i9BwJhFGhf7bevQaRx7FUDPS1Qq3j"
"QmVJjjb6KzRsuuLPJJuTVn6UeWw68ZuciyogNg6McdwMa9"
"QmTpXQjusH24xkWy7W1fYG5G6Sb5ejCx49VrUjemAWq3FC"
"QmWz4mLeupMtLAAeHYR47Q2MBntbcPf3vz1dVne8ceymfz"
"Qmaea13rwdJnnJMLM3zL1hTE25X5cpK163b2c1ktomBASN"
"QmUzSYYj75SxH8Fby4Wkztg87iP3xzL1Atgp5iWwVKQbKB"
"QmYHLbQZ78zKFeeuz5YFwPcRmEr1ZTQfhskiQveaj2PqNr"
"QmQwAnE8FuXU4jVARHNcvgBvUPUrFEb47Ztd1EXYzMHwyi"
"QmTiGWNVtQ5eam8k6gWsCr5PDJHexQH223eVTZf3M1oLHy"
"QmVJ3qk77rxyTLRdABqPub7gX6MbSWMDrXkUF9X4eMMU3o"
"QmZYtNLrBtLqhCaNsUuRSvnS8wvsCtMAGqXuNwmw4bm8fk"
"QmT3dG1WUTPuVezi9b3byxR8UXSSqTnDXEVAdU2vUpJ19e"
"QmTJfGKbWnFhzcGRrAjAve667gCwYF2meGgLuFzpJ64Mjc"
"QmfN7JcaNNE3EDaBVd98mCM7U5Noc8KGPRdGumMWcAqcju"
"QmUJi6fq2KvTVVw1YVRqTMjoKaW7HqMpRdzMyLagmHxfbL"
"QmRquD7ywERFc1NG83Qj9hupY3B7oWpqAY7G4V6MFafgXA"
"QmPtkYtpxsmDWsHP9D7cmjcgwwCVTvGvNN3VmKoPypATY8"
"QmcrRCXjdwPxfSQnZCBbCZPr595aBK28uYkhKyBZEqjo4u"
"QmNZKzmb9PWy8vPpyribzkTQcLHVFR3bA1nZaX4Ts9qoxi"
"QmZjisMmzqYZUuV8T1kvg2krUa6y2Nb84iJUNxkrsCgNBy"
"QmSkGVDEhXzu6Yw5aatMCWjb5ke1fFsA4mb4BFkBHiznAY"
"QmNeNsL6cBm15KHUdHLGmVuXqKZ6h2DJpFWw7yRasDrfpv"
"QmR9ZHxSFDjWAonfth5fJBZ4U6CfLwHTzMjc3p22xcZQaC"
"QmUJt2Z6qt5muTKLAJQhoFyWHo3bz7LYHU7n2GgUSaWDe8"
"QmQwH8u8vXYsrANgMbVRUz3vyyM6krDdsgf21HbrsqDvCQ"
"QmVmiR42KyhXzWWhxsTwdF8tTpCkpbS8VdiMroexQXHMQ5"
"QmaeWdaWaqnR6WcvN7SmUYdgz43SDPTaNYop2WYqoBDJ7D"
"QmXzrLDv18g4NvwN7HRwsx4ZrGVS1tMvLYCKMjsAhDw6aH"
"QmVKrThW9S6gLWqRMPceQPgQMKVVbrVUqT2qJmPw2v9McE"
"Qma4BVFGZMxzGwR4vf2AE3NNHQYD1A9TvYeDdtuoJCjiEy"
"QmNpHrhfriX4Yp6Zxki2xtLHE5eoDPA3s6A3NeEMVVMWZE"
"Qma2Nv3VzVAEpNAQCQQioBScWGkAuPrUL7WEdhj4Ccp5UQ"
"QmaeBhjWDYC3CSrjmn3eFFv6r5wEZqJkKBFnMvXFugj8iz"
"QmdXXoAsMp4kmWjZ8K2CmkCGeTS8PgvMJBT4oKe2xgTgAb"
"QmRmBBy8mBLuc9BXwfgKPhymoQf6VJ8aSDuU7qARGeSAJJ"
"QmeAShJfSA5oSq9o2BtPrBM7M8fNgU6bXekrcVreUuvrhm"
"QmaQqJTMcpWuFHvkQTLEe85bekHYYFjkf3uG4yjupNn7DZ"
"QmYa3d68orw4DVxC9NqtnQjJeAKR8JidsiMbRVYe1gTARc"
"QmdQj6x5L6fA3fYRoQMuEQpK2KDKZixL2CmnhccgrA5kb8"
"QmcXYoemNJ1doAPZN4n6JRQSTZFYVfhCopVDCBpiaoou7W"
"QmcrKBnQWcuUdetp1ccacYHRB99fz24eFSVTDaTGXDtdZm"
"QmZodH2xSeBLF1yYGCwEU58nFy8LVbnX65mD4SEt79KhyP"
"QmQSBE8QsFx2DDdPU3wQtNwSSWvMvZhcYoo2Nn75aJfPcV"
"QmQffizCusMJFUYw25dKPMP94hY8ATtcWSRSTkToVKw6Jw"
"QmQP6D7NC2CdsDDHnnrMJConpSXxUXudHcoGDGo3A67gXN"
"QmdgqsRen33kKBs3DpGGmmoM2L8wezyDw2zujjKeKCWJPt"
"QmXcSJu8r7gzxUwnBn4jisnXq8xQ49Sam6GudpnBafAcmQ"
"Qmf6d3eFmkGtk5PkicpQdy2K99PPTrNXT28st6byg29bGn"
"QmRBoG3mMDbXEVUGQxKDTAidJuCEvNP6BWhWEyQtCVZZFb"
"QmVR6DH8mcQJFb77RCYDD82xWnJwndnBoB3PeqHGcVw4wP"
"Qmcy6YcDdmcnCWvHqzSpKxfVoRtS2miEwc8KYTWkpouEtH"
"QmZxXW9eVCRhNgzWr3ViMvUA2G1ickwc3x4MXbs5LRNGdD"
" QmXAFDsuf3synfYnvSVz139PwYru3AEBD6SVCgw5b5waKx"
"ipfsFileCID"
The cause of this bug is as outlined in my original report. When ipfs pin remote ls gets each subsequent page of pins, it sends a get request like this
GET /ipfsapi/pins?before=2022-03-04T06%3A00%3A17Z&limit=10&status=pinned
The timestamp sent does not include the millisecond portion of time, and therefore pins created within the same second are missed. The problem is magnified by the relatively small batch size (10) which means that when there are hundreds of pins, there are many opportunities for missed records.
Even if a timestamp with millisecond precision is sent, the algorithm is still vulnerable to a situation in which two pins have exact same timestamp which could occur when (for instance) we pin an entire IPFS directory. Every pin in the directory might have the same timestamp down to the millisecond, therefore causing missing data in the ls command. I do have some pins in my service which have the exact same timestamp down to the millisecond and I think that is something that should be expected.
The solution is a little tricky if you want continue to paginate on a potentially non-unique key: I think the ultimate solution would be to use beforeOrEqual rather than before and eliminate duplicates in code. Alternatively, order the results by the requestId attribute which ought to be unique. But in the short term, I would be happy with a quick fix which specified a higher limit (say 500) and uses a timestamp that includes the millisecond portion of the date returned by the IPFS pinning service.
Moving to triage since this is more complex than I thought and is an issue with the spec itself.
We need to discuss that.
@Jorropo Any updates on this issue? We are also seeing this bug as well. We've confirmed that our HTTP API returns the full list, but ipfs pin remote ls --service=filebase | wc -l does not.
@acejam this is a systemic issue with the current way pagination is done in the pinning service API standard. There is workarrounds in both sides but neither clients neither API services like the workarround they need to do.
We had a meeting in Iceland with a few pinning service providers and comme up with an update to the spec (see https://github.com/ipfs/pinning-services-api-spec/issues/101)
And the solutions we agreeded on is:
- Remove count
- Add property for "nextToken" or similar name to act as cursor
- ~~Count could be replaced with "hasMore" boolean flag.~~
I think right now someone has to make a PR updating the spec, then we can implement it.
@Jorropo I have no doubt there may be other issues with the spec, but for this specific issue, can't we just add millisecond support back into the IPFS CLI? Pagination seems to work just fine using HTTP. It seems like providers are adhering to the spec here but the IPFS CLI client is not.
I might be missing something, but if the CLI sends exactly what the provider gives them for a timestamp, I don't see why this wouldn't work.
@acejam oh I see.
There is two issue here,
- what happen when two pins have the same timestamp ? (milisecond precision doesn't prevent this from happening) what I was talking about
- Kubo only send requests with second precision. (I don't know if we can just make it more precise the spec shows
2020-07-27T17:32:28Zas example)
@Jorropo
-
If two pins have the same timestamp, it shouldn't present any problems. For 99% of providers, the timestamp value is purely a SQL column in their database. In this case, the provider is simply running something like:
select * from pins where created_at > param. So both pins will either be included, or excluded, depending on the value. -
Based on the PSA spec, the Kubo client has a bug. It should be corrected and this problem goes away.
If two pins have the same timestamp, it shouldn't present any problems. For 99% of providers, the timestamp value is purely a SQL column in their database. In this case, the provider is simply running something like: select * from pins where created_at > param. So both pins will either be included, or excluded, depending on the value.
No most providers actually run select * from pins where created_at limit ORDER BY created_at > {req.After} LIMIT {req.Limit} OFFSET {req.Offset} which truncate the response.
Secondly we do not want provider to need SQL to implement the API, it must be implementable with a graph (maybe KV ?) DB also.
Anyway the cursor thing solve all of thoses issues.
No most providers actually run
select * from pins where created_at limit ORDER BY created_at > {req.After} LIMIT {req.Limit} OFFSET {req.Offset}which truncate the response.
@Jorropo You are correct. I was simplifying my example specifically to focus on this issue. The datastore a provider uses doesn't really matter here: the provider returned a valid timestamp in a response and the Kubo client is mutating that value, resulting in this bug.
Without this fix, the IPFS CLI will remain in a broken state. I don't see why we need to wait for an entire spec update when the client has a bug. In our case, we are having to explain to users that the IPFS CLI has a bug, and we are pushing them towards alternative clients. That results in a poor user experience.
@acejam I understand that why I said:
There is two issue here,
Pls open an other issue.
@Jorropo The title of this issue is: ipfs pin remote ls skips files that are created within a second of each other. We are having the exact same problem.
If I open a new issue, it will have the same title and the same description. This appears to be the correct issue.
If I open a new issue, it will have the same title and the same description. This appears to be the correct issue.
Yeah nvm, I don't remember what is the issue with sending more precision, let me check.
@acejam @gharriso I've implemented up to nanosecond precision here: #9206
I don't have any account to easily test this on, can you pls test #9206 for me ?
I get the same (inaccurate) result with kubo 0.15.0 and the PR, this is probably related to the missing pagination.
@MinmoTech sorry if this wasn't clear, I asked you to test a development branch, the fix is shipping in v0.16.0.
If you don't know how to build go code you can test when v0.16.0-rc1 will ship (I belive that happening next week). :slightly_smiling_face:
I think I did that :grin:
I checked out the repo and switched to your branch with gh pr checkout 9206, did make build and compared the resulting binary with my locally running instance (kubo 0.15.0).
ipfs pin remote ls --service=filebase | wc -l gives 311 as a result but ipfs pin remote service ls --stat returns
filebase https://api.filebase.io/v1/ipfs 0/0/10571/0
This is the case for both kubo 0.15.0 as well as the development branch.
I still assume this is an issue with the pagination mentioned further up in this thread as I strongly assume that there were more that 311 instances of file uploads which should result in a higher file count.
But I will also try the 0.16 rc when that releases in case I did anything wrong when building :)
@MinmoTech I have multiple questions:
ipfs pin remote lsI assume you didmake installalso (to actually use the binary you just built) ?- Did you restarted the daemon to use the new binary ? (I know it's a weird architecture but the command binary just calls into the daemon, so if you use a new CLI but with an old daemon, the old daemon code is still gonna handle the request).
If both yes I don't really know what could do this (we literally send up to nanosecond precision now).
As an update: I tried using v0.16.0-rc1 just now (I did restart the daemon) and get the same results.
I sadly haven't gotten to write code to log the requests yet, so I cannot provide more detailed logs.