filecoin-plus-large-datasets
filecoin-plus-large-datasets copied to clipboard
filecoin-project
[DataCap Application] NFT.Storage - NFT.Storage
Data Owner Name
NFT.Storage
What is your role related to the dataset
Data Preparer
Data Owner Country/Region
United States
Data Owner Industry
Life Science / Healthcare
Website
https://nft.storage
Social Media
https://twitter.com/NFTdotStorage
Total amount of DataCap being requested
15PiB
Expected size of single dataset (one copy)
500TiB and growing
Number of replicas to store
10
Weekly allocation of DataCap requested
250TiB
On-chain address for first allocation
f3watks6wyq5sakerofowyv7q4gwx4z6ukmfe2irh5zitspj6gpceudt3rrbvm5psofkcgoabev3s2mwtkcunq
Data Type of Application
Public, Open Commercial/Enterprise
Custom multisig
- [ ] Use Custom Multisig
Identifier
No response
Share a brief history of your project and organization
NFT.Storage was created in 2021 to be an "Internet Archive" of NFTs, storing off-chain NFT data as a public good using the Filecoin network.
Is this project associated with other projects/ecosystem stakeholders?
Yes
If answered yes, what are the other projects/ecosystem stakeholders
This project is associated with Protocol Labs. Projects have plans for independence.
Describe the data being stored onto Filecoin
User uploaded data that, per the NFT.Storage Terms of Service (https://nft.storage/terms), must be legitimate off-chain NFT data. In addition, NFT.Storage is likely to scrape NFT data using on-chain references in the future.
Where was the data currently stored in this dataset sourced from
Other
If you answered "Other" in the previous question, enter the details here
User uploads
If you are a data preparer. What is your location (City and Country)
Seattle, WA
If you are a data preparer, how will the data be prepared? Please include tooling used and technical details?
The data preparer is DAG House, which builds client tools and runs infrastructure for NFT.Storage and web3.storage. Users upload CAR files directly, which are then aggregated into sector sized CAR files. Our client libraries do the CAR conversion for users. One example can be found here: https://github.com/web3-storage/w3up-cli
If you are not preparing the data, who will prepare the data? (Provide name and business)
No response
Has this dataset been stored on the Filecoin network before? If so, please explain and make the case why you would like to store this dataset again to the network. Provide details on preparation and/or SP distribution.
Sort of. We previously stored NFT.Storage data using the same Datacap wallet found here (https://github.com/filecoin-project/filecoin-plus-large-datasets/issues/1838). As copies of data expire that were stored using this other wallet, we will use this wallet to re-up the number of replicas.
Please share a sample of the data
- https://ipfs.io/ipfs/bafybeid5jpdqzlb4tqsd6peoa7qstoxat3ovsg62wutyp4gnzqbqsggfsq
- https://ipfs.io/ipfs/bafybeihity6bx24npzvvkzopjbat25ekefjwmnshe7rvldy72dxngzf644
- https://ipfs.io/ipfs/bafybeicvcevx3ktiqjsfwnjguu4lnzejlhgb35brayuod5xdtn7demfdhe
Confirm that this is a public dataset that can be retrieved by anyone on the Network
- [X] I confirm
If you chose not to confirm, what was the reason
No response
What is the expected retrieval frequency for this data
Daily
For how long do you plan to keep this dataset stored on Filecoin
Permanently
In which geographies do you plan on making storage deals
Asia other than Greater China, Africa, North America, South America, Europe, Australia (continent)
How will you be distributing your data to storage providers
Cloud storage (i.e. S3)
How do you plan to choose storage providers
Others
If you answered "Others" in the previous question, what is the tool or platform you plan to use
We plan to use Spade for SP selection and deal execution. Spade is being developed by the Data Programs team at Protocol Labs (https://github.com/data-preservation-programs/spade) and supports storage client to SP mtching based on requirements like geography, size, retrievability, etc.
If you already have a list of storage providers to work with, fill out their names and provider IDs below
No response
How do you plan to make deals to your storage providers
Others/custom tool
If you answered "Others/custom tool" in the previous question, enter the details here
https://github.com/web3-storage/w3filecoin
Can you confirm that you will follow the Fil+ guideline
Yes
Thanks for your request!
Heads up, you’re requesting more than the typical weekly onboarding rate of DataCap!
![large-datacap-requests[bot] avatar](https://avatars.githubusercontent.com/in/116297?v=4 "Published by a pub.dev verified publisher"){kind=small}
@dchoi27 the bot is confused here. How much DataCap are you requesting? Need a number. If its greater than 15PiB it will go through E-Fil+ pathway. Let me know.
Got it - I changed to 15PiB. (note that greater than 15PiB is listed as an option in the issue creation form)
Yes, if you need greater than 15PiB, we can add that. Just let me know the amount. Need to do it manually so that I am triggered to work with the large data client via E-Fil processes.
Seems like for now you have 500TiB x 10 copies so ~ 5 to 6 PiB needed?
@dchoi27 can you confirm answer to question above? https://github.com/filecoin-project/filecoin-plus-large-datasets/issues/2110#issuecomment-1650326985
This one? (think the comment was edited to include this question after i adjusted the application)
Seems like for now you have 500TiB x 10 copies so ~ 5 to 6 PiB needed?
The dataset is constantly growing because of new user uploads, so we were advised to submit an application for 15PiB (so we wouldn't need to keep submitting new applications)
Datacap Request Trigger
Total DataCap requested
15PiB
Expected weekly DataCap usage rate
250TiB
Client address
f3watks6wyq5sakerofowyv7q4gwx4z6ukmfe2irh5zitspj6gpceudt3rrbvm5psofkcgoabev3s2mwtkcunq
DataCap Allocation requested
Multisig Notary address
f02049625
Client address
f3watks6wyq5sakerofowyv7q4gwx4z6ukmfe2irh5zitspj6gpceudt3rrbvm5psofkcgoabev3s2mwtkcunq
DataCap allocation requested
125TiB
Id
d467872a-eb72-48c8-babb-26c195b92111
Request Proposed
Your Datacap Allocation Request has been proposed by the Notary
Message sent to Filecoin Network
bafy2bzacebay6asyijh47npwup5bzj52xsgjtcflcmg4hxfkwxnu4myubtghq
Address
f3watks6wyq5sakerofowyv7q4gwx4z6ukmfe2irh5zitspj6gpceudt3rrbvm5psofkcgoabev3s2mwtkcunq
Datacap Allocated
125.00TiB
Signer Address
f1bwugfihrmn3iyunzyxst5nttql3dge4khwmurtq
Id
d467872a-eb72-48c8-babb-26c195b92111
You can check the status of the message here: https://filfox.info/en/message/bafy2bzacebay6asyijh47npwup5bzj52xsgjtcflcmg4hxfkwxnu4myubtghq
Request Approved
Your Datacap Allocation Request has been approved by the Notary
Message sent to Filecoin Network
bafy2bzacedx37eecvx22xgir26ws7yrxeiqyo6qeqsfu2ctjjwwwciy66f7fa
Address
f3watks6wyq5sakerofowyv7q4gwx4z6ukmfe2irh5zitspj6gpceudt3rrbvm5psofkcgoabev3s2mwtkcunq
Datacap Allocated
125.00TiB
Signer Address
f1krmypm4uoxxf3g7okrwtrahlmpcph3y7rbqqgfa
Id
d467872a-eb72-48c8-babb-26c195b92111
You can check the status of the message here: https://filfox.info/en/message/bafy2bzacedx37eecvx22xgir26ws7yrxeiqyo6qeqsfu2ctjjwwwciy66f7fa
This application has not seen any responses in the last 10 days. This issue will be marked with Stale label and will be closed in 4 days. Comment if you want to keep this application open.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
This application has not seen any responses in the last 14 days, so for now it is being closed. Please feel free to contact the Fil+ Gov team to re-open the application if it is still being processed. Thank you!
-- Commented by Stale Bot.
DataCap Allocation requested
Request number 6
Multisig Notary address
f02049625
Client address
f3watks6wyq5sakerofowyv7q4gwx4z6ukmfe2irh5zitspj6gpceudt3rrbvm5psofkcgoabev3s2mwtkcunq
DataCap allocation requested
1000.0TiB
Id
4854e052-0cfa-45d4-8435-74cbee291263
Stats & Info for DataCap Allocation
Multisig Notary address
f02049625
Client address
f3watks6wyq5sakerofowyv7q4gwx4z6ukmfe2irh5zitspj6gpceudt3rrbvm5psofkcgoabev3s2mwtkcunq
Rule to calculate the allocation request amount
400% of weekly dc amount requested
DataCap allocation requested
1000.0TiB
Total DataCap granted for client so far
1.1368683772161625e+40YiB
Datacap to be granted to reach the total amount requested by the client (15PiB)
1.1368683772161625e+40YiB
Stats
| Number of deals | Number of storage providers | Previous DC Allocated | Top provider | Remaining DC |
|---|---|---|---|---|
| 2678 | 10 | 125TiB | 59.97 | 31.40TiB |
Request Proposed
Your Datacap Allocation Request has been proposed by the Notary
Message sent to Filecoin Network
bafy2bzacedgiibl2py7mqgkxrs7drqnv5xep63b2ojpvoqszomt7tobdznnoq
Address
f3watks6wyq5sakerofowyv7q4gwx4z6ukmfe2irh5zitspj6gpceudt3rrbvm5psofkcgoabev3s2mwtkcunq
Datacap Allocated
1000.00TiB
Signer Address
f1krmypm4uoxxf3g7okrwtrahlmpcph3y7rbqqgfa
Id
4854e052-0cfa-45d4-8435-74cbee291263
You can check the status of the message here: https://filfox.info/en/message/bafy2bzacedgiibl2py7mqgkxrs7drqnv5xep63b2ojpvoqszomt7tobdznnoq
DataCap and CID Checker Report Summary[^1]
Retrieval Statistics
- Overall Graphsync retrieval success rate:
- Overall HTTP retrieval success rate: 2.13%
- Overall Bitswap retrieval success rate:
Storage Provider Distribution
✔️ Storage provider distribution looks healthy.
Deal Data Replication
⚠️ 96.29% of deals are for data replicated across less than 3 storage providers.
Deal Data Shared with other Clients[^3]
⚠️ CID sharing has been observed. (Top 3)
- 669.28 TiB - f3ugiocmlvcaixff6gisjyi3nupy4l6bwlfbrqfe6
47y3urntcba2quri3v53hie4ad65fuydpiczgz2ol
gd5a - DAGHouse and the users of our tools - 247.91 TiB - f144zep4gitj73rrujd3jw6iprljicx6vl4wbeavi - Textile
- 2.09 TiB - f3se4aw3p3tlvggkhsymdg7qmbjnxu4uudjvar6c3
scfwsge6voesbxff4lkoiqyjcl4yot7hxqmukhusx
gurq - DAGHouse and the users of our tools
[^1]: To manually trigger this report, add a comment with text checker:manualTrigger
[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger
[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...
Full report
Click here to view the CID Checker report. Click here to view the Retrieval Dashboard. Click here to view the Retrieval report.
![filplus-checker-app[bot] avatar](https://avatars.githubusercontent.com/in/267472?v=4 "Published by a pub.dev verified publisher"){kind=small}
This application has not seen any responses in the last 10 days. This issue will be marked with Stale label and will be closed in 4 days. Comment if you want to keep this application open.
-- Commented by Stale Bot.
DataCap and CID Checker Report Summary[^1]
Retrieval Statistics
- Overall Graphsync retrieval success rate:
- Overall HTTP retrieval success rate: 2.28%
- Overall Bitswap retrieval success rate:
Storage Provider Distribution
✔️ Storage provider distribution looks healthy.
Deal Data Replication
⚠️ 96.27% of deals are for data replicated across less than 3 storage providers.
Deal Data Shared with other Clients[^3]
⚠️ CID sharing has been observed. (Top 3)
- 669.34 TiB - f3ugiocmlvcaixff6gisjyi3nupy4l6bwlfbrqfe6
47y3urntcba2quri3v53hie4ad65fuydpiczgz2ol
gd5a - DAGHouse and the users of our tools - 247.91 TiB - f144zep4gitj73rrujd3jw6iprljicx6vl4wbeavi - Textile
- 55.94 TiB - f3se4aw3p3tlvggkhsymdg7qmbjnxu4uudjvar6c3
scfwsge6voesbxff4lkoiqyjcl4yot7hxqmukhusx
gurq - DAGHouse and the users of our tools
[^1]: To manually trigger this report, add a comment with text checker:manualTrigger
[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger
[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...
Full report
Click here to view the CID Checker report. Click here to view the Retrieval Dashboard. Click here to view the Retrieval report.
Request Approved
Your Datacap Allocation Request has been approved by the Notary
Message sent to Filecoin Network
bafy2bzacea5m5xuibh2lrlibyl3dxxbxiubdatxsat52q2hu6fwwnqc2x73wc
Address
f3watks6wyq5sakerofowyv7q4gwx4z6ukmfe2irh5zitspj6gpceudt3rrbvm5psofkcgoabev3s2mwtkcunq
Datacap Allocated
1000.00TiB
Signer Address
f1pszcrsciyixyuxxukkvtazcokexbn54amf7gvoq
Id
4854e052-0cfa-45d4-8435-74cbee291263
You can check the status of the message here: https://filfox.info/en/message/bafy2bzacea5m5xuibh2lrlibyl3dxxbxiubdatxsat52q2hu6fwwnqc2x73wc
This application has not seen any responses in the last 10 days. This issue will be marked with Stale label and will be closed in 4 days. Comment if you want to keep this application open.
-- Commented by Stale Bot.
This application has not seen any responses in the last 14 days, so for now it is being closed. Please feel free to contact the Fil+ Gov team to re-open the application if it is still being processed. Thank you!
-- Commented by Stale Bot.