filecoin-plus-large-datasets
filecoin-plus-large-datasets copied to clipboard
filecoin-project
[DataCap Application] <FogMeta Lab> - <Sentinel-2 Cloud-Optimized GeoTIFFs V1>
Data Owner Name
FogMeta Lab
What is your role related to the dataset
Data Preparer
Data Owner Country/Region
China
Data Owner Industry
Web3 / Crypto
Website
https://fogmetalabs.com
Social Media
Twitter: https://twitter.com/FogMeta
GitHub: https://github.com/FogMeta
Total amount of DataCap being requested
15PiB
Expected size of single dataset (one copy)
16.5 PiB
Number of replicas to store
4
Weekly allocation of DataCap requested
1PiB
On-chain address for first allocation
f1lgh5chkslwjql2fekxs3dflc6aznfkpexf5awfa
Data Type of Application
Public, Open Dataset (Research/Non-Profit)
Custom multisig
- [ ] Use Custom Multisig
Identifier
No response
Share a brief history of your project and organization
FogMeta Lab's research spans multiple levels from system technology, infrastructure, and middleware to services and solutions, and involves future systems, network technology and business, distributed systems and management, information management, and interactive and innovative services. Based on the views on and practices in the industry, FogMeta also solves the problem of business complexity through operations optimization and other technologies.
Is this project associated with other projects/ecosystem stakeholders?
No
If answered yes, what are the other projects/ecosystem stakeholders
No response
Describe the data being stored onto Filecoin
"The Sentinel-2 mission is a land monitoring constellation of two satellites that provide high resolution optical imagery and provide continuity for the current SPOT and Landsat missions. The mission provides a global coverage of the Earth's land surface every 5 days, making the data of great use in ongoing studies. This dataset is the same as the Sentinel-2 dataset, except the JP2K files were converted into Cloud-Optimized GeoTIFFs (COGs). Additionally, SpatioTemporal Asset Catalog metadata has were in a JSON file alongside the data, and a STAC API called Earth-search is freely available to search the archive. This dataset contains all of the scenes in the original Sentinel-2 Public Dataset and will grow as that does. L2A data are available from April 2017 over wider Europe region and globally since December 2018.
Update Frequency
New Sentinel data are added regularly, usually within few hours after they are available on Copernicus OpenHub."
Source: https://registry.opendata.aws/sentinel-2-l2a-cogs/
Where was the data currently stored in this dataset sourced from
AWS Cloud
If you answered "Other" in the previous question, enter the details here
No response
If you are a data preparer. What is your location (Country/Region)
China
If you are a data preparer, how will the data be prepared? Please include tooling used and technical details?
IPFS, lotus, graphsplit, others/custom tool, MetaArk(https://fogmetalabs.com/metaark)
If you are not preparing the data, who will prepare the data? (Provide name and business)
No response
Has this dataset been stored on the Filecoin network before? If so, please explain and make the case why you would like to store this dataset again to the network. Provide details on preparation and/or SP distribution.
We are THE FIRST data preparers that works on this dataset.
Please share a sample of the data
s3://sentinel-cogs/(16.4 PiB)
s3://sentinel-cogs-inventory/(3.4 TiB)
Confirm that this is a public dataset that can be retrieved by anyone on the Network
- [X] I confirm
If you chose not to confirm, what was the reason
No response
What is the expected retrieval frequency for this data
Monthly
For how long do you plan to keep this dataset stored on Filecoin
2 to 3 years
In which geographies do you plan on making storage deals
Greater China, Asia other than Greater China, Africa, North America, South America, Europe, Australia (continent), Antarctica
How will you be distributing your data to storage providers
Cloud storage (i.e. S3), HTTP or FTP server, IPFS, Shipping hard drives, Others
How do you plan to choose storage providers
Slack, Partners, Others
If you answered "Others" in the previous question, what is the tool or platform you plan to use
We'd also like to use Swan Chain (https://www.swanchain.io/) to choose storage providers who meet our requirements.
If you already have a list of storage providers to work with, fill out their names and provider IDs below
The storage providers we'd like to work with are presented below. Some of them are from the FilSwan platform.
f01955033
f02029115
f03624
f010088
f02301
f08399
f02401
f01955030
f0187709
f01163272
f01402814
f01390330
f01225882
f0717969
f03223
f01395673
f01072221
f0143858
f01786736
f0836160
f032824
f01443744
f01871352
f01907556
f01955028
f01947280
f01946551
f02012951
f01970630
f0240185
How do you plan to make deals to your storage providers
Boost client, Lotus client, Others/custom tool
If you answered "Others/custom tool" in the previous question, enter the details here
Swan Client tool
https://github.com/filswan/go-swan-client
Can you confirm that you will follow the Fil+ guideline
Yes
Thanks for your request! Everything looks good. :ok_hand:
A Governance Team member will review the information provided and contact you back pretty soon.
![large-datacap-requests[bot] avatar](https://avatars.githubusercontent.com/in/116297?v=4 "Published by a pub.dev verified publisher"){kind=small}
The new large dataset application will serve as our primary application moving forward. Consequently, all previously submitted applications have been closed, including: https://github.com/filecoin-project/filecoin-plus-large-datasets/issues/1739 https://github.com/filecoin-project/filecoin-plus-large-datasets/issues/1738; https://github.com/filecoin-project/filecoin-plus-large-datasets/issues/1740; https://github.com/filecoin-project/filecoin-plus-large-datasets/issues/1736; https://github.com/filecoin-project/filecoin-plus-large-datasets/issues/1735; https://github.com/filecoin-project/filecoin-plus-large-datasets/issues/1734; https://github.com/filecoin-project/filecoin-plus-large-datasets/issues/1733; https://github.com/filecoin-project/filecoin-plus-large-datasets/issues/1737; https://github.com/filecoin-project/filecoin-plus-large-datasets/issues/1741; https://github.com/filecoin-project/filecoin-plus-large-datasets/issues/1742
Also can you validate the SPs being used, entity and locations. How does Filswan collect and verify this information?
@hengdingy 16.5PiB x 4 = 66PiB correct? you are applying for 15
yes, we think 15PiB is a big amount datacap for us to prepare and send the data to the SPs in the first phase. After the phase, we will continue to apply the second datacap.
Also can you validate the SPs being used, entity and locations. How does Filswan collect and verify this information?
@Filplus-govteam Swan platform is a public platform, SPs can join it by validate the owner address signature. And Swan platform will auto-match and distribute the data to SPs by query-ask, fast-retrieve requirements and locations; for now, A part of the SPs information can be found here. And considering the platform's user privacy, we just give the permission to the fil+ gov team to verify. Please apply for the form's permission
@hengdingy thank you, it was reviewed. @Sunnyiscoming please trigger and we'll review deal onboarding
Datacap Request Trigger
Total DataCap requested
15PiB
Expected weekly DataCap usage rate
1PiB
Client address
f1lgh5chkslwjql2fekxs3dflc6aznfkpexf5awfa
DataCap Allocation requested
Multisig Notary address
f02049625
Client address
f1lgh5chkslwjql2fekxs3dflc6aznfkpexf5awfa
DataCap allocation requested
512TiB
Id
a103274b-d991-4323-bd58-4089a1ae6317
Please add provide ID, City, Country, Organization of each SP here according with the format https://github.com/filecoin-project/filecoin-plus-large-datasets/issues/2281#issuecomment-1833941809
@hengdingy, thanks to Filswan for the contributions to the community. I've noticed you've been active in the Fil+ community for a while, and you have rich experience in dealing with large amounts of data. For instance, this time you mentioned expecting more than 60 PiB.
I would like to ask, as a senior member of the community, if you could share any insights regarding how you verify the consistency of customer-stored data. What methods do you use for tracking them? This question has been discussed in the community for a long time, and I believe the @Filplus-govteam would also be interested in hearing from experienced users like yourself.
Thank you. :)
Nice community project, willing to support.
For the SPs location distribution, the applicant told me the verification method via slack, looking forward to see the follow-up progress.
Request Proposed
Your Datacap Allocation Request has been proposed by the Notary
Message sent to Filecoin Network
bafy2bzaceaam7i4aupsgnoin6qhgzusw35fopbvls6bd3o5uazktxbkdsvlck
Address
f1lgh5chkslwjql2fekxs3dflc6aznfkpexf5awfa
Datacap Allocated
512.00TiB
Signer Address
f1pnmzlxj7cfeo2v6oj5nco46hkg2l46wj7o4xxui
Id
a103274b-d991-4323-bd58-4089a1ae6317
You can check the status of the message here: https://filfox.info/en/message/bafy2bzaceaam7i4aupsgnoin6qhgzusw35fopbvls6bd3o5uazktxbkdsvlck
Request Approved
Your Datacap Allocation Request has been approved by the Notary
Message sent to Filecoin Network
bafy2bzacedwtufjom332wjhrdlxqxvk3oevnluukm3fbieznjhmf7wzaytuss
Address
f1lgh5chkslwjql2fekxs3dflc6aznfkpexf5awfa
Datacap Allocated
512.00TiB
Signer Address
f12mckci3omexgzoeosjvstcfxfe4vqw7owdia3da
Id
a103274b-d991-4323-bd58-4089a1ae6317
You can check the status of the message here: https://filfox.info/en/message/bafy2bzacedwtufjom332wjhrdlxqxvk3oevnluukm3fbieznjhmf7wzaytuss
Please add provide ID, City, Country, Organization of each SP here according with the format #2281 (comment)
https://github.com/filecoin-project/filecoin-plus-large-datasets/issues/1999#issuecomment-1824684456
@Sunnyiscoming please request access to the form)
DataCap and CID Checker Report[^1]
No active deals found for this client.
[^1]: To manually trigger this report, add a comment with text checker:manualTrigger
![filplus-checker-app[bot] avatar](https://avatars.githubusercontent.com/in/267472?v=4 "Published by a pub.dev verified publisher"){kind=small}
I have ask for access to the form. I will list the information of these nodes here.
This application has not seen any responses in the last 10 days. This issue will be marked with Stale label and will be closed in 4 days. Comment if you want to keep this application open.
-- Commented by Stale Bot.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
DataCap and CID Checker Report[^1]
No active deals found for this client.
[^1]: To manually trigger this report, add a comment with text checker:manualTrigger
DataCap and CID Checker Report[^1]
No active deals found for this client.
[^1]: To manually trigger this report, add a comment with text checker:manualTrigger
There is no organization name of each nodes. Can you add them in the form? @hengdingy
DataCap and CID Checker Report[^1]
No active deals found for this client.
[^1]: To manually trigger this report, add a comment with text checker:manualTrigger
checker:manualTrigger
DataCap and CID Checker Report1
No active deals found for this client.
Footnotes
- To manually trigger this report, add a comment with text
checker:manualTrigger↩
checker:manualTrigger
DataCap and CID Checker Report[^1]
No active deals found for this client.
[^1]: To manually trigger this report, add a comment with text checker:manualTrigger