filecoin-plus-large-datasets
filecoin-plus-large-datasets copied to clipboard
filecoin-project
[DataCap Application] <SXX Future Data> - LAMOST DR5 Public Dataset
Data Owner Name
SXX Future Data
What is your role related to the dataset
Data Preparer
Data Owner Country/Region
China
Data Owner Industry
Not-for-Profit
Website
https://www.sxxfuture.com/
Social Media
https://twitter.com/sxxfuture
https://medium.com/@sxxfuture.official
https://discord.gg/GFsJaUFS9K
https://weibo.com/u/7801654981
Total amount of DataCap being requested
15PiB
Weekly allocation of DataCap requested
500TiB
On-chain address for first allocation
f1j4jxcfyqkzxivzwktseo6dtbhd7mczlkg5uhrpq
Data Type of Application
Public, Open Dataset (Research/Non-Profit)
Custom multisig
- [ ] Use Custom Multisig
Share a brief history of your project and organization
Focusing on distributed big data storage, disaster data protection, encryption algorithm research, encrypted data product application and Web3.0 new-generation Internet research and development.
SXX Future Data provides government enterprises and individual users with products and service system with data value as the core to meet the ever-expanding demand for mass data storage, management and application.
Is this project associated with other projects/ecosystem stakeholders?
No
If answered yes, what are the other projects/ecosystem stakeholders
No response
Describe the data being stored onto Filecoin
The Large Sky Area Multi-Object Fiber Spectroscopic Telescope (LAMOST) is a Chinese national scientific research facility operated by the National Astronomical Observatories, Chinese Academy of Sciences. It is a special reflecting Schmidt telescope with 4000 fibers in a field of view of 20 deg2
in the sky. Until July 2017, LAMOST has completed its pilot survey which was launched in October 2011 and ended in June 2012, and the first five years of regular survey which was initiated on September 2012. After this six-year-survey, we totally obtain 9,026,365 spectra, which consist of stars, galaxies, quasars and other unknown objects[1−7]
. Now, the fifth data release (DR5) has published online (http://dr5.lamost.org/), and released data products include：
Spectra. - IIn general, there are 9,026,365 flux- and wavelength-calibrated, sky-subtracted spectra in DR5, including 8,183,160 stars, 152,863 galaxies, 52,453 quasars, and 637,889 unknown objects, and these spectra cover the wavelength range of 3690-9100 angstrom with a resolution of 1800[2−3]
at the 5500 angstrom.
Spectroscopic Parameters Catalogs. - In this data release, six spectroscopic parameters catalogs are also published，and they are the LAMOST general catalog, the A, F, G and K type star catalog, the A type star catalog, the M dwarf catalog, the observed plate information catalog, and the input catalog respectively. In the LAMOST general catalog, it includes 36 columns of basic spectroscopic information, for example, right ascension, declination, signal to noise ratio, magnitude, classification and redshift, which are also provided by the A, F, G and K type star catalog, the A type star catalog, and the M dwarf catalog. These three catalogs also provide other parameters, for example, atmospheric parameters (effective temperature, surface gravity, and metallicity), spectral line indices, line widths, the metallicity sensitive parameter, and the magnetic activity flag. In addition, the observed plate information catalog mainly contains nine basic plate information for all published plates, and the input catalog includes 24 basic fields mentioned above and three new fields which are not included in above catalogs.
http://dr5.lamost.org/v3/doc/data-production-description
Guoshoujing Telescope (the Large Sky Area Multi-Object Fiber Spectroscopic Telescope LAMOST) is a National Major Scientific Project built by the Chinese Academy of Sciences. Funding for the project has been provided by the National Development and Reform Commission. LAMOST is operated and managed by the National Astronomical Observatories, Chinese Academy of Sciences.
Where was the data currently stored in this dataset sourced from
Other
If you answered "Other" in the previous question, enter the details here
http://dr5.lamost.org/
How do you plan to prepare the dataset
lotus, singularity, graphsplit
If you answered "other/custom tool" in the previous question, enter the details here
No response
Please share a sample of the data
http://dr5.lamost.org/v3/sas/catalog/
http://dr5.lamost.org/v3/sas/fits/20111024/F5902/
http://dr5.lamost.org/v3/sas/fits/20111024/F5907/
http://dr5.lamost.org/v3/sas/fits/20111024/F5909/
http://dr5.lamost.org/v3/sas/png/20111024/F5902/
http://dr5.lamost.org/v3/sas/png/20111024/F5907/
http://dr5.lamost.org/v3/sas/png/20111024/F5909/
http://dr5.lamost.org/v3/sas/sky/20111024/
Confirm that this is a public dataset that can be retrieved by anyone on the Network
- [X] I confirm
If you chose not to confirm, what was the reason
No response
What is the expected retrieval frequency for this data
Yearly
For how long do you plan to keep this dataset stored on Filecoin
2 to 3 years
In which geographies do you plan on making storage deals
Greater China, Asia other than Greater China, North America, Europe
How will you be distributing your data to storage providers
HTTP or FTP server, Shipping hard drives
How do you plan to choose storage providers
Slack, Partners
If you answered "Others" in the previous question, what is the tool or platform you plan to use
No response
If you already have a list of storage providers to work with, fill out their names and provider IDs below
f02188881
f01986236
f02169409
f01986262
f02144581
How do you plan to make deals to your storage providers
Boost client, Lotus client, Singularity
If you answered "Others/custom tool" in the previous question, enter the details here
No response
Can you confirm that you will follow the Fil+ guideline
Yes
Thanks for your request!
Heads up, you’re requesting more than the typical weekly onboarding rate of DataCap!
![large-datacap-requests[bot] avatar](https://avatars.githubusercontent.com/in/116297?v=4 "Published by a pub.dev verified publisher"){kind=small}
Thanks for your request! Everything looks good. :ok_hand:
A Governance Team member will review the information provided and contact you back pretty soon.
It appears that there are currently two applications open for the same project: application #1992 and #1993. Going forward, kindly close one of the applications and submit a single application requesting the full DataCap amount.
@sxxfuture-official It appears that you have submitted applications for 10PiB in issue #1992 and another 10PiB in issue #1993, resulting in a combined total of 20PiB. Can you clarify the reason behind the revised application requesting a reduced capacity of 15PiB. Additionally, could you kindly provide details regarding the dataset size and the number of replicas that will be necessary?
@simonkim0515 Until July 2017, LAMOST has completed its pilot survey which was launched in October 2011 and ended in June 2012, and the first five years of regular survey which was initiated on September 2012. After this six-year-survey, we totally obtain 9,026,365 spectra. The fifth data release (DR5) has published online (http://dr5.lamost.org/), and released data products include：
- Spectra. - there are 9,026,365 flux- and wavelength-calibrated, sky-subtracted spectra.
- Spectroscopic Parameters Catalogs. - six spectroscopic parameters catalogs.
The main data occupation is spectral image data, in order to accurately retrieve the CID of each image, we will not compress the data before sealing. The uncompressed data size of each spectral data is ≈ 205MByte.
Originally we planned to save 6-7 replicas to different SPs. 205MB * 9,026,365 Spectra / 18GB per sector * 6 replicas = 601757 sectors = 18.36 PiB DataCap
So I applied for two 10PiB LDNs for the first time, but you asked me to close one of them and only keep one LDN, but the maximum capacity of a single LDN is only 15PiB, so I can only reduce the number of replicas for some data during the sealing process.
@sxxfuture-official Thank you for providing an explanation. It is worth noting that the DataCap application limit is not limited to 15PiB. Applicants have the option to apply for a higher amount by selecting the "More Than 15PiB" option, which will prompt contact from @Kevin-FF-USA.
@simonkim0515 Thank you very much for your reminder ! At the current stage, 15PiB is enough to save 5-6 replicas to avoid complicating the process. The current application quota is acceptable.
Could you send an email to [email protected] with your official domain in order to confirm your identity? Email name should includes the issue id #1992.
Datacap Request Trigger
Total DataCap requested
15PiB
Expected weekly DataCap usage rate
500TiB
Client address
f1j4jxcfyqkzxivzwktseo6dtbhd7mczlkg5uhrpq
DataCap Allocation requested
Multisig Notary address
f02049625
Client address
f1j4jxcfyqkzxivzwktseo6dtbhd7mczlkg5uhrpq
DataCap allocation requested
250TiB
Id
c88af20a-820a-4d81-b2f8-fcde9f6f8111
Hello @sxxfuture-official How much data have you prepared? How many copies do you plan to send?
@nj-steve
- Spectra. - there are 9,026,365 flux- and wavelength-calibrated, sky-subtracted spectra.
- Spectroscopic Parameters Catalogs. - six spectroscopic parameters catalogs.
The main data occupation is spectral image data, in order to accurately retrieve the CID of each image, we will not compress the data before sealing. The uncompressed data size of each spectral data is ≈ 205MByte.
Now, we planned to save 5-6 replicas to different SPs. 205MB * 9,026,365 Spectra / 18GB per sector * 6 replicas = 601757 sectors = 18.36 PiB DataCap
Request Proposed
Your Datacap Allocation Request has been proposed by the Notary
Message sent to Filecoin Network
bafy2bzacebc57vbxm5l763r3ixvyj2mwc5pecg4rbw6sqr5katcofqgs27jfq
Address
f1j4jxcfyqkzxivzwktseo6dtbhd7mczlkg5uhrpq
Datacap Allocated
250.00TiB
Signer Address
f1xx6555qijma7igpnjspyvdunc4vfxkawnpqy5ii
Id
c88af20a-820a-4d81-b2f8-fcde9f6f8111
You can check the status of the message here: https://filfox.info/en/message/bafy2bzacebc57vbxm5l763r3ixvyj2mwc5pecg4rbw6sqr5katcofqgs27jfq
Request Approved
Your Datacap Allocation Request has been approved by the Notary
Message sent to Filecoin Network
bafy2bzaceco4ln3setfcidyz2aobcm43cv4ie3kdqtzrg3r5xgjtdkmkz6gd2
Address
f1j4jxcfyqkzxivzwktseo6dtbhd7mczlkg5uhrpq
Datacap Allocated
250.00TiB
Signer Address
f1yjhnsoga2ccnepb7t3p3ov5fzom3syhsuinxexa
Id
c88af20a-820a-4d81-b2f8-fcde9f6f8111
You can check the status of the message here: https://filfox.info/en/message/bafy2bzaceco4ln3setfcidyz2aobcm43cv4ie3kdqtzrg3r5xgjtdkmkz6gd2
This application has not seen any responses in the last 10 days. This issue will be marked with Stale label and will be closed in 4 days. Comment if you want to keep this application open.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
This application has not seen any responses in the last 14 days, so for now it is being closed. Please feel free to contact the Fil+ Gov team to re-open the application if it is still being processed. Thank you!
DataCap and CID Checker Report Summary[^1]
Retrieval Statistics
⚠️ All retrieval success ratios are below 1%.
- Overall Graphsync retrieval success rate: 0.00%
- Overall HTTP retrieval success rate: 0.00%
- Overall Bitswap retrieval success rate: 0.00%
Storage Provider Distribution
✔️ Storage provider distribution looks healthy.
Deal Data Replication
✔️ Data replication looks healthy.
Deal Data Shared with other Clients[^3]
✔️ No CID sharing has been observed.
[^1]: To manually trigger this report, add a comment with text checker:manualTrigger
[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger
[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...
Full report
Click here to view the CID Checker report. Click here to view the Retrieval report.
![filplus-checker-app[bot] avatar](https://avatars.githubusercontent.com/in/267472?v=4 "Published by a pub.dev verified publisher"){kind=small}
DataCap Allocation requested
Request number 2
Multisig Notary address
f02049625
Client address
f1j4jxcfyqkzxivzwktseo6dtbhd7mczlkg5uhrpq
DataCap allocation requested
500TiB
Id
8f69ef04-a2ea-422c-a8d2-06fc8fdac3c7
Stats & Info for DataCap Allocation
Multisig Notary address
f02049625
Client address
f1j4jxcfyqkzxivzwktseo6dtbhd7mczlkg5uhrpq
Rule to calculate the allocation request amount
100% of weekly dc amount requested
DataCap allocation requested
500TiB
Total DataCap granted for client so far
250TiB
Datacap to be granted to reach the total amount requested by the client (15PiB)
14.75PiB
Stats
| Number of deals | Number of storage providers | Previous DC Allocated | Top provider | Remaining DC |
|---|---|---|---|---|
| 5943 | 3 | 250TiB | 33.65 | 59.81TiB |
Hi @sxxfuture-official
Seems you sealed 62% of this datacap with your own miners. Can you explain us why this is?
Secondly none of your data is retrievable.
As notary you need to have an example function for the community and clean sheet. What is your plan here?
@cryptowhizzard Because it is the first round of quota, our Miner packaging will be faster, and other SPs will follow the pace of packaging. Regarding the retrieval problem, we will definitely solve this problem. More SPs is on the way.
Since we have been using lotus publish storage deals before, the technical team has been studying how to support http retrieval, and I believe it will be improved in the near future.