go-data-transfer Support Multiple Transports, allows choosing per transfer

After #310 is complete, and we've implemented a second transfer with #311, we need to modify the data transfer constructor and the open channel call to support selecting transports on a per transfer basis

Mar 25 '22 22:03 hannahhoward

Thanks for your request!

Heads up, you’re requesting more than the typical weekly onboarding rate of DataCap!

May 16 '23 04:05 large-datacap-requests[bot]

Thanks for your request! Everything looks good. :ok_hand:

A Governance Team member will review the information provided and contact you back pretty soon.

May 16 '23 04:05 large-datacap-requests[bot]

May 16 '23 04:05 sxxfuture-official

The public time node table of its data. http://www.lamost.org/lmusers/

May 16 '23 04:05 sxxfuture-official

It appears that there are currently two applications open for the same project: application #1992 and #1993. Going forward, kindly close one of the applications and submit a single application requesting the full DataCap amount.

May 16 '23 12:05 simonkim0515

OK, I will modify it.

May 16 '23 13:05 sxxfuture-official

@sxxfuture-official It appears that you have submitted applications for 10PiB in issue #1992 and another 10PiB in issue #1993, resulting in a combined total of 20PiB. Can you clarify the reason behind the revised application requesting a reduced capacity of 15PiB. Additionally, could you kindly provide details regarding the dataset size and the number of replicas that will be necessary?

May 16 '23 18:05 simonkim0515

@simonkim0515 Until July 2017, LAMOST has completed its pilot survey which was launched in October 2011 and ended in June 2012, and the first five years of regular survey which was initiated on September 2012. After this six-year-survey, we totally obtain 9,026,365 spectra. The fifth data release (DR5) has published online (http://dr5.lamost.org/), and released data products include：

Spectra. - there are 9,026,365 flux- and wavelength-calibrated, sky-subtracted spectra.
Spectroscopic Parameters Catalogs. - six spectroscopic parameters catalogs.

The main data occupation is spectral image data, in order to accurately retrieve the CID of each image, we will not compress the data before sealing. The uncompressed data size of each spectral data is ≈ 205MByte.

Originally we planned to save 6-7 replicas to different SPs. 205MB * 9,026,365 Spectra / 18GB per sector * 6 replicas = 601757 sectors = 18.36 PiB DataCap

So I applied for two 10PiB LDNs for the first time, but you asked me to close one of them and only keep one LDN, but the maximum capacity of a single LDN is only 15PiB, so I can only reduce the number of replicas for some data during the sealing process.

May 17 '23 05:05 sxxfuture-official

@sxxfuture-official Thank you for providing an explanation. It is worth noting that the DataCap application limit is not limited to 15PiB. Applicants have the option to apply for a higher amount by selecting the "More Than 15PiB" option, which will prompt contact from @Kevin-FF-USA.

May 17 '23 13:05 simonkim0515

@simonkim0515 Thank you very much for your reminder ! At the current stage, 15PiB is enough to save 5-6 replicas to avoid complicating the process. The current application quota is acceptable.

May 18 '23 01:05 sxxfuture-official

Could you send an email to [email protected] with your official domain in order to confirm your identity? Email name should includes the issue id #1992.

May 20 '23 15:05 Sunnyiscoming

@Sunnyiscoming The email has been sent, please check~

May 22 '23 02:05 sxxfuture-official

Datacap Request Trigger

Total DataCap requested

15PiB

Expected weekly DataCap usage rate

500TiB

Client address

f1j4jxcfyqkzxivzwktseo6dtbhd7mczlkg5uhrpq

May 22 '23 14:05 Sunnyiscoming

DataCap Allocation requested

Multisig Notary address

f02049625

Client address

f1j4jxcfyqkzxivzwktseo6dtbhd7mczlkg5uhrpq

DataCap allocation requested

250TiB

Id

c88af20a-820a-4d81-b2f8-fcde9f6f8111

May 22 '23 14:05 large-datacap-requests[bot]

Hello @sxxfuture-official How much data have you prepared? How many copies do you plan to send?

May 24 '23 10:05 nj-steve

@nj-steve

Spectra. - there are 9,026,365 flux- and wavelength-calibrated, sky-subtracted spectra.
Spectroscopic Parameters Catalogs. - six spectroscopic parameters catalogs.

The main data occupation is spectral image data, in order to accurately retrieve the CID of each image, we will not compress the data before sealing. The uncompressed data size of each spectral data is ≈ 205MByte.

Now, we planned to save 5-6 replicas to different SPs. 205MB * 9,026,365 Spectra / 18GB per sector * 6 replicas = 601757 sectors = 18.36 PiB DataCap

May 24 '23 10:05 sxxfuture-official

Request Proposed

Your Datacap Allocation Request has been proposed by the Notary

Message sent to Filecoin Network

bafy2bzacebc57vbxm5l763r3ixvyj2mwc5pecg4rbw6sqr5katcofqgs27jfq

Address

f1j4jxcfyqkzxivzwktseo6dtbhd7mczlkg5uhrpq

Datacap Allocated

250.00TiB

Signer Address

f1xx6555qijma7igpnjspyvdunc4vfxkawnpqy5ii

Id

c88af20a-820a-4d81-b2f8-fcde9f6f8111

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacebc57vbxm5l763r3ixvyj2mwc5pecg4rbw6sqr5katcofqgs27jfq

May 24 '23 13:05 nj-steve

In support for public dataset, especially something outside of AWS S3

May 25 '23 06:05 kernelogic

Request Approved

Your Datacap Allocation Request has been approved by the Notary

Message sent to Filecoin Network

bafy2bzaceco4ln3setfcidyz2aobcm43cv4ie3kdqtzrg3r5xgjtdkmkz6gd2

Address

f1j4jxcfyqkzxivzwktseo6dtbhd7mczlkg5uhrpq

Datacap Allocated

250.00TiB

Signer Address

f1yjhnsoga2ccnepb7t3p3ov5fzom3syhsuinxexa

Id

c88af20a-820a-4d81-b2f8-fcde9f6f8111

You can check the status of the message here: https://filfox.info/en/message/bafy2bzaceco4ln3setfcidyz2aobcm43cv4ie3kdqtzrg3r5xgjtdkmkz6gd2

May 25 '23 06:05 kernelogic

This application has not seen any responses in the last 10 days. This issue will be marked with Stale label and will be closed in 4 days. Comment if you want to keep this application open.

Jun 05 '23 01:06 github-actions[bot]

This application has not seen any responses in the last 14 days, so for now it is being closed. Please feel free to contact the Fil+ Gov team to re-open the application if it is still being processed. Thank you!

Jun 09 '23 01:06 github-actions[bot]

@simonkim0515 @galen-mcandrew Data is being sealed continuously

Jun 25 '23 01:06 sxxfuture-official

Jun 25 '23 08:06 sxxfuture-official

This user’s identity has been verified through filplus.storage

Jun 27 '23 19:06 data-programs

checker:manualTrigger

Jun 28 '23 11:06 cryptowhizzard

DataCap and CID Checker Report Summary[^1]

Retrieval Statistics

⚠️ All retrieval success ratios are below 1%.

Overall Graphsync retrieval success rate: 0.00%
Overall HTTP retrieval success rate: 0.00%
Overall Bitswap retrieval success rate: 0.00%

Storage Provider Distribution

✔️ Storage provider distribution looks healthy.

Deal Data Replication

✔️ Data replication looks healthy.

Deal Data Shared with other Clients[^3]

✔️ No CID sharing has been observed.

[^1]: To manually trigger this report, add a comment with text checker:manualTrigger

[^2]: Deals from those addresses are combined into this report as they are specified with checker:manualTrigger

[^3]: To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Full report

Click here to view the CID Checker report. Click here to view the Retrieval report.

Jun 28 '23 11:06 filplus-checker-app[bot]

DataCap Allocation requested