idr-metadata idr0015-colin-taraoceans S-BIAD861

idr0015-colin-taraoceans

Feb 22 '23 12:02 will-moore

Reimport still in progress - cancelled once because of long wait on FILESET_UPLOAD_PREP. The new import in progress since 8 March, also FILESET_UPLOAD_PREP (with parallel-upload=10)

Mar 10 '23 11:03 pwalczysko

Since we will be working on that study. We should take the opportunity to also fix the location metadata

Mar 10 '23 11:03 jburel

Imported without chunks and exchanged the symlink in ManagedRepo similarly to the idr0013 case. The new Plate on pilot-idrtesting is http://localhost:1080/webclient/?show=plate-254 and the name is idr0015-nochunks. All looks good, the thumbs and full viewer work fine.

Apr 05 '23 09:04 pwalczysko

Estimate data volume...

uint8, 4 channels, Z: 20, 2048 x 2048, 22 x 18 wells, 84 plates. 11 TB

Jun 20 '23 09:06 will-moore

Starting to free-up some space...

(base) [wmoore@pilot-zarr1-dev data]$ df -h /data
Filesystem      Size  Used Avail Use% Mounted on
/dev/vdb        4.9T  4.4T  587G  89% /data

$ cd /data
$ sudo rm -rf idr0011/

$ df -h ./
Filesystem      Size  Used Avail Use% Mounted on
/dev/vdb        4.9T  3.6T  1.3T  74% /data

Jul 02 '23 04:07 will-moore

Convert 1 screen...

screen -S idr0015_ngff
/home/wmoore/bioformats2raw-0.6.0-24/bin/bioformats2raw /uod/idr/metadata/idr0015-colin-taraoceans/screenA/patterns/TARA_HCS1_H5_G100001472_G100001473--2013_09_28_19_45_25_chamber--U00--V01.screen TARA_HCS1_H5_G100001472_G100001473--2013_09_28_19_45_25_chamber--U00--V01.ome.zarr

EDIT: permission denied - chown -R wmoore idr0015 then re-ran at 13:16...

Jul 02 '23 05:07 will-moore

Make bucket...

$ aws --endpoint-url https://uk1s3.embassy.ebi.ac.uk s3 mb s3://idr0015
make_bucket: idr0015
$ aws --endpoint-url https://uk1s3.embassy.ebi.ac.uk s3api put-bucket-policy --bucket idr0015 --policy file://policy.json
$ aws --endpoint-url https://uk1s3.embassy.ebi.ac.uk s3api put-bucket-cors --bucket idr0015 --cors-configuration file://cors.json

Jul 02 '23 05:07 will-moore

Upload a previously-created plate from pilot-zarr1-dev

cd /data/idr0015
$ /home/wmoore/mc cp -r TARA_HCS1_H5_G100002411_G100002481--2013_08_28_14_46_59_chamber--U01--V01.ome.zarr uk1s3/idr0015/zarr
.../V/9/0/3/0/4/9/0/0: 53.04 GiB / 53.04 GiB ━━━━━━━━━━━━━━━ 25.02 MiB/s 36m10s

Looks good in validator

https://ome.github.io/ome-ngff-validator/?source=https://uk1s3.embassy.ebi.ac.uk/idr0015/zarr/TARA_HCS1_H5_G100002411_G100002481--2013_08_28_14_46_59_chamber--U01--V01.ome.zarr

And vizarr (although the omero rendering settings in the .zarr are different from what's in IDR). E.g. only 3 channels active, although 5 are exported OK:

Jul 02 '23 05:07 will-moore

Uploaded a 2nd plate, recently generated above:

(base) [wmoore@pilot-zarr1-dev idr0015]$ /home/wmoore/mc cp -r TARA_HCS1_H5_G100001472_G100001473--2013_09_28_19_45_25_chamber--U00--V01.ome.zarr uk1s3/idr0015/zarr
...me.zarr/V/9/0/3/0/4/9/0/0: 79.03 GiB / 79.03 GiB ━━━━━━━━━━━━━━━━━━━━ 37.09 MiB/s 36m21

Jul 02 '23 16:07 will-moore

Converting 4 more plates...

for i in TARA_HCS1_H5_G100001472_G100001473--2013_09_28_19_45_25_chamber--U01--V01 TARA_HCS1_H5_G100001988_G100001989--2013_09_23_19_42_50_chamber--U00--V01 TARA_HCS1_H5_G100001988_G100001989--2013_09_23_19_42_50_chamber--U01--V01 TARA_HCS1_H5_G100002411_G100002481--2013_08_28_14_46_59_chamber--U00--V0; do echo $i; ~/bioformats2raw-0.6.0-24/bin/bioformats2raw --memo-directory ../memo /uod/idr/metadata/idr0015-colin-taraoceans/screenA/patterns/$i.screen ${i%.*}.ome.zarr; done

Jul 02 '23 16:07 will-moore

Seeing errors writing memo files...

2023-07-02 20:38:56,486 [main] WARN  loci.formats.Memoizer - failed to save memo file: ../memo/uod/idr/metadata/idr0015-colin-taraoceans/screenA/patterns/.TARA_HCS1_H5_G100001988_G100001989--2013_09_23_19_42_50_chamber--U01--V01.screen.bfmemo
java.io.IOException: No such file or directory

Jul 02 '23 21:07 will-moore

And then... (caused by a typo in my command above: --V0 should be --V00.

Exception in thread "main" picocli.CommandLine$ExecutionException: Error while calling command (com.glencoesoftware.bioformats2raw.Converter@63a65a25): java.io.FileNotFoundException: /uod/idr/metadata/idr0015-colin-taraoceans/screenA/patterns/TARA_HCS1_H5_G100002411_G100002481--2013_08_28_14_46_59_chamber--U00--V0.screen (No such file or directory)

Jul 03 '23 04:07 will-moore

Try to avoid memo issues by not using memo directory, but allow writing to source dir...

$ cd /uod/idr/metadata/idr0015-colin-taraoceans/screenA/patterns/
$ sudo chown wmoore patterns/

$ df -h /data/
Filesystem      Size  Used Avail Use% Mounted on
/dev/vdb        4.9T  3.6T  1.4T  74% /data

Running 12 more (including repeat of last typo fix above):

for i in TARA_HCS1_H5_G100002411_G100002481--2013_08_28_14_46_59_chamber--U00--V01 TARA_HCS1_H5_G100002479_G100002163--2013_08_26_14_59_40_chamber--U00--V01 TARA_HCS1_H5_G100002479_G100002163--2013_08_26_14_59_40_chamber--U01--V01 TARA_HCS1_H5_G100002567_G100002568--2013_09_25_12_55_58_chamber--U00--V01 TARA_HCS1_H5_G100002567_G100002568--2013_09_25_12_55_58_chamber--U01--V01 TARA_HCS1_H5_G100002655_G100002656--2013_09_24_15_21_06_chamber--U00--V01 TARA_HCS1_H5_G100002655_G100002656--2013_09_24_15_21_06_chamber--U01--V01 TARA_HCS1_H5_G100002978_G100002980--2013_08_29_12_36_46_chamber--U00--V01 TARA_HCS1_H5_G100002978_G100002980--2013_08_29_12_36_46_chamber--U01--V01 TARA_HCS1_H5_G100003406_G100004906--2013_08_24_19_23_14_chamber--U00--V01 TARA_HCS1_H5_G100003406_G100004906--2013_08_24_19_23_14_chamber--U01--V01 TARA_HCS1_H5_G100003584_G100003586--2013_09_26_15_23_10_chamber--U00--V01; do echo $i; ~/bioformats2raw-0.6.0-24/bin/bioformats2raw /uod/idr/metadata/idr0015-colin-taraoceans/screenA/patterns/$i.screen ${i%.*}.ome.zarr; done

Jul 03 '23 05:07 will-moore

Started to zip some... First 4 zarrs...

screen -S idr0015_zip
for i in TARA_HCS1_H5_G100001472_G100001473--2013_09_28_19_45_25_chamber--U01--V01 TARA_HCS1_H5_G100001988_G100001989--2013_09_23_19_42_50_chamber--U00--V01 TARA_HCS1_H5_G100001988_G100001989--2013_09_23_19_42_50_chamber--U01--V01 TARA_HCS1_H5_G100002411_G100002481--2013_08_28_14_46_59_chamber--U00--V0; do zip -r "${i%/}.ome.zarr.zip" "$i.ome.zarr"; done

Jul 03 '23 09:07 will-moore

Still seeing memo issues...

2023-07-03 18:41:39,091 [pool-1-thread-4] WARN  loci.formats.Memoizer - skipping memo: directory not writeable - /uod/idr/filesets/idr0015-UNKNOWN-taraoceans/20150918-tara/RAW_DATA/TARA_HCS1_H5_G100003406_G100004906--2013_08_24_19_23_14/slide--S00/chamber--U00--V01/field--X17--Y21

Jul 03 '23 19:07 will-moore

Zip command above created 3 zips but failed with same typo as earlier! (oops again):

zip warning: name not matched: TARA_HCS1_H5_G100002411_G100002481--2013_08_28_14_46_59_chamber--U00--V0.ome.zarr

Conversion above was generating memo errors as before. Last was:

2023-07-03 21:47:48,790 [pool-1-thread-4] WARN  loci.formats.Memoizer - skipping memo: directory not writeable - /uod/idr/filesets/idr0015-UNKNOWN-taraoceans/20150918-tara/RAW_DATA/TARA_HCS1_H5_G100003584_G100003586--2013_09_26_15_23_10/slide--S00/chamber--U00--V01/field--X17--Y21

but ran to completion OK...

Current status...

$ ls -alh /data/idr0015
total 161G
drwxrwxr-x. 19 wmoore dlindner 4.0K Jul  4 05:08 .
drwxrwxr-x. 14 root   idr-data  270 Jul  2 22:53 ..
drwxrwxr-x. 25 wmoore wmoore   4.0K Jul  2 13:17 TARA_HCS1_H5_G100001472_G100001473--2013_09_28_19_45_25_chamber--U00--V01.ome.zarr
drwxrwxr-x. 25 wmoore wmoore   4.0K Jul  2 18:23 TARA_HCS1_H5_G100001472_G100001473--2013_09_28_19_45_25_chamber--U01--V01.ome.zarr
-rw-rw-r--.  1 wmoore wmoore    48G Jul  3 15:20 TARA_HCS1_H5_G100001472_G100001473--2013_09_28_19_45_25_chamber--U01--V01.ome.zarr.zip
drwxrwxr-x. 25 wmoore wmoore   4.0K Jul  2 20:03 TARA_HCS1_H5_G100001988_G100001989--2013_09_23_19_42_50_chamber--U00--V01.ome.zarr
-rw-rw-r--.  1 wmoore wmoore    63G Jul  3 22:49 TARA_HCS1_H5_G100001988_G100001989--2013_09_23_19_42_50_chamber--U00--V01.ome.zarr.zip
drwxrwxr-x. 25 wmoore wmoore   4.0K Jul  2 21:33 TARA_HCS1_H5_G100001988_G100001989--2013_09_23_19_42_50_chamber--U01--V01.ome.zarr
-rw-rw-r--.  1 wmoore wmoore    51G Jul  4 05:08 TARA_HCS1_H5_G100001988_G100001989--2013_09_23_19_42_50_chamber--U01--V01.ome.zarr.zip
drwxrwxr-x. 25 wmoore wmoore   4.0K Jul  3 06:27 TARA_HCS1_H5_G100002411_G100002481--2013_08_28_14_46_59_chamber--U00--V01.ome.zarr
drwxrwxr-x. 25 wmoore dlindner 4.0K Mar  9 11:30 TARA_HCS1_H5_G100002411_G100002481--2013_08_28_14_46_59_chamber--U01--V01.ome.zarr
-rw-rw-r--.  1 wmoore dlindner 115K Mar  9 10:13 TARA_HCS1_H5_G100002411_G100002481--2013_08_28_14_46_59_chamber--U01--V01.screen
drwxrwxr-x. 25 wmoore wmoore   4.0K Jul  3 07:54 TARA_HCS1_H5_G100002479_G100002163--2013_08_26_14_59_40_chamber--U00--V01.ome.zarr
drwxrwxr-x. 25 wmoore wmoore   4.0K Jul  3 09:33 TARA_HCS1_H5_G100002479_G100002163--2013_08_26_14_59_40_chamber--U01--V01.ome.zarr
drwxrwxr-x. 25 wmoore wmoore   4.0K Jul  3 10:44 TARA_HCS1_H5_G100002567_G100002568--2013_09_25_12_55_58_chamber--U00--V01.ome.zarr
drwxrwxr-x. 25 wmoore wmoore   4.0K Jul  3 11:55 TARA_HCS1_H5_G100002567_G100002568--2013_09_25_12_55_58_chamber--U01--V01.ome.zarr
drwxrwxr-x. 25 wmoore wmoore   4.0K Jul  3 13:03 TARA_HCS1_H5_G100002655_G100002656--2013_09_24_15_21_06_chamber--U00--V01.ome.zarr
drwxrwxr-x. 25 wmoore wmoore   4.0K Jul  3 14:14 TARA_HCS1_H5_G100002655_G100002656--2013_09_24_15_21_06_chamber--U01--V01.ome.zarr
drwxrwxr-x. 25 wmoore wmoore   4.0K Jul  3 15:24 TARA_HCS1_H5_G100002978_G100002980--2013_08_29_12_36_46_chamber--U00--V01.ome.zarr
drwxrwxr-x. 25 wmoore wmoore   4.0K Jul  3 16:29 TARA_HCS1_H5_G100002978_G100002980--2013_08_29_12_36_46_chamber--U01--V01.ome.zarr
drwxrwxr-x. 25 wmoore wmoore   4.0K Jul  3 18:42 TARA_HCS1_H5_G100003406_G100004906--2013_08_24_19_23_14_chamber--U00--V01.ome.zarr
drwxrwxr-x. 16 wmoore wmoore    232 Jul  3 20:34 TARA_HCS1_H5_G100003406_G100004906--2013_08_24_19_23_14_chamber--U01--V01.ome.zarr
drwxrwxr-x. 25 wmoore wmoore   4.0K Jul  3 21:47 TARA_HCS1_H5_G100003584_G100003586--2013_09_26_15_23_10_chamber--U00--V01.ome.zarr

Available space getting low...

$ df -h /data
Filesystem      Size  Used Avail Use% Mounted on
/dev/vdb        4.9T  4.7T  234G  96% /data

Need to delete data... $ sudo rm -rf idr0036

Upload first 3 zips...

$ ./ascp -P33001 -i ../etc/asperaweb_id_dsa.openssh -d /data/idr0015/idr0015 [email protected]:5f/13xxxxxxxx

Jul 05 '23 14:07 will-moore

Deleted 3 zips uploaded above and their .zarr dirs.

Move all remaining .zarr to batch1 for zipping and upload...

mkdir batch1
mv *.zarr batch1
cd batch1
for i in */; do zip -r "${i%/}.zip" "$i"; done

Jul 05 '23 22:07 will-moore

Converting 10 Filesets into "batch2"...

screen -r idr0015_ngff
mkdir batch2
for i in TARA_HCS1_H5_G100003584_G100003586--2013_09_26_15_23_10_chamber--U01--V01 TARA_HCS1_H5_G100003584_G100003586--2014_06_26_15_58_43_chamber--U00--V01 TARA_HCS1_H5_G100003584_G100003586--2014_06_26_15_58_43_chamber--U01--V01 TARA_HCS1_H5_G100003741_G100003739--2013_09_30_14_59_10_chamber--U00--V01 TARA_HCS1_H5_G100003741_G100003739--2013_09_30_14_59_10_chamber--U01--V01 TARA_HCS1_H5_G100004339_G100004341--2013_09_27_15_24_28_chamber--U00--V01 TARA_HCS1_H5_G100004339_G100004341--2013_09_27_15_24_28_chamber--U01--V01 TARA_HCS1_H5_G100004727_G100004940--2013_12_08_21_26_28_chamber--U00--V01 TARA_HCS1_H5_G100004727_G100004940--2013_12_08_21_26_28_chamber--U01--V01 TARA_HCS1_H5_G100004906_G100002201--2013_08_25_19_31_15_chamber--U00--V01; do echo $i; ~/bioformats2raw-0.6.0-24/bin/bioformats2raw /uod/idr/metadata/idr0015-colin-taraoceans/screenA/patterns/$i.screen batch2/${i%.*}.ome.zarr; done

Jul 05 '23 22:07 will-moore

Deleting individual ome.zarr filesets from batch1 once their ome.zarr.zip has been created.

Also upload a random (last) ome.zarr to s3 from batch1 to validate we're still good...

$ /home/wmoore/mc cp -r TARA_HCS1_H5_G100003584_G100003586--2013_09_26_15_23_10_chamber--U00--V01.ome.zarr uk1s3/idr0015/zarr
...me.zarr/A/1/0/0/0/0/5/0/0: 7.45 MiB
...me.zarr/V/9/0/3/0/4/9/0/0: 60.48 GiB / 60.48 GiB ━━━━━━━━━━━━━━━━━━━━ 28.41 MiB/s 36m19s

Jul 06 '23 14:07 will-moore

Current state of batch1: In a day of generating zips, we only have 4 zips created (5th due soon): Zips take 4-5 hours to generate, compared to only about 1.5 hours to convert the ome.zarr.

(base) [wmoore@pilot-zarr1-dev ~]$ ls -alh /data/idr0015/batch1
total 196G
drwxrwxr-x. 14 wmoore wmoore   4.0K Jul  6 20:21 .
drwxrwxr-x.  4 wmoore dlindner  138 Jul  5 22:55 ..
-rw-rw-r--.  1 wmoore wmoore    49G Jul  6 04:39 TARA_HCS1_H5_G100001472_G100001473--2013_09_28_19_45_25_chamber--U00--V01.ome.zarr.zip
-rw-rw-r--.  1 wmoore wmoore    32G Jul  6 08:48 TARA_HCS1_H5_G100002411_G100002481--2013_08_28_14_46_59_chamber--U00--V01.ome.zarr.zip
drwxrwxr-x.  8 wmoore dlindner  107 Jul  6 23:09 TARA_HCS1_H5_G100002411_G100002481--2013_08_28_14_46_59_chamber--U01--V01.ome.zarr
-rw-rw-r--.  1 wmoore wmoore    31G Jul  6 13:12 TARA_HCS1_H5_G100002411_G100002481--2013_08_28_14_46_59_chamber--U01--V01.ome.zarr.zip
drwxrwxr-x. 25 wmoore wmoore   4.0K Jul  3 07:54 TARA_HCS1_H5_G100002479_G100002163--2013_08_26_14_59_40_chamber--U00--V01.ome.zarr
-rw-rw-r--.  1 wmoore wmoore    61G Jul  6 20:14 TARA_HCS1_H5_G100002479_G100002163--2013_08_26_14_59_40_chamber--U00--V01.ome.zarr.zip
drwxrwxr-x. 25 wmoore wmoore   4.0K Jul  3 09:33 TARA_HCS1_H5_G100002479_G100002163--2013_08_26_14_59_40_chamber--U01--V01.ome.zarr
drwxrwxr-x. 25 wmoore wmoore   4.0K Jul  3 10:44 TARA_HCS1_H5_G100002567_G100002568--2013_09_25_12_55_58_chamber--U00--V01.ome.zarr
drwxrwxr-x. 25 wmoore wmoore   4.0K Jul  3 11:55 TARA_HCS1_H5_G100002567_G100002568--2013_09_25_12_55_58_chamber--U01--V01.ome.zarr
drwxrwxr-x. 25 wmoore wmoore   4.0K Jul  3 13:03 TARA_HCS1_H5_G100002655_G100002656--2013_09_24_15_21_06_chamber--U00--V01.ome.zarr
drwxrwxr-x. 25 wmoore wmoore   4.0K Jul  3 14:14 TARA_HCS1_H5_G100002655_G100002656--2013_09_24_15_21_06_chamber--U01--V01.ome.zarr
drwxrwxr-x. 25 wmoore wmoore   4.0K Jul  3 15:24 TARA_HCS1_H5_G100002978_G100002980--2013_08_29_12_36_46_chamber--U00--V01.ome.zarr
drwxrwxr-x. 25 wmoore wmoore   4.0K Jul  3 16:29 TARA_HCS1_H5_G100002978_G100002980--2013_08_29_12_36_46_chamber--U01--V01.ome.zarr
drwxrwxr-x. 25 wmoore wmoore   4.0K Jul  3 18:42 TARA_HCS1_H5_G100003406_G100004906--2013_08_24_19_23_14_chamber--U00--V01.ome.zarr
drwxrwxr-x. 16 wmoore wmoore    232 Jul  3 20:34 TARA_HCS1_H5_G100003406_G100004906--2013_08_24_19_23_14_chamber--U01--V01.ome.zarr
drwxrwxr-x. 25 wmoore wmoore   4.0K Jul  3 21:47 TARA_HCS1_H5_G100003584_G100003586--2013_09_26_15_23_10_chamber--U00--V01.ome.zarr
-rw-------.  1 wmoore wmoore    25G Jul  6 23:10 zitPNp7I

Current state of batch2 (started generating zips too):

(base) [wmoore@pilot-zarr1-dev ~]$ ls -alh /data/idr0015/batch2
total 88G
drwxrwxr-x. 12 wmoore wmoore   4.0K Jul  6 22:37 .
drwxrwxr-x.  4 wmoore dlindner  138 Jul  5 22:55 ..
drwxrwxr-x. 25 wmoore wmoore   4.0K Jul  6 00:29 TARA_HCS1_H5_G100003584_G100003586--2013_09_26_15_23_10_chamber--U01--V01.ome.zarr
-rw-rw-r--.  1 wmoore wmoore    38G Jul  6 17:24 TARA_HCS1_H5_G100003584_G100003586--2013_09_26_15_23_10_chamber--U01--V01.ome.zarr.zip
drwxrwxr-x. 25 wmoore wmoore   4.0K Jul  6 01:41 TARA_HCS1_H5_G100003584_G100003586--2014_06_26_15_58_43_chamber--U00--V01.ome.zarr
-rw-rw-r--.  1 wmoore wmoore    45G Jul  6 22:29 TARA_HCS1_H5_G100003584_G100003586--2014_06_26_15_58_43_chamber--U00--V01.ome.zarr.zip
drwxrwxr-x. 25 wmoore wmoore   4.0K Jul  6 02:53 TARA_HCS1_H5_G100003584_G100003586--2014_06_26_15_58_43_chamber--U01--V01.ome.zarr
drwxrwxr-x. 25 wmoore wmoore   4.0K Jul  6 04:06 TARA_HCS1_H5_G100003741_G100003739--2013_09_30_14_59_10_chamber--U00--V01.ome.zarr
drwxrwxr-x. 25 wmoore wmoore   4.0K Jul  6 05:11 TARA_HCS1_H5_G100003741_G100003739--2013_09_30_14_59_10_chamber--U01--V01.ome.zarr
drwxrwxr-x. 25 wmoore wmoore   4.0K Jul  6 06:27 TARA_HCS1_H5_G100004339_G100004341--2013_09_27_15_24_28_chamber--U00--V01.ome.zarr
drwxrwxr-x. 25 wmoore wmoore   4.0K Jul  6 07:45 TARA_HCS1_H5_G100004339_G100004341--2013_09_27_15_24_28_chamber--U01--V01.ome.zarr
drwxrwxr-x. 25 wmoore wmoore   4.0K Jul  6 08:59 TARA_HCS1_H5_G100004727_G100004940--2013_12_08_21_26_28_chamber--U00--V01.ome.zarr
drwxrwxr-x. 25 wmoore wmoore   4.0K Jul  6 10:16 TARA_HCS1_H5_G100004727_G100004940--2013_12_08_21_26_28_chamber--U01--V01.ome.zarr
drwxrwxr-x. 25 wmoore wmoore   4.0K Jul  6 11:37 TARA_HCS1_H5_G100004906_G100002201--2013_08_25_19_31_15_chamber--U00--V01.ome.zarr
-rw-------.  1 wmoore wmoore   5.1G Jul  6 23:14 ziBWfTRP

Jul 06 '23 23:07 will-moore

Convert another 10 in batch3...

for i in TARA_HCS1_H5_G100004906_G100002201--2013_08_25_19_31_15_chamber--U01--V01 TARA_HCS1_H5_G100006116_G100006253--2013_10_30_19_38_12_chamber--U00--V01 TARA_HCS1_H5_G100006116_G100006253--2013_10_30_19_38_12_chamber--U01--V01 TARA_HCS1_H5_G100006175_G100006179--2013_11_01_23_35_08_chamber--U00--V01 TARA_HCS1_H5_G100006175_G100006179--2013_11_01_23_35_08_chamber--U01--V01 TARA_HCS1_H5_G100007472_G100006130--2013_10_31_18_54_20_chamber--U00--V01 TARA_HCS1_H5_G100007472_G100006130--2013_10_31_18_54_20_chamber--U01--V01 TARA_HCS1_H5_G100007665_G100007576--2013_10_28_21_05_26_chamber--U00--V01 TARA_HCS1_H5_G100007665_G100007576--2013_10_28_21_05_26_chamber--U01--V01 TARA_HCS1_H5_G100007887_G100007889--2013_10_29_19_56_21_chamber--U00--V01; do echo $i; ~/bioformats2raw-0.6.0-24/bin/bioformats2raw /uod/idr/metadata/idr0015-colin-taraoceans/screenA/patterns/$i.screen batch3/${i%.*}.ome.zarr; done

Jul 07 '23 08:07 will-moore

Upload completed zips from batch1...

(base) [wmoore@pilot-zarr1-dev batch1]$ ls /data/idr0015/batch1/idr0015/
TARA_HCS1_H5_G100001472_G100001473--2013_09_28_19_45_25_chamber--U00--V01.ome.zarr.zip
TARA_HCS1_H5_G100002411_G100002481--2013_08_28_14_46_59_chamber--U00--V01.ome.zarr.zip
TARA_HCS1_H5_G100002411_G100002481--2013_08_28_14_46_59_chamber--U01--V01.ome.zarr.zip
TARA_HCS1_H5_G100002479_G100002163--2013_08_26_14_59_40_chamber--U00--V01.ome.zarr.zip
TARA_HCS1_H5_G100002479_G100002163--2013_08_26_14_59_40_chamber--U01--V01.ome.zarr.zip
TARA_HCS1_H5_G100002567_G100002568--2013_09_25_12_55_58_chamber--U00--V01.ome.zarr.zip
$ ./ascp -P33001 -i ../etc/asperaweb_id_dsa.openssh -d /data/idr0015/batch1/idr0015 [email protected]:5f/136xxxxxxxxxx
TARA_HCS1_H5_G100001472_G100001473--2013_09_2             100%   48GB  328Mb/s    23:53    
TARA_HCS1_H5_G100002411_G100002481--2013_08                  100%   31GB  364Mb/s    38:41    
TARA_HCS1_H5_G100002411_G100002481--2013_08_2.             100%   30GB  290Mb/s    53:57    
TARA_HCS1_H5_G100002479_G100002163--2013_08_2             100%   60GB  118Mb/s  1:32:42    
TARA_HCS1_H5_G100002479_G100002163--2013_08_2             100%   60GB  304Mb/s  2:03:27    
TARA_HCS1_H5_G100002567_G100002568--2013_09_2             100%   33GB 91.7Mb/s  2:20:28

Jul 07 '23 08:07 will-moore

Zipping ran out of space...

  adding: TARA_HCS1_H5_G100004906_G100002201--2013_08_25_19_31_15_chamber--U00--V01.ome.zarr/A/1/0/0/0/3/4/1/ (stored 0%)
  adding: TARA_HCS1_H5_G100004906_G100002201--2013_08_25_19_31_15_chamber--U00--V01.ome.zarr/A/1/0/0/0/3/4/1/0
zip I/O error: No space left on device
zip error: Output file write failure (write error on zip file)

Jul 07 '23 23:07 will-moore

For batch3 try zipping with -0 (no compression) and -m (move files into zip - delete originals)...

$ screen -r idr0015_zip
$ cd /data/idr0015/batch3
$ for i in */; do zip -0mr "${i%/}.zip" "$i"; done

Jul 07 '23 23:07 will-moore

Upload 5 more from batch2, then delete

$ ls -alh /data/idr0015/batch2/idr0015
total 196G
drwxrwxr-x. 2 wmoore wmoore 4.0K Jul  7 14:58 .
drwxrwxr-x. 8 wmoore wmoore 4.0K Jul  8 00:04 ..
-rw-rw-r--. 1 wmoore wmoore  38G Jul  6 17:24 TARA_HCS1_H5_G100003584_G100003586--2013_09_26_15_23_10_chamber--U01--V01.ome.zarr.zip
-rw-rw-r--. 1 wmoore wmoore  45G Jul  6 22:29 TARA_HCS1_H5_G100003584_G100003586--2014_06_26_15_58_43_chamber--U00--V01.ome.zarr.zip
-rw-rw-r--. 1 wmoore wmoore  43G Jul  7 03:30 TARA_HCS1_H5_G100003584_G100003586--2014_06_26_15_58_43_chamber--U01--V01.ome.zarr.zip
-rw-rw-r--. 1 wmoore wmoore  37G Jul  7 08:14 TARA_HCS1_H5_G100003741_G100003739--2013_09_30_14_59_10_chamber--U00--V01.ome.zarr.zip
-rw-rw-r--. 1 wmoore wmoore  34G Jul  7 12:17 TARA_HCS1_H5_G100003741_G100003739--2013_09_30_14_59_10_chamber--U01--V01.ome.zarr.zip

$ ./ascp -P33001 -i ../etc/asperaweb_id_dsa.openssh -d /data/idr0015/batch2/idr0015 [email protected]:5f/136e8d-xxxx
TARA_HCS1_H5_G100003584_G100003586--2013_09_2                                                                                       100%   37GB  313Mb/s    16:46
TARA_HCS1_H5_G100003584_G100003586--2014_06_2                                                                                       100%   44GB  177Mb/s    38:08
TARA_HCS1_H5_G100003584_G100003586--2014_06_2                                                                                       100%   42GB  271Mb/s    58:21    
TARA_HCS1_H5_G100003741_G100003739--2013_09_3                                                                                       100%   36GB  291Mb/s  1:16:30    
TARA_HCS1_H5_G100003741_G100003739--2013_09_3                                                                                       100%   33GB 64.6Mb/s  1:32:49

$ rm -rf /data/idr0015/batch2/idr0015/*

Jul 08 '23 00:07 will-moore

Looks like zipping of batch3 (with -m and -0) stopped prematurely for some reason:

screen -r idr0015_zip2
...
  adding: TARA_HCS1_H5_G100007887_G100007889--2013_10_29_19_56_21_chamber--U00--V01.ome.zarr/A/1/0/0/0/4/19/0/ (stored 0%)
  adding: TARA_HCS1_H5_G100007887_G100007889--2013_10_29_19_56_21_chamber--U00--V01.ome.zarr/A/1/0/0/0/4/19/0/0 (stored 0%)
  adding: TARA_HCS1_H5_G100007887_G100007889--2013_10_29_19_56_21_chamber--U00--V01.ome.zarr/A/1/0/0/0/4/19/0/1 (stored 0%)

Last 2 zips are smaller than expected. Probably need to regenerate

(base) [wmoore@pilot-zarr1-dev batch3]$ ls -alh
total 614G
drwxrwxr-x. 2 wmoore wmoore   4.0K Jul  9 04:32 .
drwxrwxr-x. 5 wmoore dlindner  156 Jul  7 08:03 ..
-rw-rw-r--. 1 wmoore wmoore    96G Jul  8 05:04 TARA_HCS1_H5_G100004906_G100002201--2013_08_25_19_31_15_chamber--U01--V01.ome.zarr.zip
-rw-rw-r--. 1 wmoore wmoore    80G Jul  8 09:04 TARA_HCS1_H5_G100006116_G100006253--2013_10_30_19_38_12_chamber--U00--V01.ome.zarr.zip
-rw-rw-r--. 1 wmoore wmoore    79G Jul  8 12:29 TARA_HCS1_H5_G100006116_G100006253--2013_10_30_19_38_12_chamber--U01--V01.ome.zarr.zip
-rw-rw-r--. 1 wmoore wmoore    72G Jul  8 16:00 TARA_HCS1_H5_G100006175_G100006179--2013_11_01_23_35_08_chamber--U00--V01.ome.zarr.zip
-rw-rw-r--. 1 wmoore wmoore    76G Jul  8 19:06 TARA_HCS1_H5_G100006175_G100006179--2013_11_01_23_35_08_chamber--U01--V01.ome.zarr.zip
-rw-rw-r--. 1 wmoore wmoore    75G Jul  8 22:08 TARA_HCS1_H5_G100007472_G100006130--2013_10_31_18_54_20_chamber--U00--V01.ome.zarr.zip
-rw-rw-r--. 1 wmoore wmoore    74G Jul  9 01:30 TARA_HCS1_H5_G100007472_G100006130--2013_10_31_18_54_20_chamber--U01--V01.ome.zarr.zip
-rw-rw-r--. 1 wmoore wmoore    59G Jul  9 04:09 TARA_HCS1_H5_G100007665_G100007576--2013_10_28_21_05_26_chamber--U00--V01.ome.zarr.zip
-rw-rw-r--. 1 wmoore wmoore   6.6G Jul  9 04:31 TARA_HCS1_H5_G100007665_G100007576--2013_10_28_21_05_26_chamber--U01--V01.ome.zarr.zip
-rw-rw-r--. 1 wmoore wmoore    55M Jul  9 04:32 TARA_HCS1_H5_G100007887_G100007889--2013_10_29_19_56_21_chamber--U00--V01.ome.zarr.zip

but looks like space didn't run out:

$ df -h /data
Filesystem      Size  Used Avail Use% Mounted on
/dev/vdb        4.9T  4.5T  482G  91% /data

Jul 10 '23 03:07 will-moore

Zip remaining zarrs in batch1 (and batch2)

screen -S idr0015_zip(2)
cd batch1(2)
for i in */; do zip -0mr "${i%/}.zip" "$i"; done

Jul 10 '23 03:07 will-moore

Moved all zips for batch3 and upload (except for 2 zips that appear too small - corrupted?).

(base) [wmoore@pilot-zarr1-dev batch3]$ ls -alh idr0015/
total 608G
-rw-rw-r--. 1 wmoore wmoore  96G Jul  8 05:04 TARA_HCS1_H5_G100004906_G100002201--2013_08_25_19_31_15_chamber--U01--V01.ome.zarr.zip
-rw-rw-r--. 1 wmoore wmoore  80G Jul  8 09:04 TARA_HCS1_H5_G100006116_G100006253--2013_10_30_19_38_12_chamber--U00--V01.ome.zarr.zip
-rw-rw-r--. 1 wmoore wmoore  79G Jul  8 12:29 TARA_HCS1_H5_G100006116_G100006253--2013_10_30_19_38_12_chamber--U01--V01.ome.zarr.zip
-rw-rw-r--. 1 wmoore wmoore  72G Jul  8 16:00 TARA_HCS1_H5_G100006175_G100006179--2013_11_01_23_35_08_chamber--U00--V01.ome.zarr.zip
-rw-rw-r--. 1 wmoore wmoore  76G Jul  8 19:06 TARA_HCS1_H5_G100006175_G100006179--2013_11_01_23_35_08_chamber--U01--V01.ome.zarr.zip
-rw-rw-r--. 1 wmoore wmoore  75G Jul  8 22:08 TARA_HCS1_H5_G100007472_G100006130--2013_10_31_18_54_20_chamber--U00--V01.ome.zarr.zip
-rw-rw-r--. 1 wmoore wmoore  74G Jul  9 01:30 TARA_HCS1_H5_G100007472_G100006130--2013_10_31_18_54_20_chamber--U01--V01.ome.zarr.zip
-rw-rw-r--. 1 wmoore wmoore  59G Jul  9 04:09 TARA_HCS1_H5_G100007665_G100007576--2013_10_28_21_05_26_chamber--U00--V01.ome.zarr.zip
(base) [wmoore@pilot-zarr1-dev batch3]$ ls -alh
total 6.7G
drwxrwxr-x. 2 wmoore wmoore   4.0K Jul 10 07:30 idr0015
-rw-rw-r--. 1 wmoore wmoore   6.6G Jul  9 04:31 TARA_HCS1_H5_G100007665_G100007576--2013_10_28_21_05_26_chamber--U01--V01.ome.zarr.zip
-rw-rw-r--. 1 wmoore wmoore    55M Jul  9 04:32 TARA_HCS1_H5_G100007887_G100007889--2013_10_29_19_56_21_chamber--U00--V01.ome.zarr.zip

Upload...

./ascp -P33001 -i ../etc/asperaweb_id_dsa.openssh -d /data/idr0015/batch3/idr0015 [email protected]:5f/13xxxxxxx

Jul 10 '23 07:07 will-moore

As suggested by Jason & Josh, looking at 7zip... https://www.7-zip.org/download.html

$ wget https://www.7-zip.org/a/7z2201-linux-x64.tar.xz

Jul 10 '23 10:07 will-moore

Seems that we actually need p7zip, downloading from https://sourceforge.net/projects/p7zip/files/p7zip/16.02/

wget https://sourceforge.net/projects/p7zip/files/p7zip/16.02/p7zip_16.02_src_all.tar.bz2/download

That doesn't give me anything useful. Can't actually find any install instructions!

Jul 10 '23 12:07 will-moore