idr-metadata icon indicating copy to clipboard operation
idr-metadata copied to clipboard
verified publisher icon IDR

idr0033-rohban-pathways S-BIAD848

Open will-moore opened this issue 2 years ago • 26 comments

Conversion

  • [x] 41744
  • [x] 41744_illum_corrected
  • [x] 41749
  • [x] 41749_illum_corrected
  • [x] 41754
  • [x] 41754_illum_corrected
  • [x] 41755
  • [x] 41755_illum_corrected
  • [x] 41756
  • [x] 41756_illum_corrected
  • [x] 41757
  • [x] 41757_illum_corrected

will-moore avatar Feb 22 '23 12:02 will-moore

Reimport still in progress on pilot-idrtesting (since 28th Feb...)

dominikl avatar Mar 06 '23 08:03 dominikl

Incidentally, I came across this conversion as part of a separate body of work. With the help of @melissalinkert, a workaround was found to inject additional plate metadata, let this data be recognized via the MetaXpress reader and use bioformats2raw 0.6.1 directly.

On the pilot with SSD drive

(conversion) [sbesson@pilot-zarr2-dev ~]$ time ~/bioformats2raw-0.6.1/bin/bioformats2raw -p /data/idr0033/41744/taoe005-u2os-72h-cp-a-au00044859.HTD /data/idr0033/41744.zarr
...
[3455/3] 100% │███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████│ 5/5 (0:00:00 / 0:00:00)

real    37m22.858s
user    132m59.999s
sys     9m5.163s
(conversion) [sbesson@pilot-zarr2-dev ~]$ du -csh /data/idr0033/41744.zarr/
51G     /data/idr0033/41744.zarr/
51G     total

If that's useful for the ongoing work, happy to upload the NGFF dataset for the first plate into a temporary idr0033 bucket and allow you to test the fileset swap and/or start the conversion of the rest of the study.

sbesson avatar Apr 27 '23 07:04 sbesson 

$ aws --endpoint-url https://uk1s3.embassy.ebi.ac.uk s3 mb s3://idr0033
make_bucket: idr0033
$ aws --endpoint-url https://uk1s3.embassy.ebi.ac.uk s3api put-bucket-policy --bucket idr0033 --policy file://policy.json
$ aws --endpoint-url https://uk1s3.embassy.ebi.ac.uk s3api put-bucket-cors --bucket idr0033  --cors-configuration file://cors.json

will-moore avatar Apr 27 '23 09:04 will-moore 

$ /home/wmoore/mc ls uk1s3/idr0033/zarr
[2023-04-27 10:06:54 UTC]     0B 41744.zarr/

https://ome.github.io/ome-ngff-validator/?source=https://uk1s3.embassy.ebi.ac.uk/idr0033/zarr/41744.zarr - currently partially uploaded

will-moore avatar Apr 27 '23 10:04 will-moore

https://hms-dbmi.github.io/vizarr/?source=https://uk1s3.embassy.ebi.ac.uk/idr0033/zarr/41744.zarr

Screenshot 2023-04-27 at 11 34 27

will-moore avatar Apr 27 '23 10:04 will-moore

For reference, this is the script that was used for symlinking the source TIFFs and renaming them according to the pattern expected by the reader

import os
import re

pattern=r'(.*)(_[a-p]\d\d)(.*)'

sourcedir='/uod/idr/filesets/idr0033-rohban-pathways/20170214-original/'
targetdir='/data/idr0033/sources/'

plates = ['41744', '41749', '41754', '41755', '41756', '41757']
types = ['images', 'images_illum_corrected']
def replace(s):
    return s.groups()[0]+ s.groups()[1].upper()+s.groups()[2]

for type in types:
    for plate in plates:
        source_plate = os.path.join(sourcedir, type, plate)
        target_plate = os.path.join(targetdir, type, plate)
        tiffs = os.listdir(source_plate)
        
        for tiff in tiffs:
            new_tiff = re.sub(pattern, replace, tiff)
            os.symlink(os.path.join(source_plate, tiff), os.path.join(target_plate, new_tiff))

and the content of the metadata file which must be saved as <plate_prefix>.HTD alongside the symlinks

"HTSInfoFile", Version 1.0
"Description", "BBBC037 41749"
"PlateType", 1
"TimePoints", 1
"ZSeries", FALSE
"ZSteps", 1
"ZProjection", FALSE
"XWells", 24
"YWells", 16
"WellsSelection1", TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE
"WellsSelection2", TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE
"WellsSelection3", TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE
"WellsSelection4", TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE
"WellsSelection5", TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE
"WellsSelection6", TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE
"WellsSelection7", TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE
"WellsSelection8", TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE
"WellsSelection9", TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE
"WellsSelection10", TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE
"WellsSelection11", TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE
"WellsSelection12", TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE
"WellsSelection13", TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE
"WellsSelection14", TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE
"WellsSelection15", TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE
"WellsSelection16", TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE
"Sites", TRUE
"XSites", 3
"YSites", 3
"SiteSelection1", TRUE, TRUE, TRUE
"SiteSelection2", TRUE, TRUE, TRUE
"SiteSelection3", TRUE, TRUE, TRUE
"Waves", TRUE
"NWavelengths", 5
"WaveName1", "Hoechst"
"WaveName2", "ERSyto"
"WaveName3", "ERSytoBleed"
"WaveName4", "PhGolgi"
"WaveName5", "Mito"
"WaveCollect1", 1
"WaveCollect2", 1
"WaveCollect3", 1
"WaveCollect4", 1
"WaveCollect5", 1
"UniquePlateIdentifier", "abc123"
"EndFile"

With the above, Bio-Formats correctly detects the plate as MetaXpress and the bioformats2raw conversion utility can be used natively to generate NGFF

sbesson avatar Apr 27 '23 10:04 sbesson

Started import of metadata-only plate on idr0125-pilot - uploading nearly 4 hours... - Import took > 8 hours. Following https://github.com/IDR/idr-metadata/issues/656

#  tail -f /tmp/41744.zarr.err 

2023-04-27 11:01:49,972 20637      [      main] INFO       ome.formats.OMEROMetadataStoreClient - Server: 5.6.7
2023-04-27 11:01:49,972 20637      [      main] INFO       ome.formats.OMEROMetadataStoreClient - Client: 5.6.2
2023-04-27 11:01:49,972 20637      [      main] INFO       ome.formats.OMEROMetadataStoreClient - Java Version: 11.0.13
2023-04-27 11:01:49,972 20637      [      main] INFO       ome.formats.OMEROMetadataStoreClient - OS Name: Linux
2023-04-27 11:01:49,972 20637      [      main] INFO       ome.formats.OMEROMetadataStoreClient - OS Arch: amd64
2023-04-27 11:01:49,972 20637      [      main] INFO       ome.formats.OMEROMetadataStoreClient - OS Version: 3.10.0-1160.45.1.el7.x86_64
2023-04-27 11:01:50,112 20777      [2-thread-1] INFO   ormats.importer.cli.LoggingImportMonitor - FILESET_UPLOAD_PREPARATION

...

2023-04-27 19:35:18,024 30828689   [l.Client-4] INFO   ormats.importer.cli.LoggingImportMonitor - IMPORT_DONE Imported file: /ngff/idr0033/41744.zarr/OME/METADATA.ome.xml
Other imported objects:
Fileset:5286991

==> Summary
21525 files uploaded, 1 fileset, 1 plate created, 3456 images imported, 0 errors in 8:33:28.125

will-moore avatar Apr 27 '23 13:04 will-moore 

$ python scripts/managed_repo_symlinks.py Fileset:5286991 /idr0033/zarr --report
Render Image 14778952

Fileset: 5286991 /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-25/2023-04/27/11-01-50.951/
fs_contents ['41744.zarr']
Link from /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-25/2023-04/27/11-01-50.951/41744.zarr to /idr0033/zarr/41744.zarr

Plate Images are viewable on pilot-idr0125: http://localhost:1080/webclient/?show=plate-10388

will-moore avatar Apr 28 '23 10:04 will-moore

@sbesson - you're currently converting the rest of the plates and uploading to s3 bucket?

I'll update the status of this issue... I see these ones done so far: 👍

[wmoore@pilot-zarr2-dev ~]$ /home/wmoore/mc ls uk1s3/idr0033/zarr
[2023-04-28 10:48:18 UTC]     0B 41744.zarr/
[2023-04-28 10:48:18 UTC]     0B 41749.zarr/
[2023-04-28 10:48:18 UTC]     0B 41754.zarr/
[2023-04-28 10:48:18 UTC]     0B 41755.zarr/
[2023-04-28 10:48:18 UTC]     0B 41756.zarr/
[2023-04-28 10:48:18 UTC]     0B 41757.zarr/

will-moore avatar Apr 28 '23 10:04 will-moore

Yes converting and updating the checklist as a new dataset is uploaded to the bucket

sbesson avatar Apr 28 '23 11:04 sbesson

Checked all plates on s3 with ome-ngff-validator 👍

will-moore avatar Jun 23 '23 15:06 will-moore

@sbesson Where are the .zarr for idr0033? I couldn't find them on pilot-zarr1-dev or pilot-zarr2-dev? Thx

will-moore avatar Jun 28 '23 16:06 will-moore

On idr-ftp downloaded the plates from the S3 bucket and zipped them in place with for i in $(ls .); do zip -rm $i.zip $i; done:

(base) [sbesson@idrftp-ftp idr0033]$ pwd
/data/idr0033
(base) [sbesson@idrftp-ftp idr0033]$ ls -alh
total 435G
drwxr-xr-x. 2 sbesson root    4.0K Jul  1 05:51 .
drwxr-xr-x. 6 root    root     143 Jun 29 12:00 ..
-rw-rw-r--. 1 sbesson sbesson  36G Jun 29 20:27 41744_illum_corrected.zarr.zip
-rw-rw-r--. 1 sbesson sbesson  37G Jun 29 23:27 41744.zarr.zip
-rw-rw-r--. 1 sbesson sbesson  36G Jun 30 02:31 41749_illum_corrected.zarr.zip
-rw-rw-r--. 1 sbesson sbesson  37G Jun 30 05:30 41749.zarr.zip
-rw-rw-r--. 1 sbesson sbesson  37G Jun 30 08:34 41754_illum_corrected.zarr.zip
-rw-rw-r--. 1 sbesson sbesson  37G Jun 30 11:33 41754.zarr.zip
-rw-rw-r--. 1 sbesson sbesson  37G Jun 30 14:38 41755_illum_corrected.zarr.zip
-rw-rw-r--. 1 sbesson sbesson  37G Jun 30 17:35 41755.zarr.zip
-rw-rw-r--. 1 sbesson sbesson  36G Jun 30 20:38 41756_illum_corrected.zarr.zip
-rw-rw-r--. 1 sbesson sbesson  37G Jun 30 23:42 41756.zarr.zip
-rw-rw-r--. 1 sbesson sbesson  36G Jul  1 02:48 41757_illum_corrected.zarr.zip
-rw-rw-r--. 1 sbesson sbesson  37G Jul  1 05:50 41757.zarr.zip

Ready for the next phase and the upload

sbesson avatar Jul 03 '23 07:07 sbesson

Uploading...

$ ssh idr-ftp
$ sudo /root/.aspera/cli/bin/ascp -P33001 -i /root/.aspera/cli/etc/asperaweb_id_dsa.openssh -d /data/idr0033/ [email protected]:5f/xxxxxxxx

will-moore avatar Aug 07 '23 08:08 will-moore

Created and uploaded idr0033_files.tsv

will-moore avatar Aug 07 '23 21:08 will-moore

https://www.ebi.ac.uk/biostudies/bioimages/studies/S-BIAD848

will-moore avatar Aug 17 '23 08:08 will-moore

Testing mkngff on idr0125-pilot.

idr0033/41749.ome.zarr,S-BIAD848/19d861dc-22ca-4739-9b6c-ac0fa179d8f5,23109
idr0033/41754.ome.zarr,S-BIAD848/1a0ebc8d-5969-4062-b0c9-b27913ce13a2,23111
idr0033/41757_illum_corrected.ome.zarr,S-BIAD848/1d6d3d59-ddfb-4625-819e-5d0f139de895,23122
idr0033/41744.ome.zarr,S-BIAD848/40b32e31-413f-4f2e-b034-e6fe89e30efb,23108
idr0033/41755.ome.zarr,S-BIAD848/5a2838f8-6489-4d47-a92b-5808127c170b,23112

for r in $(cat idr0033.csv); do
  biapath=$(echo $r | cut -d',' -f2)
  uuid=$(echo $biapath | cut -d'/' -f2)
  fsid=$(echo $r | cut -d',' -f3)
  omero mkngff sql --symlink_repo /data/OMERO/ManagedRepository --secret=$SECRET $fsid "/bia-integrator-data/$biapath/$uuid.zarr" > "$fsid.sql"
done

will-moore avatar Aug 25 '23 11:08 will-moore

After about 50 mins, only the first 2 Filesets are done, 3rd in progress (lots of files to walk in these Plates)...

Took nearly 2 hours in total - about 20 mins per Fileset.

Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Found prefix demo_2/2017-03/06 // 18-49-06.418 for fileset 23109
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2017-03/06/18-49-06.418
Creating dir at /data/OMERO/ManagedRepository/demo_2/2017-03/06/18-49-06.418_converted/bia-integrator-data/S-BIAD848/19d861dc-22ca-4739-9b6c-ac0fa179d8f5
Creating symlink /data/OMERO/ManagedRepository/demo_2/2017-03/06/18-49-06.418_converted/bia-integrator-data/S-BIAD848/19d861dc-22ca-4739-9b6c-ac0fa179d8f5/19d861dc-22ca-4739-9b6c-ac0fa179d8f5.zarr -> /bia-integrator-data/S-BIAD848/19d861dc-22ca-4739-9b6c-ac0fa179d8f5/19d861dc-22ca-4739-9b6c-ac0fa179d8f5.zarr
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Found prefix demo_2/2017-03/06 // 23-31-46.781 for fileset 23111
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2017-03/06/23-31-46.781
Creating dir at /data/OMERO/ManagedRepository/demo_2/2017-03/06/23-31-46.781_converted/bia-integrator-data/S-BIAD848/1a0ebc8d-5969-4062-b0c9-b27913ce13a2
Creating symlink /data/OMERO/ManagedRepository/demo_2/2017-03/06/23-31-46.781_converted/bia-integrator-data/S-BIAD848/1a0ebc8d-5969-4062-b0c9-b27913ce13a2/1a0ebc8d-5969-4062-b0c9-b27913ce13a2.zarr -> /bia-integrator-data/S-BIAD848/1a0ebc8d-5969-4062-b0c9-b27913ce13a2/1a0ebc8d-5969-4062-b0c9-b27913ce13a2.zarr
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Found prefix demo_2/2017-03/08 // 10-31-00.187 for fileset 23122
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2017-03/08/10-31-00.187
Creating dir at /data/OMERO/ManagedRepository/demo_2/2017-03/08/10-31-00.187_converted/bia-integrator-data/S-BIAD848/1d6d3d59-ddfb-4625-819e-5d0f139de895
Creating symlink /data/OMERO/ManagedRepository/demo_2/2017-03/08/10-31-00.187_converted/bia-integrator-data/S-BIAD848/1d6d3d59-ddfb-4625-819e-5d0f139de895/1d6d3d59-ddfb-4625-819e-5d0f139de895.zarr -> /bia-integrator-data/S-BIAD848/1d6d3d59-ddfb-4625-819e-5d0f139de895/1d6d3d59-ddfb-4625-819e-5d0f139de895.zarr
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Found prefix demo_2/2017-03/06 // 14-24-32.180 for fileset 23108
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2017-03/06/14-24-32.180
Creating dir at /data/OMERO/ManagedRepository/demo_2/2017-03/06/14-24-32.180_converted/bia-integrator-data/S-BIAD848/40b32e31-413f-4f2e-b034-e6fe89e30efb
Creating symlink /data/OMERO/ManagedRepository/demo_2/2017-03/06/14-24-32.180_converted/bia-integrator-data/S-BIAD848/40b32e31-413f-4f2e-b034-e6fe89e30efb/40b32e31-413f-4f2e-b034-e6fe89e30efb.zarr -> /bia-integrator-data/S-BIAD848/40b32e31-413f-4f2e-b034-e6fe89e30efb/40b32e31-413f-4f2e-b034-e6fe89e30efb.zarr
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Found prefix demo_2/2017-03/07 // 04-03-26.724 for fileset 23112
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2017-03/07/04-03-26.724
Creating dir at /data/OMERO/ManagedRepository/demo_2/2017-03/07/04-03-26.724_converted/bia-integrator-data/S-BIAD848/5a2838f8-6489-4d47-a92b-5808127c170b
Creating symlink /data/OMERO/ManagedRepository/demo_2/2017-03/07/04-03-26.724_converted/bia-integrator-data/S-BIAD848/5a2838f8-6489-4d47-a92b-5808127c170b/5a2838f8-6489-4d47-a92b-5808127c170b.zarr -> /bia-integrator-data/S-BIAD848/5a2838f8-6489-4d47-a92b-5808127c170b/5a2838f8-6489-4d47-a92b-5808127c170b.zarr

will-moore avatar Aug 25 '23 11:08 will-moore 

for r in $(cat idr0033.csv); do
  fsid=$(echo $r | cut -d',' -f3)
  psql -U omero -d idr -h 192.168.10.231 -f "$fsid.sql"
done

BEGIN
psql:23109.sql:39216: ERROR:  cannot set original repo property without secret key
CONTEXT:  PL/pgSQL function _protect_originalfile_repo_insert() line 28 at RAISE
SQL statement "insert into originalfile
          (id, permissions, creation_id, group_id, owner_id, update_id, mimetype, repo, path, name)
          values (nextval('seq_originalfile'), old_perms, new_event, old_group, old_owner, new_event,
            info[i][3], repo, info[i][1], uuid || info[i][2])
          returning id"
PL/pgSQL function mkngff_fileset(bigint,character varying,character varying,character varying,text[]) line 42 at SQL statement
ROLLBACK

But this certainly has the correct secret:

$ cat 23109.sql | grep 4b358149-af39-49f0-882d-10884fab7133
      '4b358149-af39-49f0-882d-10884fab7133',

psql -U omero -d idr -h 192.168.10.102 -c "select uuid from (select * from session where node = 0 and owner = 0 and defaulteventtype = 'Sessions' order by id desc limit 1) x order by x.id asc limit 1;"
                 uuid                 
--------------------------------------
 4b358149-af39-49f0-882d-10884fab7133
(1 row)

will-moore avatar Aug 25 '23 16:08 will-moore

As @sbesson noted, I have the wrong host in that psql query for secret!

will-moore avatar Aug 28 '23 15:08 will-moore

https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/pages/S-BIAD848.html now has 8 out of 12 plates viewable...

idr0033/41749.ome.zarr,S-BIAD848/19d861dc-22ca-4739-9b6c-ac0fa179d8f5,23109
idr0033/41754.ome.zarr,S-BIAD848/1a0ebc8d-5969-4062-b0c9-b27913ce13a2,23111
idr0033/41757_illum_corrected.ome.zarr,S-BIAD848/1d6d3d59-ddfb-4625-819e-5d0f139de895,23122
idr0033/41744.ome.zarr,S-BIAD848/40b32e31-413f-4f2e-b034-e6fe89e30efb,23108
idr0033/41755.ome.zarr,S-BIAD848/5a2838f8-6489-4d47-a92b-5808127c170b,23112
idr0033/41756.ome.zarr,S-BIAD848/68d2d088-a48b-4e78-81c1-c59b0acc1f73,23114
idr0033/41749_illum_corrected.ome.zarr,S-BIAD848/8bafd83a-e4f3-4dd5-9ddc-6094012a390f,23118
idr0033/41754_illum_corrected.ome.zarr,S-BIAD848/9c1a5073-3ebd-4e04-894c-900022e554e1,23119

forgot to update idr0033.csv so only first 5 of those got used...

$ for r in $(cat $IDRID.csv); do
  fsid=$(echo $r | cut -d',' -f3)
  psql -U omero -d idr -h $DBHOST -f "$fsid.sql"
done

BEGIN
 mkngff_fileset 
----------------
        5287482
(1 row)
COMMIT
BEGIN
 mkngff_fileset 
----------------
        5287483
(1 row)
COMMIT
BEGIN
 mkngff_fileset 
----------------
        5287484
(1 row)
COMMIT
BEGIN
 mkngff_fileset 
----------------
        5287485
(1 row)
COMMIT
BEGIN
 mkngff_fileset 
----------------
        5287486
(1 row)
COMMIT

UPDATE: after "19 hours" images are still not visible (checking plate 41749), due to memo file still being generated?

will-moore avatar Aug 30 '23 13:08 will-moore

Still checking http://localhost:1080/webclient/?show=image-3201293 and no image showing yet after > 24 hours

will-moore avatar Aug 31 '23 15:08 will-moore

Testing on idr-testing:omeroreadwrite with todays OMEZarrReader.jar including https://github.com/ome/ZarrReader/pull/64

Started mkngff on 12 Plates (see https://github.com/IDR/idr-utils/pull/56/commits/504b9d9a68687516be2baf7055fc34860e5c65e3) at 13:09...

will-moore avatar Sep 12 '23 12:09 will-moore

All sql generated overnight...

drwxr-xr-x.  2 omero-server omero-server  210 Sep 13 03:33 .
drwxr-xr-x. 12 omero-server root         4.0K Sep 12 12:09 ..
-rw-r--r--.  1 omero-server omero-server 5.0M Sep 12 17:41 23108.sql
-rw-r--r--.  1 omero-server omero-server 5.0M Sep 12 13:34 23109.sql
-rw-r--r--.  1 omero-server omero-server 5.0M Sep 12 14:57 23111.sql
-rw-r--r--.  1 omero-server omero-server 5.0M Sep 12 20:31 23112.sql
-rw-r--r--.  1 omero-server omero-server 5.0M Sep 12 21:57 23114.sql
-rw-r--r--.  1 omero-server omero-server 5.0M Sep 13 03:33 23115.sql
-rw-r--r--.  1 omero-server omero-server 5.0M Sep 13 04:57 23116.sql
-rw-r--r--.  1 omero-server omero-server 5.0M Sep 13 00:42 23118.sql
-rw-r--r--.  1 omero-server omero-server 5.0M Sep 13 02:09 23119.sql
-rw-r--r--.  1 omero-server omero-server 5.0M Sep 12 23:20 23120.sql
-rw-r--r--.  1 omero-server omero-server 5.0M Sep 12 19:04 23121.sql
-rw-r--r--.  1 omero-server omero-server 5.0M Sep 12 16:19 23122.sql

last one completed at 04:57(GMT) (~6am BST) - 18 hours - 1.5 hours each.

will-moore avatar Sep 13 '23 08:09 will-moore

After sql update, viewing Image in first plate to init memo file... http://localhost:1080/webclient/?show=image-3191231

will-moore avatar Sep 13 '23 09:09 will-moore 

[wmoore@test120-omeroreadwrite ~]$ grep -A 2 "saved memo" /opt/omero/server/OMERO.server/var/log/Blitz-0.log | grep -A 2 "14-24-32.180_mkngff"
2023-09-14 09:07:32,288 DEBUG [                   loci.formats.Memoizer] (l.Server-2) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2017-03/06/14-24-32.180_mkngff/40b32e31-413f-4f2e-b034-e6fe89e30efb.zarr/OME/.METADATA.ome.xml.bfmemo (11691040 bytes)
2023-09-14 09:07:32,288 DEBUG [                   loci.formats.Memoizer] (l.Server-2) start[1694595772741] time[86679547] tag[loci.formats.Memoizer.setId]
2023-09-14 09:07:32,288 INFO  [                ome.io.nio.PixelsService] (l.Server-2) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/2017-03/06/14-24-32.180_mkngff/40b32e31-413f-4f2e-b034-e6fe89e30efb.zarr/OME/METADATA.ome.xml Series: 0

Memo file creation (black images) took 86679547 ms which is 24 hours for a Plate.

will-moore avatar Sep 14 '23 09:09 will-moore