getObject() and getExternalInfo() load externalInfo

[will-moore]

See https://github.com/ome/omero-web/pull/613.

This tweaks the base BlitzObjectWrapper._getQueryString() to include loading externalInfo.

It also adds BlitzObjectWrapper.getExternalInfo() to return the externalInfo and load it if necessary, e.g. for cases where self._obj is NOT the object loaded by getObject().

E.g. for image:

def __loadedHotSwap__(self):
        ctx = self._conn.SERVICE_OPTS.copy()
        ctx.setOmeroGroup(self.getDetails().group.id.val)
        self._obj = self._conn.getContainerService().getImages(
            self.OMERO_CLASS, (self._oid,), None, ctx)[0]

To test:

Set some values for ExternalInfo: Using this branch:

$ omero obj ext-info-set Image:12 myLsid myEntityType

Since the JSON API uses _getQueryString() to build queries, and omero-marshal already handles externalInfo, you can check the details at E.g. /api/v0/m/images/ID/: (NB: deployed on idr-testing, e.g. /api/v0/m/images/15159664/).

"externalInfo": {
    "@type": "TBD#ExternalInfo",
    "@id": 8,
    "omero:details": {},
    "EntityId": 3,
    "EntityType": "com.glencoesoftware.ngff:multiscales",
    "Lsid": "test_dataset_extinfo"
}

[will-moore]

@jburel I added support for:

$ omero obj ext-info-set Image:12 myLsid myEntityType

so we can set externalInfo on any object now.

[snoopycrimecop]

Conflicting PR. Removed from build OMERO-python-superbuild-push#350. See the console output for more details. Possible conflicts:

• PR #299 joshmoore 'Add parents and children to omero obj'
  • src/omero/plugins/obj.py

--conflicts

[snoopycrimecop]

Conflicting PR. Removed from build OMERO-python-superbuild-push#351. See the console output for more details. Possible conflicts:

• PR #299 joshmoore 'Add parents and children to omero obj'
  • src/omero/plugins/obj.py

--conflicts

[snoopycrimecop]

Conflicting PR. Removed from build OMERO-python-superbuild-push#352. See the console output for more details. Possible conflicts:

• PR #299 joshmoore 'Add parents and children to omero obj'
  • src/omero/plugins/obj.py

--conflicts

[snoopycrimecop]

Conflicting PR. Removed from build OMERO-python-superbuild-push#353. See the console output for more details. Possible conflicts:

• PR #299 joshmoore 'Add parents and children to omero obj'
  • src/omero/plugins/obj.py

--conflicts

[snoopycrimecop]

Conflicting PR. Removed from build OMERO-python-superbuild-push#354. See the console output for more details. Possible conflicts:

• PR #299 joshmoore 'Add parents and children to omero obj'
  • src/omero/plugins/obj.py

--conflicts

[snoopycrimecop]

Conflicting PR. Removed from build OMERO-python-superbuild-push#355. See the console output for more details. Possible conflicts:

• PR #299 joshmoore 'Add parents and children to omero obj'
  • src/omero/plugins/obj.py

--conflicts

[snoopycrimecop]

Conflicting PR. Removed from build OMERO-python-superbuild-push#356. See the console output for more details. Possible conflicts:

• PR #299 joshmoore 'Add parents and children to omero obj'
  • src/omero/plugins/obj.py

--conflicts

[snoopycrimecop]

Conflicting PR. Removed from build OMERO-python-superbuild-push#357. See the console output for more details. Possible conflicts:

• PR #299 joshmoore 'Add parents and children to omero obj'
  • src/omero/plugins/obj.py

~~--conflicts~~ Conflict resolved in build OMERO-python-superbuild-push#358. See the console output for more details.

[will-moore]

@sbesson Thanks for the review.

Re: performance - for a Plate that contains lots of other objects, the ExternalInfo is only loaded for the root Plate object, not all the Wells, Images etc. So, performance is not an issue. But I guess this raises the question: "should we load ExternalInfo" for EVERY object that is loaded in the graph? This would be a lot more work and I'm not sure if it's needed? In the various graph-loading queries we have, we make a judgement call on how much of the graph to load, and this is just another of those questions.

For getting the extInfo, how about we support:

omero obj ext-info-get Image:12
entityId=3
entityType=myEntityType
id=101
lsid=myLsid
uuid=

The pair of new methods omero obj ext-info-set and omero obj ext-info-get would correspond to existing omero obj map-set and omero obj map-get.

I guess we could also support getting individual properties:

omero obj ext-info-get Image:12 lsid
myLsid

Calling omero ext-info-set <Object>: with different properties won't try to update the immutable ExternalInfo object, but will create a new ExternalInfo with those updated properties.

I'm not sure we need to support ext-info-set with field values. It just seems such an edge case. Mostly we want to update both lsid and entityType together, which you can do as above.

If this looks good, I'll implement ext-info-get?

[sbesson]

Implementing some equivalent get method would certainly help assessing this PR.

Calling omero ext-info-set : with different properties won't try to update the immutable ExternalInfo object, but will create a new ExternalInfo with those updated properties.

You're correct. What confused me during the initial code review is that extinfo is used for the retrieved the exiting ExternalInfoif applicable and later recycled for creating a new object. What is the rationale/workflow for copying properties from an existing ExternalInfo object? To reduce the number of cases to consider, would it be simpler to only consider the values passed in command-line. In that case, both entityType and entityId should be mandatory with lsid (and uuid) optional.

[will-moore]

@sbesson I think I was influenced by https://github.com/ome/omero-cli-zarr/pull/167 which allows setting of just the entityType or entityId, but this is probably not needed and it's simpler to only allow setting of both together (which is what I've done in all my usage of this PR so far). I'll update to that...

[will-moore]

@sbesson I'm actually not sure what should be mandatory / optional. I agree that uuid is optional, but NOT lsid? Is there any use for ExternalInfo if it doesn't have lsid? e.g.

entityId=3
entityType="com.glencoesoftware.ngff:multiscales"
lsid="/bia-idr/S-BIAD1483/Experiment/P30/WT/Animal_1/Image_18.ome.zarr"
uuid=

I don't know about entityId (what does this even mean). Do we want users to specify that it's 3? Or can we use 3 as a default?

I'm kinda assuming you got these attributes mixed up above "In that case, both entityType and entityId should be mandatory with lsid (and uuid) optional." ??

[sbesson]

I'm actually not sure what should be mandatory / optional. I agree that uuid is optional, but NOT lsid?

My suggestion was based on the omero-model mappings which translated into the database schema where only entityid and entitytype are not nullable

omero=> \d externalinfo;

                      Table "public.externalinfo"
   Column    |          Type          | Collation | Nullable | Default 
-------------+------------------------+-----------+----------+---------
 id          | bigint                 |           | not null | 
 permissions | bigint                 |           | not null | 
 entityid    | bigint                 |           | not null | 
 entitytype  | character varying(255) |           | not null | 
 lsid        | text                   |           |          | 
 uuid        | character varying(255) |           |          | 
 creation_id | bigint                 |           | not null | 
 external_id | bigint                 |           |          | 
 group_id    | bigint                 |           | not null | 
 owner_id    | bigint                 |           | not null | 

Is there any use for ExternalInfo if it doesn't have lsid?

In the case of omero-zarr-pixel-buffer extension, I agree no lsid would be meaningless. In general, someone could construct an extension point that only requires entitytype/entityid to be present.

I don't know about entityId (what does this even mean). Do we want users to specify that it's 3? Or can we use 3 as a default?

No we cannot assume that 3 should be the default. In the case of omero-zarr-pixel-buffer, the entityId field is really used as to communicate this is the third version of the specification to store OME-Zarr label/images in the OMERO database (the two first versions were internal only).

[will-moore]

OK, understood. Supported now:

$ omero obj ext-info-set Project:302
usage: ext-info-set OBJ entityId entityType [lsid] [uuid]

$omero obj ext-info-set Project:302 3 myEntType
Project:302

$ omero obj ext-info-get Project:302
id=108
entityId=3
entityType=myEntType
lsid=
uuid=

$ omero obj ext-info-set Project:302 3 myEntityType2 this.is.my.lsid
Project:302

$ omero obj ext-info-get Project:302
id=109
entityId=3
entityType=myEntityType2
lsid=this.is.my.lsid
uuid=

$ omero obj ext-info-get Project:302 lsid
this.is.my.lsid

$ omero obj ext-info-get Project:302 uuid


$ omero obj ext-info-get Project:302 uuid_foo
usage: ext-info-get OBJ [id|lsid|entityId|entityType|uuid]

$ omero obj ext-info-get Project:301
No ExternalInfo found: Project:301

[will-moore]

@sbesson Thanks - addressed those 2 points

[imagesc-bot]

This pull request has been mentioned on Image.sc Forum. There might be relevant details there:

https://forum.image.sc/t/compatibility-of-ome-zarr-stored-on-s3-and-omero/111380/17