No metadata being populated by pyvips from NDPI file

[skillcoyne]

I have a number of NDPI image files that I am reading using pyvips. However, while I can clearly see that these images have metadata if I open them using a viewer application.

image = pv.Image.new_from_file(slide_file, level=0, access="sequential")
pprint(vars(image))
print(image.get_fields())

However, no properties appear to be found in the image by the 'get_fields()' method in pyvips.

<function Image.__getattr__.<locals>.call_function at 0x7f1d24cd5ca0>
[]

If I instead load these with openslide, then access the properties I can see all of the metadata

os = openslide.open_slide(slide_file)
pprint(os.properties)

Metadata:

<_PropertyMap {'hamamatsu.AHEX[0]': '0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000', 'hamamatsu.AHEX[0].fluorescence': '0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000', 'hamamatsu.AHEX[0].ploidy': '0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000', 'hamamatsu.AHEX[1]': '0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000', 'hamamatsu.AHEX[1].fluorescence': '0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000', 'hamamatsu.AHEX[1].ploidy': '0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000', 'hamamatsu.AHEX[2]': '0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000', 'hamamatsu.AHEX[2].fluorescence': '0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000', 'hamamatsu.AHEX[2].ploidy': '0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000', 'hamamatsu.Created': '2017/03/02', 'hamamatsu.Macro.S/N': '000822', 'hamamatsu.NDP.S/N': '000058', 'hamamatsu.Objective.Lens.Magnificant': '35.16', 'hamamatsu.PSHV': '107', 'hamamatsu.PSHV.10x': '107', 'hamamatsu.PSHV.40x': '107', 'hamamatsu.PSHV.ploidy': '107', 'hamamatsu.PSHV.ploidy.10x': '107', 'hamamatsu.Product': 'C13239-01', 'hamamatsu.Reference': 'AD0085\tPS00.14399_B1', 'hamamatsu.SourceLens': '40', 'hamamatsu.Updated': '2020/01/06', 'hamamatsu.XOffsetFromSlideCentre': '7156667', 'hamamatsu.YOffsetFromSlideCentre': '380488', 'hamamatsu.YRNP[0]': '0,0,0,0', 'hamamatsu.YRNP[1]': '0,0,0,0', 'hamamatsu.YRNP[2]': '0,0,0,0', 'hamamatsu.calibration.version': '3515', 'hamamatsu.ccd.height': '0', 'hamamatsu.ccd.width': '8482', 'hamamatsu.ccd.width.ploidy': '8482', 'hamamatsu.coarse.focus.pitch': '10000', 'hamamatsu.colorfilterID': '0', 'hamamatsu.cube.kind': '0', 'hamamatsu.exposure.barcode.macro': '705', 'hamamatsu.exposure.slide.darkfield.macro': '10', 'hamamatsu.exposure.slide.macro': '20', 'hamamatsu.fine.focus.pitch': '250', 'hamamatsu.focalplane.leftbottom': '88540,690520,90125', 'hamamatsu.focalplane.lefttop': '88540,490520,89937', 'hamamatsu.focalplane.rightbottom': '488540,690520,89728', 'hamamatsu.focalplane.righttop': '488540,490520,89612', 'hamamatsu.lane.shift.amount': '0', 'hamamatsu.roi.barcode.macro': '976,364,1259,758', 'hamamatsu.roi.slide.macro': '64,358,1265,769', 'hamamatsu.slant.leftbottom': '0,0,0', 'hamamatsu.slant.lefttop': '0,0,0', 'hamamatsu.slant.rightbottom': '0,0,0', 'hamamatsu.slant.righttop': '0,0,0', 'hamamatsu.slide.tickness': '0', 'hamamatsu.stage.center': '178540,590520', 'hamamatsu.system.version': '1.0', 'hamamatsu.target.white.intensity': '235', 'hamamatsu.valid.DDKP': '0', 'hamamatsu.valid.DLTP': '0', 'hamamatsu.valid.DSHP': '0', 'hamamatsu.variable.exposuretime': '0', 'hamamatsu.zCoarse[0]': '0,0,0,0', 'hamamatsu.zCoarse[1]': '0,0,0,0', 'hamamatsu.zCoarse[2]': '0,0,0,0', 'hamamatsu.zFine[0]': '0,0,0,0', 'hamamatsu.zFine[1]': '0,0,0,0', 'hamamatsu.zFine[2]': '0,0,0,0', 'openslide.level-count': '9', 'openslide.level[0].downsample': '1', 'openslide.level[0].height': '107008', 'openslide.level[0].tile-height': '8', 'openslide.level[0].tile-width': '3840', 'openslide.level[0].width': '188160', 'openslide.level[1].downsample': '2', 'openslide.level[1].height': '53504', 'openslide.level[1].tile-height': '8', 'openslide.level[1].tile-width': '1920', 'openslide.level[1].width': '94080', 'openslide.level[2].downsample': '4', 'openslide.level[2].height': '26752', 'openslide.level[2].tile-height': '8', 'openslide.level[2].tile-width': '960', 'openslide.level[2].width': '47040', 'openslide.level[3].downsample': '8', 'openslide.level[3].height': '13376', 'openslide.level[3].tile-height': '8', 'openslide.level[3].tile-width': '480', 'openslide.level[3].width': '23520', 'openslide.level[4].downsample': '16', 'openslide.level[4].height': '6688', 'openslide.level[4].tile-height': '8', 'openslide.level[4].tile-width': '240', 'openslide.level[4].width': '11760', 'openslide.level[5].downsample': '32', 'openslide.level[5].height': '3344', 'openslide.level[5].tile-height': '8', 'openslide.level[5].tile-width': '120', 'openslide.level[5].width': '5880', 'openslide.level[6].downsample': '64', 'openslide.level[6].height': '1672', 'openslide.level[6].tile-height': '4', 'openslide.level[6].tile-width': '60', 'openslide.level[6].width': '2940', 'openslide.level[7].downsample': '128', 'openslide.level[7].height': '836', 'openslide.level[7].tile-height': '2', 'openslide.level[7].tile-width': '30', 'openslide.level[7].width': '1470', 'openslide.level[8].downsample': '256', 'openslide.level[8].height': '418', 'openslide.level[8].tile-height': '1', 'openslide.level[8].tile-width': '15', 'openslide.level[8].width': '735', 'openslide.mpp-x': '0.22088708252341402', 'openslide.mpp-y': '0.22088708252341402', 'openslide.objective-power': '40', 'openslide.quickhash-1': '79caa01a0bb2eb1381367de4d5f12addc9cac748c084d49fc4785428c71c10a6', 'openslide.vendor': 'hamamatsu', 'tiff.DateTime': '2020:06:06 08:22:44', 'tiff.Make': 'Hamamatsu', 'tiff.Model': 'C13239-01', 'tiff.ResolutionUnit': 'centimeter', 'tiff.Software': 'NDP.scan 3.3.0', 'tiff.XResolution': '45272', 'tiff.YResolution': '45272'}>

Is there a step I am missing or another method to access the metadata short of loading the slide file into both pyvips and openslide?

[jcupitt]

Hi @skillcoyne,

It should work. I see:

$ python3
Python 3.9.7 (default, Sep 10 2021, 14:59:43) 
[GCC 11.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pyvips
>>> x = pyvips.Image.new_from_file("CMU-1.ndpi")
>>> x.get_fields()
['width', 'height', 'bands', 'format', 'coding', 'interpretation', 
'xoffset', 'yoffset', 'xres', 'yres', 'filename', 'vips-loader',
'slide-level', 'hamamatsu.SourceLens', 'hamamatsu.XOffsetFromSlideCentre',
'hamamatsu.YOffsetFromSlideCentre', 'openslide.level-count', 
'openslide.level[0].downsample', 'openslide.level[0].height',
'openslide.level[0].tile-height', 'openslide.level[0].tile-width',
'openslide.level[0].width', 'openslide.level[1].downsample',
'openslide.level[1].height', 'openslide.level[1].tile-height',
'openslide.level[1].tile-width', 'openslide.level[1].width',
'openslide.level[2].downsample', 'openslide.level[2].height',
'openslide.level[2].tile-height', 'openslide.level[2].tile-width',
'openslide.level[2].width', 'openslide.level[3].downsample',
'openslide.level[3].height', 'openslide.level[3].tile-height',
'openslide.level[3].tile-width', 'openslide.level[3].width',
'openslide.level[4].downsample', 'openslide.level[4].height', 
'openslide.level[4].tile-height', 'openslide.level[4].tile-width',
'openslide.level[4].width', 'openslide.level[5].downsample',
'openslide.level[5].height', 'openslide.level[5].tile-height',
'openslide.level[5].tile-width', 'openslide.level[5].width',
'openslide.level[6].downsample', 'openslide.level[6].height',
'openslide.level[6].tile-height', 'openslide.level[6].tile-width',
'openslide.level[6].width', 'openslide.level[7].downsample',
'openslide.level[7].height', 'openslide.level[7].tile-height', 
'openslide.level[7].tile-width', 'openslide.level[7].width',
'openslide.level[8].downsample', 'openslide.level[8].height',
'openslide.level[8].tile-height', 'openslide.level[8].tile-width',
'openslide.level[8].width', 'openslide.mpp-x', 'openslide.mpp-y',
'openslide.objective-power', 'openslide.quickhash-1', 'openslide.vendor',
'tiff.DateTime', 'tiff.Make', 'tiff.Model', 'tiff.ResolutionUnit',
'tiff.Software', 'tiff.XResolution', 'tiff.YResolution',
'slide-associated-images']

So I'd guess it was a problem with your libvips install. What platform are you using, what libvips are you using, etc. etc.

[kleisauke]

get_fields() requires at least libvips 8.5, see: https://github.com/libvips/pyvips/blob/5f791fac824a1dee17c432c1d38985e0e527bce0/pyvips/vimage.py#L804

Perhaps your libvips version is older than that? You can check the libvips version as follows:

$ python3
>>> import pyvips
>>> '.'.join([str(pyvips.version(i)) for i in range(3)])
'8.12.0'

[skillcoyne]

So my libvips version was older, however having updated it to 8.12.0 (and checked that it's loading the correct version) pyvips is now not able to open my NDPI files at all.

pv.Image.new_from_file('/PS00.10208 L1-3.ndpi', level=0)

Error                                     Traceback (most recent call last)
/tmp/ipykernel_16100/918042653.py in <module>
----> 1 pv.Image.new_from_file('PS00.10208 L1-3.ndpi', level=0)

~/.local/lib/python3.8/site-packages/pyvips/vimage.py in new_from_file(vips_filename, **kwargs)
    226         pointer = vips_lib.vips_foreign_find_load(vips_filename)
    227         if pointer == ffi.NULL:
--> 228             raise Error('unable to load from file {0}'.format(vips_filename))
    229         name = _to_string(pointer)
    230 

Error: unable to load from file b'PS00.10208 L1-3.ndpi'
  VipsForeignLoad: "PS00.10208 L1-3.ndpi" is not a known file format
erties

I can still open the file with OpenSlide.

Tried this on both Ubuntu and OSX, however on OSX it's a different error:

Error: unable to call VipsForeignLoadOpenslideFile
  openslideload: parameter filename not set

[jcupitt]

I tried on osx and it works for me. I see:

$ brew install vips
...
$ wget https://openslide.cs.cmu.edu/download/openslide-testdata/Hamamatsu/CMU-1.ndpi
...
$ vipsheader CMU-1.ndpi 
CMU-1.ndpi: 51200x38144 uchar, 4 bands, srgb, openslideload

So homebrew libvips loads the file with openslide.

If I install pyvips with homebrew python I can see all the metadata:

$ which python3
/usr/local/bin/python3
$ which pip3
/usr/local/bin/pip3
$ pip3 install pyvips
...
$ python3
Python 3.9.8 (main, Nov 10 2021, 06:03:50) 
[Clang 12.0.0 (clang-1200.0.32.29)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pyvips
>>> x = pyvips.Image.new_from_file("CMU-1.ndpi")
>>> x
<pyvips.Image 51200x38144 uchar, 4 bands, srgb>
>>> x.get_fields()
['width', 'height', 'bands', 'format', 'coding', 'interpretation', 'xoffset', 'yoffset', 'xres', 'yres', 'filename', 'vips-loader', 'slide-level', 'hamamatsu.SourceLens', 'hamamatsu.XOffsetFromSlideCentre', ...

[jcupitt]

On Ubuntu, I think you've built libvips without openslide support. You'll need the libopenslide-dev package. Check the output of configure.

[skillcoyne]

Yep, you're so right. Sorry I thought I had done it.

On Wed, Nov 24, 2021 at 1:03 PM John Cupitt @.***> wrote:

On Ubuntu, I think you've built libvips without openslide support. You'll need the libopenslide-dev package. Check the output of configure.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/libvips/pyvips/issues/285#issuecomment-977857700, or unsubscribe https://github.com/notifications/unsubscribe-auth/AALCZGJ3CEZJQHSG2XKEX2DUNTPALANCNFSM5IQWFGKQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

[NeelKanwal]

Hi,

Is it possible to save all the NDPI metadata when using tiffsave

Forexample: using current_img_400x.tiffsave(f"{new_fname}.tiff", bigtiff=True, properties=True, tile=True, compression="jpeg",tile_width=512, tile_height=512, pyramid=True)

loses Macro/Label and other important information like pixel size from the saved tiff tiles.

Am I missing some thing? or is there any other way to write the metadata after getting from get_fields()?

Neel