Nifti volume of specific size + Shrink filter cause segmentation fault

[okmiim]

Description

The combination of NIFTI + Shrink filter causes a segmentation fault for some dimensions and sizes.

Steps to Reproduce (MVE)

1. Generate a nifti volume of the size (1034, 1034, 1020)

   import itk
   
   pixel_type = itk.US
   dimension = 3
   image_type = itk.Image[pixel_type, dimension]
   
   start = itk.Index[dimension]()
   start.Fill(0)
   
   size = itk.Size[dimension]()
   size[0] = 1034
   size[1] = 1034
   size[2] = 1020
   
   region = itk.ImageRegion[dimension]()
   region.SetIndex(start)
   region.SetSize(size)
   
   image = image_type.New(Regions=region)
   image.Allocate()
   
   itk.imwrite(image, "segmentation-fault-test.nii")
   

2. Create a pipeline with ImageFileReader + ShrinkImageFilter and update at the end

   import itk
   from itk import ShrinkImageFilter
   
   image_path = "segmentation-fault-test"
   reader = itk.ImageFileReader.New(FileName=image_path)
   shrink = ShrinkImageFilter.New(reader)
   shrink.SetShrinkFactors(4)
   
   print("Update")
   shrink.Update()
   
   print("GetOutput")
   data = shrink.GetOutput()
   
   del shrink
   del reader
   print("Done")
   
   

3. Run it (for windows add -Xfaulthandler to the python command)

Expected behavior

Script runs through without any errors and outputs all print messages.

Actual behavior

A segmentation fault (linux) / access violation (windows) after the "Update" printout.

Reproducibility

With the mentioned dimensions 100%.

Versions/Environments

Linux:

• itk 5.1.2 and 5.2.1
• Python 3.8

Windows:

• itk 5.3rc4.post2
• Python 3.9

Additional Information

• The dimensions matter, e.g.
  • 1034, 1034, 1020 (1090539120) crash
  • 4048, 2048, 132 (1094320128) works
  • 826, 843, 3314(2307597852) crash
• Other voxel types uint8 (with 1034, 1034, 2040) or float are also affected
• Seems to be related to the byte size in combination with the dimensions
• Other shrink factors (as 2) are also affected
• .nii.gz is also affected
• nrrd seems to work FINE
• In some other workflow this causes a corruption of the volume without a seg fault (I try to add a picture later)

[github-actions[bot]]

Thank you for contributing an issue!

Welcome to the ITK community!

We are glad you are here and appreciate your contribution. Please keep in mind our community participation guidelines. Also, please check existing open issues and consider discussion on the ITK Discourse.

[dzenanz]

This seems like a bug in ShrinkImageFilter. @tbirdso did you work with this filter recently? This affects a fairly recent version ac2dfcf879c0da7c67bcaf916cf21a890a8f82c8 I had built locally with Python wrapping. Here is my single repro file:

import itk

print(f'Using ITK version: {itk.__version__} from {itk.__file__}')

# Generate an image
pixel_type = itk.US
dimension = 3
image_type = itk.Image[pixel_type, dimension]

start = itk.Index[dimension]()
start.Fill(0)

size = itk.Size[dimension]()
size[0] = 1034
size[1] = 1034
size[2] = 1020

region = itk.ImageRegion[dimension]()
region.SetIndex(start)
region.SetSize(size)

image = image_type.New(Regions=region)
image.Allocate()
itk.imwrite(image, "segmentation-fault-test.nii")

# Create a pipeline with ImageFileReader + OtsuImageFilter and update at the end
image_path = "segmentation-fault-test.nii"
reader = itk.ImageFileReader.New(FileName=image_path)
shrink = itk.ShrinkImageFilter.New(reader)
shrink.SetShrinkFactors(4)

print("Update")
shrink.Update()

print("GetOutput")
data = shrink.GetOutput()

del shrink
del reader
print("Done")

[tbirdso]

Hi @dzenanz , I have not worked with ShrinkImageFilter or OtsuImageFilter.

Git history shows that this filter has had relatively few changes in recent years:

https://github.com/InsightSoftwareConsortium/ITK/commits/9cd0f2053f0cf181ec4412bf896301faf0cc2092/Modules/Filtering/ImageGrid/include/itkShrinkImageFilter.h

https://github.com/InsightSoftwareConsortium/ITK/commits/9cd0f2053f0cf181ec4412bf896301faf0cc2092/Modules/Filtering/ImageGrid/include/itkShrinkImageFilter.hxx

[dzenanz]

ShrinkImageFilter has had some problems for a long time now. Fixing the crashes using this repro case would be useful for large images, as sometimes people don't do a proper pyramid and resort to ShrinkImageFilter.

[blowekamp]

The nifti file format is required for to reproduce this bug?

[dzenanz]

Incredibly, yes. If I change filename to .nrrd, the process finishes with exit code 0.

[blowekamp]

I was able to reproduce the issue in C++:

itkShrinkImageTest2(int, char *[])
{

  using ImageType = itk::Image<unsigned short, 3>;
  ImageType::SizeType   size = { { 1034, 1034, 1020 } };
  ImageType::RegionType region;
  region.SetSize(size);

  auto image = ImageType::New();
  image->SetRegions(region);
  image->Allocate(true);

  const std::string fname{"test.nii"};
  itk::WriteImage(image, fname);


  using ReaderType = itk::ImageFileReader<ImageType>;
  auto reader = ReaderType::New();
  reader->SetFileName(fname);

  using ShrinkFilterType = itk::ShrinkImageFilter<ImageType, ImageType>;
  auto shrinker = ShrinkFilterType::New();
  shrinker->SetInput(reader->GetOutput());
  shrinker->SetShrinkFactors(4);

  shrinker->Update();

  return 0;
}

[dzenanz]

Is it NIFTI-specific in C++ too?

[dzenanz]

It crashes in nifti reader, when trying to read a region with index 3,3,2 and size 1029,1029,1017.

[blowekamp]

Yes. And if I update the reader before it is OK too.

I get the following segfault stack:

0  __memmove_avx_unaligned_erms () at ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:416
#1  0x00007ffff7add45b in __GI__IO_file_xsgetn (fp=0x555556c611e0, data=<optimized out>, n=2058) at fileops.c:1296
#2  0x00007ffff7ad0ee3 in __GI__IO_fread (buf=0x80a, size=1, size@entry=4285254, count=2058, fp=0x555556c611e0) at iofread.c:38
#3  0x0000555555fd1aac in fread (__stream=<optimized out>, __n=<optimized out>, __size=4285254, __ptr=<optimized out>) at /usr/include/x86_64-linux-gnu/bits/stdio2.h:297
#4  znzread (buf=<optimized out>, size=4285254, nmemb=<optimized out>, file=<optimized out>) at /home/blowekamp/src/ITK/Modules/ThirdParty/NIFTI/src/nifti/znzlib/znzlib.c:164
#5  0x0000555556c67e10 in ?? ()
#6  0x0000000000000004 in ?? ()
#7  0x000000000000080a in ?? ()
#8  0x000000000000080a in ?? ()
#9  0x0000555556c65740 in ?? ()
#10 0x0000555555fd06c4 in nifti_read_subregion_image (nim=0x80a, start_index=start_index@entry=0x7fffffffe0c0, region_size=region_size@entry=0x7fffffffe0e0, data=data@entry=0x7fffffffe078)
    at /home/blowekamp/src/ITK/Modules/ThirdParty/NIFTI/src/nifti/niftilib/nifti1_io.c:7113
#11 0x0000555555fb135a in itk::NiftiImageIO::Read (this=0x555556c68690, buffer=0x7ffee5414010) at /home/blowekamp/src/ITK/Modules/IO/NIFTI/src/itkNiftiImageIO.cxx:596
#12 0x0000555555b5977b in itk::ImageFileReader<itk::Image<unsigned short, 3u>, itk::DefaultConvertPixelTraits<unsigned short> >::GenerateData (this=0x555556c61a90)
    at /usr/include/c++/9/bits/stl_algobase.h:465
#13 0x0000555555b73fbd in itk::ProcessObject::UpdateOutputData (this=0x555556c61a90) at /home/blowekamp/src/ITK/Modules/Core/Common/src/itkProcessObject.cxx:1731
#14 0x0000555555b73f91 in itk::ProcessObject::UpdateOutputData (this=0x555556c67450) at /home/blowekamp/src/ITK/Modules/Core/Common/include/itkProcessObject.h:601
#15 0x0000555555b73f91 in itk::ProcessObject::UpdateOutputData (this=0x555556c66040) at /home/blowekamp/src/ITK/Modules/Core/Common/include/itkProcessObject.h:601
#16 0x0000555555b4e4e0 in itkShrinkImageTest2 () at /home/blowekamp/src/ITK/Modules/Filtering/ImageGrid/test/itkShrinkImageTest2.cxx:58
#17 0x0000555555861263 in main (ac=<optimized out>, av=<optimized out>) at Modules/Filtering/ImageGrid/test/ITKImageGridTestDriver.cxx:437

I suspect that the ShrinkImageFilter is requesting a region that is not the largest possible. That it is cropped by some amount. It may be that the NiftiImageIO has a bug related to streaming/reading partial regions.

[dzenanz]

My stack trace is:

tester.exe!znzread(void * buf, unsigned __int64 size, unsigned __int64 nmemb, znzptr * file) Line 165
	at C:\Dev\ITK-git\Modules\ThirdParty\NIFTI\src\nifti\znzlib\znzlib.c(165)
tester.exe!nifti_read_buffer(znzptr * fp, void * dataptr, unsigned __int64 ntot, nifti_image * nim) Line 4967
	at C:\Dev\ITK-git\Modules\ThirdParty\NIFTI\src\nifti\niftilib\nifti1_io.c(4967)
tester.exe!nifti_read_subregion_image(nifti_image * nim, const int * start_index, const int * region_size, void * * data) Line 7113
	at C:\Dev\ITK-git\Modules\ThirdParty\NIFTI\src\nifti\niftilib\nifti1_io.c(7113)
tester.exe!itk::NiftiImageIO::Read(void * buffer) Line 596
	at C:\Dev\ITK-git\Modules\IO\NIFTI\src\itkNiftiImageIO.cxx(596)
tester.exe!itk::ImageFileReader<itk::Image<unsigned short,3>,itk::DefaultConvertPixelTraits<unsigned short>>::GenerateData() Line 437
	at C:\Dev\ITK-git\Modules\IO\ImageBase\include\itkImageFileReader.hxx(437)
tester.exe!itk::ProcessObject::UpdateOutputData(itk::DataObject * __formal) Line 1708
	at C:\Dev\ITK-git\Modules\Core\Common\src\itkProcessObject.cxx(1708)
tester.exe!itk::DataObject::UpdateOutputData() Line 391
	at C:\Dev\ITK-git\Modules\Core\Common\src\itkDataObject.cxx(391)
tester.exe!itk::ImageBase<3>::UpdateOutputData() Line 267
	at C:\Dev\ITK-git\Modules\Core\Common\include\itkImageBase.hxx(267)
tester.exe!itk::ProcessObject::UpdateOutputData(itk::DataObject * __formal) Line 1671
	at C:\Dev\ITK-git\Modules\Core\Common\src\itkProcessObject.cxx(1671)
tester.exe!itk::DataObject::UpdateOutputData() Line 391
	at C:\Dev\ITK-git\Modules\Core\Common\src\itkDataObject.cxx(391)
tester.exe!itk::ImageBase<3>::UpdateOutputData() Line 267
	at C:\Dev\ITK-git\Modules\Core\Common\include\itkImageBase.hxx(267)
tester.exe!itk::DataObject::Update() Line 320
	at C:\Dev\ITK-git\Modules\Core\Common\src\itkDataObject.cxx(320)
tester.exe!itk::ProcessObject::Update() Line 1288
	at C:\Dev\ITK-git\Modules\Core\Common\src\itkProcessObject.cxx(1288)
tester.exe!itkShrinkImageTest2(int __formal, char * * __formal) Line 33
	at C:\Misc\Tester\tester.cpp(33)
tester.exe!main() Line 41
	at C:\Misc\Tester\tester.cpp(41)

[dzenanz]

NiftiImageIO has a bug related to streaming/reading partial regions

Agreed. I don't know whether this is in our code, or upstream nifti.

@hjmjohnson is NIFTI supposed to be able to read subregions?

[okmiim]

Great job so far!

A small addition: There are cases, where the data only gets corrupted and does not crash on loading. The image below is suppose to be a cylinder and the blocky part in the middle is "corrupted". This is visualized with itkwidgets and only has a fault on later use. size: (826, 843, 3314), type: unsigned_short

[blowekamp]

So the minimal example would be something just like:

  using ReaderType = itk::ImageFileReader<ImageType>;
  auto reader = ReaderType::New();
  reader->SetFileName(fname);
  region.SetSize( {{1029,1029,1017}} );
  reader->GetOutput()->SetRequestedRegion(region);
  reader->Update();

And may work with some existing data in ITK's test suite.

[dzenanz]

If we name the file test.nii.gz, it gets compressed down to 2MB. I doubt we can find an image that large in ITK's test data.

[dzenanz]

This reproduces it:

#include "itkImageFileWriter.h"
#include "itkImageFileReader.h"
#include "itkShrinkImageFilter.h"

int
itkShrinkImageTest2(int, char *[])
{

  using ImageType = itk::Image<unsigned short, 3>;
  ImageType::SizeType   size = { { 1034, 1034, 1020 } };
  ImageType::RegionType region;
  region.SetSize(size);

  auto image = ImageType::New();
  image->SetRegions(region);
  image->Allocate(true);

  const std::string fname{ "test.nii.gz" };
  itk::WriteImage(image, fname);


  using ReaderType = itk::ImageFileReader<ImageType>;
  auto reader = ReaderType::New();
  reader->SetFileName(fname);
  region.SetSize({ { 1029, 1029, 1017 } });
  reader->GetOutput()->SetRequestedRegion(region);
  reader->Update();

  return EXIT_SUCCESS;
}

[okmiim]

After a big of digging I think I found the bug in the upstream nifti_clib in nifti_read_subregion_image. The requested size of 1029 * 1029 * 1017 * 2 = 2 153 682 594 bytes is bigger than the signed size of a int causing an overflow. https://github.com/InsightSoftwareConsortium/ITK/blob/2f2b29cbbe3fe91474933e0194db8d581d5b3247/Modules/ThirdParty/NIFTI/src/nifti/niftilib/nifti1_io.c#L6972 The following malloc does not allocate any memory and thus causes a segmentation fault. https://github.com/InsightSoftwareConsortium/ITK/blob/2f2b29cbbe3fe91474933e0194db8d581d5b3247/Modules/ThirdParty/NIFTI/src/nifti/niftilib/nifti1_io.c#L7060

[blowekamp]

@dzenanz Has the test code been added? Should it be?

[dzenanz]

Jon said he would add a test. Let's keep the issue open until then.

[dzenanz]

Related issue: #3639.

[dzenanz]

@okmiim plans to do it, so I reassigned this issue to him.

[jhlegarreta]

@okmiim Do you need any help with this?