droid icon indicating copy to clipboard operation
droid copied to clipboard

Published 20 hours ago • verified publisher icon digital-preservation

NoProfileMode: STDERR Output, ambiguous messaging

Open ross-spencer opened this issue 8 years ago • 2 comments

I'm seeing this message in STDERR:

  2016-07-21 11:32:52,381 ERROR No reader was obtained for the current container file. This may be due to low memory conditions. Try running with a larger heap size!
  2016-07-21 11:32:52,392 ERROR No reader was obtained for the current container file. This may be due to low memory conditions. Try running with a larger heap size!
  2016-07-21 11:41:49,890 ERROR No reader was obtained for the current container file. This may be due to low memory conditions. Try running with a larger heap size!
  2016-07-21 11:42:22,052 ERROR No reader was obtained for the current container file. This may be due to low memory conditions. Try running with a larger heap size!

I've increased the heap argument to -Xmx1000 and will try upping it further as it hasn't helped.

Two points about this message:

  • Is it possible to have the name of the specific container file that is affected.
  • Can the ambiguity in the message be reduced at all, i.e. it may be due to low memory conditions, but are there other influencing factors the user should be aware of?

ross-spencer avatar Jul 20 '16 23:07 ross-spencer

Because it can help with isolating errors, I've isolated the files from the govdocs-selected collection causing the issue and run them through Siegfried (SF). If there are any errors being reported by POIFS/DROID's OLE2 Handler that DROID could access and send back to STDERR then that might be more useful. It might be the case that going beyond the memory limits of DROID/Java is a symptom of the more detailed error, the liked of which we're seeing in SF.

CSV here:

image

DROID STDERR and STDOUT:

  2016-07-21 12:42:43,673 ERROR No reader was obtained for the current container file. This may be due to low memory conditions. Try running with a larger heap size!
  C:\Working\droid-test\govdocs\govdocs_selected\govdocs_selected\DOC_138\873410.doc,fmt/111
  2016-07-21 12:42:43,677 ERROR No reader was obtained for the current container file. This may be due to low memory conditions. Try running with a larger heap size!
  C:\Working\droid-test\govdocs\govdocs_selected\govdocs_selected\DOC_138\881859.doc,fmt/111
  2016-07-21 12:42:43,683 ERROR No reader was obtained for the current container file. This may be due to low memory conditions. Try running with a larger heap size!
  C:\Working\droid-test\govdocs\govdocs_selected\govdocs_selected\DOC_138\881860.doc,fmt/111
  2016-07-21 12:42:43,687 ERROR No reader was obtained for the current container file. This may be due to low memory conditions. Try running with a larger heap size!
  C:\Working\droid-test\govdocs\govdocs_selected\govdocs_selected\DOC_138\889338.doc,fmt/111
  2016-07-21 12:42:43,693 ERROR No reader was obtained for the current container file. This may be due to low memory conditions. Try running with a larger heap size!
  C:\Working\droid-test\govdocs\govdocs_selected\govdocs_selected\DOC_138\889339.doc,fmt/111
  2016-07-21 12:42:43,695 ERROR No reader was obtained for the current container file. This may be due to low memory conditions. Try running with a larger heap size!
  C:\Working\droid-test\govdocs\govdocs_selected\govdocs_selected\DOC_138\961938.doc,fmt/111
  2016-07-21 12:42:43,697 ERROR No reader was obtained for the current container file. This may be due to low memory conditions. Try running with a larger heap size!
  C:\Working\droid-test\govdocs\govdocs_selected\govdocs_selected\DOC_138\973796.doc,fmt/111
  016-07-21 12:43:03,517 ERROR No reader was obtained for the current container file. This may be due to low memory conditions. Try running with a larger heap size!
  C:\Working\droid-test\govdocs\govdocs_selected\govdocs_selected\DOC_39\160706.doc,fmt/126
  2016-07-21 12:43:03,519 ERROR No reader was obtained for the current container file. This may be due to low memory conditions. Try running with a larger heap size!
  C:\Working\droid-test\govdocs\govdocs_selected\govdocs_selected\DOC_39\254076.doc,fmt/111
  2016-07-21 12:43:03,550 ERROR No reader was obtained for the current container file. This may be due to low memory conditions. Try running with a larger heap size!
  C:\Working\droid-test\govdocs\govdocs_selected\govdocs_selected\DOC_39\279367.doc,fmt/111
  2016-07-21 12:43:03,553 ERROR No reader was obtained for the current container file. This may be due to low memory conditions. Try running with a larger heap size!
  C:\Working\droid-test\govdocs\govdocs_selected\govdocs_selected\DOC_39\317595.doc,fmt/40
  2016-07-21 12:43:03,555 ERROR No reader was obtained for the current container file. This may be due to low memory conditions. Try running with a larger heap size!
  C:\Working\droid-test\govdocs\govdocs_selected\govdocs_selected\DOC_39\367762.doc,fmt/111
  2016-07-21 12:43:03,558 ERROR No reader was obtained for the current container file. This may be due to low memory conditions. Try running with a larger heap size!
  C:\Working\droid-test\govdocs\govdocs_selected\govdocs_selected\DOC_39\657091.doc,fmt/40
  2016-07-21 12:43:03,562 ERROR No reader was obtained for the current container file. This may be due to low memory conditions. Try running with a larger heap size!
  C:\Working\droid-test\govdocs\govdocs_selected\govdocs_selected\DOC_39\874330.doc,fmt/40
  2016-07-21 12:43:03,566 ERROR No reader was obtained for the current container file. This may be due to low memory conditions. Try running with a larger heap size!
  C:\Working\droid-test\govdocs\govdocs_selected\govdocs_selected\DOC_39\876259.doc,fmt/40
  2016-07-21 12:43:03,570 ERROR No reader was obtained for the current container file. This may be due to low memory conditions. Try running with a larger heap size!
  C:\Working\droid-test\govdocs\govdocs_selected\govdocs_selected\DOC_39\876751.doc,fmt/111
  2016-07-21 12:43:03,573 ERROR No reader was obtained for the current container file. This may be due to low memory conditions. Try running with a larger heap size!
  C:\Working\droid-test\govdocs\govdocs_selected\govdocs_selected\DOC_39\933792.doc,fmt/111
  2016-07-21 12:43:23,192 ERROR No reader was obtained for the current container file. This may be due to low memory conditions. Try running with a larger heap size!
  C:\Working\droid-test\govdocs\govdocs_selected\govdocs_selected\HTML_57\095885.html,fmt/111
  2016-07-21 12:43:23,208 ERROR No reader was obtained for the current container file. This may be due to low memory conditions. Try running with a larger heap size!
  C:\Working\droid-test\govdocs\govdocs_selected\govdocs_selected\HTML_57\363940.html,fmt/111
  2016-07-21 12:43:23,237 ERROR No reader was obtained for the current container file. This may be due to low memory conditions. Try running with a larger heap size!
  C:\Working\droid-test\govdocs\govdocs_selected\govdocs_selected\HTML_57\444941.html,fmt/111
  2016-07-21 12:43:35,952 ERROR No reader was obtained for the current container file. This may be due to low memory conditions. Try running with a larger heap size!
  C:\Working\droid-test\govdocs\govdocs_selected\govdocs_selected\OBD_1\594136.unk,fmt/40
  2016-07-21 12:43:35,963 ERROR No reader was obtained for the current container file. This may be due to low memory conditions. Try running with a larger heap size!
  2016-07-21 12:52:16,669 ERROR No reader was obtained for the current container file. This may be due to low memory conditions. Try running with a larger heap size!
  C:\Working\droid-test\govdocs\govdocs_selected\govdocs_selected\PPS_9\261990.pps,fmt/111
  2016-07-21 12:52:52,822 ERROR No reader was obtained for the current container file. This may be due to low memory conditions. Try running with a larger heap size!
  C:\Working\droid-test\govdocs\govdocs_selected\govdocs_selected\PPT_20\032833.doc,fmt/111
  2016-07-21 12:52:52,840 ERROR No reader was obtained for the current container file. This may be due to low memory conditions. Try running with a larger heap size!
  C:\Working\droid-test\govdocs\govdocs_selected\govdocs_selected\PPT_20\166335.doc,fmt/111
  2016-07-21 12:52:52,857 ERROR No reader was obtained for the current container file. This may be due to low memory conditions. Try running with a larger heap size!
  C:\Working\droid-test\govdocs\govdocs_selected\govdocs_selected\PPT_20\404339.doc,fmt/111
  2016-07-21 12:52:52,914 ERROR No reader was obtained for the current container file. This may be due to low memory conditions. Try running with a larger heap size!
  C:\Working\droid-test\govdocs\govdocs_selected\govdocs_selected\PPT_20\862280.doc,fmt/111
  2016-07-21 12:52:52,923 ERROR No reader was obtained for the current container file. This may be due to low memory conditions. Try running with a larger heap size!
  C:\Working\droid-test\govdocs\govdocs_selected\govdocs_selected\PPT_20\885451.doc,fmt/111
  2016-07-21 12:52:52,962 ERROR No reader was obtained for the current container file. This may be due to low memory conditions. Try running with a larger heap size!
  C:\Working\droid-test\govdocs\govdocs_selected\govdocs_selected\PPT_20\886223.doc,fmt/111
  2016-07-21 12:52:52,978 ERROR No reader was obtained for the current container file. This may be due to low memory conditions. Try running with a larger heap size!
  C:\Working\droid-test\govdocs\govdocs_selected\govdocs_selected\PPT_20\891502.doc,fmt/111
  2016-07-21 12:52:53,110 ERROR No reader was obtained for the current container file. This may be due to low memory conditions. Try running with a larger heap size!
  C:\Working\droid-test\govdocs\govdocs_selected\govdocs_selected\PPT_20\948342.doc,fmt/111
  2016-07-21 12:52:53,150 ERROR No reader was obtained for the current container file. This may be due to low memory conditions. Try running with a larger heap size!
  C:\Working\droid-test\govdocs\govdocs_selected\govdocs_selected\PPT_20\973419.doc,fmt/111
  2016-07-21 12:52:53,191 ERROR No reader was obtained for the current container file. This may be due to low memory conditions. Try running with a larger heap size!
  C:\Working\droid-test\govdocs\govdocs_selected\govdocs_selected\PPT_20\979573.doc,fmt/111
  2016-07-21 12:52:56,774 ERROR No reader was obtained for the current container file. This may be due to low memory conditions. Try running with a larger heap size!
  C:\Working\droid-test\govdocs\govdocs_selected\govdocs_selected\PPT_30\124583.ppt,fmt/125
  2016-07-21 12:52:58,829 ERROR No reader was obtained for the current container file. This may be due to low memory conditions. Try running with a larger heap size!
  C:\Working\droid-test\govdocs\govdocs_selected\govdocs_selected\PPT_34\958518.ppt,fmt/126
  2016-07-21 12:53:17,923 ERROR No reader was obtained for the current container file. This may be due to low memory conditions. Try running with a larger heap size!
  C:\Working\droid-test\govdocs\govdocs_selected\govdocs_selected\PPT_6\089485.ppt,fmt/111
  2016-07-21 12:53:17,937 ERROR No reader was obtained for the current container file. This may be due to low memory conditions. Try running with a larger heap size!
  C:\Working\droid-test\govdocs\govdocs_selected\govdocs_selected\PPT_6\221537.ppt,fmt/126
  2016-07-21 12:53:17,955 ERROR No reader was obtained for the current container file. This may be due to low memory conditions. Try running with a larger heap size!
  C:\Working\droid-test\govdocs\govdocs_selected\govdocs_selected\PPT_6\390311.ppt,fmt/111
  2016-07-21 12:53:19,689 ERROR No reader was obtained for the current container file. This may be due to low memory conditions. Try running with a larger heap size!
  C:\Working\droid-test\govdocs\govdocs_selected\govdocs_selected\PPT_6\665446.ppt,fmt/111
  C:\Working\droid-test\govdocs\govdocs_selected\govdocs_selected\PPT_6\761539.ppt,fmt/126
  2016-07-21 12:53:19,758 ERROR No reader was obtained for the current container file. This may be due to low memory conditions. Try running with a larger heap size!
  C:\Working\droid-test\govdocs\govdocs_selected\govdocs_selected\PPT_6\785388.ppt,fmt/111
  2016-07-21 12:53:19,775 ERROR No reader was obtained for the current container file. This may be due to low memory conditions. Try running with a larger heap size!
  C:\Working\droid-test\govdocs\govdocs_selected\govdocs_selected\PPT_6\788610.ppt,fmt/126
  2016-07-21 12:53:19,802 ERROR No reader was obtained for the current container file. This may be due to low memory conditions. Try running with a larger heap size!
  C:\Working\droid-test\govdocs\govdocs_selected\govdocs_selected\PPT_6\808724.ppt,fmt/126
  2016-07-21 12:53:19,812 ERROR No reader was obtained for the current container file. This may be due to low memory conditions. Try running with a larger heap size!
  C:\Working\droid-test\govdocs\govdocs_selected\govdocs_selected\PPT_6\864888.ppt,fmt/126
  2016-07-21 12:53:19,876 ERROR No reader was obtained for the current container file. This may be due to low memory conditions. Try running with a larger heap size!
  C:\Working\droid-test\govdocs\govdocs_selected\govdocs_selected\PPT_6\887937.ppt,fmt/111
  2016-07-21 12:53:25,533 ERROR No reader was obtained for the current container file. This may be due to low memory conditions. Try running with a larger heap size!
  C:\Working\droid-test\govdocs\govdocs_selected\govdocs_selected\PPT_8\986608.ppt,fmt/111
  2016-07-21 12:54:17,921 ERROR No reader was obtained for the current container file. This may be due to low memory conditions. Try running with a larger heap size!
  C:\Working\droid-test\govdocs\govdocs_selected\govdocs_selected\XLS_29\024891.xls,fmt/61
  2016-07-21 12:54:17,988 ERROR No reader was obtained for the current container file. This may be due to low memory conditions. Try running with a larger heap size!
  C:\Working\droid-test\govdocs\govdocs_selected\govdocs_selected\XLS_29\085904.xls,fmt/61
  2016-07-21 12:54:18,037 ERROR No reader was obtained for the current container file. This may be due to low memory conditions. Try running with a larger heap size!
  C:\Working\droid-test\govdocs\govdocs_selected\govdocs_selected\XLS_29\188343.xls,fmt/61
  2016-07-21 12:54:18,079 ERROR No reader was obtained for the current container file. This may be due to low memory conditions. Try running with a larger heap size!
  C:\Working\droid-test\govdocs\govdocs_selected\govdocs_selected\XLS_29\577248.xls,fmt/59
  2016-07-21 12:54:18,103 ERROR No reader was obtained for the current container file. This may be due to low memory conditions. Try running with a larger heap size!
  C:\Working\droid-test\govdocs\govdocs_selected\govdocs_selected\XLS_29\596737.xls,fmt/61
  2016-07-21 12:54:18,124 ERROR No reader was obtained for the current container file. This may be due to low memory conditions. Try running with a larger heap size!
  C:\Working\droid-test\govdocs\govdocs_selected\govdocs_selected\XLS_29\599284.xls,fmt/61
  2016-07-21 12:54:18,201 ERROR No reader was obtained for the current container file. This may be due to low memory conditions. Try running with a larger heap size!
  C:\Working\droid-test\govdocs\govdocs_selected\govdocs_selected\XLS_29\601878.xls,fmt/61
  C:\Working\droid-test\govdocs\govdocs_selected\govdocs_selected\XLS_29\601878.xls,fmt/62
  2016-07-21 12:54:18,226 ERROR No reader was obtained for the current container file. This may be due to low memory conditions. Try running with a larger heap size!
  C:\Working\droid-test\govdocs\govdocs_selected\govdocs_selected\XLS_29\609175.xls,fmt/61
  C:\Working\droid-test\govdocs\govdocs_selected\govdocs_selected\XLS_29\609175.xls,fmt/62
  2016-07-21 12:54:18,258 ERROR No reader was obtained for the current container file. This may be due to low memory conditions. Try running with a larger heap size!
  C:\Working\droid-test\govdocs\govdocs_selected\govdocs_selected\XLS_29\614604.xls,fmt/61
  C:\Working\droid-test\govdocs\govdocs_selected\govdocs_selected\XLS_29\614604.xls,fmt/62
  2016-07-21 12:54:18,269 ERROR No reader was obtained for the current container file. This may be due to low memory conditions. Try running with a larger heap size!
  C:\Working\droid-test\govdocs\govdocs_selected\govdocs_selected\XLS_29\617761.xls,fmt/61

Five sample files attached.

broken-container-samples.zip

ross-spencer avatar Jul 21 '16 02:07 ross-spencer

Please see https://github.com/digital-preservation/droid/issues/67 - in particular my 6 January 2016 comment. DROID's OLE2 identifier engine uses Apache POIFS - it calls the POIFSFileSystem constructor to obtain a reader. This sometimes throws an error internally but unfortunately the error is not propogated to the calling code, rather the caller ends up with a null POIFSFileSystem reader reference. The latest version of DROID specifically checks for a null reader and outputs this message if one is found. I found in testing that increasing the heap size would sometimes help, but this may not be the case for all files and operating conditions - as appears to be the case with these examples.

Previously, the null reference was not handled by the caller - leading to a NullReferenceExcexption. This was what caused the "skipping files" issue discussed under the above link and elsewhere.

The name of the container file is output when this issue arises whilst running in profile mode. Unfortunately it is not available at equivalent point in the "no profile" code - hence the generic wording. There are several differences in the code between the two modes and from initial investigation the no profile code would need some refactoring to be able to access the container name.

Brian-O-TNA avatar Aug 01 '16 13:08 Brian-O-TNA