PythonQt wrapping randomly fails because missing character in include path

[lassoan]

PythonQt wrapping randomly fails on many computers, for example:

32>6>16> File "C:/Projects/Slicer4/Slicer-build/CTK/CMake/ctkWrapPythonQt.py", line 226, in 32>6>16> ctk_wrap_pythonqt(options.target, options.namespace, options.output_dir, args, options.extra_verbose) 32>6>16> File "C:/Projects/Slicer4/Slicer-build/CTK/CMake/ctkWrapPythonQt.py", line 77, in ctk_wrap_pythonqt 32>6>16> with open(input_file) as f: 32>6>16>IOError: [Errno 2] No such file or directory: 'C:/Projects/Slicer4/Slicer-build/CTK/Libs/Widgets/ctkPopupWidgetEventlayer.h' 32>6>16>Project : error PRJ0019: A tool returned an error code from "PythonQt Wrapping - Generating generated_cpp/org_commontk_CTKWidgets/org_commontk_CTKWidgets_init.cpp"

The problem is that Looks to me that somewhere

ctkPopupWidgetEventlayer.h

is being included instead of

ctkPopupWidgetEventPlayer.h

[lassoan]

It seems that there may be a clean and definitive solution!

=> Problem

Maximum command line length on Windows for cmd.exe is 8192 (32767 for CreateProcess). If you pass longer than 8192 then the behavior is undefined (actually, 8192th character is removed and the rest of the string is passed). CMake passes parameters to the compiler or linker using a parameter file (instead of command-line arguments) when it finds that the command-line would be too long.

However, CMake cannot implement this mechanism for custom commands, that's up to the developer who adds a custom command!

=> Example

The particular issue that caused build error for Jim is due to a custom command defined in ctkMacroWrapPythonQt.cmake:

  add_custom_command(
    OUTPUT
      ${wrapper_init_cpp_filename}
      ${wrapper_h_filename}
    DEPENDS
      ${SOURCES_TO_WRAP}
      ${CTK_CMAKE_DIR}/ctkWrapPythonQt.py
    COMMAND ${PYTHON_EXECUTABLE} ${CTK_CMAKE_DIR}/ctkWrapPythonQt.py
      --target=${TARGET}
      --namespace=${WRAPPING_NAMESPACE}
      --output-dir=${CMAKE_CURRENT_BINARY_DIR}/${wrap_int_dir} ${extra_args}
      ${SOURCES_TO_WRAP}
    COMMENT "PythonQt Wrapping - Generating ${wrapper_init_cpp_filename}"
    VERBATIM
    ) 

The ${SOURCES_TO_WRAP} variable can contain hundreds of full paths, which of course can easily grow over 8192. For example, a command like this can be generated (see in the generated CTKWidgetsPythonQt.vcxproj file):

    <CustomBuild Include="..\..\CMakeFiles\d4e9ed124a08aa4e815590aff2c25a3c\org_commontk_CTKWidgets_init.cpp.rule">
      <Message Condition="'$(Configuration)|$(Platform)'=='Debug|x64'">PythonQt Wrapping - Generating generated_cpp/org_commontk_CTKWidgets/org_commontk_CTKWidgets_init.cpp</Message>
      <Command Condition="'$(Configuration)|$(Platform)'=='Debug|x64'">setlocal
C:\S4D\python-install\bin\SlicerPython.exe C:/S4D/CTK/CMake/ctkWrapPythonQt.py --target=CTKWidgets --namespace=org.commontk --output-dir=C:/S4D/CTK-build/CTK-build/Libs/Widgets/generated_cpp/org_commontk_CTKWidgets/ C:/S4D/CTK/Libs/Widgets/ctkActionsWidget.h C:/S4D/CTK/Libs/Widgets/ctkAddRemoveComboBox.h C:/S4D/CTK/Libs/Widgets/ctkAxesWidget.h C:/S4D/CTK/Libs/Widgets/ctkBasePopupWidget.h C:/S4D/CTK/Libs/Widgets/ctkButtonGroup.h C:/S4D/CTK/Libs/Widgets/ctkCheckableComboBox.h C:/S4D/CTK/Libs/Widgets/ctkCheckableHeaderView.h C:/S4D/CTK/Libs/Widgets/ctkCheckableModelHelper.h C:/S4D/CTK/Libs/Widgets/ctkCheckBox.h C:/S4D/CTK/Libs/Widgets/ctkCheckBoxPixmaps.h C:/S4D/CTK/Libs/Widgets/ctkCheckablePushButton.h C:/S4D/CTK/Libs/Widgets/ctkComboBox.h C:/S4D/CTK/Libs/Widgets/ctkCompleter.h C:/S4D/CTK/Libs/Widgets/ctkCollapsibleButton.h C:/S4D/CTK/Libs/Widgets/ctkCollapsibleGroupBox.h C:/S4D/CTK/Libs/Widgets/ctkColorDialog.h C:/S4D/CTK/Libs/Widgets/ctkColorPickerButton.h C:/S4D/CTK/Libs/Widgets/ctkConsole.h C:/S4D/CTK/Libs/Widgets/ctkCoordinatesWidget.h C:/S4D/CTK/Libs/Widgets/ctkCrosshairLabel.h C:/S4D/CTK/Libs/Widgets/ctkDateRangeWidget.h C:/S4D/CTK/Libs/Widgets/ctkDirectoryButton.h 
…
many many more similar lines here
…
C:/S4D/CTK/Libs/Widgets/ctkMenuComboBoxEventTranslator.h C:/S4D/CTK/Libs/Widgets/ctkPathLineEditEventPlayer.h C:/S4D/CTK/Libs/Widgets/ctkPathLineEditEventTranslator.h C:/S4D/CTK/Libs/Widgets/ctkPopupWidgetEventPlayer.h C:/S4D/CTK/Libs/Widgets/ctkPopupWidgetEventTranslator.h C:/S4D/CTK/Libs/Widgets/ctkRangeSliderEventPlayer.h C:/S4D/CTK/Libs/Widgets/ctkRangeSliderEventTranslator.h C:/S4D/CTK/Libs/Widgets/ctkTreeComboBoxEventPlayer.h C:/S4D/CTK/Libs/Widgets/ctkTreeComboBoxEventTranslator.h
if %errorlevel% neq 0 goto :cmEnd
:cmEnd
endlocal &amp; call :cmErrorLevel %errorlevel% &amp; goto :cmDone
:cmErrorLevel
exit /b %1
:cmDone
if %errorlevel% neq 0 goto :VCEnd</Command>

=> Solution

Fortunately, there is a nice, clean solution to this issue: pass the ${SOURCES_TO_WRAP} list to the python script through a file, not through command-line.

There may be other instances where long lists are attempted to pass through command-line parameters, so fixing this issue may not allow us to use arbitrarily long paths yet, but if we fix these issues one by one as they pop up, finally we may end up fixing all of them.

[pieper]

+1!

On Tue, Dec 16, 2014 at 9:41 PM, Andras Lasso [email protected] wrote:

It seems that there may be a clean and definitive solution!

=> Problem

Maximum command line length on Windows for cmd.exe is 8192 (32767 for CreateProcess). If you pass longer than 8192 then the behavior is undefined (actually, 8192th character is removed and the rest of the string is passed). CMake passes parameters to the compiler or linker using a parameter file (instead of command-line arguments) when it finds that the command-line would be too long.

However, CMake cannot implement this mechanism for custom commands, that's up to the developer who adds a custom command!

=> Example The particular issue that caused build error for Jim is due to a custom command defined in ctkMacroWrapPythonQt.cmake:

add_custom_command( OUTPUT ${wrapper_init_cpp_filename} ${wrapper_h_filename} DEPENDS ${SOURCES_TO_WRAP} ${CTK_CMAKE_DIR}/ctkWrapPythonQt.py COMMAND ${PYTHON_EXECUTABLE} ${CTK_CMAKE_DIR}/ctkWrapPythonQt.py --target=${TARGET} --namespace=${WRAPPING_NAMESPACE} --output-dir=${CMAKE_CURRENT_BINARY_DIR}/${wrap_int_dir} ${extra_args} ${SOURCES_TO_WRAP} COMMENT "PythonQt Wrapping - Generating ${wrapper_init_cpp_filename}" VERBATIM )

The ${SOURCES_TO_WRAP} variable can contain hundreds of full paths, which of course can easily grow over 8192. For example, a command like this can

be generated (see in the generated CTKWidgetsPythonQt.vcxproj file):

PythonQt Wrapping - Generating generated_cpp/org_commontk_CTKWidgets/org_commontk_CTKWidgets_init.cpp setlocal

C:\S4D\python-install\bin\SlicerPython.exe C:/S4D/CTK/CMake/ctkWrapPythonQt.py --target=CTKWidgets --namespace=org.commontk --output-dir=C:/S4D/CTK-build/CTK-build/Libs/Widgets/generated_cpp/org_commontk_CTKWidgets/ C:/S4D/CTK/Libs/Widgets/ctkActionsWidget.h C:/S4D/CTK/Libs/Widgets/ctkAddRemoveComboBox.h C:/S4D/CTK/Libs/Widgets/ctkAxesWidget.h C:/S4D/CTK/Libs/Widgets/ctkBasePopupWidget.h C:/S4D/CTK/Libs/Widgets/ctkButtonGroup.h C:/S4D/CTK/Libs/Widgets/ctkCheckableComboBox.h C:/S4D/CTK/Libs/Widgets/ctkCheckableHeaderView.h C:/S4D/CTK/Libs/Widgets/ctkCheckableModelHelper.h C:/S4D/CTK/Libs/Widgets/ctkCheckBox.h C:/S4D/CTK/Libs/Widgets/ctkCheckBoxPixmaps.h C:/S4D/CTK/Libs/Widgets/ctkCheckablePushButton.h C:/S4D/CTK/Libs/Widgets/ctkComboBox.h C:/S4D/CTK/Libs/Widgets/ctkCompleter.h C:/S4D/CTK/Libs/Widgets/ctkCollapsibleButton.h C:/S4D/CTK/Libs/Widgets/ctkCollapsibleGroupBox.h C:/S4D/CTK/Libs/Widgets/ctkColorDialog.h C:/S4D/CTK/Libs/Widgets/ctkColorPickerButton.h C:/S4D/CTK/Libs/Widgets/ctkConsole.h C:/S4D/CTK/Libs/Widgets/ctkCoordinatesWidget.h C:/S4D/CTK/Libs/Widgets/ctkCrosshairLabel.h C:/S4D/CTK/Libs/Widgets/ctkDateRangeWidget.h C:/S4D/CTK/Libs/Widgets/ctkDirectoryButton.h … many many more similar lines here … C:/S4D/CTK/Libs/Widgets/ctkMenuComboBoxEventTranslator.h C:/S4D/CTK/Libs/Widgets/ctkPathLineEditEventPlayer.h C:/S4D/CTK/Libs/Widgets/ctkPathLineEditEventTranslator.h C:/S4D/CTK/Libs/Widgets/ctkPopupWidgetEventPlayer.h C:/S4D/CTK/Libs/Widgets/ctkPopupWidgetEventTranslator.h C:/S4D/CTK/Libs/Widgets/ctkRangeSliderEventPlayer.h C:/S4D/CTK/Libs/Widgets/ctkRangeSliderEventTranslator.h C:/S4D/CTK/Libs/Widgets/ctkTreeComboBoxEventPlayer.h C:/S4D/CTK/Libs/Widgets/ctkTreeComboBoxEventTranslator.h if %errorlevel% neq 0 goto :cmEnd :cmEnd endlocal & call :cmErrorLevel %errorlevel% & goto :cmDone :cmErrorLevel exit /b %1 :cmDone if %errorlevel% neq 0 goto :VCEnd

=> Solution

Fortunately, there is a nice, clean solution to this issue: pass the ${SOURCES_TO_WRAP} list to the python script through a file, not through command-line.

There may be other instances where long lists are attempted to pass through command-line parameters, so fixing this issue may not allow us to use arbitrarily long paths yet, but if we fix these issues one by one as they pop up, finally we may end up fixing all of them.

— Reply to this email directly or view it on GitHub https://github.com/commontk/CTK/issues/525#issuecomment-67269513.

The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Partners Compliance HelpLine at http://www.partners.org/complianceline . If the e-mail was sent to you in error but does not contain patient information, please contact the sender and properly dispose of the e-mail.