Have install-devtools.py support building CMake with OpenSSL support for https submission

[bkochuna]

Hey Ross,

We're transitioning to https for our CDash submissions. I had some trouble figuring this out, but eventually was able to get it to work (with version 3.3.2).

What I had to do was configure and build CMake with OpenSSL support.

After some cursory examination of tribits/devtools_install/install-cmake.py it looks like enabling this option is not supported by the command that configure's cmake. Specifically in install-cmake.py there is:

  def doConfigure(self):
    createDir(self.cmakeBuildBaseDir, True, True)
    echoRunSysCmnd(
      "../"+self.cmakeSrcDir+"/configure "+\
      " "+self.inOptions.extraConfigureOptions+\
      getParallelOpt(self.inOptions, "--parallel=")+\
      " --prefix="+self.inOptions.installDir,
      extraEnv={"CXXFLAGS":"-O3", "CFLAGS":"-O3"},
      )

In examining the CMake-3.3.2 tarball, the help from configure and bootstrap do not seem to list any options for OpenSSL. The help is:

$ ./configure --help
Options: [defaults in brackets after descriptions]
Configuration:
  --help                  print this message
  --version               only print version information
  --verbose               display more information
  --parallel=n            bootstrap cmake in parallel, where n is
                          number of nodes [1]
  --enable-ccache         Enable ccache when building cmake
  --init=FILE             load FILE as script to populate cache
  --system-libs           use all system-installed third-party libraries
                          (for use only by package maintainers)
  --no-system-libs        use all cmake-provided third-party libraries
                          (default)
  --system-curl           use system-installed curl library
  --no-system-curl        use cmake-provided curl library (default)
  --system-expat          use system-installed expat library
  --no-system-expat       use cmake-provided expat library (default)
  --system-jsoncpp        use system-installed jsoncpp library
  --no-system-jsoncpp     use cmake-provided jsoncpp library (default)
  --system-zlib           use system-installed zlib library
  --no-system-zlib        use cmake-provided zlib library (default)
  --system-bzip2          use system-installed bzip2 library
  --no-system-bzip2       use cmake-provided bzip2 library (default)
  --system-libarchive     use system-installed libarchive library
  --no-system-libarchive  use cmake-provided libarchive library (default)

  --qt-gui                build the Qt-based GUI (requires Qt >= 4.2)
  --no-qt-gui             do not build the Qt-based GUI (default)
  --qt-qmake=<qmake>      use <qmake> as the qmake executable to find Qt

  --sphinx-man            build man pages with Sphinx
  --sphinx-html           build html help with Sphinx
  --sphinx-qthelp         build qch help with Sphinx
  --sphinx-build=<sb>     use <sb> as the sphinx-build executable
  --sphinx-flags=<flags>  pass <flags> to sphinx-build executable

Directory and file names:
  --prefix=PREFIX         install files in tree rooted at PREFIX
                          [/usr/local]
  --datadir=DIR           install data files in PREFIX/DIR
                          [share/cmake-3.3]
  --docdir=DIR            install documentation files in PREFIX/DIR
                          [doc/cmake-3.3]
  --mandir=DIR            install man pages files in PREFIX/DIR/manN
                          [man]

In the base CMakeLists.txt file in the tarball there's:

$ grep OPENSSL ./CMakeLists.txt
    option(CMAKE_USE_OPENSSL "Use OpenSSL." OFF)
    mark_as_advanced(CMAKE_USE_OPENSSL)
    if(CMAKE_USE_OPENSSL)
    set(ENABLE_OPENSSL ${CMAKE_USE_OPENSSL} CACHE INTERNAL "Enable use of OpenSSL")

So if a non-configre/bootstrap (e.g. a pure CMake configure) is done then the CMAKE_USE_OPENSSL option can be enabled.

This also adds a dependency that the target platform have OpenSSL libraries already installed.

Bottom line is, it would be nice to figure out a way to tell the install_devtools.py script to build CMake with OpenSSL support. Maybe this is the case and I just don't know enough python to figure it out!

[bartlettroscoe]

@bkochuna, is there an option for this in newer CMake versions? CASL VERA needs to upgrade CMake anyway so if a newer version of CMake supports this, let's just make that the default and then take advantage of that in the install-cmake.py script (automatically if it can detect that openssl is present on that machine).

[bkochuna]

Ross,

Yes, but it appears to only be supported when building CMake with CMake, rather than with the configure or bootstrap scripts. If building CMake with CMake then it is just the CMake configure option -DCMAKE_USE_OPENSSL=ON

[bartlettroscoe]

Yes, but it appears to only be supported when building CMake with CMake, rather than with the configure or bootstrap scripts. If building CMake with CMake then it is just the CMake configure option -DCMAKE_USE_OPENSSL=ON

Okay, I think what that means is that install-cmake.py would need to be upgraded to have a mode where it looks to see if there is already some cmake version on the system (such as in the current path) and if so, then it would use that cmake to configure the target cmake from source and then build.

Would you mind taking a crack at revising install-cmake.py on a topic branch and seeing how that goes? I could then review it on the branch in a PR and give it a try to see if it works. Then we should just make install-cmake.py use the cmake configure and always configure with Yes, but it appears to only be supported when building CMake with CMake, rather than with the configure or bootstrap scripts. If building CMake with CMake then it is just the CMake configure option -DCMAKE_USE_OPENSSL=ON if we can determine if it is on the system (or just provide a --with-openssl option to make this explicit).

I kind of like the auto detect since then we would not have to modify the install_devtools.py script at all. It would just call install-cmake.py like it does now and if some cmake version was already in the path and openssl was on the system, then it would try to configure with cmake with -DCMAKE_USE_OPENSSL=ON and the installed cmake would magically support https submits to CDash.

What do you think?

[bkochuna]

Yeah that's a nice way to handle it, and similar to what I was thinking. I can try and take a crack at this. Probably can have something for review next week.

[bartlettroscoe]

@bkochuna,

Yeah that's a nice way to handle it, and similar to what I was thinking. I can try and take a crack at this. Probably can have something for review next week.

Sounds good.

If you don't mind, could you give the process described at:

• https://github.com/TriBITSPub/TriBITS/blob/master/CONTRIBUTING.md

at try? Basically it just requests that you put the issue ID (#220) in the commits on the branch. However, I don't expect automated tests for install-cmake.py. That would be hard to do and risk associated with a broken script is low. We can just manually test it.

[bkochuna]

Hi Ross,

I'm trying to push my changes for this to a new branch not sure if I have permissions to create/push a branch. I see:

$ git push -u origin issue_220
Username for 'https://github.com': bkochuna
Password for 'https://[email protected]':
remote: Permission to TriBITSPub/TriBITS.git denied to bkochuna.
fatal: unable to access 'https://github.com/TriBITSPub/TriBITS.git/': The requested URL returned error: 403

Once I push the branch I figure we continue the discussion of the changes in the pull request?

[bartlettroscoe]

@bkochuna,

I'm trying to push my changes for this to a new branch not sure if I have permissions to create/push a branch.

Can you just push the branch to your GitHub fork of TriBITS and then submit a PR in this GitHub repo? If you check the to allow maintainers to modify the branch (which I think is the default), then I can edit it just as well as if it was in the main TriBITS git repo.

[bkochuna]

Yep! I'll give that a shot.