RFC: PDL installation - install optionally, segregate other "optional" perl modules

[wwlwpd]

I just tried and with the master and the latest perl that is installed (5.34.0) PDL seems to install fine.

But given it's broken before and is a rather complex bundle, I was thinking about defining an optional set of Perl modules that can be installed altogether from the asgsh environment.

Another benefit is that we can pull out some things from PERL-MODULES that may not necessarily be needed for operational reasons, but more so for development. E.g., App::Perltidy, Perl::Critic.

[jasonfleming]

I like this idea a lot. One concern would be the complexity and maintainability of core/default Perl modules as well as optional Perl modules. If it can be done without a lot of extra code (for example, just commenting or uncommenting things in PERL-MODULES) then I am all for it! :-)

[wwlwpd]

Hmmm....currently we have,

• PERL-MODULES - always installed via cpanm and fully tested
• PERL-MODULES.notest - installed via cpanm --no-test because some modules have flaky tests (verified)
• PERL-MODULES.wget - modules that are grabbed via upstream .tar.gz with wget--no-check-certificate due to the letsencrypt root CA issue

To avoid having a separate set of modules files I do think it'd be a fine idea to differentiate between "required/base" modules and "optional" modules - then whatever installed optional modules as a step once inside of the shell environment of asgsh would use all the files above,but grep for only ones that are "optional"...how's that sound?

FWIW, take a look at how PERL-MODULES.wget is structured; it's not so simple but we can apply I think what you're suggesting in the way that I've outlined.

[jasonfleming]

I like the idea of having a base set of Perl modules for ordinary production deployments. I am also thinking about alternative uses of the ASGS codebase, specifically subsets of functionality acting as microservices. It would be great if we could isolate Perl module requirements to their appropriate use cases, particularly for complex modules like PDL. This will contain the impact of pm build failures, test failures, cert failures etc. Modules can be moved into the base set if/when they are needed in production.

[wwlwpd]

Sounds good, for this issue I will create a way to install modules not essential for deployment but are useful because it's handy to use asgsh as a development environment - some modules off the top of my head for this include:

• App::perltidy (perltidy)
• Perl::Tidy
• Dist::Zilla (CPAN module author helper, but HUGE and not required for downstream use of any module)
• PDL

This will also perhaps beg the usefulness of other script that install other "things" (utilities, libraries, etc) - I want to be careful to not stray into "package manager" territory, but I can envision a day of "install" scripts that we "officially" maintain. I can see something like ffmpeg and similar post production tools falling into this catagory.

For this I'll create some bundle scripts for perl modules based on a large umbrella (e.g., developer-budle (App::perltidy, Dist::Zilla, Perl::Critic, etc); math (PDL), etc).

[wwlwpd]

Removing "needs triage" flag since I think for the most part this is pretty well worked out, feel free to continue comments on specifics or thoughts.

[wwlwpd]

Bump - yes, let's discuss what we'd like to do here. Most of these are available trivially via cpanm <MODULENAME>; I think there is a benefit to having a way to install a set of "developer" related modules - or when installing something via build or fetch, we would necessarily install required modules. For example, if we had a tool that was optional but required PDL, then we'd include cpanm PDL as a dependency step.

[jasonfleming]

If things keep going they way they are going, then I think PDL will end up being a required module. I am hoping to use it to replace the quality control functionality that we can't get from ncap2 due to its unresolved dependencies (#1367). Looking at PERL-MODULES*, I'm actually not sure which ones we are using. And although ImageMagick or gnuplot are not perl modules, those packages could be optional because we are not using them right now.

[wwlwpd]

If things keep going they way they are going, then I think PDL will end up being a required module. I am hoping to use it to replace the quality control functionality that we can't get from ncap2 due to its unresolved dependencies (#1367). Looking at PERL-MODULES*, I'm actually not sure which ones we are using. And although ImageMagick or gnuplot are not perl modules, those packages could be optional because we are not using them right now.

I am all for that, we may want to manage the version (i.e., pin to version known to compile); they don't often break, but sometimes they do. For good measure, I'd like to throw in the OpenMP modules and start making use of them as well. PDL has some auto-threading features that are much touted, so there is a lot to explore there. In the meantime, you may want to subscribe to their mailing list, https://sourceforge.net/projects/pdl/lists/pdl-general. And they have a free book from a long time ago - https://master.dl.sourceforge.net/project/pdl/PDL_2013/PDL-Book/PDL-Book-20130322.pdf?viasf=1