Deprecation of `lxplus7`

[DickyChant]

qiansitian@sqmbp16 ~> ssh lxplus7
ssh: Could not resolve hostname lxplus7.cern.ch: nodename nor servname provided, or not known

Today I realized that lxplus7 it no longer there...

• So we can still run gridpack generation in "local" mode with containers
• But I foresee some issues if we try to use condor on lxplus with el7 OS.

For MadGraph gridpack generation, the issue is that we need to setup a CMSSW as working environment on the fly, which means

1. if we use the run in one go option, we better need to run everything inside an environment matched to the target scram_arch, saying "el7" for ul.
2. we could split them and try some trick to submit condor jobs from a environment different from target scram_arch, saying we run "CODEGEN" first in a container, then exit and submit... but this option seems to be not working due to the current implementation (https://github.com/cms-sw/genproductions/blob/3c15d3baaa44018390c75b685cf07c9b2988774e/bin/MadGraph5_aMCatNLO/gridpack_generation.sh#L429, at least i cannot make it work)

I have 3 solutions in mind right now:

• We could use https://github.com/cms-sw/cmssw/pull/44900 that actually runs event generation from gridpack with a container, this is already there and we just need to bump up versions, no action needed.
• I started a container today built upon 'dask-lxplus' that allows us to submit condor jobs to cern condor pool from it https://gitlab.cern.ch/cms-genprod-containers/lxplus_genprod_condor with necessary libraries. We can submit jobs from it, I can give detailed instructions if anyone is interested. But I got weird issues when querying jobs from inside the container due to some IP issue (typical container thing and there should be a workaround)
• We could also rely on CMSConnect, @celia-lo made some progress, but again that is also not quite reliable...

Those are the options that I feel are feasible (some are already available, some need a little bit of work), but I'd like to go with recommendation from GEN since some of them are not really fitting the roadmap.