Pre Commands before pulling plugins

[hype8912]

Is there a way to run pre commands before megalinter attempts to pull the plugins?

I am using MegaLinter behind a proxy that requires setting the proxy address variables and certificates before communicating with the internet. For the linters I can do this with Pre commands but when MegaLinter pulls the plugins there isn't a way to execute anything before, that I've found in the documentation.

[hype8912]

I'd like to expand this to also having a remote MEGALINTER_CONFIG file. The call to pull this file is executed before any certificate information is applied. Maybe the ability to ignore SSL errors would be a good work around option.

[nvuillam]

@hype8912 I fear to create breaking changes if I change the execution order :/

But I'm trying to add a tag system that will allow your commands to be especially run before loading plugins :) (if not tag, it's still run after so no breaking change)

I'll let you know when released in beta version so you can test :)

Example:

  - command: echo "Some command called before loading MegaLinter plugins"
    cwd: workspace   # Will be run at the root of the workspace (usually your repository root)
    continue_if_failed: False  # Will stop the process if command is failed (return code > 0)
    tag: before_plugins # Tag indicating that the command will be run before loading plugins

[hype8912]

I'm finally back to work. I'm attempting to run the tag with a plugin but it still throws an SSL Certificate Verify error when attempting to make the call to GitHub for pulling the plugin. When I run it in DEBUG log level I can see the [pre] calls in the log setting the certificates with no errors but then the python call to pull the plugin gets the error.

[nvuillam]

@hype8912 the call to load the remote plugin descriptor is a simple https call

            if plugin.startswith("https://"):
                r = requests.get(plugin, allow_redirects=True).content

Do you think it would require more ?

[hype8912]

I'm confused with it because I'm setting the environment variables REQUESTS_CA_BUNDLE and CURL_CA_BUNDLE which is supposed to work with urllib3. From my understanding of python is requests is supposed to use urllib3. I wonder if you did a check for REQUESTS_CA_BUNDLE being set. If set you pass the path to the verify value of the request otherwise just call it as you have above. Below is some code to show an idea. Feel free to do whatever you want to do with it.

if plugin.startswith("https://"):
  if "REQUESTS_CA_BUNDLE" in os.environ:
    r = requests.get(plugin, allow_redirects=True, verify=os.getenv("REQUESTS_CA_BUNDLE")).content
  else
    r = requests.get(plugin, allow_redirects=True).content

A better option if you want to continue with it is maybe a new CONFIG variable for a string to pass to the requests.get verify argument if it's set. If a user want to pass "False" to ignore SSL requests they can or if they have a path as I do I could pass the path into the variable. If the CONFIG variable is not set then it just calls requests.get as you currently have it. Some ideas.

[nvuillam]

@hype8912 do you need something like lines 82 to 110 here ?

https://github.com/oxsecurity/megalinter/blob/8ec5e3ef2d860b1bdf705e60f0e38a36d75c3e55/megalinter/reporters/GitlabCommentReporter.py#L82

[hype8912]

That looks like it would work for calling plugins and the remote config file

[nvuillam]

Hmmm ok I'll have to factorize that... Not very available currently, so meanwhile maybe you should host your plugin on GitHub This is the first time that someone wants to use a private plugin ^^ Out of curiosity, what is it about ? :)

[hype8912]

It's not a private plugin. Because the GitLab runners are behind a corporate proxy, any communication to the internet requires the proxy environment variables and certificates be set on the image. Since the plugin is hosted on GitHub, you have to set all the proxy information before the python calls in MegaLinter can work to request the files from the internet.

If it would break anything or mess up the architecture then we can close the issue and move on. I don't want to cause significant issues for something that is probably rarely used.

[nvuillam]

@hype8912 no no , it's a valid use case , we'll make it work :) Another workaround idea before we make the update: host within your network the descriptor of the plugin ? MegaLinter does not need more than a single file to load a plugin :)

And again another workaround: if you copy the plugin descriptor in your repo, you can refer to it using file://

Example: file://path/to/mega-linter-plugin-custom/custom.megalinter-descriptor.yml

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.

If you think this issue should stay open, please remove the O: stale 🤖 label or comment on the issue.

[hype8912]

@nvuillam I figured out an alternative to this that allows me to run my pre-scripts first to configure the image with the proper certs and proxy information before megalinter runs. I essentially pull the image and override the entrypoint setting it to "". This allows me to run my before script stuff. Then in the script section I can call /entrypoint.sh to execute megalinter.

gitlab-ci.yml Example

  my-megalinter-job:
    image:
      name: oxsecurity/megalinter-ci_light:v8
      entrypoint: [""]
    before_script:
      - set-proxy-and-certs-script
    script:
      - /entrypoint.sh

[echoix]

Nice workaround! Where does the before_script executes? If its inside the docker container, would creating a new entrypoint script wrapper, that ends by calling the original entrypoint work too?

[hype8912]

Yes. The before_script executes inside the container. A wrapper script would be very cool. I know calling the entrypoint.sh was a gamble and could miss some setup stuff by doing it this way. A wrapper script would make sure it's executed the way you would expect it to run.

[echoix]

Like if your set-proxy-and-certs-script.sh file ends with

./entrypoint.sh

Is it enough to work?

Then, set your set-proxy-and-certs-script.sh as the entrypoint, or change the names

[hype8912]

I'll give that idea a shot. I appreciate the idea of the new entrypoint script.