ARM64 docker images

[bdovaz]

At the moment it is a draft as "the only" thing I have done was to edit the existing Dockerfile files to make them linux/arm64 compatible.

Please note:

• The changes in the branch does not take into account anything multi arch, I have simply made changes to make it compile on arm64. Someone with experience could tell how to edit the Dockerfile to include the 2 targets (amd64 and arm64) at the same time or if you decide to duplicate everything then duplicate it ....
• I have changed the version of .NET to LTS
• I changed the powershell binary from linux-alpine-x64 to linux-arm64
• Chromium doesn't work in arm so I had to set PUPPETEER_SKIP_CHROMIUM_DOWNLOAD=true
• shellcheck changed from shellcheck-stable.linux.x86_64 to shellcheck-stable.linux.aarch64
• I could not get dotenv-linter to work (no alpine-arm64 binary): https://github.com/dotenv-linter/dotenv-linter/pull/458
• In order to run "Run pwsh" I had to make a change and include this image: https://github.com/PowerShell/PowerShell-Docker/issues/520#issuecomment-1121468388
• I couldn't get misspell to work as it doesn't target arm (apart from the fact that the repo seems abandoned): https://github.com/client9/misspell
• I have not been able to get rakudo to work as it does not target arm: https://github.com/nxadm/rakudo-pkg
• I have created a simple workflow to test the flavors first but I have not been able to test it from the fork (I think it is ok).

Also I could have missed something in the build log because for example rakudo's build didn't fail but I have seen errors.

I have executed it in the following way (from repo root with Raspberry Pi 4 with Raspberry Pi OS 64 bits):

docker buildx build --platform linux/amd64,linux/arm64 -f Dockerfile .

docker buildx build --platform linux/amd64,linux/arm64 -f flavors/ci_light/Dockerfile .
docker buildx build --platform linux/amd64,linux/arm64 -f flavors/dart/Dockerfile .
docker buildx build --platform linux/amd64,linux/arm64 -f flavors/documentation/Dockerfile .
docker buildx build --platform linux/amd64,linux/arm64 -f flavors/dotnet/Dockerfile .
docker buildx build --platform linux/amd64,linux/arm64 -f flavors/go/Dockerfile .
docker buildx build --platform linux/amd64,linux/arm64 -f flavors/java/Dockerfile .
docker buildx build --platform linux/amd64,linux/arm64 -f flavors/javascript/Dockerfile .
docker buildx build --platform linux/amd64,linux/arm64 -f flavors/php/Dockerfile .
docker buildx build --platform linux/amd64,linux/arm64 -f flavors/python/Dockerfile .
docker buildx build --platform linux/amd64,linux/arm64 -f flavors/ruby/Dockerfile .
docker buildx build --platform linux/amd64,linux/arm64 -f flavors/rust/Dockerfile .
docker buildx build --platform linux/amd64,linux/arm64 -f flavors/salesforce/Dockerfile .
docker buildx build --platform linux/amd64,linux/arm64 -f flavors/scale/Dockerfile .
docker buildx build --platform linux/amd64,linux/arm64 -f flavors/swift/Dockerfile .
docker buildx build --platform linux/amd64,linux/arm64 -f flavors/terraform/Dockerfile .

Pending tasks:

• [ ] Create a CASE in "install > docker" for the cases where you only vary the architecture string in the binary (case of at least bash and powershell at the moment).
• [ ] Add the supported platforms in each descriptor and in each linter of each descriptor.
• [ ] Create the necessary code to read the new properties created in the descriptors and linters to add to the resulting Dockerfile. They are not yet being used.
• [ ] With buildx test each image in each architecture and be sure that build is done correctly.
• [ ] Modify the necessary github workflows to build the images in the required architectures.
• [ ] Create or use a workflow that runs all linters to check that they run on the target platforms correctly.
• [ ] Modify the script that generates the flavors/linters documentation to indicate per architecture what works and what does not.

[echoix]

Thanks for the first draft! What I understood from working with this project the last time is that the Dockerfiles are generated by script, so if there are changes to make in the Dockerfiles, we must change the .megalinter-descriptor.yml like megalinter/descriptors/kubernetes.megalinter-descriptor.yml. Then run the bash build.sh script and see if the Dockerfiles produced match what is expected.

Since you have found the changes needed, a part of the job is done!

With my secondary testing a couple months ago, that didn't finish nor end up with a pull request (after I passed around my raspberry pi 4 wasn't running in a 64 bit OS), I learned that it was possible with the buildx to cross build on my windows amd64 computer, which was quite faster with more cores and way more memory to build simultaneously the aarch64 and amd64. I didn't manage to compile up to the end though, and hadn't had a chance to look at the next problems (solving a problem one layer at the time).

The problem that I didn't have a good idea to fix, that remains, is how to change the .megalinter-descriptor.yml schema to account for the multitude of different conventions of getting and including tools for many architectures. Some might need to be in another format maybe (installed in amd64 vs Docker in aarch64). Sometimes it was a different url scheme, and not everyone names the arm 64 bit architecture the same way. Maybe one of the new active collaborators have some ideas architecture-wise?

[nvuillam]

Thanks for this POC !

That looks nice, but as Dockerfiles are not written but generated (from build.py) , such enhancement must be done with an evolution of the architecture.

I think that having a Dockerfile_arm at the root and Dockerfile_arm files in flavors could be a good way to implement that. And to generate such ARM-specific dockerfiles, I think that adding a new property install_arm in YML descriptor would be the way, and when generating dockerfile:

• If in the linter_descriptor there is an install_arm property, use it
• else use install property

[bdovaz]

@nvuillam I've been analyzing it and from what I understand we have to:

• megalinter/descriptors/*: add the "install_arm" field where it differs from "normal" and add whatever changes.
• megalinter/descriptors/schemas/megalinter-descriptor.jsonschema.json: add the "install_arm" field.
• .automation/build.py: somehow we have to make that script generate the Dockerfile and Dockerfile_arm files (root and flavors) in the same execution no? And in the "normal" case ignore the "install_arm" field but in the case of ARM we will have to search if the "install_arm" field exists and if not use the default "install" field.

[echoix]

For

• megalinter/descriptors/*: add the "install_arm" field where it differs from "normal" and add whatever changes. and
• And in the "normal" case ignore the "install_arm" field but in the case of ARM we will have to search if the "install_arm" field exists and if not use the default "install" field.

I think we need to account that not every tool can or could be supported on first try, so if we could have a way to "check" (like in a checkbox) which tools to include in this arm flavor. This way, a compatibility table could be created showing which tools are ported.

[nvuillam]

@echoix we could add a property arm-compliant on linter descriptors :)

[echoix]

@nvuillam Shouldn't we be not short-sighted and inverse the logic a bit, to say we comply with these platforms, instead of considering a dichotomy amd64/aarch64(arm64)? Especially in docker architectures, I'm pretty sure some tools (flavors) like python-only could use armv7l, which is still the default Raspbian OS (most Raspberry Pis can run in 64 bit mode, but when there was no >4Gb option, there wasn't any advantage to use 64bit).

[echoix]

@bdovaz I could be there for help with this PR if you need :)

[bdovaz]

@echoix I'll appreciate that.

Specifically:

• Adjust the descriptor schema to add the 2 fields because I have no experience creating schemas:
  • arm_compliant: I would make it optional and set it to true by default because there is only one descriptor that doesn't work (dot-env).
  • install_arm: Optional also with a fallback to install if it doesn't exist.
• Make the settings of the "build.py" file because I have tried but I don't understand well how the Dockerfiles generation system works and which part is automatic and which is manual.
  • I have disabled validation temporarily until the json schema changes are done.
  • I have tried to create the Dockerfile_arm file of each flavor "in theory" so that it is generated as the original but I have not managed to generate more than the header.

I also see that the main Dockerfile does not use descriptors no? Because just running the original build.py mashes the changes I made and changes are lost.

[echoix]

For sure @bdovaz, I'm looking into that this afternoon :)

[Kurt-von-Laven]

I agree with @echoix that it may be better long-term to have a supported_architectures (or arches) list. I wonder how many of the linters we use have an asdf plugin with multi-architecture support. I know many of the tools we use have an asdf plugin (or, more often, a package manager with an asdf plugin).

[echoix]

Could I have a little recap on asdf, or how it is used or planned to be used in Megalinter? I receive most of the issues of this repo by email, but I still don't understand it in relation to this project.

I'm glad you think the same way as I thought, as I have an idea of how Megalinter could make use of buildx and multi-arch build efficiently if we know how which platform each tool supports.

[Kurt-von-Laven]

buildx does seem like a good way to go with the caveat that I always optimistically hope there is some approach that involves less work ha ha ha. Is this the same multiarch you are referring to? #870 contains some of my thoughts on asdf's relevance to MegaLinter. asdf is capable of installing all of these tools. It is basically a version manager manager, but it can manage versions of most anything. asdf has expressed interest in supporting multiple architectures in the core if necessary (c.f., asdf-vm/asdf#62), but I don't know which of the plugins presently do. One of the challenges MegaLinter inherently faces is managing many dependencies from many different package ecosystems plus some with bespoke installation methods. It will simplify maintenance if we can eventually install everything that has an asdf plugin (or, better yet, a package manager that has an asdf plugin) through asdf, but the first step would be to simply install one package manager this way. I suppose one way of marrying the ideas would be to use your buildx approach to improve or create asdf plugins since then MegaLinter could leverage that uniformity to install everything with as much multi-architecture support as the plugin (and underlying tool) allows. This might also be a good way to get more hands on deck since the asdf community is presently much larger than MegaLinter's, and I assume many people who don't use MegaLinter share your woes getting things working on various architectures. If there is some tool other than asdf that is already better suited to your goal, I imagine someone there will educate us, particularly since it brings together a diverse cross-section of people using a huge variety of toolchains.

[echoix]

Is this the same multiarch you are referring to?

Nope, the multi arch in Docker, where there an image's manifest contains different images for many cpu architectures. https://docs.docker.com/desktop/multi-arch/

[bdovaz]

For sure @bdovaz, I'm looking into that this afternoon :)

Have you finally started working on it?

Likewise you would have to clearly define how you want to make the changes.

As far as I have understood it would be to put a "supported_architectures" to be able to distinguish the installation commands and that for the future it is better designed.

My doubt is at what level would it be? Below "install", above "install"? I also have doubts with the name "amd64" and "arm64", opinions?

Example 1:

linters:
  - linter_name: ansible-lint
    install:
      supported_architectures:
        amd64:
          pip:
            - ansible-lint==6.0.2
        arm64:
          pip:
            - ansible-lint==6.0.2

Example 2:

linters:
  - linter_name: ansible-lint
    supported_architectures:
      install:
          amd64:
            pip:
              - ansible-lint==6.0.2
          arm64:
            pip:
              - ansible-lint==6.0.2

[echoix]

(Sorry, I started this message on Monday but never had the opportunity to finish it)

For the name of the architecture used, it's sure that we will be using the names used by Docker, it greatly simplifies writing multi-architecture Dockerfiles. Ideally, the same Dockerfile can be used for multiple architectures. That way, buildx build --target can be used directly for all architectures (like many multi-arch image https://docs.docker.com/desktop/multi-arch/)

I drafted out scenarios (without committing anything yet), but couldn't find a way that would be a breaking change. The most common problem is when there is a URL that doesn't use the same architecture names.

One of the scenarios that wouldn't be as breaking, would be that we keep the install property with their pip, docker, apk, etc children fields, but add some architecture-specific overrides. So in this case, supported_architectures, could have the architectures, and any overrides in the same format as install. One simple override I think could be used to make architecture-independent Dockerfiles is having a mapping of the architecture name, for mapping the shellcheck architecture URL that uses aarch64. Otherwise, we could always use inline shell case/esac construct, in the original RUN command.

A scenario that would be breaking, is explicitly having all instructions for each platform, but doesn't help reuse layer caching.

In the same line of thought, common multi-arch Dockerfile patterns use multi-stage Dockerfiles to be able to execute some build in the builder with one architecture, and for the final layer(s), have them in the target instruction set, so this speeds up building.

[echoix]

I completed thinking out the JSON schema, and this should cover everything. I opened a PR to your branch, so it can me merged into this PR.

I started one converrsion of megalinter descriptor, and another day I would need to work with the build.py. We shouldn't need to have multiple files if using buildx and proper multi-stage builds, but in the meantime, with some tricks like the following, we could not even need any overrides, and just adapt the descriptor's install commands.

https://github.com/BretFisher/multi-platform-docker-build/blob/cdd1bb1135554c41f5a4b0f09c8e912eae6a5022/example-tini/Dockerfile#L11-L20

https://github.com/crazy-max/docker-docker/blob/2d6870eecee16f15b7f3ef8140ed975d38cb2866/Dockerfile-20.10#L26-L37

(from https://github.com/BretFisher/multi-platform-docker-build)

As for the changes required in build.py, apart from the logic of building or not the flavors for each platform, to be able to efficiently use caching and parallelizing parts of builds that dont change, the idea would be to have small stages, perhaps having empty stages for tools unavailable on a platform where we could split the fetches and unpack, and/or merge back with two possible paths, one per target (the part where there is a series of copies). If the concept of little independent stages could be implemented, this would allow to have the cache still valid even if one tool has a version upgraded, and only in the copy stages at the end that the cache would be invalidated. See the idea below from this post https://itnext.io/building-multi-cpu-architecture-docker-images-for-arm-and-x86-1-the-basics-2fa97869a99b

Multi-Arch Dockerfile with Multi-stage Build

You might not always find the base image that supports all target platforms that you plan to build. With the multi-stage build, it’s possible to conditionally select different base images depending on the current building platform (by checking TARGETOS & TARGETARCH ARG) in your Dockerfile.

Here is a complete Dockefile example:

# Declare stage using linux/amd64 base image
FROM --platform=linux/amd64 docker.elastic.co/elasticsearch/elasticsearch:6.8.22 as stage-amd64
# Commands run only for linux/amd64 target platform
RUN yum -y install sudo zip
RUN find modules -type d -name "x-pack-*" -exec rm -r {} +
COPY --chown=elasticsearch:elasticsearch component/elasticsearch.yml /usr/share/elasticsearch/config/
# Declare stage using linux/arm64 base image
FROM --platform=linux/arm64 data61/elasticsearch:6.8.22 as stage-arm64
# Commands run only for linux/arm64 target platform
RUN apt-get update && apt-get install -y --no-install-recommends sudo zip && rm -rf /var/lib/apt/lists/*
COPY --chown=elasticsearch:elasticsearch component/elasticsearch-arm64.yml /usr/share/elasticsearch/config/elasticsearch.yml 
# Declare TARGETARCH to make it available
ARG TARGETARCH
# Select final stage based on TARGETARCH ARG
FROM stage-${TARGETARCH} as final
# Commands run at final stage for all target platforms
ADD component/setup.sh /setup.shCOPY --chown=elasticsearch:elasticsearch component/wn_s.pl /usr/share/elasticsearch/config/analysis/
COPY --chown=elasticsearch:elasticsearch component/regionSynonyms.txt /usr/share/elasticsearch/config/analysis/
CMD ["/setup.sh"]

This reading kinda has diagrams of what I'm trying to explain too https://www.docker.com/blog/faster-multi-platform-builds-dockerfile-cross-compilation-guide/ or this that explains more what is allowed to write and why https://www.docker.com/blog/advanced-dockerfiles-faster-builds-and-smaller-images-using-buildkit-and-multistage-builds/

The changes needed for this is simply handling the way the array of dockerfile commands in the descriptor are split, so that multiple froms (with all their commands) could stay together instead of checking "if starts with FROM, bump the complete line at the dockerfile's header, else if check if the line starts with arg, else treat it normally".

[bdovaz]

@echoix I have merged your PR and updated my changes.

I am encountering a problem and that is that there are in several cases where the changes that need to be made is not at the linter level but at the descriptor level.

Look at this change:

https://github.com/oxsecurity/megalinter/pull/1553/files#diff-614f3cce8a8bb565c2d879cc11efa6e21100fb2d32ba39ad2ab25aa446a67be8

Right now it obviously does not comply with the schema because the changes you made only apply to the level of a linter, but somehow we have to accommodate the whole descriptor.

Can you help? Thanks.

[echoix]

Oh, I see now how it is used there, it wasn't clear why there was a semi-duplicate. With this, I understand how it is used and why we would need it there too.

I'll be able to update it this way. Leave me a bit of time though, I'm not at my computer :)

[bdovaz]

I have also noticed that in many cases the change is just modifying the architecture in the binary file name and it is quite ugly to have to duplicate all the commands just for that minimal change.

If somehow there was a way to pass arguments to the install commands like:

install:
  dockerfile:
    command:
      - ARG PWSH_VERSION='latest'
      - ARG PWSH_DIRECTORY='/opt/microsoft/powershell'
      - |
        RUN mkdir -p ${PWSH_DIRECTORY} \
            && curl --retry 5 --retry-delay 5 -s https://api.github.com/repos/powershell/powershell/releases/${PWSH_VERSION} \
                | grep browser_download_url \
                | grep linux-{{arch}} \
                | cut -d '"' -f 4 \
                | xargs -n 1 wget -O - \
                | tar -xzC ${PWSH_DIRECTORY} \
            && ln -sf ${PWSH_DIRECTORY}/pwsh /usr/bin/pwsh
supported_platforms:
  platform:
    - linux/amd64
    - linux/arm64
  install_override:
    - platform: linux/amd64
      install:
        dockerfile:
          args:
            - arch=alpine-x64
    - platform: linux/arm64
      install:
        dockerfile:
          args:
            - arch=arm64

Where those arguments are replaced by the templating engine used (if one is used) when copying these descriptor and linter fragments to the resulting Dockerfile.

It's a matter of analyzing what pays off in terms of maintenance in the future, because duplicating all the code for a minimal change is very error prone in the future that you make the change in one place but not in the other.

This will not be the case in all cases, in other cases there will probably be big changes, so this system of command / args would not be applicable in all cases.

[echoix]

I see very much where the problem is. When working on the schema, I tried to find a way to accept two types of data, like a list and optional objects. What you suggest (that I think is quite natural, but not easy on the parsing part), is exactly that. The dockerfile parameter in the install node is a list of string. To keep the existing schema (and thus compatibility intact), I decided to leave it as is.

However, nothing stops us from use a case instruction inside the run command to do that. It's even better, since there is only one place to update. I left the override for extreme cases where a completely different type of install would be needed, or some kind of workarounds could be used. I had two links with examples in my previous comment:

https://github.com/BretFisher/multi-platform-docker-build/blob/cdd1bb1135554c41f5a4b0f09c8e912eae6a5022/example-tini/Dockerfile#L11-L20

# translating Docker's TARGETPLATFORM into tini download names
RUN case ${TARGETPLATFORM} in \
         "linux/amd64")  TINI_ARCH=amd64  ;; \
         "linux/arm64")  TINI_ARCH=arm64  ;; \
         "linux/arm/v7") TINI_ARCH=armhf  ;; \
         "linux/arm/v6") TINI_ARCH=armel  ;; \
         "linux/386")    TINI_ARCH=i386   ;; \
    esac \
 && wget -q https://github.com/krallin/tini/releases/download/${TINI_VERSION}/tini-static-${TINI_ARCH} -O /tini \
 && chmod +x /tini

and https://github.com/crazy-max/docker-docker/blob/2d6870eecee16f15b7f3ef8140ed975d38cb2866/Dockerfile-20.10#L26-L37

# translating Docker's TARGETPLATFORM into tini download names
RUN case ${TARGETPLATFORM} in \
         "linux/amd64")  TINI_ARCH=amd64  ;; \
         "linux/arm64")  TINI_ARCH=arm64  ;; \
         "linux/arm/v7") TINI_ARCH=armhf  ;; \
         "linux/arm/v6") TINI_ARCH=armel  ;; \
         "linux/386")    TINI_ARCH=i386   ;; \
    esac \
 && wget -q https://github.com/krallin/tini/releases/download/${TINI_VERSION}/tini-static-${TINI_ARCH} -O /tini \
 && chmod +x /tini

(see readme of https://github.com/BretFisher/multi-platform-docker-build)

For example, it could be something like:

descriptor_id: POWERSHELL
descriptor_type: language
descriptor_flavors:
  - dotnet
file_extensions:
  - ".ps1"
  - ".psm1"
  - ".psd1"
  - ".ps1xml"
  - ".pssc"
  - ".psrc"
  - ".cdxml"
# Reference: https://docs.microsoft.com/en-us/powershell/scripting/install/installing-powershell-core-on-linux?view=powershell-7
# Slightly modified to always retrieve latest stable Powershell version
# If changing PWSH_VERSION='latest' to a specific version, use format PWSH_VERSION='tags/v7.0.2'
install:
  dockerfile:
    - ARG PWSH_VERSION='latest'
    - ARG PWSH_DIRECTORY='/opt/microsoft/powershell'
    - \# Use BuildKit to help translate architecture names
    - ARG TARGETPLATFORM
    - |
      RUN ML_PWSH_ARCH=$(case ${TARGETPLATFORM:-linux/amd64} in \
            "linux/amd64")   echo "linux-alpine-x64"  ;; \
            "linux/arm64")   echo "linux-arm64"       ;; \
            "linux/arm/v7")  echo "linux-arm32"       ;; \
            "linux/arm")     echo "linux-arm32"       ;; \
            *)               echo "linux-x64"         ;; esac) \
          && echo "ML_PWSH_ARCH=$ML_PWSH_ARCH" \
          mkdir -p ${PWSH_DIRECTORY} \
          && curl --retry 5 --retry-delay 5 -s https://api.github.com/repos/powershell/powershell/releases/${PWSH_VERSION} \
              | grep browser_download_url \
              | grep ${ML_PWSH_ARCH} \
              | cut -d '"' -f 4 \
              | xargs -n 1 wget -O - \
              | tar -xzC ${PWSH_DIRECTORY} \
          && ln -sf ${PWSH_DIRECTORY}/pwsh /usr/bin/pwsh
supported_platforms:
  platform:
    - linux/amd64
    - linux/arm64
    - linux/arm/v7
    - linux/arm
linters:
  # Internal powershell linter
  - class: PowershellLinter
    linter_name: powershell
    linter_url: https://github.com/PowerShell/PSScriptAnalyzer
    linter_image_url: https://github.com/PowerShell/PSScriptAnalyzer/raw/master/logo.png
    linter_rules_configuration_url: https://github.com/PowerShell/PSScriptAnalyzer#explicit
    linter_rules_inline_disable_url: https://github.com/PowerShell/PSScriptAnalyzer#suppressing-rules
    config_file_name: .powershell-psscriptanalyzer.psd1
    version_extract_regex: "(\\d+) *(\\d+) *(\\d+)"
    examples:
      - 'pwsh -NoProfile -NoLogo -Command "Invoke-ScriptAnalyzer -EnableExit -Path myfile.ps1"'
      - 'pwsh -NoProfile -NoLogo -Command "Invoke-ScriptAnalyzer -EnableExit -Settings .powershell-psscriptanalyzer.psd1 -Path myfile.ps1"'
    # If changing PWSH_VERSION='latest' to a specific version, use format PWSH_VERSION='tags/v7.0.2'
    install:
      dockerfile:
        - ARG PSSA_VERSION='latest'
        - RUN pwsh -c 'Install-Module -Name PSScriptAnalyzer -RequiredVersion ${PSSA_VERSION} -Scope AllUsers -Force'
    supported_platforms:
      platform:
        - linux/amd64
        - linux/arm64
    ide:
      vscode:
        - name: VsCode PowerShell extension
          url: https://marketplace.visualstudio.com/items?itemName=ms-vscode.PowerShell

It sets an env variable inline, just some plain shell tricks.

[bdovaz]

@echoix I have merged your two PRs, thanks!

I also don't know how you tested the bash one as you mentioned, plus I see you listed more platforms than amd64, arm64 and arm/v7.

[echoix]

I also don't know how you tested the bash one as you mentioned,

That one (the bash case in the RUN instruction) I didn't formally test, since it will be possible to test more thoroughly when we will be doing the build tests. There is no apparent reason to not work, it's theoretically correct.

plus I see you listed more platforms than amd64, arm64 and arm/v7.

I consider you're talking about bash-exec, where I listed more architectures. Here I listed all the architectures that the image supports, since it's native bash in this linter's case. My reasoning was to include all the supported architectures that the linters support in the descriptors support, so that a user with who would want a different platform could build it. We could (or not) create the Dockerfile(s) for every architecture each time, even if we only build the containers for the platforms and flavours that are common enough. (I suspect that the Python flavour could be such a candidate, so we could leave the door open). Having more supported_platforms doesn't mean that we will build containers for each. Since to know the installation steps we go open each tool's website/repo and look at their releases, we know by the same time what platforms it supports.

[bdovaz]

@echoix I just made a commit adding the supported platforms and the CASE in the cases that it is possible.

It is true that my experience is in the world of .NET so I have several doubts in the descriptors/linters that add dependencies through npm, pip and some other cases because I do not know on which platforms nodejs or python works.

I also have another doubt and that is that initially when I managed to build the Dockerfile in arm64 with the few necessary changes is actually a lie because there are cases in which the linters only download a file, unzip it and copy it to a directory but that does not mean that in execution on that platform that binary is correct and will work.

That is why I think we should have a workflow that runs all the linters on each platform to be sure that they have the correct binary and it is compatible.

Finally, I have moved the task list to the first post and added some more tasks that I have mentioned. @echoix can you help me with the tasks? I can see that this is quite a lot of work.

@nvuillam now that you have already released v6 I don't know if you are more relaxed and you can also give us a hand.

Thanks!

[SoumayaMauthoorMOJ]

I don't have the skills for making changes but as I told @echoix happy to do some testing on my M1 once ready! I've been using the oxsecurity/megalinter-python:v6 docker image and linting+formatting works as expected apart from the speed (I estimate about 5 times slower than on Mac Intel)

[github-actions[bot]]

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.

If you think this pull request should stay open, please remove the O: stale 🤖 label or comment on the pull request.

[bdovaz]

As I said, I need support in this PR to unlock it because it is quite a lot of work and I am not able to tackle it alone.

[Kurt-von-Laven]

Is there a way this PR can be decomposed into more bite-sized pieces that don't need to all be landed atomically? My experience in the open-source world has been that single large atomic units of work often tend to be indefinitely deprioritized.

[echoix]

I've been working on the build.py this weekend. I thought of an element of this PR (that is still left to be done) that could be part of a separate PR and be used before this is ready that could reduce the risk of changing too much.

If the project could already switch to use BuildKit (the Docker backend) either with a env variable or changing the places where docker build is called to docker buildx (preferable, as we can use the --platform argument), it would help, as it will be necessary to cross-build the containers for other architectures. Like if it could be running with it for a week or two only for the amd64 images it would make it easier to add another platform to it.

[nvuillam]

The docker build part is inherited from super-linter, good old ugly bash, so if you want to make propositions to enhance it, it will be greatly appreciated :)

[github-actions[bot]]

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.

If you think this pull request should stay open, please remove the O: stale 🤖 label or comment on the pull request.