Look on how to retain older images in case latest is broken

[fzipi]

From #62 , we should investigate if we can have old images laying around in hub.

Unfortunately, all images on docker-hub (see: https://hub.docker.com/r/owasp/modsecurity-crs/tags) have just been overridden and it is not possible to pull an older variant. This should definitely be fixed to allow (automatic) rollbacks in case a newer image is broken.

[arnegroskurth]

The images have been overridden again and caused an outage of one of our services due to changed behaviour (not yet diagnosed).

This makes the images unusable for any productive scenario.

[fzipi]

Hi @arnegroskurth ! Sorry to hear that. We can try maybe adding a timestamp to the images, like 3.3.2_2022041401. But then "latest 3.3.2" should point to the one with the last date.

[arnegroskurth]

I think the root of the problem here is that you are trying to re-use the the version from something else (maybe ModSecurity itself?) to also version the container images. I think they have to be versioned separately as there is a whole lot of configuration-files, image definitions and versions of other involved software (apache, nginx, etc.) that has an influence on the API of the images. Having an independent version solves these problems and I guess is also what people generally expect.

[matthiasbaldi]

We had the same problem on our side - We got cached by the fact, that the underlying Nginx Docker image also changed and the new template paths came in. May you could also try to decouple the two versions. It should be allowed to add i.e. 3.3.2-134, then you have in the first part the modsecurity version and behind a buildnumber/releasenumber for the Dockerfile. Or decouple it completely and set your own version numbers and you change the minor whenever the version of modsecurity changes. And then write it into every changelog which modsecurity version is included. Any solution would help to stabilize the deployment procedure.

[fzipi]

Thanks all for the comments again. I will try to do some tests with my personal repos and see if this can be achieved easily.

The solution I'll try is to add a timestamp like nginx-2022041901, etc. Any tag that can be referred to in a static way, and let people control when to update.

[arnegroskurth]

I think even though that would already been an improvement from the current situation, it is not really ideal as the update process is much less straight forward with that solution. Semantic versioning is all about categorizing and communicating changes to your software component which is completely gone when you effectively version the component by date. With just a meaningless date as versioning information, I, as a user of the component, have to go through all changes when updating the component to be able to decide whether I also have to change my system due to a breaking change or can rely on the component still function like before. And for what?

So may I ask what your reasoning for that is?

[fzipi]

Sorry, the reasoning for the tag, or for updating in general?

[arnegroskurth]

For preferring date-strings added to versions from another component (the CRS, I presume), rather then semantically versioning this component independently.

[fzipi]

Sure!

[fzipi]

Semantic versioning is why we would try 🤷

[fzipi]

In any case, we receive PRs if anyone here wants to help.

[matthiasbaldi]

A PR is a bit hard while it is not clear in which direction you wanna go. Personally I think a dedicated versioning for the images is better. This is also the way @arnegroskurth would prefer, when I understood him correctly. But maybe I can show with the following lines how to solve the problem with Github Actions. Just as an idea 💡.

To reach something like this you could replace the static version in the Github Action with a "dev" flag for example. https://github.com/coreruleset/modsecurity-crs-docker/blob/79b4c70970baad8c18eb6866c3622b230bb0bc24/.github/workflows/buildimage.yml#L16-L17

Then you can create a new Github Action with more or less the similar content but changed triggers. In my opinion something like the following example to build only new Github releases:

on:
  release:
    types: [published]

https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#release

Then in the action itself you could checkout the source from the created Git tag

steps:
    - name: Checkout
      uses: actions/checkout@v2
      with:
        ref: ${{ github.event.release.tag_name }}

And then you can build the docker images with the given tag_name instead of the fixed version as before. Do not forget to also build and push the latest tag for those guys which want to setup their dev/test infrastructure on latest.

Then every time you want to release a new version you can create a new Github Release in which you can describe which CRS/Modsecurity release was used and then the pipeline would automatically run the release container build/push procedure. In this way everybody can check the release list and can check if the new release needs some internal changes.

To improve the transparency more, may also consider to not use the latest image from Modsecurity. https://github.com/coreruleset/modsecurity-crs-docker/blob/79b4c70970baad8c18eb6866c3622b230bb0bc24/nginx/Dockerfile#L1 In that way you will have a commit in which you change the the Modsecurity version to an newer tag. But maybe that is something you may want to simplify the update procedure - here I don't understand your workflow or why it is on latest per default.

@arnegroskurth @fzipi wdyt? 🙃 To complicated? Or would that help?

[fzipi]

Ok, let me think about this for a bit, and will circle back. Using just latest was deprecated because we have "two" latest (one for apache and one for nginx)

[fzipi]

I still need to process this, and we will get a solution. In the meantime, looks like this workaround might be useful.

[fzipi]

Didn't forgot about this, we are kind of overwhelmed with the bugbounty program on CRS these couple of weeks.

[fzipi]

Ok, started with something. This is the PR I have for the base docker: https://github.com/coreruleset/modsecurity-docker/pull/140. If you have any comments, please add them there! the idea is to replicate it here in a similar way.

[fzipi]

So, the corresponding fix in this repo is linked now: https://github.com/coreruleset/modsecurity-crs-docker/pull/88.

Now we are building on top of a specific version from the base repo. So each build will be unique and long term, if you use the proper tag.

Let me know what you think @matthiasbaldi @arnegroskurth.

[matthiasbaldi]

Nice. Thank you. I this will help to get a more stable setup 👍👌

[fzipi]

Changes are live now. Can you test them and see if they work for your use case?