Bug: Keycloak container fails with 'HTTPS required'

[bvanelli]

Describe the bug

The basic example from Keycloak does not run.

There is actually another "solvable" bug that the library python-keycloak does not work on version 5.7.1 due to a missing import (KeycloakClient), and must be downgraded to 5.6.0

To Reproduce

I have used the exact example provided in the source code here:

from testcontainers.keycloak import KeycloakContainer

def test_keycloak_setup():
    with KeycloakContainer(f"quay.io/keycloak/keycloak:25.0.4") as keycloak:
        counts = keycloak.get_client().users_count()
        print(counts)

I have also tried the example on the repo example_basic.py, and this one also does not run, but fails early since the API for the library has changed.

I understand that keycloak is sometimes a pain to setup with insecure options, and probably it needs extra options, but it's unclear from the docs that that is the case. Docs here:

https://testcontainers-python.readthedocs.io/en/latest/modules/keycloak/README.html

I could offer my help to tackle the issue for a working solution, the question is if the change is for documentation (keycloak is not expected to work out of the box) or configuration. It's also possible that it needs a realm import.

Runtime environment

# Get the operating system information (on a unix os).
$ uname -a
Darwin
# Get the python version.
$ python --version
3.10.11
# Get the docker version and other docker information.
$ docker info
Client:
 Version:    28.3.2
 Context:    desktop-linux
 Debug Mode: false
# Get all python packages.
$ pip freeze
testcontainers==4.12.0
python-keycloak==5.6.0

[walsha2]

I never had an issue with the KeycloakContainer, but recently this is causing a problem. Unclear what changed, but I suspect it might be something related to MacOS and specifically the user's computer. No one else on my team has an issue and it just randomly stopped working for me with no changes to the code. Tested on all versions of >= 4.10.0 and still have the issue.

I only see the problem when using testcontainers to run the Keycloak image. If I run the image (for the same version) on my own with docker, everything works fine and I do not get the HTTPS required error.

Just a hunch: There is some configuration on the user machine that is causing a problem for testcontainers to configure/launch Keycloak container. It has been really frustrating to debug.

Example:

from testcontainers.keycloak import KeycloakContainer


def test_keycloak_setup():
    with KeycloakContainer() as keycloak:
        counts = keycloak.get_client().users_count()

if __name__ == "__main__":
    test_keycloak_setup()

Results in:

keycloak.exceptions.KeycloakPostError: 403: b'{"error":"invalid_request","error_description":"HTTPS required"}'

Configuration:

MacOS Sequoia Version 15.5 (24F74)

>>> python --version
Python 3.12.4

>>> docker --version
Docker version 28.3.2, build 578ccf6

[bvanelli]

Hmmm, that is a good point, I've setup keycloak multiples times before and this time I really struggled (and failed) to find a reason. I'll try tomorrow on Linux to see if the problem persists, and maybe try to find a workaround.

[walsha2]

I'll try tomorrow on Linux to see if the problem persists

I bet you will not see the problem. But, yes please try.

I tested on three other machines (Linux and Mac) with the same codebase, and did not have this issue. So it seems to be machine specific 🤷

Root Cause

After a bunch of snooping into the running docker container that testcontainers bootstraps, I determined the issue. From within the running docker container, I dumped the master realm and see this:

>>> /opt/keycloak/bin/kcadm.sh get realms/master

{
  "realm" : "master",
   ...
  "sslRequired" : "external",
   ...
}

When this is set to external Keycloak enforces SSL for all incoming requests originating from external IP addresses. Requests from private IP addresses are permitted over HTTP.

For some reason, my machine is triggering this.

Workaround

The solution here is to try to patch the master realm to set sslRequired=NONE before the running container tries to perform any operations. We can do this via the following. It is ugly, but it works.

Of course, if you have a realm import already configured you can just patch it there.

Here, test and test are the default username/password that testcontainers uses.

from testcontainers.keycloak import KeycloakContainer


def test_keycloak_setup():
    with KeycloakContainer() as keycloak:
        container = keycloak.get_wrapped_container()
        container.exec_run(
            "/opt/keycloak/bin/kcadm.sh update realms/master -s sslRequired=NONE --server http://localhost:8080 --realm master --user test --password test"
        )
        counts = keycloak.get_client().users_count()
        print(f"Number of users: {counts}")


if __name__ == "__main__":
    test_keycloak_setup()

Comments

Still need to determine why this is not reproducible across machines and only started happening recently, without any code changes.

@bvanelli can you give this a shot and confirm it resolves your issue as well?

[bvanelli]

Works for me @walsha2 👍

[bvanelli]

Interesting enough, just like you said, the issue cannot be reproduced anymore after fixing it once (or coincidence, since I also had to clear the docker cache because of disk space).

[walsha2]

Interesting enough, just like you said, the issue cannot be reproduced anymore after fixing it once (or coincidence, since I also had to clear the docker cache because of disk space).

I was getting all sorts of odd behavior:

• Sometimes a restart of my computer worked, sometimes it did not
• Sometimes clearing anything docker cache related worked, sometimes it did not
• Sometimes the issue happened on the second time, but not the first
• Sometimes it happened every single time

This issue was an absolute pain to debug. Still dont know what about the user's machine state causes this, but I spent way too much time to care anymore. We added the sslRequired=NONE patch to our pytest fixtures because we don't need SSL during tests. So explicitly turning it off for tests is actually preferred (for our functional tests).

[bvanelli]

We have tried reproducing for a couple of hours on a #hackergarten with @eintim, @RobertAsmussen and @Brooks-Alexander, and the only explanation we could reach is that the docker network is misconfigured in certain situations.

Here is how we managed to reproduce it:

The default configuration for keycloak is not to allow external requests without HTTPS:

According to the docs:

External requests Users can interact with Keycloak without SSL so long as they stick to private IPv4 addresses such as localhost, 127.0.0.1, 10.x.x.x, 192.168.x.x, 172.16.x.x or IPv6 link-local and unique-local addresses. If you try to access Keycloak without SSL from a non-private IP address, you will get an error.

This means that if you, by any reason, have launched a container using a different IP address than the ones allowed, you will get the same error.

What we have tried so far offline (did not lead to the error):

• Running the container in multiple Mac Architechtures (M1, M4 and Intel)
• Running in multiple docker versions (24.0.6 and 28.3.2)
• Running multiple Python versions (3.10 to 3.12)
• Running multiple docker platform versions on the same Mac (arch linux/amd64 and linux/arm64)
• Having a pizza break did not help either

If you are reading this and had this issue in your machine, please report what your IP addresses are, in this case you can find the host ip by doing a command similar to:

ifconfig -a | grep bridge100 -A 20 | grep "inet "