Secure communication standard for IaaS infastructure

[markus-hentsch]

As a customer I expect the communication channels to and between infrastructure components of an SCS cloud to be secured appropriately, so that in-transit data is protected. As a CSP I want to establish secure communication channels adhering to the corresponding SCS standard the users can rely on.

Contents

• [x] Overview and classification of communication channels
• [x] References to the OpenStack Security Guide where applicable
• [x] TLS version and cipher restrictions

Definition of Done:

Please refer to scs-0001-v1 for details.

• [x] Proposal has been written with name of the form scs-xxxx-v1-slug.md (only substitute slug)
• [x] Proposal has the fields status, type, track set
• [ ] Proposal has been voted upon in the corresponding team
• [ ] Status has been changed into Draft, file renamed: xxxx replaced by document number
• [ ] If applicable: test script has been written (this item may be moved into a separate issue so long as the state is Draft)

[markus-hentsch]

Major communication channels that need to be secured:

#	Classification	Details	Solution
1	Database backend traffic	Replication and sync between database instances	SSL/TLS
2	Database frontend traffic	Communication between OpenStack services and databases	SSL/TLS
3	Message queue traffic	Message queue communication between OpenStack components as provided by oslo.messaging	SSL/TLS
4	Internal API communication	HTTP traffic to services registered as internal endpoints in the Keystone service catalog	SSL/TLS
5	External API communication	HTTP traffic to services registered as external endpoints in the Keystone service catalog	SSL/TLS
6	Nova VM migration	Nova VM migration data transfer traffic between compute nodes	QEMU-native TLS

Resources:

• 1, 2: OpenStack Security Guide: Database transport security
• 3: OpenStack Security Guide: Messaging transport security
• 4, 5: OpenStack Security Guide: Secure communication
• 6: Nova: Secure live migration with QEMU-native TLS

[markus-hentsch]

Most of the configuration adjustments proposed here cannot be verified via external tests unfortunately. Only the public-facing APIs can be checked for TLS by inspecting Keystone's service catalog, contacting each public endpoint and verifying its TLS handshake.

All the internal stuff would be up to dedicated audits with access to the infrastructure.

Speaking of TLS handshakes, I think we should also forbid using older/weak TLS versions and ciphers in the standard.

[markus-hentsch]

I began adding TLS specifics to the standard based on the official guidelines on ssl.com^1.

I think we should align this to the most commonly and broadly accepted current recommendations around TLS.

[artificial-intelligence]

I began adding TLS specifics to the standard based on the official guidelines on ssl.com1.

I think we should align this to the most commonly and broadly accepted current recommendations around TLS.

Footnotes

1. https://www.ssl.com/guide/tls-standards-compliance/ [↩](#user-content-fnref-1-cfaa4efba83e4b7037c3fc3cf637ce1e)

Might I ask why you choose this specific guide?

E.g. I think this guide is better: https://ssl-config.mozilla.org/ because it does not focus on compliance but on security.

[markus-hentsch]

Might I ask why you choose this specific guide?

E.g. I think this guide is better: https://ssl-config.mozilla.org/ because it does not focus on compliance but on security.

No particular reason. I was just looking for some form of official consensus on current SSL/TLS recommendations to base the standard on. I guess the Mozilla ones^1 would be a good alternative.

@martinmo had the same thought apparently: https://github.com/SovereignCloudStack/standards/pull/548#issuecomment-2046931970 :)

I will have a look.

[markus-hentsch]

I've had a look and it seems that sslyze does indeed support checking against the Mozilla recommendations directly. We could simplify the maintenance effort for this standard by referencing Mozilla's recommendations. However, this would mean referencing recommendations which are also a moving target themselves potentially (how often does Mozilla update this?). Also not sure how this would relate to the standard versioning and what a CSP has to test against at a given point in time exactly (version-pin sslyze? always upgrade sslyze before execution? etc.).

I've put it on the agenda for the next IaaS call.

[markus-hentsch]

I've put it on the agenda for the next IaaS call.

We discussed this on the SIG Standardization / Certification call: SovereignCloudStack/minutes/sig-standardization/20240516.md#tracking-moving-targets-tls-recommendations-from-mozilla

---

I updated the standard and test script to use the Mozilla TLS recommendations instead after figuring out how that works with the SSLyze library.

Sadly, the library only ships the single most recent Mozilla TLS preset version at the time of the library version release^1. So, when executing, you cannot choose between the versions of the recommendations. This means the recommendation version is coupled to the library version in a non-obvious way (different version numbers, requiring manual lookup).

We should pin the SSLyze library version for the test together with the JSON of the recommendation.

[markus-hentsch]

Problem with parameterizing the SSLyze version using SovereignCloudStack/standards/pull/595

I looked at the implementation of https://github.com/SovereignCloudStack/standards/pull/595 and tried to adjust the PR of this issue accordingly but it seems that the problem is not properly addressed yet.

Due to

Sadly, the library only ships the single most recent Mozilla TLS preset version at the time of the library version release.

... as stated above, parameterization is necessary when implementing the standard and executing tests.

This had been mentioned in https://github.com/SovereignCloudStack/standards/issues/592 which was implemented in https://github.com/SovereignCloudStack/standards/pull/595. However this did not consider the requirements.txt dilemma:

• the tests use a Docker image which installs all library dependencies at build-time
• the Docker image is built automatically in a pipeline
• the parameterized YAML is only provided at runtime in a separate action
• the version of the SSLyze library determines the version of the Mozilla TLS preset used, hence the requirements.txt entry is decisive here
• due to the decoupling of build and execution and the non-existence of the parameterized YAML during the former (build), the SSLyze library version cannot be influenced during the latter (execution) anymore

As a result, specifying the SSLyze library version in the scs-compatible-iaas.yaml would be "too late" in the process, as the SSLyze library will already have been installed during the Docker image build.

There seem to be dirty hacks to install pip packages from within Python code but I'm not a fan.

[mbuechse]

That's unfortunate. I'm not sure there's a good solution for that, because we need to have a requirements.txt that works for all tests, so it should not be dependent on the parameter. Therefore, I would assume that this test needs to create a new virtual environment, install the correct library, and then run python in a subprocess with some script, or something similar.

[mbuechse]

Another idea would be to vendor sslyze somehow and do so for multiple versions at a time, so we could select the version that we want at runtime (import time) and import that one.

[markus-hentsch]

Solution ideas for the SSLyze versioning dilemma

Context: https://github.com/SovereignCloudStack/standards/issues/547#issuecomment-2194781213

a) Modifying the call to the SSLyze library to change version

• does not seem possible: since the JSON is pretty much hardcoded^1 the library calls are unable to be modified sufficiently

b) Changing the virtualenv from within the script

• idea: from within the virtualenv, overwrite the installed SSLyze package with the desired version
• there seem to be possible pip calls that can be made from within a Python script
• problem: this requires internet connection from within the execution environment which is currently not a requirement for the tests afaict as they seem self-contained so far

c) Vendoring the necessary package versions

• as suggested by Matthias
• idea:
  • at built time crawl the YAMLs for SSLyze versions and package them into the Docker image
    • alternative to simplify the process: simply vendor the last N versions of SSLyze (removes need for YAML parsing)
  • at runtime: install the desired SSLyze version by using method b explained above
• problems:
  • the SSLyze library has dependencies; to properly install a specific version, we would need to vendor all dependencies in the correct version as well
  • SSLyze version is decoupled from the Mozilla TLS JSON versioning; this means that we would also need to record a mapping between both to select the correct SSLyze version[^3]

[^3]: when parameterizing the Secure Connections standard in the YAML, we want to specify the Mozilla JSON version first and foremost because this is the crucial information for the CSP, the SSLyze version is only relevant for testing purposes

d) Monkey patching a different Mozilla JSON in

• idea:
  • at built time: only vendor the last N JSON files of the Mozilla TLS recommendations
  • at runtime: monkey patch the MozillaTlsConfigurationChecker.get_default() method to point to the desired JSON or replace the hardcoded JSON file's contents directly
• problems:
  • when the version pin of the SSLyze library is updated, checks need to be done to ensure that the monkey patching still works and the SSLyze implementation has not changed at this part

e) Upstream contribution or fork

• idea:
  • as in method d, vendor the Mozilla JSONs at built time
  • instead of monkey patching the JSON path in like described in method d, fork or contribute to SSLyze to introduce an optional call argument which allows changing the path to the JSON
• problems:
  • fork maintainership is hard long-term
  • success/acceptance of upstream contribution is not guaranteed

Thoughts

• options a, b and c seem too brittle
• option d could provide a short-term solution while we attempt an upstream contribution as per e
  • if the upstream contribution is unsuccessful, option d could also be kept long-term as a compromise
• a SSLyze fork maintainership is out of the question I think

[mbuechse]

this requires internet connection from within the execution environment which is currently not a requirement for the tests afaict as they seem self-contained so far

Good point!

I think we can work around the following problem mentioned in c:

the SSLyze library has dependencies; to properly install a specific version, we would need to vendor all dependencies in the correct version as well

as follows:

We could indeed prepare one venv per sslyze version that we want to have (maybe prepare during build time, but it could also happen during "dev time"). Then we use subprocess to run something like /bin/bash -c ". venv-sslyze-vX && python3 check.py"

The script check.py would use sslyze (and not much more, if possible) and produce some output on stdout or stderr that the actual test script can then parse and turn into a verdict.

[markus-hentsch]

a) Modifying the call to the SSLyze library to change version

• does not seem possible: since the JSON is pretty much hardcoded the library calls are unable to be modified sufficiently

Au contraire! After digging and experimenting a bit more, the following has proven to work:

from sslyze.mozilla_tls_profile.mozilla_config_checker import (
    MozillaTlsConfigurationChecker,
    _MozillaTlsProfileAsJson)

json_file = open(...)
mozilla_json_preset = json.load(json_file)
parsed_profile = _MozillaTlsProfileAsJson(**mozilla_json_preset)
mozilla_checker = MozillaTlsConfigurationChecker(parsed_profile)
mozilla_checker.check_server(...)

It is the custom alternative to:

mozilla_checker = MozillaTlsConfigurationChecker.get_default()

and mimicks its behavior regarding loading the JSON file.

We don't need to fiddle with the SSLyze version at all (as long as the implementation does not change the imported classes and functions) and only need to ship and reference the Mozilla JSONs.

I vastly prefer this over the other alternatives now. EDIT: implemented changes in the test script.

[martinmo]

+1 from me for solution (a) from https://github.com/SovereignCloudStack/standards/issues/547#issuecomment-2196830436

I think this is a good compromise, even if _MozillaTlsProfileAsJson doesn't seem to be intended as public API (mind the _ prefix). At least in the past three years there haven't been too many or drastic changes to the mozilla_tls_profile (sub)package (see https://github.com/nabla-c0d3/sslyze/commits/release/sslyze/mozilla_tls_profile).

Just as a side note, for the venv per sslyze approach you could also use pipx to install several versions in parallel and avoid having to manage the venv stuff manually. Something like this (not really tested, but in principle this worked as I have used this --suffix machinery to test multiple poetry versions in parallel):

pipx install --suffix 6.0 sslyze==6.0.0  # installs as sslyze6.0
pipx install --suffix 5.2 sslyze==5.2.0  # installs as sslyze5.2

[artificial-intelligence]

fwiw my patch to unify the haproxy TLS settings and to update them to a modern standard was finally merged today :https://review.opendev.org/c/openstack/kolla-ansible/+/915403

so on new openstack releases you should only see properly secured TLS API endpoints (if TLS is enabled).