Create a standard for the security of iaas service software

[josephineSei]

closes https://github.com/SovereignCloudStack/standards/issues/749

[josephineSei]

We discussed today in the IaaS meeting, how to proceed. The Conclusions were:

• expecting security patches to be deployed is reasonable for SCS-Compatible, but may be not easily possible to test -> this would need self-reports from CSPs in the reasonable timeframe, that they have updated their deployments
• SBOMs are part of SCS-Open and will give hints on both: overall software version and security patches
• behavior changes and/or must-have new API-endpoints should be done through standards with reasonable timeframe to become "mandatory" -> this means that a CSP will have to look through the standards and check, whether their used software version fulfill all standards -> the standards will have to be adjusted ob a regular basis -> we should think about also enabling another test for future requirements of standards (this means nearly all SHOULDS !)

This means I will rename the issue/PR and standard and will discard work on the tests, because the presence of security patches will not be easily testable.

[markus-hentsch]

Good first draft! I added some minor spelling/phrasing correction suggestions and some questions. I think the topic is not easy and we will need to be careful about two matters in particular:

1. the timeframe/deadline imposed on CSPs to apply updates
2. the way CSPs report back to SCS/OSBA about their compliance concerning software versions

No. 1 needs to be a sweet spot between giving CSPs enough room to implement/test and being strict enough to ensure security in a meaningful way.

Regarding no. 2 we maybe could be a bit more precise or provide a template. Unless I have missed something this could currently range from hand-written letters to SBOM XML files that are gigabytes in size which no human can look through in a timely manner.

[artificial-intelligence]

For now I just want to comment on something I don't see explained or discussed neither in the proposal nor in any of the current commentary:

This standard - in it's current form - talks broadly about "(the) software" comprising the IaaS layer, and I assume this mostly is meant to mean Openstack software.

However I would find it important that we clarify this, because a typical IaaS Deployment is comprised of the following - very much different - software layers, especially when it comes to security updates.

• Firmware needed for bare metal components, e.g. NIC and SSD firmware, CPU microcode
• basic operating system software needed to boot and manage hardware, e.g. grub bootloader, linux kernel, initrd etc.
• basic operating system - usually linux - packages to support a base layer of software and to start the next layer: systemd, glibc, a package manager (apt/dpkg), network configuration software (Netplan), a container runtime (docker, podman, k8s), others
• some kind of container, which often contains it's own layer of basic operating system packages again (see above point), minus the container runtime stuff itself
• the actual openstack services, usually python files (these are mostly covered by the openinfra security advisories/ OSSAs)
• direct openstack service dependencies also developed under the openinfra umbrella - also covered by the openinfra security process, e.g. oslo libraries
• (in)direct openstack service dependencies developed outside openinfra security process, e.g. docker-py, various low level python librariers, like python-ldap (for keystone)
• adjacent services deployed as containers alongside openstack infrastructure, e.g. logging and monitoring (fluentd, prometheus, ELK stack, keycloak) which are also not part of the openinfra security process

So I think it is important to at least distinguish which software from the above list the CSP should patch. If the answer to this is: "all of it" then we must be clear that only a tiny fraction of it is covered by openinfra security processes!

e.g. even direct dependencies of core openstack services are actually not monitored by openinfra for security issues if these are not developed under the openinfra umbrella!

of course, if issues are reported these are fixed, but e.g. the requirements files listing tested versions which are known to work with various openstack services are no indicator that these releases are secure!

So I think this is important to highlight to the user. This is the reason upstream provided kolla docker images are only provided for testing purposes and not for production, because these are not regularly updated for security fixes.

So imho we should keep in mind that CSPs need to monitor way more than just OSSA announcements if they want to be secure.

Thanks

[josephineSei]

For now I just want to comment on something I don't see explained or discussed neither in the proposal nor in any of the current commentary:

This standard - in it's current form - talks broadly about "(the) software" comprising the IaaS layer, and I assume this mostly is meant to mean Openstack software.

Indeed this standard should be applied on the IaaS Level, meaning all services with user-accesible APIs on that layer (this includes OpenStack and S3 if used).

Nevertheless we definitely should talk about security on all layers from Metal to KaaS. We may discuss this in the IAM call next week. But I think putting all of it into a single standard may be too complex?

However I would find it important that we clarify this, because a typical IaaS Deployment is comprised of the following - very much different - software layers, especially when it comes to security updates.

* Firmware needed for bare metal components, e.g. NIC and SSD firmware, CPU microcode

* basic operating system software needed to boot and manage hardware, e.g. grub bootloader, linux kernel, initrd etc.

* basic operating system - usually linux - packages to support a base layer of software and to start the next layer: systemd, glibc, a package manager (apt/dpkg), network configuration software (Netplan), a container runtime (docker, podman, k8s), others

* some kind of container, which often contains it's own layer of basic operating system packages again (see above point), minus the container runtime stuff itself

* the actual openstack services, usually python files (these are _mostly_ covered by the openinfra security advisories/ OSSAs)

* direct openstack service dependencies also developed under the openinfra umbrella - also covered by the openinfra security process, e.g. oslo libraries

* (in)direct openstack service dependencies developed outside openinfra security process, e.g. docker-py, various low level python librariers, like python-ldap (for keystone)

* adjacent services deployed as containers alongside openstack infrastructure, e.g. logging and monitoring (fluentd, prometheus, ELK stack, keycloak) which are also not part of the openinfra security process

So I think it is important to at least distinguish which software from the above list the CSP should patch. If the answer to this is: "all of it" then we must be clear that only a tiny fraction of it is covered by openinfra security processes!

This is a good point that needs to be integrated into the standard regardless of how detailed the standard itself is.

e.g. even direct dependencies of core openstack services are actually not monitored by openinfra for security issues if these are not developed under the openinfra umbrella!

of course, if issues are reported these are fixed, but e.g. the requirements files listing tested versions which are known to work with various openstack services are no indicator that these releases are secure!

So I think this is important to highlight to the user. This is the reason upstream provided kolla docker images are only provided for testing purposes and not for production, because these are not regularly updated for security fixes.

So imho we should keep in mind that CSPs need to monitor way more than just OSSA announcements if they want to be secure.

Thanks

Thank you for your feedback. I will integrate the focus on dependencies that are not monitored by OpenStack. For the rest I think we should discuss, whether this standard should have a very broad scope or just focus on the IaaS Layer. When we decide for the latter, we will most likely need standards for the other layers too.

[josephineSei]

@anjastrunk and I discussed @artificial-intelligence comment. We both think, that we should have a standard for security and updates on each Layer from Hardware to OS to IaaS and up to KaaS. I wrote 3 other issues, to reflect this: Hardware Layer: https://github.com/SovereignCloudStack/standards/issues/790 OS Layer: https://github.com/SovereignCloudStack/standards/issues/791 KaaS Layer: https://github.com/SovereignCloudStack/standards/issues/792

@artificial-intelligence I hope this reflects your concerns.

This would also narrow done the scope of this PR to exactly the IaaS Layer and dependencies of it. If you miss some software or want to explicitly call out some in the standard, please let me know.

[fkr]

@Adri2000 @neuroserve @matfechner please review this - I think it is important that the view from CSPs is incorporated.