caasp-services
caasp-services copied to clipboard
SUSE
Request: Expand further on Portus TLS implementation
I've been working on trying to get the Portus TLS portion working for secure communication between the resources. And, I'm hoping that someone can clarify the docs a little bit.
To run Portus securely you'll need an SSL key and certificate that cover the domain names that the Portus, Docker registry and Nginx services use.
Check. I have a cert that covers the fqdn ex foo.bar.com
To use TLS you must set portus.tls.enabled to true.
Check. Option is configured to true in the values.yaml
You will need to generate an appropriate key and certificate if you would like to install the chart securely.
Check. see the first quote above
If you would like to enable TLS for the Portus installation you will need to store your key in the value portus.tls.key and the certificate in the value portus.tls.cert
This is where I'm a bit confused. Perhaps this is because I'm new to helm, but, is this expecting a file or a C&P of the crt & key. I've tried both with no avail. I've renamed my cert to portus.crt and portus.key and I've tried it a number of different ways and I end up in the same spot.
2017/12/26 22:16:31 [emerg] 1#1: PEM_read_bio_X509_AUX("/certificates/portus.crt") failed (SSL: error:0906D06C:PEM routines:PEM_read_bio:no start line:Expecting: TRUSTED CERTIFICATE) nginx: [emerg] PEM_read_bio_X509_AUX("/certificates/portus.crt") failed (SSL: error:0906D06C:PEM routines:PEM_read_bio:no start line:Expecting: TRUSTED CERTIFICATE)
checking my certs with "openssl x509 -in certs/portus.crt -text -noout" returns expected output.
So, I guess my question/request is, what is the proper way to setup TLS in portus.
Also, side note. How would one do this leveraging kube-lego? I have my own cert that I bought from cert authority, however it would be cool to get this working with kube-lego moving forward.
Hey @Nick-Harvey , thanks for reaching out.
Sorry, I'm not sure what part of you first question "or a C&P of the crt & key", means?
For TLS support the Portus key and cert values need to use the full contents of both the key and cert files that you want to use. For example if you update the values.yaml file it should look something like:
portus:
...
tls:
key: |
-----BEGIN RSA PRIVATE KEY-----
...
-----BEGIN RSA PRIVATE KEY-----
cert: |
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
I hope this clarifies things a bit. Your comment makes me realize that the documentation is not very clear for the TLS values, so I think that will need to be updated to avoid confusion in the future.
If TLS still does not work for you after using the format above please let me know. If there are any bugs in the TLS support for this chart I will definitely look into it.
And as for kube-lego, I agree that leveraging that would be nice going forward. This chart still needs some work, we have not quite gotten to the point where we are ready to propose it into the official charts repo, so that might have to be something added at a later point. But I have seen some charts in the official repo offer kube-lego support, so I think that could potentially be something that could be offered in the Portus chart as well.
Found out that I need to add a "|" to keep the cert formatting in place, but upon deployment, the portus container is stuck in a waiting failed state. My guess is that it has something to do with the cert having to cover all the hostnames, but since helm generates names upon deployment it's going to be difficult to generate a cert that covers it.
The services use the chart and release name combo as names, which become the host names. You can specify a release name when installing, I think something like helm install . --name="test" would end up generating the host names:
test-portus test-portus-registry test-portus-nginx test-portus-mariadb
Not ideal, but I believe the naming of services has to be done this way for a chart in order for it to be accepted into the official repo.
Speaking of the official repo, it just came to my attention that other people have been working on a Portus chart, and have proposed it into the charts repo: https://github.com/kubernetes/charts/pull/2766
We may start collaborating with them, to get this chart to a more stable place.
I am a bit of a fan of collaboration. Getting some official SUSE input into the Chart would be awesome as well. I am not sure if this is the best place to put these comments but the Portus Chart in PR into the official Charts repo is failing tests because it is expecting the TLS secret to exist before the Chart can be deployed so this seems relevant. We have started using cert-manager in our setup. It's pretty good at deploying those certs but we would need a Helm resource setup to create them. We have a Chart we use internally to do that so I am going to put in a PR to kube charts to add it. We could then use that as a way of generating those certs. I noticed the Chart you are building creates the TLS certs if they don't exist. We could use that as the default method if cert-manager isn't available. If we need to split this out into it's own issue I am happy to add my input and help with the solution.
Hey @rendhalver we are definitely still up for collaboration- I have been working locally on the chart you are proposing into the official repo, and would like to make a few updates in order to get it passing, if you are on board.
So we do generate a key for Portus here if one is not specified when installing, but that was only done because I was receiving an error in my deployment when testing with no key, even with an insecure deployment. The default setting for this chart is actually no TLS- both Portus and the Registry are setup insecurely unless TLS is turned on by setting portus.tls.enabled to true. An insecure deployment is just for testing of course, but we noticed most other charts with TLS support have TLS turned off by default, I think because offering TLS requires some extra work from the user, so secure setups do not usually meet the chart repos guideline of being deployable with the base configuration. Our chart does have full support for TLS of course, but it has to be turned on.
I propose we do something similar in the chart you have pending- have TLS off for a default insecure deployment. I'd be happy to make this change, I've been experimenting with your chart, and I have been able to achieve this.
I do have one more question about your chart- it looks like it will always create an ingress resource. We decided to make ingress optional in our chart, so that an ingress resource is not created unless the user wants it, since not everyone has setup an ingress controller. We allow them to setup the Portus endpoint as a NodePort as well. Not a big deal, but I was wondering if making ingress optional might help the process as well?
Cheers
Setting the default to no TLS sounds good @bonhamjp. That would fix the testing in that PR as well. As for the ingress we can default that to off as well.
I will make a start at getting those setup this evening.
@rendhalver how would you like me to add the TLS changes to your PR? I pulled down your branch that is being proposed into the charts repo, so I could just commit to that branch, and push up my changes, if that sounds good?
Yeah sure thing, I'll do one of the two once I get it wrapped up. I'm still doing some finishing touches
Getting to the bottom of this is going to be the death of me ;)
After lots of trial and error I still can't get it to work. And I'm pretty sure it has to do with the cert having to cover the internal container names :) which isn't possible afaik.
In the values file you need to plug in a cert in the tls location
tls:
enabled: true
## must include key if using tls
##
cert: |
-----BEGIN CERTIFICATE-----
MIIFAzCCA+ugAwIBAgISBNt4Kz67vJOybe7VIDR4SOshMA0GCSqGSIb3DQEBCwUA
MEoxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MSMwIQYDVQQD
ExpMZXQncyBFbmNyeXB0IEF1dGhvcml0eSBYMzAeFw0xODAxMDgxMzI5MjFaFw0x
ODA0MDgxMzI5MjFaMBsxGTAXBgNVBAMTEHByb2plY3RpY2FydXMuYWkwggEiMA0G
CSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDAm3OccbGzBL/VrmKA4jG0IjEkffkB
lmwS6BFZdHCmPMhrJbpLpqMAlehzuMsnxXGTXtV4m6SFVWx14eTIk2sXdbEwJyfK
ltCzSxe2xswdJTefhc6gqJR2tHyVVUjE9MxK8DNZnWHhAuYNFW3yThd6RbdE/eIS
RxUArF5mbx1+0+iThz25or1Sld0McfGaqWXIteSbawkgjTf/ZZFl1wNX2czDaEoK
sZ43vLriBKRi8px5Ob+qZx5d87EhcWvs8NwFuzTenpja9byPVGRGPKzdT5/F21nH
icaR8G3pfDnCIBfpjZUUIjk6Lnei4QOsqsDxcQLWmTUEHNRrvRAXCmn1AgMBAAGj
ggIQMIICDDAOBgNVHQ8BAf8EBAMCBaAwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsG
AQUFBwMCMAwGA1UdEwEB/wQCMAAwHQYDVR0OBBYEFDu5pZkNALC3D1x+Q0vNktBm
fDRUMB8GA1UdIwQYMBaAFKhKamMEfd265tE5t6ZFZe/zqOyhMG8GCCsGAQUFBwEB
BGMwYTAuBggrBgEFBQcwAYYiaHR0cDovL29jc3AuaW50LXgzLmxldHNlbmNyeXB0
Lm9yZzAvBggrBgEFBQcwAoYjaHR0cDovL2NlcnQuaW50LXgzLmxldHNlbmNyeXB0
Lm9yZy8wGwYDVR0RBBQwEoIQcHJvamVjdGljYXJ1cy5haTCB/gYDVR0gBIH2MIHz
MAgGBmeBDAECATCB5gYLKwYBBAGC3xMBAQEwgdYwJgYIKwYBBQUHAgEWGmh0dHA6
Ly9jcHMubGV0c2VuY3J5cHQub3JnMIGrBggrBgEFBQcCAjCBngyBm1RoaXMgQ2Vy
dGlmaWNhdGUgbWF5IG9ubHkgYmUgcmVsaWVkIHVwb24gYnkgUmVseWluZyBQYXJ0
aWVzIGFuZCBvbmx5IGluIGFjY29yZGFuY2Ugd2l0aCB0aGUgQ2VydGlmaWNhdGUg
UG9saWN5IGZvdW5kIGF0IGh0dHBzOi8vbGV0c2VuY3J5cHQub3JnL3JlcG9zaXRv
cnkvMA0GCSqGSIb3DQEBCwUAA4IBAQBHs9x7IWduOaYgkrZBls7mNAxxK6ZTMemz
wmPTuvfC+zkslIQN+OpiOTu+TiAhgLkOHd85ukYeaCIW0DbrX0OdsVr7smD03n1e
qu5KK59uUqeSR6bbRgAFKW0YqgOJlUTPVIyogmT3IfC9MBEkbCm0e45TVq86ODSD
WU2blEcxEMBFESTTUYGfE+gAbdmtUWQcOrKhAh3LHh0sHpbOXi3ql9kQHNappnwR
vl811nnvVEh2L3V4IA6BXLrHYNiWTDnPRXsqijIBMqeo++kQ3tBgmsyjg+WosMA+
I6OU+h1T44YcVVmvbZNA+y02ZKkSke4svkBhqd62HjU258FcdaOW
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIIEkjCCA3qgAwIBAgIQCgFBQgAAAVOFc2oLheynCDANBgkqhkiG9w0BAQsFADA/
MSQwIgYDVQQKExtEaWdpdGFsIFNpZ25hdHVyZSBUcnVzdCBDby4xFzAVBgNVBAMT
DkRTVCBSb290IENBIFgzMB4XDTE2MDMxNzE2NDA0NloXDTIxMDMxNzE2NDA0Nlow
SjELMAkGA1UEBhMCVVMxFjAUBgNVBAoTDUxldCdzIEVuY3J5cHQxIzAhBgNVBAMT
GkxldCdzIEVuY3J5cHQgQXV0aG9yaXR5IFgzMIIBIjANBgkqhkiG9w0BAQEFAAOC
AQ8AMIIBCgKCAQEAnNMM8FrlLke3cl03g7NoYzDq1zUmGSXhvb418XCSL7e4S0EF
q6meNQhY7LEqxGiHC6PjdeTm86dicbp5gWAf15Gan/PQeGdxyGkOlZHP/uaZ6WA8
SMx+yk13EiSdRxta67nsHjcAHJyse6cF6s5K671B5TaYucv9bTyWaN8jKkKQDIZ0
Z8h/pZq4UmEUEz9l6YKHy9v6Dlb2honzhT+Xhq+w3Brvaw2VFn3EK6BlspkENnWA
a6xK8xuQSXgvopZPKiAlKQTGdMDQMc2PMTiVFrqoM7hD8bEfwzB/onkxEz0tNvjj
/PIzark5McWvxI0NHWQWM6r6hCm21AvA2H3DkwIDAQABo4IBfTCCAXkwEgYDVR0T
AQH/BAgwBgEB/wIBADAOBgNVHQ8BAf8EBAMCAYYwfwYIKwYBBQUHAQEEczBxMDIG
CCsGAQUFBzABhiZodHRwOi8vaXNyZy50cnVzdGlkLm9jc3AuaWRlbnRydXN0LmNv
bTA7BggrBgEFBQcwAoYvaHR0cDovL2FwcHMuaWRlbnRydXN0LmNvbS9yb290cy9k
c3Ryb290Y2F4My5wN2MwHwYDVR0jBBgwFoAUxKexpHsscfrb4UuQdf/EFWCFiRAw
VAYDVR0gBE0wSzAIBgZngQwBAgEwPwYLKwYBBAGC3xMBAQEwMDAuBggrBgEFBQcC
ARYiaHR0cDovL2Nwcy5yb290LXgxLmxldHNlbmNyeXB0Lm9yZzA8BgNVHR8ENTAz
MDGgL6AthitodHRwOi8vY3JsLmlkZW50cnVzdC5jb20vRFNUUk9PVENBWDNDUkwu
Y3JsMB0GA1UdDgQWBBSoSmpjBH3duubRObemRWXv86jsoTANBgkqhkiG9w0BAQsF
AAOCAQEA3TPXEfNjWDjdGBX7CVW+dla5cEilaUcne8IkCJLxWh9KEik3JHRRHGJo
uM2VcGfl96S8TihRzZvoroed6ti6WqEBmtzw3Wodatg+VyOeph4EYpr/1wXKtx8/
wApIvJSwtmVi4MFU5aMqrSDE6ea73Mj2tcMyo5jMd6jmeWUHK8so/joWUoHOUgwu
X4Po1QYz+3dszkDqMp4fklxBwXRsW10KXzPMTZ+sOPAveyxindmjkW8lGy+QsRlG
PfZ+G6Z6h7mjem0Y+iWlkYcV4PIWL1iwBi8saCbGS5jN2p8M+X+Q7UNKEkROb3N6
KOqkqm57TH2H3eDJAkSnh6/DNFu0Qg==
-----END CERTIFICATE-----
## must include certificate if using tls
##
key: |
<private key>
Then you deploy via helm
helm install --name reg \
--set nginx.ingress.host=foobar.com \
./
The chart deploys and creates the follow names:
- reg-foobar-registry
- reg-foobar-nginx
- reg-foobar-crono
- reg-foobar
Portus loads just fine, but since the registry is deployed at reg-foobar-registry:5000 your cert needs to cover that name which isn't possible due to hostnames covered under the ssl cert needing to be externally addressible....
Could you use DNS alternate names or something like puppet certs used to do? I am not sure if that is part of puppet or built into x509.
That's an excellent point, totally forgot about that.
What I like about this chart is that both portus and the reg are accessible from the same domain, due to the reverse proxy portion. Ideally, this would be better for my use case as my devs would only have to use the one domain instead of having to remember two different domains.
---- Scratch that --- Sans have to use externally addressable names as well, so puppet must have built them into the x509 which I'm not sure how to do
@Nick-Harvey correct me if I misinterpreted anything.
What domains are covered in portus.tls? With the name you specified above it should cover "reg-foobar" "reg-foobar-registry" and "reg-foobar-nginx". The TLS credentials used in portus.tls are only used internally to setup a secure registry, and portus deployment for communication within the cluster. When using ingress, you will need to setup a key/cert for the endpoint you are connecting externally to as well, so for "foobar" in your example. You will need to setup a secret with a key and cert value that covers the ingress host name, which you then specify here: https://github.com/kubic-project/caasp-services/blob/master/contrib/helm-charts/portus/values.yaml#L161. Make sure to turn on TLS for ingress as well, by setting nginx.ingress.tls.enabled to true.
lol, maybe I got this all wrong.....
if I wanted to have Portus and the registry go through https://foo.bar.com what would I need to do?
would it be something like:
ingress:
enabled: true
## Anntations to be added to the web ingress
##
annotations:
kubernetes.io/ingress.class: "nginx"
#ingress.kubernetes.io/ssl-passthrough: "true"
kubernetes.io/tls-acme: "true"
#kubernetes.io/ssl-passthrough: "true"
## TLS configuration
## the ingress host must be covered by the key/cert in order for TLS to work properly
##
tls:
enabled: true
## Secrets containing SSL key and cert must be manually created in the namespace
##
secretName: portus-tls
host:
- foo.bar.com
Because it looks like all the charts require the tls section up at the top to be true in order for them to use ssl.
Yes :)
I think "ingress.kubernetes.io/ssl-passthrough" might need to be uncommented as well.
I realize it is confusing setting up TLS twice, but I really couldn't think of a better way, since both the internal host names, and an external domain all need to be supported.
So yes if you use ingress the setup you posted above would allow you to cover whatever domain you want to use externally. I set up the reverse proxy to make this as simple as possible, as you mentioned above, so externally you will only need to cover the one domain, which maps to both Portus and the registry.
You could even turn TLS on for ingress and have insecure communication between services internally by leaving the portus.tls off.
Using a key/cert for internal communications between Portus the registry and the NGINX server is done for two reason 1.) Internal security. If anyone gets access within your cluster, then they would have access to the unsecured apps. Probably not a huge concern, but something to be aware of. 2.) Supporting more than one user within the registry. This one is a little odd. While it is possible to setup the Docker registry without a key/cert, that setup requires using an authentication mode called "Silly". This mode requires that the username that interacts with the registry is named "Silly". This obviously ruins the whole value of Portus, since it was built to allow many users to interact with the registry.
See item #2 is crucial for me. I need multiple developers to use both portus and the registry and their accounts be unique. Similar functionality to hub.docker.com but for an internal company registry. Is that possible with this chart? or will I have to leverage ldap to separate the accounts out?
Oh sorry for any confusion- yes it is most definitely possible with this chart!
I was just trying to clarify why TLS is needed for the internal services as well as for the ingress resource.
So, to be clear, with this chart you deploy a fully secured Portus and Registry pair, with as many users as you need.
Dude!!! That was it..... I can't believe how I missed that!!! BTW I know open source projects like this can often times be a thankless job (I know from experience) and I just wanted to say thanks for putting this together, it's an incredibly well done chart for an awesome project like Portus. I look forward to contributing where I can.
Hey @rendhalver I got an updated version of your chart working with a base non-TLS default, which is passing the chart test locally for me.
I ran into some issues- were you guys testing with Portus 2.1? It seems like that is what the environment variables were set up for. Which is totally fine, but running Portus 2.1 insecurely doesn't work. So I created an image in dockerhub using a working version of 2.2, which I had been using to build my chart. It doesn't change too much, although you will see some new env variables, and changed env variable names in the Portus deployment file. It sounds like Portus 2.3 will be done soon, and I believe a lot of the env vars are about to change again there.
I added the chart to my cloned charts repo: https://github.com/bonhamjp/charts/tree/feature/portus-default-insecure
Awesome news @bonhamjp! We couldn't get 2.2 working. 2.1 worked so we have been running that. We run Tectonic and there seems to be a bug that stopped 2.2 working there. I did find an issue about it when I was setting it up but I can't find it now. I will check that code out when I get a chance.
Hey @rendhalver just checking in to see if you have had a chance to look over my version of the chart? I did just get word the Portus 2.3 image is available! So I plan updating to the 2.3 image in the branch I linked to above. I will respond here once I've done that.
Sorry @bonhamjp It's been a busy week and that slipped out of my brain. I checked out your code locally and had a brief look but then got distracted. I will have a look as soon as I can.
No worries, I'm currently working on getting that branch working with 2.3 as well. I have it mostly operational, I've just run into a problem getting pushed repositories showing up, which I'm trying to work through. I'll push up the changes there once I have the chart fully working with 2.3
I've pushed up changes to the branch in my charts repo, so it is now using Portus 2.3. The background processing is a little different now, so I ended up adding a second Portus container to the deployment to do the background processing- otherwise changes to the registry were not showing up on the web site.
@bonhamjp
I have went through all the discussion above, and now understand better on tls setup for portus.
i am now able to deploy insecure portus, ingress tls enalbed portus. Now i am trying to deploy portus with tls enabled all the way, but i met problem to create cert and key for the internal pods, like below: test-portus test-portus-registry test-portus-nginx test-portus-mariadb
Could you help to share some points on how to create crt and key for all these pods?
Thanks, yzha
