Add CA/Cert/SSL Support for ETCD/Consul/Patroni/Postgres/Pgbouncer

[Vonng]

trafficstars

Need further local sandbox testing before merging

https://github.com/Vonng/pigsty/pull/131

[Vonng]

Sorry for keeping you waiting. I'm available for local testing and validating in 2 days.

It looks great since the default configuration file works perfectly.

I wonder if there could be a new configuration example file like pigsty-sec.yml to illustrate SSL/CA best practices.😆

[alemacci]

Sure, Can you give me an example of what to do? Thanks

[Vonng]

e.g a config template file in https://github.com/Vonng/pigsty/blob/master/files/conf/pigsty-dcs3.yml

which enables all security features by default? :D

[alemacci]

Ohh Ok I'll do it :)

[Vonng]

I made some modifications to ca & certs

refactor role ca

• now ca are execute on meta nodes only
• generate self-signed ca on primary meta node (meta_ip)
• generate ca.crt & ca.key to /etc/pigsty/ca by default
• copy ca.key & ca.cert to all meta nodes by default
• move role ca to infra boot (after environ) section

trust ca on role node

• trust self-signed ca-cert in os level during role node

As for the meta service ca & postgres ca, maybe we can put it into role node and postgres to keep role self-sustain

[alemacci]

It feels good!

Would you like me to implement the pg/dcs cert generation or are you working on this?

[Vonng]

That would be great and thank you, (I'm still occupied by some chores >_<)

since we have introduced new variable meta_ip which point to primary meta node address. we can delegate csr request to meta_ip node and deliver it to consul/etcd conf dir or postgres data dir maybe.

Another tricky point is about cert identity. nodename is the identity of node (which is set to {{ pg_cluster }}-{{ pg_seq }} by default for postgres cases), and {{ dcs_servers.key }} would be nice to use as the dcs server's cert subject.

[alemacci]

I will try to implement in next days :)

[Vonng]

Security Enhancement Summary:

• CA issue on primary meta node: /etc/pki/ca.{crt,key}
• CA key cert sync to all meta nodes: /etc/pki/ca.{crt,key}
• meta nodes have their own certs: /etc/pki/meta.{crt,key}
• all consul agent have their own certs: /etc/consul.d/ssl/consul.{crt,key}
• etcd server have their own certs: /etc/etcd/etcd.{crt,key}
• pgsql have their certs: /pg/cert/server.{crt,key}, duplicate to /pg/data/ too

etcd now bump to 3.5.4 with V2 API enabled & client cert enabled and used by:

• etcd metrics endpoint for prometheus scrape
• vip-manager inquiry
• patroni inquiry
• meta node etcdctl inquiry

patroni_ssl_enabled are disabled by default since health check with ssl would be too bulky. instead, patroni allowlist and basic authentication is used. you can use safe health check and status inquiry API as usually, while unsafe API can only be accessed from meta nodes with credentials. e.g pg command.

Example security enhancement config: https://github.com/Vonng/pigsty/blob/master/files/conf/pigsty-el9.yml

Thanks for your great contribution @alemacci