moco
moco copied to clipboard
Support require_secure_transport=ON
What
A user wants to enforce encrypted communication within a k8s cluster using the require_secure_transport=ON
option.
However, the current MOCO implementation does not suppose encrypted communication. Some updates are needed.
How
Describe how to address the issue.
Checklist
- [ ] Finish implementation of the issue
- [ ] Test all functions
- [ ] Have enough logs to trace activities
- [ ] Notify developers of necessary actions
First, as @daichimukai pointed out in https://github.com/cybozu-go/moco-agent/pull/83, the agent
container gets an error.
- Apply the follwoing manifests.
apiVersion: moco.cybozu.com/v1beta1
kind: MySQLCluster
metadata:
name: test
spec:
mysqlConfigMapName: test
replicas: 3
podTemplate:
spec:
containers:
- name: mysqld
image: quay.io/cybozu/mysql:8.0.30
volumeClaimTemplates:
- metadata:
name: mysql-data
spec:
accessModes: [ "ReadWriteOnce" ]
resources:
requests:
storage: 5Gi
---
apiVersion: v1
kind: ConfigMap
metadata:
name: test
data:
require_secure_transport: "ON"
- Then the
moco-agent
does not start.
$ kubectl get pod
NAME READY STATUS RESTARTS AGE
moco-test-0 1/3 CrashLoopBackOff 4 (80s ago) 3m19s
moco-test-1 1/3 CrashLoopBackOff 4 (80s ago) 3m19s
moco-test-2 1/3 CrashLoopBackOff 4 (76s ago) 3m19s
$ kubectl logs -n sandbox moco-test-0 -c agent -p
Error: Error 3159: Connections using insecure transport are prohibited while --require_secure_transport=ON.
Usage:
moco-agent [flags]
Flags:
--address string Listening address and port for gRPC API. (default ":9080")
--connection-timeout duration Dial timeout (default 5s)
--grpc-cert-dir string gRPC certificate directory (default "/grpc-cert")
-h, --help help for moco-agent
--log-rotation-schedule string Cron format schedule for MySQL log rotation (default "*/5 * * * *")
--logfile string Log filename
--logformat string Log format [plain,logfmt,json]
--loglevel string Log level [critical,error,warning,info,debug]
--max-delay duration Acceptable max commit delay considering as ready; the zero value accepts any delay (default 1m0s)
--max-idle-time duration The maximum amount of time a connection may be idle (default 30s)
--metrics-address string Listening address and port for metrics. (default ":8080")
--probe-address string Listening address and port for mysqld health probes. (default ":9081")
--read-timeout duration I/O read timeout (default 30s)
--socket-path string Path of mysqld socket file. (default "/run/mysqld.sock")
Error 3159: Connections using insecure transport are prohibited while --require_secure_transport=ON.
Next, the cluster-manager will cause errors.
{"level":"error","ts":"2023-07-24T07:33:49Z","logger":"cluster-manager.default/test","msg":"failed to get mysqld status","operationId":"op-m4ffp","error":"failed to get global variables: pod=moco-test-2, namespace=default: failed to get mysql global variables: Error 3159: Connections using insecure transport are prohibited while --require_secure_transport=ON.","stacktrace":"github.com/cybozu-go/moco/clustering.(*managerProcess).GatherStatus.func2\n\t/work/clustering/status.go:206"}
Even if the above errors are resolved, replication will fail, and the MySQLCluster will not be HEALTHY.
$ kubectl get pod -l app.kubernetes.io/name=mysql
NAME READY STATUS RESTARTS AGE
moco-test-0 3/3 Running 0 4m1s
moco-test-1 2/3 Running 0 4m1s
moco-test-2 2/3 Running 0 4m1s
$ kubectl get mysqlcluster
NAME AVAILABLE HEALTHY PRIMARY SYNCED REPLICAS ERRANT REPLICAS LAST BACKUP
test False False 0 1 <no value>
Log of agent
container in moco-test-1
.
{"level":"info","ts":1690186048.8364568,"logger":"agent","caller":"server/mysqld_health.go:63","msg":"replication threads are stopped"}
2023-07-24T08:07:28.836730Z moco-test-1 moco-agent error: "well: access" http_host="10.244.2.14:9081" http_method="GET" http_status_code=503 http_user_agent="kube-probe/1.26" protocol="HTTP/1.1" remote_ipaddr="10.244.2.1" request_id="872f3224-6f0c-3d78-99c5-2650c609d452" request_size=0 response_size=31 response_time=0.003915732 type="access" url="/readyz"
Log of mysqld
container in moco-test-1
.
2023-07-24T08:10:28.830865Z 5238 [Note] [MY-010581] [Repl] Slave SQL thread for channel '' initialized, starting replication in log 'FIRST' at position 0, relay log './moco-test-1-relay-bin.000001' position: 4
2023-07-24T08:10:29.061344Z 5238 [Note] [MY-010596] [Repl] Error reading relay log event for channel '': slave SQL thread was killed
2023-07-24T08:10:29.061977Z 5238 [Note] [MY-010587] [Repl] Slave SQL thread for channel '' exiting, replication stopped in log 'FIRST' at position 0
2023-07-24T08:10:29.070286Z 5237 [Note] [MY-010563] [Repl] Slave I/O thread for channel '' killed while connecting to master
2023-07-24T08:10:29.070353Z 5237 [Note] [MY-010570] [Repl] Slave I/O thread exiting for channel '', read up to log 'FIRST', position 4
2023-07-24T08:10:29.077741Z 5244 [System] [MY-010597] [Repl] 'CHANGE MASTER TO FOR CHANNEL '' executed'. Previous state master_host='moco-test-0.moco-test.default.svc', master_port= 3306, master_log_file='', master_log_pos= 4, master_bind=''. New state master_host='moco-test-0.moco-test.default.svc', master_port= 3306, master_log_file='', master_log_pos= 4, master_bind=''.
2023-07-24T08:10:29.141562Z 5245 [Note] [MY-011179] [Repl] Slave I/O thread: Start semi-sync replication to master '[email protected]:3306' in log 'FIRST' at position 4.
2023-07-24T08:10:29.141620Z 5245 [Warning] [MY-010897] [Repl] Storing MySQL user name or password information in the master info repository is not secure and is therefore not recommended. Please consider using the USER and PASSWORD connection options for START SLAVE; see the 'START SLAVE Syntax' in the MySQL Manual for more information.
2023-07-24T08:10:29.143689Z 5245 [ERROR] [MY-010584] [Repl] Slave I/O for channel '': error connecting to master '[email protected]:3306' - retry-time: 60 retries: 1 message: Connections using insecure transport are prohibited while --require_secure_transport=ON. Error_code: MY-003159