k8s中部署使用外部mysql，mysql会中断，portal打不开

[zlingqu]

我是在k8s中部署了apollo，mysql使用k8s外部的。 mysql使用mycat做读写分离 jdbc配置：spring.datasource.url = jdbc:mysql://mysql:3306/apollo_portaldb?characterEncoding=utf8&useLocalSessionState=true

在k8s中创建了service使用外部mysql，mysql.yaml文件如下

apiVersion: v1
kind: Service
metadata:
  name: mysql
  namespace: apollo
spec:
  selector:
    k8s-app: mysql
  ports:
    - port: 3306
      targetPort: 8066
      protocol: TCP
---
kind: Endpoints
apiVersion: v1
metadata:
  name: mysql
  namespace: apollo
subsets:
  - addresses:
      - ip: 10.11.12.201
    ports:
      - port: 8066
        protocol: TCP

使用时一切正常。 但每隔一段时间就会出现portal无法登陆的情况，日志报错

2020-04-17 11:23:25.790 WARN 46 --- [http-nio-3000-exec-10] com.zaxxer.hikari.pool.PoolBase : HikariPool-1 - Failed to validate connection com.mysql.jdbc.JDBC4Connection@4f51fa19 (No operations allowed after connection closed.)

此时我需要重新kubectl apply -f msyql.yaml，即可恢复。

我使用 k8s1.15.5 docker-ce-18.06.3 apollo1.4.0

[ljluestc]

#!/bin/bash

# Step 1: Verify network connectivity
# Check network connectivity between Apollo and MySQL
ping <mysql-ip> -c 5

# Step 2: Check MySQL configuration
# Verify MyCat configuration for read and write separation
# Check MySQL configuration files for any errors or misconfigurations

# Step 3: Check Kubernetes service and endpoint configuration
# Verify the Kubernetes service and endpoint configuration for MySQL
kubectl get services -n apollo
kubectl get endpoints -n apollo

# Step 4: Monitor resource usage
# Check resource usage of Apollo and MySQL deployments
kubectl top pods -n apollo
kubectl top nodes

# Step 5: Enable additional logging
# Update Apollo and MySQL configuration to enable detailed logging
# Restart Apollo and MySQL deployments

# Step 6: Upgrade Kubernetes and Docker
# Consider upgrading Kubernetes and Docker to the latest stable versions
# Follow upgrade guides and perform necessary steps

# Step 7: Perform connection and operation tests
# Test connectivity to MySQL from Apollo deployment
kubectl exec -it <apollo-pod> -n apollo -- bash
# Inside the Apollo pod, test connection to MySQL using mysql client
mysql -h <mysql-ip> -P 3306 -u <username> -p

# Perform read and write operations in Apollo to verify MySQL connectivity

# Step 8: Analyze logs and error messages
# Collect logs from Apollo and MySQL deployments
kubectl logs <apollo-pod> -n apollo
kubectl logs <mysql-pod> -n apollo

# Analyze logs for any error messages or warnings related to connection issues

# Step 9: Seek assistance from community forums or support channels
# If the issue persists, seek help from Apollo or Kubernetes community forums
# Share relevant logs and details to get assistance from experts

# Step 10: Document findings and solutions
# Document the troubleshooting process, findings, and any solutions implemented
# This will help in future reference and troubleshooting similar issues