blog
blog copied to clipboard
penglongli
基于 ingress-nginx 实现灰度发布和蓝绿发布
本文描述如何基于 https://github.com/kubernetes/ingress-nginx 实现灰度发布和蓝绿发布功能：
- 灰度发布：
- 基于Request Header的流量切分
- 基于Cookie的流量切分
- 基于Query Param的流量切分
- 蓝绿发布：
- 根据权重进行流量切分
本文基于 controller-0.32.0 版本
-
修改后的代码仓库地址：https://github.com/penglongli/ingress-nginx/tree/controller-0.32.0
-
提供编译后的镜像使用：docker pull pelin/nginx-ingress-controller:0.32.0
概述
ingress-nginx 目前其实已经做了灰度发布、蓝绿发布，但是在使用方式上需要建多个 ingress 的方式来实现。我们目前希望能够在一个 Ingress 中即实现上述功能，则需要做一部分的开发工作。
目标
此处的目标是实现与阿里云 相同的方式，阿里云的使用方式参考：Kubernetes 集群中通过 Ingress 实现灰度发布和蓝绿发布
灰度发布
在 Ingress 建立的时候，使用如下 annotation 来实现灰度发布：
nginx.ingress.kubernetes.io/service-match: |
# 请求匹配到 cookie 存在 c=test，则转发到 new-nginx
# 如：curl -H 'Cookie: c=test' canary.example.com
new-nginx: cookie("c", "test")
# 请求匹配到 header 存在 h=test，则转发到 new-nginx
# 如：curl -H 'h: test' canary.example.com
new-nginx: header("h", "test")
# 请求匹配到 查询参数 存在 q=test，则转发到 new-nginx
# 如：curl http://canary.example.com?q=test
new-nginx: query("q", "test")
蓝绿发布
在 Ingress 建立时，使用如下 annotation 来实现灰度发布：
nginx.ingress.kubernetes.io/service-weight: |
new-nginx: 20, old-nginx: 80
则会出现 20% 的请求量被分配到 new-ingress。（nginx-ingress此处的请求权重分配非绝对分配，并且分配不均匀）
对比阿里云
阿里云的实现更多的是对 balancer.lua 的修改，有兴趣可以看一下其修改的 LUA 模块。
ingress-nginx 原理
后端
-
开启
syncIngress同步任务[https://github.com/penglongli/ingress-nginx/blob/controller-0.32.0/internal/ingress/controller/nginx.go#L310] -
初始化同步任务[https://github.com/penglongli/ingress-nginx/blob/controller-0.32.0/internal/ingress/controller/nginx.go#L139]
-
获取并组装 Ingress 信息[https://github.com/penglongli/ingress-nginx/blob/controller-0.32.0/internal/ingress/controller/controller.go#L135]
- 此步骤会将 Ingress 的信息拆解开，并通过 kubernetes API 获取 Service 对应的 Endpoint 地址，将其组装进来
-
判断本次是否需要 Reload Nginx 配置文件[https://github.com/penglongli/ingress-nginx/blob/controller-0.32.0/internal/ingress/controller/controller.go#L146]
LUA 模块
-
Nginx 配置里的动态 Upstream [https://github.com/penglongli/ingress-nginx/blob/controller-0.32.0/rootfs/etc/nginx/template/nginx.tmpl#L443]
-
LUA 动态 Upstream [https://github.com/penglongli/ingress-nginx/blob/controller-0.32.0/rootfs/etc/nginx/lua/balancer.lua#L274]
- 后端会开启一个 10246 端口，LUA 模块会请求 http://127.0.0.1:10246/configuration/backends 获取配置(此为获取 HTTP Backend 配置)
实现过程
代码修改提交：https://github.com/penglongli/ingress-nginx/commit/ad38db951224a1d2696c7d6d2298832b09becdaf
灰度
controller.go：
func (n *NGINXController) getBackendServers(ingresses []*ingress.Ingress) ([]*ingress.Backend, []*ingress.Server) {
for _, ing := range ingresses {
..............................
..............................
// 解析 nginx.ingress.kubernetes.io/service-match 注解，抽出灰度发布的策略，组装成 GrayStrategy 对象
svcStrategy := make(map[string]*ingress.GrayStrategy)
if serviceMatch := ing.Annotations["nginx.ingress.kubernetes.io/service-match"]; strings.TrimSpace(serviceMatch) != "" {
for _, annotation := range strings.Split(serviceMatch, "\n") {
splitServiceMatch(annotation, svcStrategy)
}
}
for _, rule := range ing.Spec.Rules {
..............................
..............................
// 设置每个 Rule(如：canary.example.com)下灰度策略的：灰度 Backend、灰度 Service
var graySvcSlice []string
for _, path := range rule.HTTP.Paths {
if strategy := svcStrategy[path.Backend.ServiceName]; strategy != nil {
graySvcSlice = append(graySvcSlice, path.Backend.ServiceName)
strategy.Service = path.Backend.ServiceName
strategy.Backend = upstreamName(ing.Namespace, path.Backend.ServiceName, path.Backend.ServicePort)
}
}
for pathIndex, path := range rule.HTTP.Paths {
..............................
..............................
for _, loc := range server.Locations {
// 为第一个 Location 设置灰度策略
if pathIndex == 0 {
for _, svc := range graySvcSlice {
loc.GrayStrategies = append(loc.GrayStrategies, svcStrategy[svc])
}
}
}
}
}
}
return aUpstreams, aServers
}
nginx.tmpl 模板
为每个域名（如：canary.example.com）判断是否需要灰度策略，并通过 generateTmplate() 将策略生成到 nginx.conf 配置中。
优先级：Cookie > Header > Query
{{ range $k, $v := $location.GrayStrategies }}
{{ $condition := (buildGrayIfCondition $v) }}
# 判断是否存在 Query 灰度策略
{{ if (ne $condition.QueryCondition "" )}}
if ({{ $condition.QueryCondition }}) {
set $service_name {{ $v.Service }};
set $proxy_upstream_name {{ $v.Backend }};
}
{{ end }}
# 判断是否存在 Header 灰度策略
{{ if (ne $condition.HeaderCondition "" )}}
if ({{ $condition.HeaderCondition }}) {
set $service_name {{ $v.Service }};
set $proxy_upstream_name {{ $v.Backend }};
}
{{ end }}
# 判断是否存在 Cookie 灰度策略
{{ if (ne $condition.CookieCondition "" )}}
if ({{ $condition.CookieCondition }}) {
set $service_name {{ $v.Service }};
set $proxy_upstream_name {{ $v.Backend }};
}
{{ end }}
{{ end }}
权重
controller.go
// createUpstreams creates the NGINX upstreams (Endpoints) for each Service
// referenced in Ingress rules.
func (n *NGINXController) createUpstreams(data []*ingress.Ingress, du *ingress.Backend) map[string]*ingress.Backend {
upstreams := make(map[string]*ingress.Backend)
upstreams[defUpstreamName] = du
for _, ing := range data {
anns := ing.ParsedAnnotations
var defBackend string
// 解析 nginx.ingress.kubernetes.io/service-weight 权重策略注解
serviceWeight := splitServiceWeight(anns)
for _, rule := range ing.Spec.Rules {
if rule.HTTP == nil {
continue
}
for i, path := range rule.HTTP.Paths {
// 为所有非 第一个 的 Path 设置权重参数
// 为什么“非第一个”？因为第一个会被映射到 nginx.conf 文件中的 $serviceName
if i != 0 {
weight := serviceWeight[path.Backend.ServiceName]
if weight > 0 && weight < 100 {
upstreams[name].NoServer = true
upstreams[name].TrafficShapingPolicy = ingress.TrafficShapingPolicy{
Weight: weight,
Header: anns.Canary.Header,
HeaderValue: anns.Canary.HeaderValue,
HeaderPattern: anns.Canary.HeaderPattern,
Cookie: anns.Canary.Cookie,
}
}
}
}
// 为第一个 Path 设置 alternativeService（此 Service 即为“替代 Service”，用于权重策略）
if len(rule.HTTP.Paths) >= 2 {
path := rule.HTTP.Paths[0]
name := upstreamName(ing.Namespace, path.Backend.ServiceName, path.Backend.ServicePort)
for i := 1; i < len(rule.HTTP.Paths); i++ {
if serviceWeight[path.Backend.ServiceName] != 0 {
upstreams[name].AlternativeBackends = append(
upstreams[name].AlternativeBackends,
upstreamName(ing.Namespace, rule.HTTP.Paths[i].Backend.ServiceName, rule.HTTP.Paths[i].Backend.ServicePort),
)
}
}
}
}
}
return upstreams
}
调试
在 nginx-ingress-controller 服务启动后，修改代码，使用如下脚本来编译运行：
#!/bin/bash
# 编译
make build
# 拷贝 nginx-ingress-controller
docker ps | grep "/usr/bin/dumb-ini" | awk '{print $1}' | xargs -I {} docker cp bin/amd64/nginx-ingress-controller {}:/nginx-ingress-controller
# 拷贝 nginx.tmpl
docker ps | grep "/usr/bin/dumb-ini" | awk '{print $1}' | xargs -I {} docker cp rootfs/etc/nginx/template/nginx.tmpl {}:/etc/nginx/template/
# 重启 nginx-ingress-controller 容器
docker ps | grep "/usr/bin/dumb-ini" | awk '{print $1}' | xargs docker restart
使用
部署服务
首先，部署两个服务：test-python、test-nginx
# test-python 服务
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: test-python
labels:
app: python
namespace: kube-system
spec:
replicas: 1
selector:
matchLabels:
app: python
template:
metadata:
labels:
app: python
spec:
containers:
- name: centos
image: centos:7
command: ["python"]
args:
- "-m"
- "SimpleHTTPServer"
- "8080"
---
apiVersion: v1
kind: Service
metadata:
name: test-python
namespace: kube-system
spec:
ports:
- port: 8080
protocol: TCP
targetPort: 8080
selector:
app: python
type: ClusterIP
# test-nginx 服务
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: test-nginx
labels:
app: nginx
namespace: kube-system
spec:
replicas: 2
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.18.0
imagePullPolicy: IfNotPresent
---
apiVersion: v1
kind: Service
metadata:
name: test-nginx
namespace: kube-system
spec:
ports:
- port: 8080
protocol: TCP
targetPort: 80
selector:
app: nginx
type: ClusterIP
创建 Ingress
权重
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/service-weight: |
test-python: 50, test-nginx: 50
labels:
app: demo
name: demo-ingress
namespace: kube-system
spec:
rules:
- host: canary.example.com
http:
paths:
- backend:
serviceName: test-python
servicePort: 8080
path: /
- backend:
serviceName: test-nginx
servicePort: 8080
path: /
请求域名判断是否能够按照 50% 分配到两个应用
灰度
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/service-match: |
test-nginx: cookie("c", "test")
labels:
app: demo
name: demo-ingress
namespace: kube-system
spec:
rules:
- host: canary.example.com
http:
paths:
- backend:
serviceName: test-python
servicePort: 8080
path: /
- backend:
serviceName: test-nginx
servicePort: 8080
path: /
使用 curl http://canary.example.com -H 'Cookie: c=test' 确定请求是否能够一直落在 test-nginx 上
