Contents
1 ingress介绍
Ingress就是一组基于DNS名称(host)或URL路径把请求转发至指定的Service资源的规则,用于将集群外部的请求流量转发至集群内部完成服务发布。然而,Ingress资源自身并不能进行“流量穿透”,它仅是一组路由规则的集合,这些规则要想真正发挥作用还需要其他功能的辅助,如监听某套接字,然后根据这些规则的匹配机制路由请求流量。这种能够为Ingress资源监听套接字并转发流量的组件称为Ingress控制器(Ingress Controller)。 Ingress控制器并不直接运行为kube-controller-manager的一部分,它是Kubernetes集群的一个重要附件,类似于CoreDNS,需要在集群上单独部署。 Ingress控制器可以由任何具有反向代理(HTTP/HTTPS)功能的服务程序实现,如Nginx、Envoy、HAProxy、Vulcand和Traefik等。Ingress控制器自身也是运行于集群中的Pod资源对象,它与被代理的运行为Pod资源的应用运行于同一网络中 使用Ingress资源进行流量分发时,Ingress控制器可基于某Ingress资源定义的规则将客户端的请求流量直接转发至与Service对应的后端Pod资源之上,这种转发机制会绕过Service资源,从而省去了由kube-proxy实现的端口代理开销。
2 pod和ingress的关系
- 通过label-selector相关联
- 通过Ingress Controller实现Pod的负载均衡
- 支持TCP/UDP 4层和HTTP 7层
3 ingress流程图

4 部署ingress-nginx controller
本次实验使用ingress-nginx部署 其他主流控制器 traefik: http反向代理,负载均衡工具 istio: 服务治理,控制入口流量.

下载对应的yaml
wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.26.1/deploy/static/mandatory.yaml
内容如下
apiVersion: v1
kind: Namespace
metadata:
name: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
---
kind: ConfigMap
apiVersion: v1
metadata:
name: nginx-configuration
namespace: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
---
kind: ConfigMap
apiVersion: v1
metadata:
name: tcp-services
namespace: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
---
kind: ConfigMap
apiVersion: v1
metadata:
name: udp-services
namespace: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: nginx-ingress-serviceaccount
namespace: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
name: nginx-ingress-clusterrole
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
rules:
- apiGroups:
- ""
resources:
- configmaps
- endpoints
- nodes
- pods
- secrets
verbs:
- list
- watch
- apiGroups:
- ""
resources:
- nodes
verbs:
- get
- apiGroups:
- ""
resources:
- services
verbs:
- get
- list
- watch
- apiGroups:
- ""
resources:
- events
verbs:
- create
- patch
- apiGroups:
- "extensions"
- "networking.k8s.io"
resources:
- ingresses
verbs:
- get
- list
- watch
- apiGroups:
- "extensions"
- "networking.k8s.io"
resources:
- ingresses/status
verbs:
- update
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: Role
metadata:
name: nginx-ingress-role
namespace: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
rules:
- apiGroups:
- ""
resources:
- configmaps
- pods
- secrets
- namespaces
verbs:
- get
- apiGroups:
- ""
resources:
- configmaps
resourceNames:
# Defaults to "<election-id>-<ingress-class>"
# Here: "<ingress-controller-leader>-<nginx>"
# This has to be adapted if you change either parameter
# when launching the nginx-ingress-controller.
- "ingress-controller-leader-nginx"
verbs:
- get
- update
- apiGroups:
- ""
resources:
- configmaps
verbs:
- create
- apiGroups:
- ""
resources:
- endpoints
verbs:
- get
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
name: nginx-ingress-role-nisa-binding
namespace: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: nginx-ingress-role
subjects:
- kind: ServiceAccount
name: nginx-ingress-serviceaccount
namespace: ingress-nginx
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: nginx-ingress-clusterrole-nisa-binding
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: nginx-ingress-clusterrole
subjects:
- kind: ServiceAccount
name: nginx-ingress-serviceaccount
namespace: ingress-nginx
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: nginx-ingress-controller
namespace: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
spec:
selector:
matchLabels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
template:
metadata:
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
annotations:
prometheus.io/port: "10254"
prometheus.io/scrape: "true"
spec:
hostNetwork: true
# wait up to five minutes for the drain of connections
terminationGracePeriodSeconds: 300
serviceAccountName: nginx-ingress-serviceaccount
containers:
- name: nginx-ingress-controller
image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.26.1
args:
- /nginx-ingress-controller
- --configmap=$(POD_NAMESPACE)/nginx-configuration
- --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
- --udp-services-configmap=$(POD_NAMESPACE)/udp-services
- --publish-service=$(POD_NAMESPACE)/ingress-nginx
- --annotations-prefix=nginx.ingress.kubernetes.io
securityContext:
allowPrivilegeEscalation: true
capabilities:
drop:
- ALL
add:
- NET_BIND_SERVICE
# www-data -> 33
runAsUser: 33
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
ports:
- name: http
containerPort: 80
- name: https
containerPort: 443
livenessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 10254
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 10
readinessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 10254
scheme: HTTP
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 10
lifecycle:
preStop:
exec:
command:
- /wait-shutdown
---
修改
spec:
hostNetwork: true
# wait up to five minutes for the drain of connections
terminationGracePeriodSeconds: 300
serviceAccountName: nginx-ingress-serviceaccount
containers:
- name: nginx-ingress-controller
image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.26.1
args:
创建ingress controller
[root@k8s-master01 ~]# kubectl apply -f mandatory.yaml
namespace/ingress-nginx created
configmap/nginx-configuration created
configmap/tcp-services created
configmap/udp-services created
serviceaccount/nginx-ingress-serviceaccount created
clusterrole.rbac.authorization.k8s.io/nginx-ingress-clusterrole created
role.rbac.authorization.k8s.io/nginx-ingress-role created
rolebinding.rbac.authorization.k8s.io/nginx-ingress-role-nisa-binding created
clusterrolebinding.rbac.authorization.k8s.io/nginx-ingress-clusterrole-nisa-binding created
deployment.apps/nginx-ingress-controller created
[root@k8s-master01 ~]#
查看pod创建情况
[root@k8s-master01 ~]# kubectl get pod -n ingress-nginx
NAME READY STATUS RESTARTS AGE
nginx-ingress-controller-6d8d7c675b-r7bxl 1/1 Running 0 89s
5 ingress的使用
单Service资源型Ingress
使用Ingress来暴露服务,此时只需要为Ingress指定“default backend”即可
[root@k8s-master01 ~]# cat my-ingress_default.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: my-ingress
spec:
backend:
serviceName: web
servicePort: 80
[root@k8s-master01 ~]# kubectl apply -f my-ingress_default.yaml
ingress.extensions/my-ingress created
访问查看

5.1 基于URL路径进行流量分发
[root@k8s-master01 ~]# cat myapp03.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: svc3
labels:
app: svc3
spec:
replicas: 1
selector:
matchLabels:
app: svc3
template:
metadata:
labels:
app: svc3
spec:
containers:
- name: svc3
image: cnych/example-web-service
env:
- name: APP_SVC
value: svc3
ports:
- containerPort: 8080
protocol: TCP
kind: Service
apiVersion: v1
metadata:
labels:
app: svc3
name: svc3
spec:
type: ClusterIP
ports:
- port: 8080
name: http
selector:
app: svc3
创建测试的svc和pods
[root@k8s-master01 ~]# kubectl apply -f myapp01.yaml
service/svc1 created
[root@k8s-master01 ~]# kubectl apply -f myapp02.yaml
service/svc2 created
[root@k8s-master01 ~]# kubectl apply -f myapp03.yaml
service/svc3 created
查看svc
[root@k8s-master01 ~]# kubectl get svc |grep svc
svc1 ClusterIP 10.110.207.5 <none> 8080/TCP 2m54s
svc2 ClusterIP 10.110.235.231 <none> 8080/TCP 87s
svc3 ClusterIP 10.98.100.177 <none> 8080/TCP 83s
查看pods
[root@k8s-master01 ~]# kubectl get pods |grep svc
svc1-5ff98455f6-8g78g 1/1 Running 0 19m
svc2-678c5fb65-rnt8t 1/1 Running 0 56s
svc3-55569d594-59zjf 1/1 Running 0 23s
创建一个 ingress 对象来访问上面的3个服务
[root@k8s-master01 ~]# cat ingress_app.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: ingress-web-url
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /
spec:
rules:
- host: ingress.devopstack.com
http:
paths:
- path: /s1
backend:
serviceName: svc1
servicePort: 8080
- path: /s2
backend:
serviceName: svc2
servicePort: 8080
- path: /s3
backend:
serviceName: svc3
servicePort: 8080
创建对应的ingress
[root@k8s-master01 ~]# kubectl apply -f ingress_app.yaml
ingress.extensions/ingress-web-url created
验证
[root@k8s-master01 ~]# kubectl get ingress
NAME CLASS HOSTS ADDRESS PORTS AGE
ingress-web-url <none> ingress.devopstack.com 80 6m45s
查看具体规则内容
[root@k8s-master01 ~]# kubectl describe ingress ingress-web-url
Name: ingress-web-url
Namespace: default
Address:
Default backend: default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
Rules:
Host Path Backends
---- ---- --------
ingress.devopstack.com
/s1 svc1:8080 (10.244.2.181:8080)
/s2 svc2:8080 (10.244.2.218:8080)
/s3 svc3:8080 (10.244.2.221:8080)
Annotations: nginx.ingress.kubernetes.io/rewrite-target: /
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal CREATE 7m47s nginx-ingress-controller Ingress default/ingress-web-url
Normal CREATE 7m47s nginx-ingress-controller Ingress default/ingress-web-url
访问验证下
upsmart@upsmart-System-Product-Name:~$ cat /etc/hosts
192.168.10.72 ingress.devopstack.com



5.2 基于主机名称的虚拟主机
主机名通配符
主机名可以是精确匹配(例如“foo.bar.com”)或者使用通配符来匹配 (例如“_.foo.com”)。 精确匹配要求 HTTP host 头部字段与 host 字段值完全匹配。 通配符匹配则要求 HTTP host 头部字段与通配符规则中的后缀部分相同。
主机 host 头部 匹配与否?
_.foo.com bar.foo.com 基于相同的后缀匹配
_.foo.com baz.bar.foo.com 不匹配,通配符仅覆盖了一个 DNS 标签
_.foo.com foo.com 不匹配,通配符仅覆盖了一个 DNS 标签
实例:
[root@k8s-master01 ~]# cat ingress_app_host.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: ingress-web-host
spec:
rules:
- host: svc1.devopstack.cn
http:
paths:
- pathType: Prefix
path: "/"
backend:
serviceName: svc1
servicePort: 8080
- host: svc2.devopstack.cn
http:
paths:
- pathType: Prefix
path: "/"
backend:
serviceName: svc2
servicePort: 8080
- host: svc3.devopstack.cn
http:
paths:
- pathType: Prefix
path: "/"
backend:
serviceName: svc3
servicePort: 8080
执行yaml生成对应的ingress策略
[root@k8s-master01 ~]# kubectl apply -f ingress_app_host.yaml
ingress.extensions/ingress-web-host created
查看ingress
[root@k8s-master01 ~]# kubectl get ingress
NAME CLASS HOSTS ADDRESS PORTS AGE
ingress-web-host <none> svc1.devopstack.cn,svc2.devopstack.cn,svc3.devopstack.cn 80 7s
查看 ingress策略详情
[root@k8s-master01 ~]# kubectl describe ingress ingress-web-host
Name: ingress-web-host
Namespace: default
Address:
Default backend: default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
Rules:
Host Path Backends
---- ---- --------
svc1.devopstack.cn
/ svc1:8080 (10.244.2.181:8080)
svc2.devopstack.cn
/ svc2:8080 (10.244.2.218:8080)
svc3.devopstack.cn
/ svc3:8080 (10.244.2.221:8080)
Annotations: Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal CREATE 2m21s nginx-ingress-controller Ingress default/ingress-web-host
Normal CREATE 2m21s nginx-ingress-controller Ingress default/ingress-web-host
查看nginx-crontroller具体策略
kubectl exec -it nginx-ingress-controller-b5p69 bash -n ingress-nginx
绑定 cat /etc/hosts
192.168.10.73 svc1.devopstack.cn
192.168.10.73 svc2.devopstack.cn
192.168.10.73 svc3.devopstack.cn
浏览器访问测试



5.3 Ingress https代理
TLS 认证
在现在大部分场景下面我们都会使用 https 来访问我们的服务,这节课我们将使用一个自签名的证书,当然你有在一些正规机构购买的 CA 证书是最好的,这样任何人访问你的服务的时候都是受浏览器信任的证书。使用下面的 openssl 命令生成 CA 证书:
openssl req -newkey rsa:2048 -nodes -keyout tls.key -x509 -days 365 -out tls.crt
现在我们有了证书,创建 secret 资源:
$ kubectl create secret generic ingressnginxcert --from-file=tls.crt --from-file=tls.key -n kube-system
查看:
[root@k8s-master01 ~]# kubectl describe secret ingressnginxcert
Name: ingressnginxcert
Namespace: default
Labels: <none>
Annotations: <none>
Type: Opaque
Data
====
tls.crt: 1220 bytes
tls.key: 1704 bytes
[root@k8s-master01 ~]# kubectl get ing
NAME CLASS HOSTS ADDRESS PORTS AGE
ingressssl <none> blogs.devopstack.com 80, 443 12h
绑定hosts
cat /etc/hosts
192.168.10.72 blogs.devopstack.com
验证结果:

查看证书

5.4 annotations对ingress个性化配置
名称 | 描述 | 值 |
---|---|---|
nginx.ingress.kubernetes.io/rewrite-target | 必须重定向流量的目标URL | value |
nginx.ingress.kubernetes.io/ssl-redirect | 指示位置部分是否仅可访问SSL(当ingress包含证书时,默认为True) | 布尔 |
nginx.ingress.kubernetes.io/force-redirect | 即使Ingress未启用TLS,也强制重定向到HTTPS | 布尔 |
nginx.ingress.kubernetes.io/app-root | 定义Controller必须重定向的应用程序根,如果它在"/"上下文中 | value |
nginx.ingress.kubernetes.io/use-regex | 指示 Ingress 上定义的路径是否使用正则表达式 | 布尔 |
参考: https://kubernetes.github.io/ingress-nginx/examples/rewrite/ |
annotations用法:
nginx配置
https://github.com/kubernetes/ingress-nginx/blob/master/docs/user-guide/nginx-configuration/index.md
实例:
my-ingress_default.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: my-ingress
annotations:
kubernetes.io/ingress.class: "nginx" #选择ingress控制器类型
nginx.ingress.kubernetes.io/proxy-connect-timeout: "600"
nginx.ingress.kubernetes.io/proxy-send-timeout: "600"
nginx.ingress.kubernetes.io/proxy-read-timeout: "600"
nginx.ingress.kubernetes.io/proxy-body-size: "100m"
nginx.ingress.kubernetes.io/ssl-redirect: "true" #http->https
nginx.ingress.kubernetes.io/server-snippet: | ##可以通过这个参数加nginx具体配置
set $agentflag 0;
if ($http_user_agent ~* "(Mobile)"){
set $agentflag 1;
}
if ($agentflag =1 ){
return 301 https://default.devopstack.cn;
}
spec:
backend:
serviceName: web
servicePort: 80
6 ingress controller高可用方案
1.固定ingress controller 到两个node上(daemonset+nodeselector)->user-> 域名-> vip(keepalive)ha->pod 2. 固定ingress controller 到两个node上(daemonset+nodeselector)->user-> 域名-> lB(nginx,lvs,haproxy)-> ingress controller -> pod
7 Traefik
- 我的微信
- 这是我的微信扫一扫
-
- 我的微信公众号
- 我的微信公众号扫一扫
-