Pod自动扩容/缩容：HPA介绍

Horizontal Pod Autoscaler（HPA，Pod水平自动伸缩）：根据资源利用率或者自定义指

标自动调整Deployment的Pod副本数量，提供应用并发。HPA不适于无法缩放的对象，例

如DaemonSet。

Pod自动扩容/缩容：HPA基本工作原理

Kubernetes 中的 Metrics Server 持续采集所有 Pod 副本的指标数据。HPA 控制器通过 Metrics-Server 的 API（聚合 API）获取这些数据，基于用户定义的扩缩容规则进行计算，得到目标 Pod 副本数量。当目标 Pod 副本数量与当前副本数量不同时，HPA 控制器就像 Pod 的Deployment控制器发起scale 操作，调整 Pod 的副本数量，完成扩缩容操作。

Pod自动扩容/缩容：使用HPA前提条件

使用HPA，确保满足以下条件：

• 启用Kubernetes API聚合层

• 相应的API已注册：

• 对于资源指标（例如CPU、内存），将使用metrics.k8s.io API，一般由metrics-server提供。

• 对于自定义指标（例如QPS），将使用custom.metrics.k8s.io API，由相关适配器（Adapter）服务提供。

已知适配器列表：

https://github.com/kubernetes/metrics/blob/master/IMPLEMENTATIONS.md#custom-metrics-api

Kubernetes API聚合层：

在 Kubernetes 1.7 版本引入了聚合层，允许第三方应用程序通过将自己注册到kube-apiserver上，仍然通过 API Server 的 HTTP URL 对新的 API 进行访问和操作。为了实现这个机制，Kubernetes 在 kube-apiserver 服务中引入了一个API 聚合层（API Aggregation Layer），用于将扩展 API 的访问请求转发到用户服务的功能。

启用聚合层：

如果你使用kubeadm部署的，默认已开启。

如果你使用二进制方式部署的话，需要在kubeAPIServer中添加启动参数，增加以下配置：

# vim /opt/kubernetes/cfg/kube-apiserver.conf

..

--requestheader-client-ca-file=/opt/kubernetes/ssl/ca.pem 

--proxy-client-cert-file=/opt/kubernetes/ssl/server.pem 

--proxy-client-key-file=/opt/kubernetes/ssl/server-key.pem 

--requestheader-allowed-names=kubernetes 

--requestheader-extra-headers-prefix=X-Remote-Extra- 

--requestheader-group-headers=X-Remote-Group 

--requestheader-username-headers=X-Remote-User 

--enable-aggregator-routing=true 

...

Pod自动扩容/缩容：基于资源指标

Metrics Server：是一个数据聚合器，从kubelet收集资源指标，并通过Metrics API在Kubernetes apiserver暴露，以供HPA使用。

项目地址：https://github.com/kubernetes-sigs/metrics-server

Metrics Server部署：

wget https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.3.7/components.yaml

[root@master-1 yaml]# vim components.yaml

apiVersion: v1

kind: ServiceAccount

metadata:

labels:

k8s-app: metrics-server

name: metrics-server

namespace: kube-system

---

apiVersion: rbac.authorization.k8s.io/v1

kind: ClusterRole

metadata:

labels:

k8s-app: metrics-server

rbac.authorization.k8s.io/aggregate-to-admin: "true"

rbac.authorization.k8s.io/aggregate-to-edit: "true"

rbac.authorization.k8s.io/aggregate-to-view: "true"

name: system:aggregated-metrics-reader

rules:

- apiGroups:

- metrics.k8s.io

resources:

- pods

- nodes

verbs:

- get

- list

- watch

---

apiVersion: rbac.authorization.k8s.io/v1

kind: ClusterRole

metadata:

labels:

k8s-app: metrics-server

name: system:metrics-server

rules:

- apiGroups:

- ""

resources:

- pods

- nodes

- nodes/stats

- namespaces

- configmaps

verbs:

- get

- list

- watch

---

apiVersion: rbac.authorization.k8s.io/v1

kind: RoleBinding

metadata:

labels:

k8s-app: metrics-server

name: metrics-server-auth-reader

namespace: kube-system

roleRef:

apiGroup: rbac.authorization.k8s.io

kind: Role

name: extension-apiserver-authentication-reader

subjects:

- kind: ServiceAccount

name: metrics-server

namespace: kube-system

---

apiVersion: rbac.authorization.k8s.io/v1

kind: ClusterRoleBinding

metadata:

labels:

k8s-app: metrics-server

name: metrics-server:system:auth-delegator

roleRef:

apiGroup: rbac.authorization.k8s.io

kind: ClusterRole

name: system:auth-delegator

subjects:

- kind: ServiceAccount

name: metrics-server

namespace: kube-system

---

apiVersion: rbac.authorization.k8s.io/v1

kind: ClusterRoleBinding

metadata:

labels:

k8s-app: metrics-server

name: system:metrics-server

roleRef:

apiGroup: rbac.authorization.k8s.io

kind: ClusterRole

name: system:metrics-server

subjects:

- kind: ServiceAccount

name: metrics-server

namespace: kube-system

---

apiVersion: v1

kind: Service

metadata:

labels:

k8s-app: metrics-server

name: metrics-server

namespace: kube-system

spec:

ports:

- name: https

port: 443

protocol: TCP

targetPort: https

selector:

k8s-app: metrics-server

---

apiVersion: apps/v1

kind: Deployment

metadata:

labels:

k8s-app: metrics-server

name: metrics-server

namespace: kube-system

spec:

selector:

matchLabels:

k8s-app: metrics-server

strategy:

rollingUpdate:

maxUnavailable: 0

template:

metadata:

labels:

k8s-app: metrics-server

spec:

containers:

- args:

- --cert-dir=/tmp

- --secure-port=4443

- --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname

- --kubelet-use-node-status-port

- --kubelet-insecure-tls

#不验证kubelet提供的https证书

image: feixiangkeji974907/metrics-server:v0.4.1

#官方给的镜像地址国内无法下载，我下载后传到了dockerhub上了

imagePullPolicy: IfNotPresent

livenessProbe:

failureThreshold: 3

httpGet:

path: /livez

port: https

scheme: HTTPS

periodSeconds: 10

name: metrics-server

ports:

- containerPort: 4443

name: https

protocol: TCP

readinessProbe:

failureThreshold: 3

httpGet:

path: /readyz

port: https

scheme: HTTPS

periodSeconds: 10

securityContext:

readOnlyRootFilesystem: true

runAsNonRoot: true

runAsUser: 1000

volumeMounts:

- mountPath: /tmp

name: tmp-dir

nodeSelector:

kubernetes.io/os: linux

priorityClassName: system-cluster-critical

serviceAccountName: metrics-server

volumes:

- emptyDir: {}

name: tmp-dir

---

apiVersion: apiregistration.k8s.io/v1

kind: APIService

metadata:

labels:

k8s-app: metrics-server

name: v1beta1.metrics.k8s.io

spec:

group: metrics.k8s.io

groupPriorityMinimum: 100

insecureSkipTLSVerify: true

service:

name: metrics-server

namespace: kube-system

version: v1beta1

versionPriority: 100

[root@master-1 yaml]# kubectl apply -f components.yaml

[root@master-1 yaml]# kubectl get all -n kube-system

NAME READY STATUS RESTARTS AGE

pod/coredns-5ffbfd976d-txzpq 1/1 Running 5 7d3h

pod/kube-flannel-ds-amd64-5tnrj 1/1 Running 11 7d3h

pod/kube-flannel-ds-amd64-fx7vc 1/1 Running 11 7d3h

pod/kube-flannel-ds-amd64-kwfxn 1/1 Running 2 5d

pod/kube-flannel-ds-amd64-l4g4t 1/1 Running 7 7d3h

pod/metrics-server-664489867d-5zwpf 1/1 Running 20 7d3h



NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE

service/kube-dns ClusterIP 10.0.0.2 <none> 53/UDP,53/TCP,9153/TCP 7d3h

service/metrics-server ClusterIP 10.0.0.42 <none> 443/TCP 7d3h



NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE

daemonset.apps/kube-flannel-ds-amd64 4 4 4 4 4 <none> 7d3h



NAME READY UP-TO-DATE AVAILABLE AGE

deployment.apps/coredns 1/1 1 1 7d3h

deployment.apps/metrics-server 1/1 1 1 7d3h


NAME DESIRED CURRENT READY AGE

replicaset.apps/coredns-5ffbfd976d 1 1 1 7d3h

replicaset.apps/metrics-server-664489867d 1 1 1 7d3h

测试metrics-server 是否可用：

[root@master-1 yaml]# kubectl get apiservice | grep metric

v1beta1.metrics.k8s.io kube-system/metrics-server True 15s

[root@master-1 yaml]# kubectl get –raw /apis/metrics.k8s.io/v1beta1/nodes

{"kind":"NodeMetricsList","apiVersion":"metrics.k8s.io/v1beta1","metadata":{"selfLink":"/apis/metrics.k8s.io/v1beta1/nodes"},"items":[{"metadata":{"name":"master-1","selfLink":"/apis/metrics.k8s.io/v1beta1/nodes/master-1","creationTimestamp":"2021-01-03T16:03:35Z"},"timestamp":"2021-01-03T16:02:35Z","window":"30s","usage":{"cpu":"433676959n","memory":"1127920Ki"}},{"metadata":{"name":"node-1","selfLink":"/apis/metrics.k8s.io/v1beta1/nodes/node-1","creationTimestamp":"2021-01-03T16:03:35Z"},"timestamp":"2021-01-03T16:02:38Z","window":"30s","usage":{"cpu":"211126782n","memory":"595564Ki"}},{"metadata":{"name":"node-2","selfLink":"/apis/metrics.k8s.io/v1beta1/nodes/node-2","creationTimestamp":"2021-01-03T16:03:35Z"},"timestamp":"2021-01-03T16:02:35Z","window":"30s","usage":{"cpu":"170548141n","memory":"366576Ki"}},{"metadata":{"name":"node-3","selfLink":"/apis/metrics.k8s.io/v1beta1/nodes/node-3","creationTimestamp":"2021-01-03T16:03:35Z"},"timestamp":"2021-01-03T16:02:32Z","window":"30s","usage":{"cpu":"120488736n","memory":"425804Ki"}}]}

[root@master-1 yaml]# kubectl get –raw /apis/metrics.k8s.io/v1beta1/pods

{"kind":"PodMetricsList","apiVersion":"metrics.k8s.io/v1beta1","metadata":{"selfLink":"/apis/metrics.k8s.io/v1beta1/pods"},"items":[{"metadata":{"name":"web-test-5cdbd79b55-87pqt","namespace":"default","selfLink":"/apis/metrics.k8s.io/v1beta1/namespaces/default/pods/web-test-5cdbd79b55-87pqt","creationTimestamp":"2021-01-03T16:04:02Z"},"timestamp":"2021-01-03T16:03:28Z","window":"30s","containers":[{"name":"nginx","usage":{"cpu":"0","memory":"6420Ki"}}]},{"metadata":{"name":"web-test-5cdbd79b55-p54nq","namespace":"default","selfLink":"/apis/metrics.k8s.io/v1beta1/namespaces/default/pods/web-test-5cdbd79b55-p54nq","creationTimestamp":"2021-01-03T16:04:02Z"},"timestamp":"2021-01-03T16:03:35Z","window":"30s","containers":[{"name":"nginx","usage":{"cpu":"0","memory":"5160Ki"}}]},{"metadata":{"name":"web-test-5cdbd79b55-r9swh","namespace":"default","selfLink":"/apis/metrics.k8s.io/v1beta1/namespaces/default/pods/web-test-5cdbd79b55-r9swh","creationTimestamp":"2021-01-03T16:04:02Z"},"timestamp":"2021-01-03T16:03:27Z","window":"30s","containers":[{"name":"nginx","usage":{"cpu":"0","memory":"6040Ki"}}]},{"metadata":{"name":"web-test-5cdbd79b55-t8pcx","namespace":"default","selfLink":"/apis/metrics.k8s.io/v1beta1/namespaces/default/pods/web-test-5cdbd79b55-t8pcx","creationTimestamp":"2021-01-03T16:04:02Z"},"timestamp":"2021-01-03T16:03:34Z","window":"30s","containers":[{"name":"nginx","usage":{"cpu":"0","memory":"5996Ki"}}]},{"metadata":{"name":"nginx-ingress-controller-766fb9f77-zzwmp","namespace":"ingress-nginx","selfLink":"/apis/metrics.k8s.io/v1beta1/namespaces/ingress-nginx/pods/nginx-ingress-controller-766fb9f77-zzwmp","creationTimestamp":"2021-01-03T16:04:02Z"},"timestamp":"2021-01-03T16:03:30Z","window":"30s","containers":[{"name":"nginx-ingress-controller","usage":{"cpu":"21249411n","memory":"96340Ki"}}]},{"metadata":{"name":"coredns-5ffbfd976d-txzpq","namespace":"kube-system","selfLink":"/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/coredns-5ffbfd976d-txzpq","creationTimestamp":"2021-01-03T16:04:02Z"},"timestamp":"2021-01-03T16:03:30Z","window":"30s","containers":[{"name":"coredns","usage":{"cpu":"7583457n","memory":"19704Ki"}}]},{"metadata":{"name":"kube-flannel-ds-amd64-5tnrj","namespace":"kube-system","selfLink":"/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/kube-flannel-ds-amd64-5tnrj","creationTimestamp":"2021-01-03T16:04:02Z"},"timestamp":"2021-01-03T16:03:39Z","window":"30s","containers":[{"name":"kube-flannel","usage":{"cpu":"4707209n","memory":"16228Ki"}}]},{"metadata":{"name":"kube-flannel-ds-amd64-fx7vc","namespace":"kube-system","selfLink":"/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/kube-flannel-ds-amd64-fx7vc","creationTimestamp":"2021-01-03T16:04:02Z"},"timestamp":"2021-01-03T16:03:34Z","window":"30s","containers":[{"name":"kube-flannel","usage":{"cpu":"4736628n","memory":"16096Ki"}}]},{"metadata":{"name":"kube-flannel-ds-amd64-kwfxn","namespace":"kube-system","selfLink":"/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/kube-flannel-ds-amd64-kwfxn","creationTimestamp":"2021-01-03T16:04:02Z"},"timestamp":"2021-01-03T16:03:30Z","window":"30s","containers":[{"name":"kube-flannel","usage":{"cpu":"5035329n","memory":"19940Ki"}}]},{"metadata":{"name":"kube-flannel-ds-amd64-l4g4t","namespace":"kube-system","selfLink":"/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/kube-flannel-ds-amd64-l4g4t","creationTimestamp":"2021-01-03T16:04:02Z"},"timestamp":"2021-01-03T16:03:33Z","window":"30s","containers":[{"name":"kube-flannel","usage":{"cpu":"5185835n","memory":"15388Ki"}}]},{"metadata":{"name":"metrics-server-664489867d-5zwpf","namespace":"kube-system","selfLink":"/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/metrics-server-664489867d-5zwpf","creationTimestamp":"2021-01-03T16:04:02Z"},"timestamp":"2021-01-03T16:03:39Z","window":"30s","containers":[{"name":"metrics-server","usage":{"cpu":"7712269n","memory":"26064Ki"}}]},{"metadata":{"name":"prometheus-5fc97df657-w4qcm","namespace":"kube-system","selfLink":"/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/prometheus-5fc97df657-w4qcm","creationTimestamp":"2021-01-03T16:04:02Z"},"timestamp":"2021-01-03T16:03:26Z","window":"30s","containers":[{"name":"prometheus-server-configmap-reload","usage":{"cpu":"0","memory":"2844Ki"}},{"name":"prometheus-server","usage":{"cpu":"15569447n","memory":"124600Ki"}}]},{"metadata":{"name":"dashboard-metrics-scraper-6b4884c9d5-g8dx8","namespace":"kubernetes-dashboard","selfLink":"/apis/metrics.k8s.io/v1beta1/namespaces/kubernetes-dashboard/pods/dashboard-metrics-scraper-6b4884c9d5-g8dx8","creationTimestamp":"2021-01-03T16:04:02Z"},"timestamp":"2021-01-03T16:03:23Z","window":"30s","containers":[{"name":"dashboard-metrics-scraper","usage":{"cpu":"224169n","memory":"12228Ki"}}]},{"metadata":{"name":"kubernetes-dashboard-7f99b75bf4-ljhtn","namespace":"kubernetes-dashboard","selfLink":"/apis/metrics.k8s.io/v1beta1/namespaces/kubernetes-dashboard/pods/kubernetes-dashboard-7f99b75bf4-ljhtn","creationTimestamp":"2021-01-03T16:04:02Z"},"timestamp":"2021-01-03T16:03:33Z","window":"30s","containers":[{"name":"kubernetes-dashboard","usage":{"cpu":"1175979n","memory":"21788Ki"}}]}]}

也可以使用kubectl top访问Metrics API：

#查看Node资源消耗

[root@master-1 yaml]# kubectl top node

NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%

master-1 584m 14% 1101Mi 19%

node-1 219m 5% 567Mi 9%

node-2 175m 4% 363Mi 6%

node-3 146m 3% 408Mi 2%

#查看Pod资源消耗：

[root@master-1 yaml]# kubectl top pod

NAME CPU(cores) MEMORY(bytes)

web-test-5cdbd79b55-x6snq 0m 2Mi

模拟部署一个nginx pod 测试

#创建deployment

[root@master-1 yaml]# kubectl create deployment web –image=nginx –dry-run=client -o yaml > deployment.yaml

[root@master-1 yaml]# vim deployment.yaml

#修改yaml，修改副本数为2，增加resources.requests.cpu

.....

spec:

replicas: 2

.....

spec:

containers:

- image: nginx

name: nginx

resources:

requests:

cpu: "250m"

........

#创建service

[root@master-1 yaml]# kubectl expose deployment web –port=80 –target-port=80

[root@master-1 yaml]# kubectl get pods,service,deployment

NAME READY STATUS RESTARTS AGE

pod/web-6fc98bbd69-dbbxx 1/1 Running 0 21s

pod/web-6fc98bbd69-xzrbm 1/1 Running 0 3m15s


NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE

service/kubernetes ClusterIP 10.0.0.1 <none> 443/TCP 7d4h

service/web ClusterIP 10.0.0.205 <none> 80/TCP 7d3h


NAME READY UP-TO-DATE AVAILABLE AGE

deployment.apps/web 2/2 2 2 7m46s

创建HPA,设置指标

[root@master-1 yaml]# kubectl autoscale deployment web –min=2 –max=5 –cpu-percent=80

horizontalpodautoscaler.autoscaling/web autoscaled

[root@master-1 yaml]# kubectl get hpa

NAME 	REFERENCE			TARGETS 		MINPODS 	MAXPODS		REPLICAS 	AGE

web 	Deployment/web 			0%/80% 			2 		5		2 		39s

说明：为名为web的deployment创建一个HPA对象，目标CPU使用率为80%，副本数量配置为2到5之间。

压力测试

使用ac 压力测试

总10w请求，并发1000，从1循环到100

[root@master-1 yaml]# for i in {1..100}

> do

> ab -n 100000 -c 1000 http://10.0.0.205/index.html

> done

观察扩容状态：

另起一个终端，查看CPU 使用率，以及扩容状态

#可以看到CPU 使用率超过了191%, pod也成功扩容到了5个

[root@master-1 ~]# kubectl get hpa

NAME	REFERENCE 		TARGETS		MINPODS 	MAXPODS 	REPLICAS      AGE

web	Deployment/web 		191%/80% 	2 		5 		4	      5m43s

[root@master-1 ~]# kubectl get pods -o wide

NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES

web-6fc98bbd69-dbbxx 1/1 Running 0 9m8s 10.244.0.13 node-2 <none> <none>

web-6fc98bbd69-lplz8 1/1 Running 0 2m51s 10.244.0.14 node-2 <none> <none>

web-6fc98bbd69-q5pcs 1/1 Running 0 3m6s 10.244.1.30 node-1 <none> <none>

web-6fc98bbd69-vncjr 1/1 Running 0 3m6s 10.244.3.15 node-3 <none> <none>

web-6fc98bbd69-xzrbm 1/1 Running 0 12m 10.244.2.28 master-1 <none> <none>

web-test-5cdbd79b55-x6snq 1/1 Running 0 22m 10.244.3.14 node-3 <none> <none>

Pod自动扩容/缩容：

冷却周期

在弹性伸缩中，冷却周期是不能逃避的一个话题，由于评估的度量标准是动态特性，副本的数量可能会不断波动，

造成丢失流量，所以不应该在任意时间扩容和缩容。

在 HPA 中，为缓解该问题，默认有一定控制：

– –horizontal-pod-autoscaler-downscale-delay 当前操作完成后等待多次时间才能执行缩容操作，默认5分钟

– –horizontal-pod-autoscaler-upscale-delay 当前操作完成后等待多长时间才能执行扩容操作，默认3分钟

可以通过调整kube-controller-manager组件启动参数调整，一般使用默认参数即可

为了方便测试，我们先停止ab压力测试，记录下时间

[root@master-1 yaml]# date

Mon Jan 4 00:44:16 CST 2021

等待冷却周期（默认5分钟）

[root@master-1 ~]# date
Mon Jan  4 00:49:30 CST 2021

[root@master-1 ~]# kubectl get pods
NAME                        READY   STATUS    RESTARTS   AGE
web-6fc98bbd69-vncjr        1/1     Running   0          25m
web-6fc98bbd69-xzrbm        1/1     Running   0          34m

可以看到 pod 又变回来原先的2个副本数

HPA 指标也显示正常

[root@master-1 ~]# kubectl get hpa
NAME   REFERENCE        TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
web    Deployment/web   0%/80%    2         5         2          31m