K8s binary node expansion scheme

K8s binary node expansion scheme

Based on the following linked documents, the k8s environment is expanded
Link: https://pan.baidu.com/s/1_DopTLhdWujC3_6Ed3gyxg Extraction code: eu2g

There is no document on the network that you can directly copy and use. However, most authors do it according to their own needs. Binary is not as simple as kubeadm. One or two commands are basically done, but it is basically a one-time copy of the deployment document of the cluster at that time. When expanding the node node, we should first consider several issues.

1. Mandatory services on node

docker, cni plug-in (calio, fan), kubelet, Kube proxy

2. if they directly establish a relationship with the k8s master

That is, how do kubelet and Kube proxy establish a connection with apiserver?

From the following example, we can see that the configuration files of all components are recorded to establish a connection with apiserver, and Kube proxy and kubelet are no exception.

[root@master kubernetes]# find . | grep -nr 6443
scheduler.kubeconfig:5:    server:
kubelet.kubeconfig:5:    server:
kube-proxy.kubeconfig:5:    server:
bootstrap-kubelet.kubeconfig:5:    server:
admin.kubeconfig:5:    server:
controller-manager.kubeconfig:5:    server:

In addition, the certificate file is required to establish a connection, because we can see that , HTTPS protocol is used here. In addition, we all know the communication principle of k8s binary. ssl certificate and tls protocol. If you don't understand them, please Baidu first.

3. what if a single master node is expanded to multiple master nodes

It's also simple. It's all in one. I'll add the solution another day. It's not very different from the previous deployment method of a single master node. It's just a question before capacity expansion. Why should we have more masters? I'm sure everyone knows what high availability is. What's the core problem of high availability? We need to understand the two core components, apiserver and etcd. Apiserver is the hub of all components, and etcd stores the metadata of k8s cluster

The apiserver needs to consider that all components and any interaction information must be carried out through the apiserver. If the apiserver is overwhelmed and dies, the cluster will collapse. Therefore, the benefits of multiple masters are obvious. In fact, how to do it is very simple. For example, I use haproxy+keeplived to realize polling through the ip:6443 port of the master node. Will the problem be solved soon.

Etcd is mainly responsible for data redundancy. Even if I hang up a master node, at least one node's etcd stores cluster related information. Etcd currently provides HTTP through port 2379 by default API Service, 2380 port and peer communication. You can directly establish the etcd cluster relationship in the configuration, as shown below

adopt etcd The configuration file of

initial-cluster: 'k8s-master01=,k8s-master02=,k8s-master03=

1. initialize system configuration

1.1 configuring hosts   master1   node1    node2    node3

1.2 configure yum source

curl -o /etc/yum.repos.d/CentOS-Base.repo https://mirrors.aliyun.com/repo/Centos-7.repo
yum install -y yum-utils device-mapper-persistent-data lvm2
yum-config-manager --add-repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
sed -i -e '/mirrors.cloud.aliyuncs.com/d' -e '/mirrors.aliyuncs.com/d' /etc/yum.repos.d/CentOS-Base.repo

1.3 installation of necessary tools

yum install wget jq psmisc vim net-tools telnet yum-utils device-mapper-persistent-data lvm2 git  wget jq psmisc -y

1.4 node shutdown firewalld, dnsmasq, selinux

systemctl disable --now firewalld 
systemctl disable --now dnsmasq
systemctl disable --now NetworkManager

setenforce 0
sed -i 's#SELINUX=enforcing#SELINUX=disabled#g' /etc/sysconfig/selinux
sed -i 's#SELINUX=enforcing#SELINUX=disabled#g' /etc/selinux/config

1.5 the node closes the swap partition and fstab notes swap

swapoff -a && sysctl -w vm.swappiness=0
sed -ri '/^[^#]*swap/s@^@#@' /etc/fstab

1.6 node synchronization time

rpm -ivh http://mirrors.wlnmp.com/centos/wlnmp-release-centos.noarch.rpm
yum install ntpdate -y

ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
echo 'Asia/Shanghai' >/etc/timezone
ntpdate time2.aliyun.com
# Join crontab
*/5 * * * * /usr/sbin/ntpdate time2.aliyun.com

1.7 node configuration limit:

ulimit -SHn 65535

vim /etc/security/limits.conf
# Add the following at the end
* soft nofile 65535
* hard nofile 65535
* soft nproc 65535
* hard nproc 65535
* soft memlock unlimited
* hard memlock unlimited 

1.8 secret free login

#master node execution

ssh-copy-id -i .ssh/id_rsa.pub 

1.9 copying node installation files

#master node execution

scp -r /root/k8s-ha-install/ node3:/root/

1.10 kernel upgrade

yum update -y --exclude=kernel* 

cd /root
cd /root && yum localinstall -y kernel-ml*

grub2-set-default  0 && grub2-mkconfig -o /etc/grub2.cfg
grubby --args="user_namespace.enable=1" --update-kernel="$(grubby --default-kernel)"
grubby --default-kernel


uname -a

1.11 change host name

hostname node3
hostnamectl set-hostname node3

1.12 installing ipvsadm

#All nodes are configured with ipvs module, and NF in kernel 4.19+ version_ conntrack_ IPv4 has been changed to nf_conntrack, NF is used below 4.18_ conntrack_ IPv4 is enough

yum install ipvsadm ipset sysstat conntrack libseccomp -y

modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe -- nf_conntrack
vim /etc/modules-load.d/ipvs.conf 
	# Add the following

#Start service
systemctl enable --now systemd-modules-load.service

#Check if it is loaded
[root@node3 ~]#  lsmod | grep -e ip_vs -e nf_conntrack
nf_conntrack_netlink    40960  0 
nfnetlink              16384  3 nf_conntrack_netlink,ip_set
ip_vs_ftp              16384  0 
nf_nat                 32768  2 nf_nat_ipv4,ip_vs_ftp
ip_vs_sed              16384  0 
ip_vs_nq               16384  0 
ip_vs_fo               16384  0 
ip_vs_dh               16384  0 
ip_vs_lblcr            16384  0 
ip_vs_lblc             16384  0 
ip_vs_wlc              16384  0 
ip_vs_lc               16384  0 
ip_vs_sh               16384  0 
ip_vs_wrr              16384  0 
ip_vs_rr               16384  13 
ip_vs                 151552  37 ip_vs_wlc,ip_vs_rr,ip_vs_dh,ip_vs_lblcr,ip_vs_sh,ip_vs_fo,ip_vs_nq,ip_vs_lblc,ip_vs_wrr,ip_vs_lc,ip_vs_sed,ip_vs_ftp
nf_conntrack          143360  6 xt_conntrack,nf_nat,ipt_MASQUERADE,nf_nat_ipv4,nf_conntrack_netlink,ip_vs
nf_defrag_ipv6         20480  1 nf_conntrack
nf_defrag_ipv4         16384  1 nf_conntrack
libcrc32c              16384  3 nf_conntrack,nf_nat,ip_vs

1.13 enable some necessary kernel parameters in the k8s cluster, and configure the k8s kernel on all nodes:

cat <<EOF > /etc/sysctl.d/k8s.conf
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
fs.may_detach_mounts = 1

net.ipv4.tcp_keepalive_time = 600
net.ipv4.tcp_keepalive_probes = 3
net.ipv4.tcp_keepalive_intvl =15
net.ipv4.tcp_max_tw_buckets = 36000
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_max_orphans = 327680
net.ipv4.tcp_orphan_retries = 3
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_max_syn_backlog = 16384
net.ipv4.ip_conntrack_max = 65536
net.ipv4.tcp_max_syn_backlog = 16384
net.ipv4.tcp_timestamps = 0
net.core.somaxconn = 16384

#Execute command
sysctl --system && sysctl -p

2. install Docker

yum install docker-ce-19.03.* -y

[root@node3 ~]# vim /etc/docker/daemon.json 
#Change the text as follows. It is recommended to copy from other nodes
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
      "max-size": "100m"
  "insecure-registries": ["https://hub.test.com:18443"]

#Start docker
systemctl daemon-reload && systemctl enable --now docker

3. preparation before installing core services

mkdir -p /var/lib/kubelet /var/log/kubernetes /etc/systemd/system/kubelet.service.d /etc/kubernetes/manifests/  /etc/kubernetes/pki /opt/cni/bin

#Execute on the master node
scp /usr/local/bin/kube{let,-proxy} node3:/usr/local/bin/ 

#Copy related certificates to the node node
[root@master1 ~]# for FILE in etcd-ca.pem etcd.pem etcd-key.pem;do  scp /etc/etcd/ssl/$FILE node3:/etc/etcd/ssl/; done
etcd-ca.pem                                                                                                                                                  100% 1367   672.2KB/s   00:00    
etcd.pem                                                                                                                                                     100% 1452     1.3MB/s   00:00    
etcd-key.pem                                                                                                                                                 100% 1675     1.3MB/s   00:00    
[root@master1 ~]# for FILE in pki/ca.pem pki/ca-key.pem pki/front-proxy-ca.pem bootstrap-kubelet.kubeconfig; do  scp /etc/kubernetes/$FILE node3:/etc/kubernetes/${FILE}; done
ca.pem                                                                                                                                                       100% 1411   966.2KB/s   00:00    
ca-key.pem                                                                                                                                                   100% 1675     1.3MB/s   00:00    
front-proxy-ca.pem                                                                                                                                           100% 1143   899.7KB/s   00:00    
bootstrap-kubelet.kubeconfig                                                                                                                                 100% 2300   139.2KB/s   00:00   

4. configure and install kubelet

vim  /usr/lib/systemd/system/kubelet.service

Description=Kubernetes Kubelet




vim /etc/systemd/system/kubelet.service.d/10-kubelet.conf

Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig --kubeconfig=/etc/kubernetes/kubelet.kubeconfig"
Environment="KUBELET_SYSTEM_ARGS=--network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin"
Environment="KUBELET_CONFIG_ARGS=--config=/etc/kubernetes/kubelet-conf.yml --pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/google_containers/pause-amd64:3.2"
Environment="KUBELET_EXTRA_ARGS=--node-labels=node.kubernetes.io/node='' "

vim /etc/kubernetes/kubelet-conf.yml
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
port: 10250
readOnlyPort: 10255
    enabled: false
    cacheTTL: 2m0s
    enabled: true
    clientCAFile: /etc/kubernetes/pki/ca.pem
  mode: Webhook
    cacheAuthorizedTTL: 5m0s
    cacheUnauthorizedTTL: 30s
cgroupDriver: systemd
cgroupsPerQOS: true
clusterDomain: cluster.local
containerLogMaxFiles: 5
containerLogMaxSize: 10Mi
contentType: application/vnd.kubernetes.protobuf
cpuCFSQuota: true
cpuManagerPolicy: none
cpuManagerReconcilePeriod: 10s
enableControllerAttachDetach: true
enableDebuggingHandlers: true
- pods
eventBurst: 10
eventRecordQPS: 5
  imagefs.available: 15%
  memory.available: 100Mi
  nodefs.available: 10%
  nodefs.inodesFree: 5%
evictionPressureTransitionPeriod: 5m0s
failSwapOn: true
fileCheckFrequency: 20s
hairpinMode: promiscuous-bridge
healthzPort: 10248
httpCheckFrequency: 20s
imageGCHighThresholdPercent: 85
imageGCLowThresholdPercent: 80
imageMinimumGCAge: 2m0s
iptablesDropBit: 15
iptablesMasqueradeBit: 14
kubeAPIBurst: 10
kubeAPIQPS: 5
makeIPTablesUtilChains: true
maxOpenFiles: 1000000
maxPods: 110
nodeStatusUpdateFrequency: 10s
oomScoreAdj: -999
podPidsLimit: -1
registryBurst: 10
registryPullQPS: 5
resolvConf: /etc/resolv.conf
rotateCertificates: true
runtimeRequestTimeout: 2m0s
serializeImagePulls: true
staticPodPath: /etc/kubernetes/manifests
streamingConnectionIdleTimeout: 4h0m0s
syncFrequency: 1m0s
volumeStatsAggPeriod: 1m0s

#Start service

systemctl daemon-reload
systemctl enable --now kubelet

#master execution
[root@master1 ~]# kubectl get nodes
master1   Ready    <none>   29d     v1.20.0
node1     Ready    <none>   29d     v1.20.0
node2     Ready    <none>   29d     v1.20.0
node3     Ready    <none>   3m      v1.20.0

Problem sorting:

[root@node3 ~]# journalctl -xefu kubelet | grep fai
Jun 02 19:33:39 node3 kubelet[24182]: F0602 19:33:39.786231   24182 server.go:269] failed to run Kubelet: misconfiguration: kubelet cgroup driver: "systemd" is different from docker cgroup driver: "cgroupfs"
Jun 02 19:33:39 node3 systemd[1]: Unit kubelet.service entered failed state.
Jun 02 19:33:39 node3 systemd[1]: kubelet.service failed.
Jun 02 19:33:54 node3 kubelet[24431]: E0602 19:33:54.914011   24431 kubelet.go:1271] Image garbage collection failed once. Stats initialization may not have completed yet: failed to get imageFs info: unable to find data in memory cache
Jun 02 19:33:54 node3 kubelet[24431]: E0602 19:33:54.922192   24431 nodelease.go:49] failed to get node "node3" when trying to set owner ref to the node lease: nodes "node3" not found
Jun 02 19:33:54 node3 kubelet[24431]: E0602 19:33:54.993972   24431 eviction_manager.go:260] eviction manager: failed to get summary stats: failed to get node info: node "node3" not found
Jun 02 19:33:55 node3 kubelet[24431]: W0602 19:33:55.518361   24431 driver-call.go:149] FlexVolume: driver call failed: executable: /usr/libexec/kubernetes/kubelet-plugins/volume/exec/nodeagent~uds/uds, args: [init], error: fork/exec /usr/libexec/kubernetes/kubelet-plugins/volume/exec/nodeagent~uds/uds: no such file or directory, output: ""
##Check /etc/docker/daemon JSON file configuration, preferably copied directly from other healthy node nodes
Jun 02 19:36:34 node3 kubelet[26467]: E0602 19:36:34.012055   26467 pod_workers.go:191] Error syncing pod 5a0942a7-1b9e-46a1-b4d0-e25afc5eabd5 ("calico-node-7zvxw_kube-system(5a0942a7-1b9e-46a1-b4d0-e25afc5eabd5)"), skipping: failed to "StartContainer" for "calico-node" with CrashLoopBackOff: "back-off 10s restarting failed container=calico-node pod=calico-node-7zvxw_kube-system(5a0942a7-1b9e-46a1-b4d0-e25afc5eabd5)"
Jun 02 19:36:42 node3 kubelet[26467]: E0602 19:36:42.628358   26467 pod_workers.go:191] Error syncing pod 5a0942a7-1b9e-46a1-b4d0-e25afc5eabd5 ("calico-node-7zvxw_kube-system(5a0942a7-1b9e-46a1-b4d0-e25afc5eabd5)"), skipping: failed to "StartContainer" for "calico-node" with CrashLoopBackOff: "back-off 10s restarting failed container=calico-node pod=calico-node-7zvxw_kube-system(5a0942a7-1b9e-46a1-b4d0-e25afc5eabd5)"
Jun 02 19:36:59 node3 kubelet[26467]: E0602 19:36:59.097043   26467 pod_workers.go:191] Error syncing pod 5a0942a7-1b9e-46a1-b4d0-e25afc5eabd5 ("calico-node-7zvxw_kube-system(5a0942a7-1b9e-46a1-b4d0-e25afc5eabd5)"), skipping: failed to "StartContainer" for "calico-node" with CrashLoopBackOff: "back-off 20s restarting failed container=calico-node pod=calico-node-7zvxw_kube-system(5a0942a7-1b9e-46a1-b4d0-e25afc5eabd5)"
Jun 02 19:37:02 node3 kubelet[26467]: E0602 19:37:02.628171   26467 pod_workers.go:191] Error syncing pod 5a0942a7-1b9e-46a1-b4d0-e25afc5eabd5 ("calico-node-7zvxw_kube-system(5a0942a7-1b9e-46a1-b4d0-e25afc5eabd5)"), skipping: failed to "StartContainer" for "calico-node" with CrashLoopBackOff: "back-off 20s restarting failed container=calico-node pod=calico-node-7zvxw_kube-system(5a0942a7-1b9e-46a1-b4d0-e25afc5eabd5)"

#It doesn't matter because I didn't install Kube proxy, so the cni plug-in CrashLoopBackOff

#There are some strange logs, but if kubelet can be started normally, don't worry about them first, because the dependency problems like Kube proxy and cni can be solved by restarting after installation

5. configure and install Kube proxy

master node operation

 scp /etc/kubernetes/kube-proxy.kubeconfig node3:/etc/kubernetes/kube-proxy.kubeconfig
 scp kube-proxy/kube-proxy.conf node3:/etc/kubernetes/kube-proxy.conf
 scp kube-proxy/kube-proxy.service node3:/usr/lib/systemd/system/kube-proxy.service

Node3 node starts Kube proxy

systemctl daemon-reload
systemctl enable --now kube-proxy


I use calico here. Because similar system plug-ins are deployed in the daemonset mode, you can directly apply

kubectl apply -f calico-etcd.yaml

kubectl  get po -n kube-system

If the container status is abnormal, you can use kubectl describe or logs to view the container log

Tags: Linux Docker Kubernetes Container

Posted by mlschutz on Thu, 02 Jun 2022 22:06:27 +0530