Kubernetes high availability cluster deployment

Original link: https://www.cpweb.top/2396

1, Environment

All three nodes of kubernetes are in the master role. The high availability scheme adopts kept + haproxy. If the server is in the cloud, SLB + haproxy is recommended.

Server information table:

|Hostname | system | configuration | IP | installation software|
|-|-|
| k8s1 | Centos 7.6.1810 | 2C4G | 10.0.0.101 | Kubernetes |
| k8s2 | Centos 7.6.1810 | 2C4G | 10.0.0.102 | Kubernetes |
| k8s3 | Centos 7.6.1810 | 2C4G | 10.0.0.103 | Kubernetes |
| nginx1 | Centos 7.6.1810 | 1C1G | 10.0.0.104 | keepalived,haproxy,openresty |
| nginx2 | Centos 7.6.1810 | 1C1G | 10.0.0.105 | keepalived,haproxy,openresty |

Software version:

Softwareedition
Kubernetes1.22.2
haproxy2.2.22
keepalived2.0.20
openresty1.19.3.1

2, Environment initialization

1. All nodes initialization

#!/bin/bash
set -e

# Turn off the firewall and selinux (not required for alicloud environment, please note)
systemctl stop firewalld && systemctl disable firewalld
sed -i '/^SELINUX=/c SELINUX=disabled' /etc/selinux/config
setenforce 0

# Source change (not required for Alibaba cloud environment, please note)
mkdir /etc/yum.repos.d/centos
mv /etc/yum.repos.d/*.repo /etc/yum.repos.d/centos/
curl -o /etc/yum.repos.d/CentoS-7.repo http://mirrors.aliyun.com/repo/Centos-7.repo    
curl -o /etc/yum.repos.d/epel7.repo http://mirrors.aliyun.com/repo/epel-7.repo
yum install -y gcc gcc-c++ vim net-tools telnet dos2unix iptraf ntp tree sysstat lrzsz bash-completion chrony wget unzip

# DNS Optimization (not required for Alibaba cloud environment, please note)
sed -i 's/#PermitRootLogin yes/PermitRootLogin yes/g' /etc/ssh/sshd_config
sed -i 's/#UseDNS yes/UseDNS no/g' /etc/ssh/sshd_config
sed -i 's/GSSAPIAuthentication yes/GSSAPIAuthentication no/g' /etc/ssh/sshd_config
systemctl restart sshd

# Time synchronization (not required in Alibaba cloud environment, please note)
timedatectl set-timezone Asia/Shanghai
sed -i 's/^server/#&/' /etc/chrony.conf
#echo "server cn.pool.ntp.org iburst" >> /etc/chrony.conf
cat >> /etc/chrony.conf << EOF
server ntp1.aliyun.com minpoll 4 maxpoll 10 iburst
server ntp2.aliyun.com minpoll 4 maxpoll 10 iburst
server ntp3.aliyun.com minpoll 4 maxpoll 10 iburst
server ntp4.aliyun.com minpoll 4 maxpoll 10 iburst
server ntp5.aliyun.com minpoll 4 maxpoll 10 iburst
server ntp6.aliyun.com minpoll 4 maxpoll 10 iburst
EOF
systemctl restart chronyd.service
systemctl enable chronyd.service

# File handles and thread restrictions
cat >> /etc/security/limits.conf << EOF 
root            soft    nofile          655350
root            hard    nofile          655350
*               soft    nofile          655350
*               hard    nofile          655350
*               soft    nofile          655350
*               hard    nofile          655350
EOF
# Change to: * soft nproc unlimited
sed -i '/^\*/s/4096/unlimited/' /etc/security/limits.d/20-nproc.conf

# Kernel parameter optimization
cat >> /etc/sysctl.conf << EOF 
net.ipv4.tcp_tw_reuse = 1
net.ipv4.ip_local_port_range = 10000 65000
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_max_tw_buckets = 36000
net.ipv4.tcp_max_syn_backlog = 16384
net.ipv4.tcp_keepalive_time = 600
net.ipv4.tcp_fin_timeout = 30
vm.swappiness=0
vm.max_map_count = 262144
kernel.pid_max = 65535
net.core.somaxconn=33268
EOF
sysctl -p

# Close swap 
swapoff -a
sed -i 's/^.*centos-swap/#&/g' /etc/fstab

# Modify file and process restrictions for the service
echo "DefaultLimitNOFILE=65535" >> /etc/systemd/system.conf 
echo "DefaultLimitNPROC=65535" >> /etc/systemd/system.conf
systemctl daemon-reexec

# Configure host mapping
cat << EOF >> /etc/hosts
10.0.0.101 k8s1
10.0.0.102 k8s2
10.0.0.103 k8s3
10.0.0.104 nginx1
10.0.0.105 nginx2
EOF

2. k8s node initialization

#!/bin/bash
set -e

# Activate br_netfilter module
modprobe br_netfilter
cat << EOF > /etc/modules-load.d/k8s.conf
br_netfilter
EOF

# Allow iptables to process the data of the bridge
cat << EOF > /etc/sysctl.d/k8s.conf 
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
EOF
sysctl --system

# Install docker
yum install -y yum-utils device-mapper-persistent-data lvm2
yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
yum install -y docker-ce
systemctl enable docker && systemctl start docker 
cat << EOF > /etc/docker/daemon.json
{
  "data-root": "/data/docker",
  "exec-opts": ["native.cgroupdriver=systemd"]
}
EOF
systemctl enable docker && systemctl restart docker 

# Configure k8s source
cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=http://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=http://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg http://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

# Install kubeadm, kubectl, kubelet
yum install -y --nogpgcheck kubeadm-1.22.2 kubectl-1.22.2 kubelet-1.22.2
systemctl enable kubelet
kubeadm completion bash > /etc/bash_completion.d/kubeadm
kubectl completion bash >/etc/bash_completion.d/kubectl
source /etc/bash_completion.d/kubeadm
source /etc/bash_completion.d/kubectl

3, keepalived and haproxy deployment

Reference documents: https://github.com/kubernetes/kubeadm/blob/main/docs/ha-considerations.md

1,haprxoy

Compile and install:

wget https://www.haproxy.org/download/2.2/src/haproxy-2.2.22.tar.gz
yum install -y systemd-devel

tar -xf haproxy-2.2.22.tar.gz -C /usr/local/
cd /usr/local/haproxy-2.2.22/ 
make TARGET=linux-glibc USE_SYSTEMD=1
make install

Register to systemd:

cat  > /usr/lib/systemd/system/haproxy.service << EOF
[Unit]
Description=HAProxy Load Balancer
After=syslog.target network.target
 
[Service]
ExecStartPre=/usr/local/sbin/haproxy -f /etc/haproxy/haproxy.cfg   -c -q
ExecStart=/usr/local/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg  -p /run/haproxy.pid
ExecReload=/bin/kill -USR2 $MAINPID
 
[Install]
WantedBy=multi-user.target
EOF

Prepare profile:

mkdir /etc/haproxy/
cat > /etc/haproxy/haproxy.cfg << EOF
#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
    # log /dev/log local0
    # log /dev/log local1 notice
    log 127.0.0.1 local2 info
    daemon
 
#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
    mode                    http
    log                     global
    option                  httplog
    option                  dontlognull
    option http-server-close
    option forwardfor       except 127.0.0.0/8
    option                  redispatch
    retries                 1
    timeout http-request    10s
    timeout queue           20s
    timeout connect         5s
    timeout client          20s
    timeout server          20s
    timeout http-keep-alive 10s
    timeout check           10s
 
#---------------------------------------------------------------------
# apiserver frontend which proxys to the control plane nodes
#---------------------------------------------------------------------
frontend apiserver
    bind *:6443
    mode tcp
    option tcplog
    default_backend apiserver
 
#---------------------------------------------------------------------
# round robin balancing for apiserver
#---------------------------------------------------------------------
backend apiserver
    option httpchk GET /healthz
    http-check expect status 200
    mode tcp
    option ssl-hello-chk
    balance     roundrobin
        server k8s1 10.0.0.101:6443 check
        server k8s2 10.0.0.102:6443 check
        server k8s3 10.0.0.103:6443 check

listen admin_stats      # Configuration status page
    mode http           # Operating mode
    bind *:1080         # Listening port
    stats enable        # Enable status page
    stats hide-version  # Hidden version
    maxconn 1024        # maximum connection
    stats refresh 30s   # Automatic page refresh time 30s
    stats uri /stats    # Access path
    #stats admin if TRUE # Enable management functions
    stats auth admin:000000  # Account and password
EOF

systemctl enable haproxy
systemctl start haproxy

2,keepalived

Compile and install:

wget --no-check-certificate https://www.keepalived.org/software/keepalived-2.0.20.tar.gz 
yum install -y openssl-devel libnl libnl-devel

tar -xf keepalived-2.0.20.tar.gz -C /usr/local/
cd /usr/local/keepalived-2.0.20
./configure --prefix=/usr/local/keepalived/
make USE_SYSTEMD=1
make install

To modify a profile:

ln -s /usr/local/keepalived/etc/keepalived /etc/keepalived
cp /etc/keepalived/keepalived.con{f,f.bak}

Configuration file of nignx1:

cat > /etc/keepalived/keepalived.conf << EOF
! /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
    router_id nginx1
}
vrrp_script check_apiserver {
  script "/etc/keepalived/check_apiserver.sh"
  interval 3
  weight -2
  fall 10
  rise 2
}

vrrp_instance VI_1 {
    state MASTER
    interface ens33
    virtual_router_id 1
    priority 101
    authentication {
        auth_type PASS
        auth_pass 0000
    }
    virtual_ipaddress {
        10.0.0.100
    }
    track_script {
        check_apiserver
    }
}
EOF
systemctl enable keepalived
systemctl start keepalived

Configuration file of nginx2:

cat > /etc/keepalived/keepalived.conf << EOF
! /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
    router_id nginx2
}
vrrp_script check_apiserver {
  script "/etc/keepalived/check_apiserver.sh"
  interval 3
  weight -2
  fall 10
  rise 2
}

vrrp_instance VI_1 {
    state BACKUP
    interface ens33
    virtual_router_id 1
    priority 100
    authentication {
        auth_type PASS
        auth_pass 0000
    }
    virtual_ipaddress {
        10.0.0.100
    }
    track_script {
        check_apiserver
    }
}
EOF
systemctl enable keepalived
systemctl start keepalived

To prepare a health check script:

cat > /etc/keepalived/check_apiserver.sh << EOF
#!/bin/sh

errorExit() {
    echo "*** $*" 1>&2
    exit 1
}

curl --silent --max-time 2 --insecure https://localhost:6443/ -o /dev/null || errorExit "Error GET https://localhost:${APISERVER_DEST_PORT}/"
if ip addr | grep -q 10.0.0.100; then
    curl --silent --max-time 2 --insecure https://10.0.0.100:6443/ -o /dev/null || errorExit "Error GET https://10.0.0.100:6443/"
fi
EOF

4, Deploy kubernetes cluster

kubeadm here is recompiled, and the certificate period is 10 years.

kubeadm init --control-plane-endpoint 10.0.0.100:6443 --upload-certs --image-repository=registry.aliyuncs.com/google_containers --kubernetes-version=v1.22.2 --service-cidr=10.96.0.0/16 --pod-network-cidr=10.244.0.0/16 --ignore-preflight-errors=Swap

The following information can be seen after successful initialization:

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of the control-plane node running the following command on each as root:

  kubeadm join 10.0.0.100:6443 --token m8d5kv.v90803q5327gjqgz \
	--discovery-token-ca-cert-hash sha256:7d1e394443e4aea3b27919d90d25c63bf79edaba003bce454fb349f1c6b09b78 \
	--control-plane --certificate-key ca3c8e52b99fc9ea32964b4ed52a6e74603b5b7c4acfa99016196979f5c20773

Please note that the certificate-key gives access to cluster sensitive data, keep it secret!
As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use
"kubeadm init phase upload-certs --upload-certs" to reload certs afterward.

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 10.0.0.100:6443 --token m8d5kv.v90803q5327gjqgz \
	--discovery-token-ca-cert-hash sha256:7d1e394443e4aea3b27919d90d25c63bf79edaba003bce454fb349f1c6b09b78 

  follow the prompts. Since we are three master nodes, we need to choose the command to add the control plane node to the other two servers.

[root@k8s1 ~]# kubectl get node
NAME   STATUS     ROLES                  AGE     VERSION
k8s1   NotReady   control-plane,master   24m     v1.22.2
k8s2   NotReady   control-plane,master   2m30s   v1.22.2
k8s3   NotReady   control-plane,master   2m27s   v1.22.2

# Modify the stain to allow Pod scheduling on the master node
kubectl taint nodes --all node-role.kubernetes.io/master-

5, Network component calico

Reference documents: https://projectcalico.docs.tigera.io/archive/v3.18/getting-started/kubernetes/self-managed-onprem/onpremises

1. Installing calico

Download and install the yaml file. We will customize the file later. Since etcd turns on tls, specify etcd certificate

curl https://docs.projectcalico.org/archive/v3.18/manifests/calico-etcd.yaml -o calico.yaml

Modify the etcd cluster address. Since etcd enables tls, we need to specify the certificate.

kind: ConfigMap
apiVersion: v1
metadata:
  name: calico-config
  namespace: kube-system
data:
  # Configure this with the location of your etcd cluster.
  etcd_endpoints: "https://10.0.0.101:2379,https://10.0.0.102:2379,https://10.0.0.103:2379"
  # If you're using TLS enabled etcd uncomment the following.
  # You must also populate the Secret below with these files.
  etcd_ca: "/calico-secrets/etcd-ca"      # "/calico-secrets/etcd-ca"
  etcd_cert: "/calico-secrets/etcd-cert"  # "/calico-secrets/etcd-cert"
  etcd_key: "/calico-secrets/etcd-key"    # "/calico-secrets/etcd-key"

First obtain the base64 code of the certificate, and then in the secret calico etcd secrets, uncomment etcd_ca,etcd_key and etcd_cert fills in the base64 encoding obtained earlier.

# Get certificate base64 encoding
cat /etc/kubernetes/pki/etcd/ca.crt | base64 -w 0               # calico-etcd-secrets etcd-ca
cat /etc/kubernetes/pki/apiserver-etcd-client.crt | base64 -w 0 # calico-etcd-secrets etcd-cert
cat /etc/kubernetes/pki/apiserver-etcd-client.key | base64 -w 0 # calico-etcd-secrets etcd-key

# Fill in certificate
apiVersion: v1
kind: Secret
type: Opaque
metadata:
  name: calico-etcd-secrets
  namespace: kube-system
data:
  # Populate the following files with etcd TLS configuration if desired, but leave blank if
  # not using TLS for etcd.
  # This self-hosted install expects three files with the following names.  The values
  # should be base64 encoded strings of the entire contents of each file.
  etcd-key: xxx
  etcd-cert: xxx
  etcd-ca: xxx

Modify the secret file permission of deployment calico Kube controllers:

      volumes:
        # Mount in the etcd TLS secrets with mode 400.
        # See https://kubernetes.io/docs/concepts/configuration/secret/
        - name: etcd-certs
          secret:
            secretName: calico-etcd-secrets
            defaultMode: 0440

Modify the IP auto discovery method. The IP segment is the IP segment of the host. Add the following environment variable in the DaemonSet.

            - name: IP_AUTODETECTION_METHOD
              value: "cidr=10.0.0.0/24"

After modification, execute the installation:

kubectl apply -f calico.yaml

2. Configure dual RR

To install calicoctl:

# Install calicoctl
curl -O -L  https://github.com/projectcalico/calicoctl/releases/download/v3.18.6/calicoctl
chmod a+x calicoctl
mv calicoctl /usr/local/bin/

# Prepare profile
mkdir /etc/calico
cat > /etc/calico/calicoctl.cfg << EOF
apiVersion: projectcalico.org/v3
kind: CalicoAPIConfig
metadata:
spec:
  etcdEndpoints: "https://10.0.0.101:2379,https://10.0.0.102:2379,https://10.0.0.103:2379"
  etcdKeyFile: /etc/kubernetes/pki/etcd/server.key
  etcdCertFile: /etc/kubernetes/pki/etcd/server.crt
  etcdCACertFile: /etc/kubernetes/pki/etcd/ca.crt
EOF

[root@k8s1 ~]# calicoctl get node
NAME   
k8s1   
k8s2   
k8s3 

Configure bgp and disable full mesh:

[root@k8s1 calico]# vim bgpconfig_default.yaml 
apiVersion: projectcalico.org/v3
kind: BGPConfiguration
metadata:
  name: default
spec:
  logSeverityScreen: Info
  nodeToNodeMeshEnabled: false
  
[root@k8s1 calico]# calicoctl create -f bgpconfig_default.yaml
Successfully created 1 'BGPConfiguration' resource(s)

[root@k8s1 calico]# calicoctl get bgpconfig
NAME      LOGSEVERITY   MESHENABLED   ASNUMBER   
default   Info          false         - 

Configure the connection establishment rules between BGP node and Route Reflector (rr). Set rules to distinguish node roles by labels:

[root@k8s1 calico]# vim bgppeer.yaml 
kind: BGPPeer
apiVersion: projectcalico.org/v3
metadata:
  name: peer-to-rrs
spec:
  # Rule 1: ordinary BGP nodes establish connections with RRS
  nodeSelector: "!has(route-reflector)"
  peerSelector: has(route-reflector)
---
kind: BGPPeer
apiVersion: projectcalico.org/v3
metadata:
  name: rr-mesh
spec:
  # Rule 2: establish connection between RRS
  nodeSelector: has(route-reflector)
  peerSelector: has(route-reflector)
  
[root@k8s1 calico]# calicoctl create -f bgppeer.yaml
Successfully created 2 'BGPPeer' resource(s)
[root@k8s1 calico]# calicoctl get bgpPeer
NAME          PEERIP   NODE                    ASN   
peer-to-rrs            !has(route-reflector)   0     
rr-mesh                has(route-reflector)    0 

Add rr:

# Export node configuration
[root@k8s1 calico]# calicoctl get node k8s2 --export -o yaml > rr-k8s2.yaml
[root@k8s1 calico]# calicoctl get node k8s3 --export -o yaml > rr-k8s3.yaml

# Modify the configuration and add the following configuration under the corresponding block
metadata:
  labels:
    # Add a tag and set the rr tag to true
    route-reflector: "true"
 
spec:
  bgp:
    # Add a cluster ID. rr-k8s2 is consistent with rr-k8s3 for redundancy and ring protection.
    routeReflectorClusterID: 10.255.11.11

[root@k8s1 calico]# calicoctl apply -f rr-k8s2.yaml 
Successfully applied 1 'Node' resource(s)
[root@k8s1 calico]# calicoctl apply -f rr-k8s3.yaml 
Successfully applied 1 'Node' resource(s)
[root@k8s1 calico]# calicoctl node status
Calico process is running.

IPv4 BGP status
+--------------+---------------+-------+----------+-------------+
| PEER ADDRESS |   PEER TYPE   | STATE |  SINCE   |    INFO     |
+--------------+---------------+-------+----------+-------------+
| 10.0.0.102   | node specific | up    | 06:55:25 | Established |
| 10.0.0.103   | node specific | up    | 06:55:28 | Established |
+--------------+---------------+-------+----------+-------------+

IPv6 BGP status
No IPv6 peers found.

We can perform calicoctl node status verification on each node. We can see that one node RR1 and RR2 establish connections, while other nodes establish connections with RR1 and RR2 respectively. They are not directly connected to each other.

6, Pass through Pod for servers outside the cluster

Add an IPIP tunnel network card to the two nginx machines. Note: since the one to many IPIP tunnels are controlled by the routing rules, the network card name is fixed as tunl0.
nginx1 is configured as 10.222.1.2, and nginx2 is configured as 10.222.1.3.

# Create tunl0 network card
ip tunnel add tunl0 mode ipip
ip link set tunl0 up

# Configure the IPIP tunnel network card IP. This IP should not be a cluster IP and does not conflict with other private IP
ip addr add 10.222.1.2/32 dev tunl0

Add a k8s cluster route. The gateway is the k8s node instance IP. Note that the network segment and its corresponding gateway need to execute calicoctl ipam check to obtain.

# k8s node execution
[root@k8s1 calico]# calicoctl ipam check
Checking IPAM for inconsistencies...

Loading all IPAM blocks...
Found 3 IPAM blocks.
 IPAM block 10.244.109.64/26 affinity=host:k8s2:
 IPAM block 10.244.166.192/26 affinity=host:k8s1:
 IPAM block 10.244.219.0/26 affinity=host:k8s3:
IPAM blocks record 5 allocations.

# nginx node execution
[root@nginx1 ~]# ip route add 10.244.109.64/26 via 10.0.0.102 dev tunl0 onlink
[root@nginx1 ~]# ip route add 10.244.166.192/26 via 10.0.0.101 dev tunl0 onlink
[root@nginx1 ~]# ip route add 10.244.219.0/26 via 10.0.0.103 dev tunl0 onlink

k8s add backhaul routes to all nodes:

ip route add 10.222.1.2/32 via 10.0.0.104 dev tunl0 onlink
ip route add 10.222.1.3/32 via 10.0.0.105 dev tunl0 onlink

Finally, the calico configuration needs to be modified, and the external node list needs to be modified:

[root@k8s1 calico]# cat FelixConfiguration_default.yaml 
apiVersion: projectcalico.org/v3
kind: FelixConfiguration
metadata:
  name: default
spec:
  bpfLogLevel: ""
  ipipEnabled: true
  logSeverityScreen: Info
  reportingInterval: 0s
  externalNodesList: ["10.0.0.104", "10.0.0.105"]

[root@k8s1 calico]# calicoctl apply -f FelixConfiguration_default.yaml 
Successfully applied 1 'FelixConfiguration' resource(s)

Overall command execution:

# nginx1 execution
ip tunnel add tunl0 mode ipip
ip link set tunl0 up
ip addr add 10.222.1.2/32 dev tunl0
ip route add 10.244.109.64/26 via 10.0.0.102 dev tunl0 onlink
ip route add 10.244.166.192/26 via 10.0.0.101 dev tunl0 onlink
ip route add 10.244.219.0/26 via 10.0.0.103 dev tunl0 onlink

# nginx2 execution
ip tunnel add tunl0 mode ipip
ip link set tunl0 up
ip addr add 10.222.1.3/32 dev tunl0
ip route add 10.244.109.64/26 via 10.0.0.102 dev tunl0 onlink
ip route add 10.244.166.192/26 via 10.0.0.101 dev tunl0 onlink
ip route add 10.244.219.0/26 via 10.0.0.103 dev tunl0 onlink

# All k8s nodes execute
ip route add 10.222.1.2/32 via 10.0.0.104 dev tunl0 onlink
ip route add 10.222.1.3/32 via 10.0.0.105 dev tunl0 onlink

Add route after startup:

# All nginx nodes execute (note the IP set for tunl0)
cat >> /etc/rc.local << EOF
ip link set tunl0 up
ip addr add 10.222.1.2/32 dev tunl0
ip route add 10.244.109.64/26 via 10.0.0.102 dev tunl0 onlink
ip route add 10.244.166.192/26 via 10.0.0.101 dev tunl0 onlink
ip route add 10.244.219.0/26 via 10.0.0.103 dev tunl0 onlink
EOF
chmod a+x /etc/rc.local

# All k8s nodes execute
cat >> /etc/rc.local << EOF
ip link set tunl0 up
ip route add 10.222.1.2/32 via 10.0.0.104 dev tunl0 onlink
ip route add 10.222.1.3/32 via 10.0.0.105 dev tunl0 onlink
EOF
chmod a+x /etc/rc.local

verification:

[root@k8s1 ~]# kubectl run test --image=nginx
pod/test created
[root@k8s1 ~]# kubectl get pod -o wide
NAME   READY   STATUS    RESTARTS   AGE    IP             NODE   NOMINATED NODE   READINESS GATES
test   1/1     Running   0          115s   10.244.219.1   k8s3   <none>           <none>

[root@nginx1 ~]# ping -c1 10.244.219.1
PING 10.244.219.1 (10.244.219.1) 56(84) bytes of data.
64 bytes from 10.244.219.1: icmp_seq=1 ttl=63 time=0.312 ms

--- 10.244.219.1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.312/0.312/0.312/0.000 ms
[root@nginx1 ~]# curl 10.244.219.1
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
......

7, Deployment of other components

1,ingress-nginx

Official documents: https://kubernetes.github.io/ingress-nginx/deploy/
Get installation yaml file

wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.2.0/deploy/static/provider/baremetal/deploy.yaml

First, set a label for the node to control the ingress nginx controller to run on the specified node.

kubectl label nodes k8s1 ingress-nginx-node=true
kubectl label nodes k8s2 ingress-nginx-node=true

Modify the deployment ingress nginx controller, set the number of copies to 2, enable the hostNetwork, and add the nodeSelector configuration.

  replicas: 2
......
      hostNetwork: true
      nodeSelector:
        ingress-nginx-node: "true" 

Deployment:

[root@k8s1 ingress-nginx]# kubectl apply -f deploy.yaml
[root@k8s1 ingress-nginx]# kubectl get pod -n ingress-nginx -o wide
NAME                                       READY   STATUS      RESTARTS   AGE     IP             NODE   NOMINATED NODE   READINESS GATES
ingress-nginx-admission-create--1-xfz7k    0/1     Completed   0          38m     10.244.219.8   k8s3   <none>           <none>
ingress-nginx-controller-577876995-8wgzc   1/1     Running     0          4m27s   10.0.0.101     k8s1   <none>           <none>
ingress-nginx-controller-577876995-wggm4   1/1     Running     0          4m28s   10.0.0.102     k8s2   <none>           <none>

A ladder may be required to install the ingress nginx kubectl plug-in. Here I install krew and plug-ins, and package /root/ krew/ directory. Just unzip it to the corresponding machine.

[root@k8s1 ~]# tar -xf krew.tar.gz
[root@k8s1 ~]# export PATH="${KREW_ROOT:-$HOME/.krew}/bin:$PATH"
[root@k8s1 ~]# kubectl krew version
OPTION            VALUE
GitTag            v0.4.3
GitCommit         dbfefa5
IndexURI          https://github.com/kubernetes-sigs/krew-index.git
BasePath          /root/.krew
IndexPath         /root/.krew/index/default
InstallPath       /root/.krew/store
BinPath           /root/.krew/bin
DetectedPlatform  linux/amd64

[root@k8s1 ~]# kubectl krew list
PLUGIN         VERSION
images         v0.3.7
ingress-nginx  v0.31.0
krew           v0.4.3

2,dashboard

Official documents: https://github.com/kubernetes/dashboard

[root@k8s1 dashboard]# wget https://raw.githubusercontent.com/kubernetes/dashboard/v2.5.1/aio/deploy/recommended.yaml
[root@k8s1 dashboard]# kubectl apply -f recommended.yaml 

[root@k8s1 dashboard]# kubectl get pod -n kubernetes-dashboard
NAME                                        READY   STATUS    RESTARTS   AGE
dashboard-metrics-scraper-c45b7869d-spc8w   1/1     Running   0          3m45s
kubernetes-dashboard-79b5779bf4-m8kpp       1/1     Running   0          3m45s

Create a user, bind the cluster role, and obtain a token.

[root@k8s1 dashboard]# vim dashboard-user.yaml 
apiVersion: v1
kind: ServiceAccount 
metadata:
  name: admin-user 
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: admin-user
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: admin-user
  namespace: kube-system

[root@k8s1 dashboard]# kubectl apply -f dashboard-user.yaml 

[root@k8s1 dashboard]# kubectl -n kube-system get secret $(kubectl -n kube-system get sa/admin-user -o jsonpath="{.secrets[0].name}") -o go-template="{{.data.token | base64decode}}" > dashboard-token.txt

3,Metrics Server

Official documents: https://github.com/kubernetes-sigs/metrics-server

[root@k8s1 ~]# wget https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/high-availability.yaml
[root@k8s1 ~]# vim high-availability   # Add variable
......
    spec:
      containers:
      - args:
        - --kubelet-insecure-tls    # newly added
......
[root@k8s1 ~]# kubectl apply -f high-availability.yaml
serviceaccount/metrics-server created

[root@k8s1 ~]# kubectl top node
NAME   CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%   
k8s1   156m         7%     1437Mi          39%       
k8s2   205m         10%    1207Mi          44%       
k8s3   187m         9%     1234Mi          45%     

8, Optimization

1. kubelet reserved resources

Official documents: https://kubernetes.io/docs/tasks/administer-cluster/reserve-compute-resources/
Reserved according to actual conditions:

[root@k8s1 ~]# vim /var/lib/kubelet/config.yaml
# Reserve resources for the system
systemReserved:
  cpu: "100m"
  memory: "256Mi"
# Reserve resources for kubernetes system Daemons
kubeReserved:
  cpu: "100m"
  memory: "256Mi"
# Configure the hard eviction threshold. If the memory is lower than the threshold, the pod will be evicted
evictionHard:
  memory.available: "128Mi"
  
[root@k8s1 ~]# systemctl restart kubele
[root@k8s1 ~]# kubectl describe nodes k8s1 | grep -E -A 5 'Capacity:|Allocatable:'
Capacity:
  cpu:                2
  ephemeral-storage:  17394Mi
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             3861512Ki
--
Allocatable:
  cpu:                1800m
  ephemeral-storage:  17394Mi
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             3206152Ki

allocatale = capacity - system_reserved - kube_reserved - eviction_hard

Tags: Linux Kubernetes

Posted by phpcat on Fri, 03 Jun 2022 01:23:26 +0530