Kuberntes Version 1.24 Deploy MetalLB Load Balancer

1. Introduction to MetalLB
MetalLB is a load balancer implemented for bare-metal Kubernetes clusters using standard routing protocol ARP or BGP. Metallb supports two modes:

  • Layer2:Layer2 mode must be a bare metal server or a network that supports arp broadcasting. Due to the special arp mechanism of the cloud, the cloud platform cannot support Layer2 mode.
  • BGP:bgp mode and Calico networking mode coexist, other networking modes are required to use BGP mode

Official website: https://metallb.universe.tf
Project Address: https://github.com/metallb/metallb

2. MetalLB Installation
1. Deploy k8s cluster
See: https://blog.csdn.net/lic95/article/details/125044136
Deployment is complete with the following details:

[root@master01 ~]# kubectl get nodes -o wide
NAME       STATUS   ROLES           AGE     VERSION   INTERNAL-IP    EXTERNAL-IP   OS-IMAGE                    KERNEL-VERSION              CONTAINER-RUNTIME
master01   Ready    control-plane   2d23h   v1.24.1   192.168.3.31   <none>        AlmaLinux 8.6 (Sky Tiger)   4.18.0-372.9.1.el8.x86_64   containerd://1.6.4
master02   Ready    control-plane   2d22h   v1.24.1   192.168.3.32   <none>        AlmaLinux 8.6 (Sky Tiger)   4.18.0-372.9.1.el8.x86_64   containerd://1.6.4
master03   Ready    control-plane   2d22h   v1.24.1   192.168.3.33   <none>        AlmaLinux 8.6 (Sky Tiger)   4.18.0-372.9.1.el8.x86_64   containerd://1.6.4
node01     Ready    <none>          2d22h   v1.24.1   192.168.3.41   <none>        AlmaLinux 8.6 (Sky Tiger)   4.18.0-372.9.1.el8.x86_64   containerd://1.6.4
node02     Ready    <none>          2d22h   v1.24.1   192.168.3.42   <none>        AlmaLinux 8.6 (Sky Tiger)   4.18.0-372.9.1.el8.x86_64   containerd://1.6.4
node03     Ready    <none>          2d22h   v1.24.1   192.168.3.43   <none>        AlmaLinux 8.6 (Sky Tiger)   4.18.0-372.9.1.el8.x86_64   containerd://1.6.4
node04     Ready    <none>          2d22h   v1.24.1   192.168.3.44   <none>        AlmaLinux 8.6 (Sky Tiger)   4.18.0-372.9.1.el8.x86_64   containerd://1.6.4
node05     Ready    <none>          2d22h   v1.24.1   192.168.3.45   <none>        AlmaLinux 8.6 (Sky Tiger)   4.18.0-372.9.1.el8.x86_64   containerd://1.6.4

2. Deploy test application echoserver

# Deploy echoserver
kubectl create deploy echoserver --image=cilium/echoserver

# View Deployment
[root@master01 manifests]# kubectl get pods
NAME                          READY   STATUS    RESTARTS   AGE
echoserver-8585bfb456-v6cx8   1/1     Running   0          8m42s

#Modify to LoadBalancer
kubectl expose deployment echoserver --port=80 --type=LoadBalancer

#View services, status not pending because MetalLB is not deployed
[root@master01 ~]# kubectl get svc
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
echoserver   LoadBalancer   172.18.149.83   <pending>     8080:30926/TCP   2m
kubernetes   ClusterIP      172.18.0.1      <none>        443/TCP          36m

3. Modify kube-proxy to enable strict ARP mode

# Modify mode l and strictARP values and proofread them
kubectl get configmap kube-proxy -n kube-system -o yaml | \
sed -e "s/strictARP: false/strictARP: true/" | \
sed -e 's/mode: ""/mode: "iptv"/' | \
kubectl diff  -f - -n kube-system

# Perform modifications
kubectl get configmap kube-proxy -n kube-system -o yaml | \
sed -e "s/strictARP: false/strictARP: true/" | \
sed -e 's/mode: ""/mode: "iptv"/' | \
kubectl apply  -f - -n kube-system

#View modification results
kubectl edit configmap -n kube-system kube-proxy

4. Install Metallb

#Metallb supports a variety of installation methods for yaml files, Kustomize, Helm, and MetalLB Operator, which are installed using yaml.

# Download the release version and unzip it into the manifests directory
wget https://github.com/metallb/metallb/archive/refs/tags/v0.12.1.tar.gz  
tar vzxf v0.12.1.tar.gz
cd metallb-0.12.1/manifests/

# Perform yaml file installation
kubectl apply -f namespace.yaml
kubectl apply -f metallb.yaml

# View running status
[root@master01 manifests]# kubectl -n metallb-system get pods 
NAME                          READY   STATUS    RESTARTS   AGE
controller-7476b58756-mkxcn   1/1     Running   0          4m12s
speaker-79r6v                 1/1     Running   0          4m12s
speaker-dwlnf                 1/1     Running   0          4m12s
speaker-ppgmj                 1/1     Running   0          4m12s
speaker-zgnjz                 1/1     Running   0          4m12s
speaker-zp6fm                 1/1     Running   0          4m12s

[root@master01 manifests]# kubectl -n metallb-system get deploy
NAME         READY   UP-TO-DATE   AVAILABLE   AGE
controller   1/1     1            1           5m19s

[root@master01 manifests]# kubectl -n metallb-system get ds
NAME      DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE
speaker   5         5         5       5            5           kubernetes.io/os=linux   6m7s

3. Configuring Layer2 mode
Note: Layer2 mode must be a bare metal server or a network that supports arp broadcasting. Due to the special arp mechanism of the cloud, Layer2 mode cannot be supported by the cloud platform.
1. Modify the default configuration file to assign part of the IP address for MetalLB (must be the same segment)

[root@master01 manifests]# cat example-layer2-config.yaml 
apiVersion: v1
kind: ConfigMap
metadata:
  namespace: metallb-system
  name: config
data:
  config: |
    address-pools:
    - name: my-ip-space
      protocol: layer2
      addresses:
      - 192.168.3.80-192.168.3.90 #Must be the same segment

2. Execute yaml file

[root@master01 manifests]# kubectl apply -f example-layer2-config.yaml 
configmap/config configured

3. View echoserver LoadBalancer address

[root@master01 manifests]# kubectl get svc
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP    PORT(S)          AGE
echoserver   LoadBalancer   172.18.149.83   192.168.3.80   80:32563/TCP     26m
kubernetes   ClusterIP      172.18.0.1      <none>         443/TCP          60m

4. View test results, OK

[root@master01 manifests]# curl http://192.168.3.80


Hostname: echoserver-8585bfb456-v6cx8

Pod Information:
        -no pod information available-

Server values:
        server_version=nginx: 1.13.3 - lua: 10008

Request Information:
        client_address=::ffff:10.244.241.64
        method=GET
        real path=/
        query=
        request_version=1.1
        request_scheme=http
        request_uri=http://192.168.3.80:80/

Request Headers:
        accept=*/*
        host=192.168.3.80
        user-agent=curl/7.61.1

Request Body:
        -no body in request-

5. About nodeport
_When creating the LoadBalancer service, k8s helps us automatically create a nodeport service by default. This operation can define switches by specifying the allocateLoadBalancerNodePorts field in the service, which is true by default. If nodeport needs to be turned off, you can modify the spec.allocateLoadBalancerNodePorts field in the service to set it to false, Noeport will not be assigned when svc is created.

3. Configuring BGP mode
_BGP mode requires communication with external routers to work properly. For convenience, we use frr to build external router and k8s cluster communication, router address 192.168.3.254, set up as container
The BGP mode and the Calico networking mode cannot coexist. Other networking modes are required to use the BGP mode.

1. Modify network mode to flannel mode when building clusters

#Use flannel networking when clustering
kubectl delete -f https://docs.projectcalico.org/manifests/calico.yaml
kubectl apply -f https://raw.githubusercontent.com/flannel-io/flannel/master/Documentation/kube-flannel.yml

2. Install frr container on external machine

# Start frr container with default configuration
[root@AlmaLinux-BGP ~]# docker run  -d --name AlmaLinux-BGP  frrouting/frr
4f0097d103d78dbfbb728771885fec4c46d50cbe64a3cd44bb7ba58cfd35021a

# Extract mirror default configuration to/etc/frr to prepare for persistent storage configuration
mkdir -p /etc/frr
docker cp AlmaLinux-BGP:/etc/frr/daemons /etc/frr/daemons

# Modify configuration file to start BGP
sed -i 's#bgpd=no#bgpd=yes#g' /etc/frr/daemons

# Close the default configuration frr container
docker stop AlmaLinux-BGP && docker rm AlmaLinux-BGP

# Persistent Storage Start frr Container
docker run  -d --restart=always \
        --name AlmaLinux-BGP  \
        -v /etc/frr:/etc/frr:Z \
        --net=host \
        --privileged \
        frrouting/frr
        

3. Configure BGP by entering the frr container with the command docker exec-it AlmaLinux-BGP vtysh

# Configuration is as follows
frr# show running-config 
Building configuration...

Current configuration:
!
frr version 8.1_git
frr defaults traditional
hostname git-nas
no ipv6 forwarding
hostname frr
!
router bgp 65000
 bgp router-id 192.168.3.254
 no bgp ebgp-requires-policy
 neighbor k8s peer-group
 neighbor k8s remote-as 65001
 neighbor k8s password metallb
 neighbor 192.168.3.41 peer-group k8s
 neighbor 192.168.3.41 password metallb
 neighbor 192.168.3.42 peer-group k8s
 neighbor 192.168.3.42 password metallb
 neighbor 192.168.3.43 peer-group k8s
 neighbor 192.168.3.43 password metallb
 neighbor 192.168.3.44 peer-group k8s
 neighbor 192.168.3.44 password metallb
 neighbor 192.168.3.45 peer-group k8s
 neighbor 192.168.3.45 password metallb
exit
!
end
frr# 

4. Install Metallb and test pod

# Deploy echoserver
kubectl create deploy echoserver --image=cilium/echoserver
#Modify to LoadBalancer
kubectl expose deployment echoserver --port=80 --type=LoadBalancer
# Download the release version and unzip it into the manifests directory
wget https://github.com/metallb/metallb/archive/refs/tags/v0.12.1.tar.gz  
tar vzxf v0.12.1.tar.gz
cd metallb-0.12.1/manifests/

# Perform yaml file installation
kubectl apply -f namespace.yaml
kubectl apply -f metallb.yaml

# View pod status
[root@master01 ~]# kubectl get pods -n metallb-system
NAME                          READY   STATUS    RESTARTS   AGE
controller-7476b58756-zjlnh   1/1     Running   0          10m
speaker-4qbh6                 1/1     Running   0          10m
speaker-9bwv8                 1/1     Running   0          10m
speaker-b4vcb                 1/1     Running   0          10m
speaker-hp9qk                 1/1     Running   0          10m
speaker-wdb78                 1/1     Running   0          10m

5. Configure BPG mode configmap

[root@master01 manifests]# cat example-config.yaml 
apiVersion: v1
kind: ConfigMap
metadata:
  namespace: metallb-system
  name: config
data:
  config: |
    # The peers section tells MetalLB what BGP routers to connect too. There
    # is one entry for each router you want to peer with.
    peers:
    - peer-address: 192.168.3.254
      peer-asn: 65000
      my-asn: 65001
      password: "metallb"
    address-pools:
    - name: default
      protocol: bgp
      addresses:
      - 192.168.0.100-192.168.0.200

6. Perform operations

[root@master01 manifests]# kubectl apply -f example-config.yaml 
configmap/config configured

7. Delete Metallb after modifying configuration Delete controller original pod,k8s restart new pod automatically

[root@master01 manifests]# kubectl delete pod -n metallb-system -l app=metallb,component=controller

8. View the test pod and assign it to the newly planned IP

[root@master01 ~]# kubectl get svc -n default
NAME         TYPE           CLUSTER-IP    EXTERNAL-IP     PORT(S)        AGE
echoserver   LoadBalancer   172.18.3.41   192.168.0.100   80:30600/TCP   12m
kubernetes   ClusterIP      172.18.0.1    <none>          443/TCP        22m

9. View frr status

# Viewing frr BGP status, BGP neighborhood relationship has been established with node1 to node5
frr# show ip bgp peer-group 

BGP peer-group k8s, remote AS 65001
  Peer-group type is external
  Configured address-families: IPv4 Unicast;
  Peer-group members:
    192.168.3.41  Established 
    192.168.3.42  Established 
    192.168.3.43  Established 
    192.168.3.44  Established 
    192.168.3.45  Established 
frr# 

#View Routing Status
frr# show ip route bgp 
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR,
       f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup
       t - trapped, o - offload failure

B>* 192.168.0.100/32 [20/0] via 192.168.3.41, ens3, weight 1, 00:08:52
  *                         via 192.168.3.42, ens3, weight 1, 00:08:52
  *                         via 192.168.3.43, ens3, weight 1, 00:08:52
  *                         via 192.168.3.44, ens3, weight 1, 00:08:52
  *                         via 192.168.3.45, ens3, weight 1, 00:08:52
frr# 

#View Routing Table
frr# show  ip bgp 
BGP table version is 3, local router ID is 192.168.3.40, vrf id 0
Default local pref 100, local AS 65000
Status codes:  s suppressed, d damped, h history, * valid, > best, = multipath,
               i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes:  i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

   Network          Next Hop            Metric LocPrf Weight Path
*= 192.168.0.100/32 192.168.3.43                           0 65001 ?
*=                  192.168.3.45                           0 65001 ?
*=                  192.168.3.42                           0 65001 ?
*>                  192.168.3.41                           0 65001 ?
*=                  192.168.3.44                           0 65001 ?

Displayed  1 routes and 5 total paths
frr# 

8. View test results, OK

root@git-nas:~# curl http://192.168.0.100


Hostname: echoserver-8585bfb456-8vn9r

Pod Information:
        -no pod information available-

Server values:
        server_version=nginx: 1.13.3 - lua: 10008

Request Information:
        client_address=::ffff:10.244.5.0
        method=GET
        real path=/
        query=
        request_version=1.1
        request_scheme=http
        request_uri=http://192.168.0.100:80/

Request Headers:
        accept=*/*
        host=192.168.0.100
        user-agent=curl/7.81.0

Tags: Docker Kubernetes Load Balance

Posted by giannis_athens on Tue, 31 May 2022 22:25:54 +0530