08 Inress and Ingress Controller of kubernetes

Introduction to Ingress

What is ingress?

In the previous article, when introducing service, it was said that the three service modes ClusterIP, NodePort and LoadBalance are exposed. These modes are provided in the service dimension. The role of service is reflected in two aspects. For the internal cluster, it constantly tracks the changes of pod, updates the objects corresponding to pod in the endpoint, and provides a service discovery mechanism for pod with changing ip. For the external cluster, it is similar to a load balancer, The pod can be accessed inside and outside the cluster. However, the method of exposing services by using services alone is not appropriate in the actual production environment:

The ClusterIP mode can only be accessed within the cluster

In NodePort mode, the test environment is OK. When there are dozens or hundreds of services running in the cluster, NodePort port management is a disaster.

The LoadBalance method is limited by the cloud platform, and ELB deployment on the cloud platform usually requires additional fees.

Fortunately, k8s also provides a way to expose services from the cluster dimension, that is, ingress. Ingress can be simply understood as the service of a service. It uses independent ingress objects to formulate request forwarding rules and route requests to one or more services. In this way, the service and request rules are decoupled, and the business exposure can be considered from the business dimension, rather than for each service separately.

For example, the cluster now has three service s: api, file store and front end. The request forwarding in the figure can be realized through an ingress object

The Ingress rule is very flexible. It can forward requests to different service s according to different domain names and path s, and supports https/http

Ingress and ingress controller

To understand ingress, you need to distinguish between two concepts, ingress and ingress controller:

Ingress object:

It refers to an api object in k8s, which is generally configured in yaml. Its function is to define rules for how requests are forwarded to service s, which can be understood as configuring templates;

Ingress-controller

The specific program of reverse proxy and load balancing, parsing the rules defined by ingress, and forwarding the request according to the configured rules;
Simply put, the Ingress controller is the component responsible for forwarding. It is exposed to the cluster portal in various ways. The external request traffic to the cluster will reach the Ingress controller first, and the Ingress object is used to tell the Ingress controller how to forward requests, such as the domain names and path s to which services

Ingress-controller

# Unlike Deployment and other controllers, other controllers belong to one of the three major components of the Master, and Ingress is an independent controller  
# Nginx  
# Trefik (micro service)  
# Envoy  

The Ingress controller is not a built-in component of k8s. In fact, Ingress controller is just a general term. Users can choose different Ingress controllers to implement. At present, there are only two Ingress controllers maintained by k8s, namely, the GCE and Ingress nginx of google cloud, and the other Ingress controllers maintained by a third party. However, no matter which Ingress controller is used, the implementation mechanism is similar, but there are differences in specific configurations, Generally speaking, the progress controller is a Pod, in which the daemon program and reverse proxy program run. The daemon is responsible for continuously monitoring the changes of the cluster, generating configurations according to the progress object and applying new configurations to the reverse proxy. For example, nginx Ingress dynamically generates nginx configurations, dynamically updates the upstream, and applies new configurations to the reload program as needed. For convenience, The following examples have been taken as nginx Ingress officially maintained by k8s

Ingress application

Ingress is an API object. Like other objects, it is configured through the yaml file. Ingress exposes the internal services of the cluster through http or https, and provides the services with external URL, load balancing, SSL/TLS capabilities and host based direction proxy. Ingress relies on ingress controller to specifically implement the above functions The configuration is as follows:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: abc-ingress
  annotations: 
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/use-regex: "true"
spec:
  tls:
  - hosts:
    - api.abc.com
    secretName: abc-tls
  rules:
  - host: api.abc.com
    http:
      paths:
      - backend:
          serviceName: apiserver
          servicePort: 80
  - host: www.abc.com
    http:
      paths:
      - path: /image/*
        backend:
          serviceName: fileserver
          servicePort: 80
  - host: www.abc.com
    http:
      paths:
      - backend:
          serviceName: feserver
          servicePort: 8080

Like other k8s objects, the ingress configuration also contains key fields such as apiVersion, kind, meatdata, and spec. among the spec fields of interest, tls is used to define https keys and certificates. Rule is used to specify the request routing rule, which refers to metadata Annotations field. Annotations are very important in the ingress configuration. As mentioned earlier, there are many different implementations of the ingress controller, and different ingress controllers can determine which ingress configurations to use according to "kubernetes.io/ingress.class". At the same time, different ingress controllers also have corresponding annotations configurations to customize some parameters, such as the above configured'nginx Ingress Kubernetes IO / use regex: "true" ', finally, in generating nginx configuration, location ~ will be used to represent regular matching

Ingress deployment

Two aspects should be considered for the deployment of Ingress:

# 1 The ingress controller is run as a Pod. How to deploy it is better
# 2 Ingress solves the problem of how to route requests to the inside of the cluster. How can it expose itself to the outside

Some common deployment and exposure methods are listed below. The specific method to be used should be determined according to actual needs

Deployment+LoadBalancer mode Service

If you want to deploy ingress in the public cloud, this method is more appropriate. Deploy the ingress controller with Deployment, and create a service of type LoadBalancer to associate this group of pod s. Most public clouds automatically create a load balancer for the LoadBalancer's service, and usually bind the public address. As long as the domain name resolution points to the address, the external exposure of the cluster service is realized.

Deployment+NodePort mode Service

Similarly, use the deployment mode to deploy the ingress controller and create the corresponding service, but the type is nodeport. In this way, ingress will be exposed to the specific port of the cluster node ip. Since the ports exposed by nodeport are random ports, a set of load balancers will be built in front to forward requests. This method is generally used in scenarios where the host is relatively fixed and the environment ip address remains unchanged.
Although it is simple and convenient to expose ingress in NodePort mode, NodePort has an additional layer of NAT, which may have a certain impact on the performance when the request magnitude is large.

DaemonSet+HostNetwork+nodeSelector

Use the DaemonSet and nodeselector to deploy the ingress controller to a specific node, and then use the HostNetwork to directly connect the pod with the network of the node of the host machine, and directly use the 80/433 port of the host machine to access the service. At this time, the node machine where the ingress controller is located is very similar to the edge node of the traditional architecture, such as the nginx server at the machine room entrance. This mode is the simplest for the entire request link, and its performance is better than that of NodePort mode. The disadvantage is that because the network and port of the host node are directly used, one node can only deploy one ingress controller pod. It is suitable for large concurrent production environments.

for file in configmap.yaml namespace.yaml mandatory.yaml rbac.yaml with-rbac.yaml; do wget https://github.com/kubernetes/ingress-nginx/tree/master/deploy/static/$file ;done

cat mandatory.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx

---

kind: ConfigMap
apiVersion: v1
metadata:
  name: nginx-configuration
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx

---
kind: ConfigMap
apiVersion: v1
metadata:
  name: tcp-services
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx

---
kind: ConfigMap
apiVersion: v1
metadata:
  name: udp-services
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: nginx-ingress-serviceaccount
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: nginx-ingress-clusterrole
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
rules:
  - apiGroups:
      - ""
    resources:
      - configmaps
      - endpoints
      - nodes
      - pods
      - secrets
    verbs:
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - nodes
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - services
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - events
    verbs:
      - create
      - patch
  - apiGroups:
      - "extensions"
      - "networking.k8s.io"
    resources:
      - ingresses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - "extensions"
      - "networking.k8s.io"
    resources:
      - ingresses/status
    verbs:
      - update

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: Role
metadata:
  name: nginx-ingress-role
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
rules:
  - apiGroups:
      - ""
    resources:
      - configmaps
      - pods
      - secrets
      - namespaces
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - configmaps
    resourceNames:
      # Defaults to "<election-id>-<ingress-class>"
      # Here: "<ingress-controller-leader>-<nginx>"
      # This has to be adapted if you change either parameter
      # when launching the nginx-ingress-controller.
      - "ingress-controller-leader-nginx"
    verbs:
      - get
      - update
  - apiGroups:
      - ""
    resources:
      - configmaps
    verbs:
      - create
  - apiGroups:
      - ""
    resources:
      - endpoints
    verbs:
      - get

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
  name: nginx-ingress-role-nisa-binding
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: nginx-ingress-role
subjects:
  - kind: ServiceAccount
    name: nginx-ingress-serviceaccount
    namespace: ingress-nginx

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: nginx-ingress-clusterrole-nisa-binding
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: nginx-ingress-clusterrole
subjects:
  - kind: ServiceAccount
    name: nginx-ingress-serviceaccount
    namespace: ingress-nginx

---

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-ingress-controller
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: ingress-nginx
      app.kubernetes.io/part-of: ingress-nginx
  template:
    metadata:
      labels:
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
      annotations:
        prometheus.io/port: "10254"
        prometheus.io/scrape: "true"
    spec:
      # wait up to five minutes for the drain of connections
      terminationGracePeriodSeconds: 300
      serviceAccountName: nginx-ingress-serviceaccount
      nodeSelector:
        kubernetes.io/os: linux
      containers:
        - name: nginx-ingress-controller
          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.26.1
          args:
            - /nginx-ingress-controller
            - --configmap=$(POD_NAMESPACE)/nginx-configuration
            - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
            - --udp-services-configmap=$(POD_NAMESPACE)/udp-services
            - --publish-service=$(POD_NAMESPACE)/ingress-nginx
            - --annotations-prefix=nginx.ingress.kubernetes.io
          securityContext:
            allowPrivilegeEscalation: true
            capabilities:
              drop:
                - ALL
              add:
                - NET_BIND_SERVICE
            # www-data -> 33
            runAsUser: 33
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
          ports:
            - name: http
              containerPort: 80
              protocol: TCP
            - name: https
              containerPort: 443
              protocol: TCP
          livenessProbe:
            failureThreshold: 3
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            initialDelaySeconds: 10
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 10
          readinessProbe:
            failureThreshold: 3
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 10
          lifecycle:
            preStop:
              exec:
                command:
                  - /wait-shutdown

---

apiVersion: v1
kind: LimitRange
metadata:
  name: ingress-nginx
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
spec:
  limits:
  - default:
    min:
      memory: 90Mi
      cpu: 100m
    type: Container


kubectl apply -f mandatory.yaml 
kubectl get pod -n ingress-nginx
NAME                                        READY   STATUS    RESTARTS   AGE
nginx-ingress-controller-568867bf56-wwgqx   1/1     Running   0          4m10s

If it is a bare metal machine, you need to create a Nodeport exposed port

https://kubernetes.github.io/ingress-nginx/deploy/#bare-metal

wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/provider/baremetal/service-nodeport.yaml

# In order to make the exposed ports not random, we specify the following
cat service-nodeport.yaml 
apiVersion: v1
kind: Service
metadata:
  name: ingress-nginx
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
spec:
  type: NodePort
  ports:
    - name: http
      port: 80
      targetPort: 80
      protocol: TCP
      nodePort: 30080
    - name: https
      port: 443
      targetPort: 443
      protocol: TCP
      nodePort: 30443
  selector:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
---
kubectl apply -f service-nodeport.yaml
kubectl get svc -n ingress-nginx
NAME            TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
ingress-nginx   NodePort   10.96.120.168   <none>        80:30080/TCP,443:30443/TCP   6m19s

# Let's use a machine outside the cluster to access

Next, we will publish the previous myapp application

cat ingress-myapp.yaml 
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ingress-myapp
  namespace: default
  annotations:
    kubernetes.io/ingress.class: "nginx"
spec:
  rules:
  - host: myapp.youmen.com
    http:
      paths:
      - path:
        backend:
          serviceName: redis
          servicePort: 80 
kubectl apply -f ingress-myapp.yaml
# Once we create ingress, we will immediately inject the environment into nginx
kubectl exec -n ingress-nginx -it nginx-ingress-controller-568867bf56-wwgqx /bin/bash
cat nginx.conf  |grep youmen
	## start server myapp.youmen.com
		server_name myapp.youmen.com ;
	## end server myapp.youmen.com

# Next, we modify the hosts file of the local host
Windows:  C:\Windows\System32\drivers\etc
Linux: /etc/hosts
172.19.0.26    myapp.youmen.com
# The following page will appear after browser access

Publish Tomcat application (http)

Create tomcatDeployment and service
cat tomcat-deploy.yaml 
apiVersion: v1
kind: Service
metadata:
  name: tomcat
  namespace: default
spec:
  selector:
    app: tomcat
    release: canary
  ports:
  - name: http
    targetPort: 8080
    port: 8080

  - name: ajp
    targetPort: 8009
    port: 8009

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: tomcat-deploy
  namespace: default
spec:
  replicas: 3
  selector:
    matchLabels:
      app: tomcat
      release: canary
  template:
    metadata:
      labels:
        app: tomcat
        release: canary
    spec:
      containers:
      - name: tomcat 
        image: tomcat:8.5.32-jre8-alpine
        ports:
        - name: http
          containerPort: 8080
        - name: ajp
          containerPort: 8009
Create ingress tomcat Yaml
cat ingress-tomcat.yaml 
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ingress-tomcat
  namespace: default
  annotations:
    kubernetes.io/ingress.class: "nginx"
spec:
  rules:
  - host: tomcat.youmen.com
    http:
      paths:
      - path:
        backend:
          serviceName: tomcat
          servicePort: 8080
# Perform domain name resolution as in the above steps, and then access the browser IP:30080

https encryption tomcat

# mkdir encryption
# cd encryption
openssl genrsa -out tls.key 2048
# Generating RSA private key, 2048 bit long modulus
# .............................+++
# ...+++
# e is 65537 (0x10001)
openssl req -new -x509 -key tls.key -out tls.crt \
 -subj /C=CN/ST=Beijing/L=Beijing/O=DevOps/CN=tomcat.youmen.com
ls
# tls.crt  tls.key

# The certificate we use for https encryption cannot be directly put into Pod. It needs to be converted into secret format. It is also a k8s object
kubectl create secret tls tomcat-ingress-secret --cert=tls.crt --key=tls.key

kubectl get secret
NAME                    TYPE                                  DATA   AGE
default-token-j9thc     kubernetes.io/service-account-token   3      5d23h
tomcat-ingress-secret   kubernetes.io/tls                     2      6s

kubectl describe secret tomcat-ingress-secret
Name:         tomcat-ingress-secret
Namespace:    default
Labels:       <none>
Annotations:  <none>
Type:  kubernetes.io/tls
Data
====
tls.crt:  1294 bytes
tls.key:  1675 bytes

cat ingress-tomcat-tls.yaml 
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ingress-tomcat-tls
  namespace: default
  annotations:
    kubernetes.io/ingress.class: "nginx"
spec:
  tls:
  - hosts:
    - tomcat.youmen.com
    secretName: tomcat-ingress-secret
  rules:
  - host: tomcat.youmen.com
    http:
      paths:
      - path:
        backend:
          serviceName: tomcat
          servicePort: 8080

apply Ingress-tomcat-tls.yaml

kubectl describe ingress ingress-tomcat-tls
Name:             ingress-tomcat-tls
Namespace:        default
Address:          10.96.120.168
Default backend:  default-http-backend:80 (<none>)
TLS:
  tomcat-ingress-secret terminates tomcat.youmen.com
Rules:
  Host               Path  Backends
  ----               ----  --------
  tomcat.youmen.com  
                        tomcat:8080 (10.244.1.45:8080,10.244.1.46:8080,10.244.2.35:8080)
Annotations:
  kubectl.kubernetes.io/last-applied-configuration:  {"apiVersion":"extensions/v1beta1","kind":\
"Ingress","metadata":{"annotations":{"kubernetes.io/ingress.class":"nginx"},"name":"\
ingress-tomcat-tls","namespace":"default"},"spec":{"rules":[{"host":"tomcat.youmen.com",\
"http":{"paths":[{"backend":{"serviceName":"tomcat","servicePort":8080},"path":null}]}}],\
"tls":[{"hosts":["tomcat.youmen.com"],"secretName":"tomcat-ingress-secret"}]}}

  kubernetes.io/ingress.class:  nginx
Events:
  Type    Reason  Age        From                      Message
  ----    ------  ----       ----                      -------
  Normal  CREATE  <invalid>  nginx-ingress-controller  Ingress default/ingress-tomcat-tls
  Normal  UPDATE  <invalid>  nginx-ingress-controller  Ingress default/ingress-tomcat-tls

kubectl get secret
NAME                    TYPE                                  DATA   AGE
default-token-j9thc     kubernetes.io/service-account-token   3      5d23h
tomcat-ingress-secret   kubernetes.io/tls                     2      10m

# Let's go to ingress nginx to check whether the certificate has been injected
kubectl exec -n ingress-nginx -it  nginx-ingress-controller-568867bf56-wwgqx  /bin/bash
www-data@nginx-ingress-controller-5688f56-wwgqx:/etc/nginx$ cat nginx.conf |grep ssl_certificate_*
	ssl_certificate     /etc/ingress-controller/ssl/default-fake-certificate.pem;
	ssl_certificate_key /etc/ingress-controller/ssl/default-fake-certificate.pem;

# Next, the browser accesses https / / Domain Name: 30443

Tags: Kubernetes

Posted by ry4n0wnz on Tue, 31 May 2022 10:44:02 +0530