Two places and three centers disaster recovery scheme of ACK One application system

Author: Yu Hui, Zhuang Huai, Pioneer

summary

Two places and three centers refer to the deployment of three business processing centers in two cities, namely: production center, local disaster recovery center and remote disaster recovery center. Two sets of environments are deployed in one city to form two centers in the same city, process services at the same time, realize data synchronization through high-speed links, and switch operation. Deploy a set of environment in another city as a remote disaster recovery center for data backup. When the two centers fail at the same time, the remote disaster recovery center can switch the processing business. The disaster recovery scheme of two places and three centers can ensure the continuous operation of business to a great extent.

Using the multi cluster management application distribution function of ACK One can help enterprises uniformly manage three K8s clusters, realize the rapid deployment and upgrading of applications in three K8s clusters, and realize the differential configuration of applications in three K8s clusters. With the use of GTM (global traffic management), the automatic switching of business traffic in three K8s clusters can be realized in case of failure. Data replication at RDS data level is not specifically introduced in this practice, but can refer to DTS data transmission service.

Scheme architecture

prerequisite

Open the master instance of multi cluster management [1]

By managing the associated cluster [2], add three K8s clusters to the master instance to build two places and three centers. In this practice, as an example, two K8s clusters (cluster1 Beijing and cluster2 Beijing) are deployed in Beijing and one K8s cluster (cluster1 Hangzhou) is deployed in Hangzhou.

Create GTM instance [3]

Application deployment

Distribute applications in three K8s clusters through the application distribution function [4] of the ACK One master instance. Compared with traditional script deployment, application distribution using ACK One can obtain the following benefits.

In this practice, the example application is a web application, including K8s Deployment/Service/Ingress/Configmap resources, Service/Ingress external exposure services, and deployment reads the configuration parameters in configmap. By creating application distribution rules, applications are distributed to three K8s clusters, including two Beijing clusters and one Hangzhou cluster, so as to realize two places and three centers. In the distribution process, the deployment and configmap resources are differentially configured to adapt to clusters without locations. At the same time, the distribution process realizes the gray control of manual audit and limits the wrong explosion radius.

Execute the following command to create a namespace demo.

kubectl create namespace demo

Use the following to create app meta Yaml file.

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: web-demo
  name: web-demo
  namespace: demo
spec:
  replicas: 5
  selector:
    matchLabels:
      app: web-demo
  template:
    metadata:
      labels:
        app: web-demo
    spec:
      containers:
      - image: acr-multiple-clusters-registry.cn-hangzhou.cr.aliyuncs.com/ack-multiple-clusters/web-demo:0.4.0
        name: web-demo
        env:
        - name: ENV_NAME
          value: cluster1-beijing
        volumeMounts:
        - name: config-file
          mountPath: "/config-file"
          readOnly: true
      volumes:
      - name: config-file
        configMap:
          items:
          - key: config.json
            path: config.json
          name: web-demo
---
apiVersion: v1
kind: Service
metadata:
  name: web-demo
  namespace: demo
  labels:
    app: web-demo
spec:
  selector:
    app: web-demo
  ports:
    - protocol: TCP
      port: 80
      targetPort: 8080
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: web-demo
  namespace: demo
  labels:
    app: web-demo
spec:
  rules:
    - host: web-demo.example.com
      http:
        paths:
        - path: /
          pathType: Prefix
          backend:
            service:
              name: web-demo
              port:
                number: 80
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: web-demo
  namespace: demo
  labels:
    app: web-demo
data:
  config.json: |
    {
      database-host: "beijing-db.pg.aliyun.com"
    }

Execute the following command to deploy the Application web demo on the master instance. Note: creating a kube resource on the master instance will not be distributed to the sub cluster. This kube resource is referenced in the subsequent Application (step 4b) as the original data.

kubectl apply -f app-meta.yaml

Create and apply distribution rules.

a. Execute the following command to view the associated clusters managed by the master instance and determine the distribution target of the application

kubectl amc get managedcluster

Expected output:

Name                                               Alias               HubAccepted
managedcluster-cxxx   cluster1-hangzhou   true
managedcluster-cxxx   cluster2-beijing    true
managedcluster-cxxx   cluster1-beijing    true

b. Use the following to create an application distribution rule app yaml. Replace managedcluster Cxxx in the example with the actual cluster name to be published. The best practices for the definition of distribution rules are described in the notes.

On app Yaml contains the following resource types: Policy (type:topology) distribution target, Policy (type: override) differentiation rules, Workflow, and Application. For details, please refer to: Application replication distribution [5], Application distribution differentiation configuration [6] and gray distribution between Application clusters [7].

apiVersion: core.oam.dev/v1alpha1
kind: Policy
metadata:
  name: cluster1-beijing
  namespace: demo
type: topology
properties:
  clusters: ["<managedcluster-cxxx>"] #Distribution target cluster 1 cluster1 Beijing
---
apiVersion: core.oam.dev/v1alpha1
kind: Policy
metadata:
  name: cluster2-beijing
  namespace: demo
type: topology
properties:
  clusters: ["<managedcluster-cxxx>"] #Distribution target cluster 2 cluster2 Beijing
---
apiVersion: core.oam.dev/v1alpha1
kind: Policy
metadata:
  name: cluster1-hangzhou
  namespace: demo
type: topology
properties:
  clusters: ["<managedcluster-cxxx>"] #Distribution target cluster 3 cluster1 Hangzhou
---
apiVersion: core.oam.dev/v1alpha1
kind: Policy
metadata:
  name: override-env-cluster2-beijing
  namespace: demo
type: override
properties:
  components:
  - name: "deployment"
    traits:
    - type: env
      properties:
        containerName: web-demo
        env:
          ENV_NAME: cluster2-beijing #Differentiated configuration of environment variables for the deployment of cluster cluster2 Beijing
---
apiVersion: core.oam.dev/v1alpha1
kind: Policy
metadata:
  name: override-env-cluster1-hangzhou
  namespace: demo
type: override
properties:
  components:
  - name: "deployment"
    traits:
    - type: env
      properties:
        containerName: web-demo
        env:
          ENV_NAME: cluster1-hangzhou #Differentiated configuration of environment variables for the deployment of cluster cluster1 Hangzhou
---
apiVersion: core.oam.dev/v1alpha1
kind: Policy
metadata:
  name: override-replic-cluster1-hangzhou
  namespace: demo
type: override
properties:
  components:
  - name: "deployment"
    traits:
    - type: scaler
      properties:
        replicas: 1          #Differential configuration of the number of replicas for the deployment of cluster cluster1 Hangzhou
---
apiVersion: core.oam.dev/v1alpha1
kind: Policy
metadata:
  name: override-configmap-cluster1-hangzhou
  namespace: demo
type: override
properties:
  components:
  - name: "configmap"
    traits:
    - type: json-merge-patch  #Make differential configuration of configmap for the deployment of cluster cluster1 Hangzhou
      properties:
        data:
          config.json: |
            {
              database-address: "hangzhou-db.pg.aliyun.com"
            }
---
apiVersion: core.oam.dev/v1alpha1
kind: Workflow
metadata:
  name: deploy-demo
  namespace: demo
steps:       #Deploy cluster1 Beijing, cluster2 Beijing, cluster1 Hangzhou in sequence.
  - type: deploy
    name: deploy-cluster1-beijing
    properties:
      policies: ["cluster1-beijing"]  
  - type: deploy
    name: deploy-cluster2-beijing
    properties:
      auto: false   #Manual review is required before deploying cluster2 Beijing
      policies: ["override-env-cluster2-beijing", "cluster2-beijing"] #Differentiate environment variables when deploying cluster2 Beijing
  - type: deploy
    name: deploy-cluster1-hangzhou
    properties:
      policies: ["override-env-cluster1-hangzhou", "override-replic-cluster1-hangzhou", "override-configmap-cluster1-hangzhou", "cluster1-hangzhou"]
      #When deploying cluster2 Beijing, do the differentiation of environment variables, number of copies and configmap
---
apiVersion: core.oam.dev/v1beta1
kind: Application
metadata:
  annotations:
    app.oam.dev/publishVersion: version8
  name: web-demo
  namespace: demo
spec:
  components:
    - name: deployment  #Independently reference deployment to facilitate differentiated configuration
      type: ref-objects
      properties:
        objects:
          - apiVersion: apps/v1
            kind: Deployment
            name: web-demo
    - name: configmap   #Independently reference configmap to facilitate differentiated configuration
      type: ref-objects
      properties:
        objects:
          - apiVersion: v1
            kind: ConfigMap
            name: web-demo
    - name: same-resource  #No differentiated configuration
      type: ref-objects
      properties:
        objects:
          - apiVersion: v1
            kind: Service
            name: web-demo
          - apiVersion: networking.k8s.io/v1
            kind: Ingress
            name: web-demo
  workflow:
    ref: deploy-demo

Execute the following command to deploy the distribution rule app on the master instance yaml.

kubectl apply -f app.yaml

View the deployment status of the application.

kubectl get app web-demo -n demo

Expected output. workflowSuspending indicates that the deployment is suspended

NAME       COMPONENT    TYPE          PHASE                HEALTHY   STATUS   AGE
web-demo   deployment   ref-objects   workflowSuspending   true               47h

View the running status of the application on each cluster

kubectl amc get deployment web-demo -n demo -m all

Expected output:

Run on ManagedCluster managedcluster-cxxx (cluster1-hangzhou)
No resources found in demo namespace    #For the first time, the workflow has not started to deploy cluster1 Hangzhou
Run on ManagedCluster managedcluster-cxxx (cluster2-beijing)
No resources found in demo namespace     #For the first time to deploy a new application, the workflow has not started to deploy cluster2 Beijing, waiting for manual review
Run on ManagedCluster managedcluster-cxxx (cluster1-beijing)
NAME       READY   UP-TO-DATE   AVAILABLE   AGE
web-demo   5/5     5            5           47h   #The Deployment runs normally on the cluster1 Beijing cluster

After passing the manual review, deploy clusters cluster2 Beijing and cluster1 Hangzhou.

kubectl amc workflow resume web-demo -n demo
Successfully resume workflow: web-demo

View the deployment status of the application.

kubectl get app web-demo -n demo

Expected output, running indicates that the application is running normally

NAME       COMPONENT    TYPE          PHASE     HEALTHY   STATUS   AGE
web-demo   deployment   ref-objects   running   true               47h

View the running status of the application on each cluster

kubectl amc get deployment web-demo -n demo -m all

Expected output:

Run on ManagedCluster managedcluster-cxxx (cluster1-hangzhou)
NAME       READY   UP-TO-DATE   AVAILABLE   AGE
web-demo   1/1     1            1           47h
Run on ManagedCluster managedcluster-cxxx (cluster2-beijing)
NAME       READY   UP-TO-DATE   AVAILABLE   AGE
web-demo   5/5     5            5           2d
Run on ManagedCluster managedcluster-cxxx (cluster1-beijing)
NAME       READY   UP-TO-DATE   AVAILABLE   AGE
web-demo   5/5     5            5           47h

View the progress status of the application on each cluster

kubectl amc get ingress -n demo -m all

The expected result is that the Ingress of each cluster runs normally and the public IP allocation is successful.

Run on ManagedCluster managedcluster-cxxx (cluster1-hangzhou)
NAME       CLASS   HOSTS                  ADDRESS         PORTS   AGE
web-demo   nginx   web-demo.example.com   47.xxx.xxx.xxx   80      47h
Run on ManagedCluster managedcluster-cxxx (cluster2-beijing)
NAME       CLASS   HOSTS                  ADDRESS         PORTS   AGE
web-demo   nginx   web-demo.example.com   123.xxx.xxx.xxx   80      2d
Run on ManagedCluster managedcluster-cxxx (cluster1-beijing)
NAME       CLASS   HOSTS                  ADDRESS          PORTS   AGE
web-demo   nginx   web-demo.example.com   182.xxx.xxx.xxx   80      2d

Flow management

By configuring global traffic management, the application running state is automatically detected, and the traffic is automatically switched to the monitoring cluster in case of abnormality.

Configure the global traffic management instance, web demo example. COM is the domain name of the example application. Please replace it with the domain name of the actual application, and set DNS to resolve to the CNAME access domain name of global traffic management.

In the created GTM example, create 2 address pools:
1. Pool Beijing: contains the progress IP addresses of two Beijing clusters. The load balancing strategy is to return all addresses to realize the load balancing of two Beijing clusters. The Ingress IP address can be obtained by running "kubectl amc get ingress -n demo -m all" on the master instance.
2. Pool Hangzhou: contains the Ingress IP address of one Hangzhou cluster.

Start the health check in the address pool. The address that fails the check will be removed from the address pool and will no longer receive traffic.

Configure the access policy and set the primary address pool as Beijing address pool and the standby address pool as Hangzhou address pool. Normal traffic is handled by Beijing cluster application. When all Beijing cluster applications are unavailable, it will automatically switch to Hangzhou cluster application processing.

Deployment validation

Normally, all traffic is handled by the application on two clusters in Beijing, and each cluster handles 50% of the traffic.

for i in {1..50}; do curl web-demo.example.com; sleep 3;  done
This is env cluster1-beijing !
Config file is {
  database-host: "beijing-db.pg.aliyun.com"
}


This is env cluster1-beijing !
Config file is {
  database-host: "beijing-db.pg.aliyun.com"
}

This is env cluster2-beijing !
Config file is {
  database-host: "beijing-db.pg.aliyun.com"
}

This is env cluster1-beijing !
Config file is {
  database-host: "beijing-db.pg.aliyun.com"
}

This is env cluster2-beijing !
Config file is {
  database-host: "beijing-db.pg.aliyun.com"
}

This is env cluster2-beijing !
Config file is {
  database-host: "beijing-db.pg.aliyun.com"
}

When the application on cluster cluster1 Beijing is abnormal, GTM routes all traffic to cluster 2 Beijing for processing.

for i in {1..50}; do curl web-demo.example.com; sleep 3;  done
...
<html>
<head><title>503 Service Temporarily Unavailable</title></head>
<body>
<center><h1>503 Service Temporarily Unavailable</h1></center>
<hr><center>nginx</center>
</body>
</html>

This is env cluster2-beijing !
Config file is {
  database-host: "beijing-db.pg.aliyun.com"
}

This is env cluster2-beijing !
Config file is {
  database-host: "beijing-db.pg.aliyun.com"
}

This is env cluster2-beijing !
Config file is {
  database-host: "beijing-db.pg.aliyun.com"
}

This is env cluster2-beijing !
Config file is {
  database-host: "beijing-db.pg.aliyun.com"
}

When the applications on cluster1 Beijing and cluster2 Beijing are abnormal at the same time, GTM routes the traffic to cluster1 Hangzhou cluster for processing.

for i in {1..50}; do curl web-demo.example.com; sleep 3;  done
<head><title>503 Service Temporarily Unavailable</title></head>
<body>
<center><h1>503 Service Temporarily Unavailable</h1></center>
<hr><center>nginx</center>
</body>
</html>
<html>
<head><title>503 Service Temporarily Unavailable</title></head>
<body>
<center><h1>503 Service Temporarily Unavailable</h1></center>
<hr><center>nginx</center>
</body>
</html>
This is env cluster1-hangzhou !
Config file is {
  database-address: "hangzhou-db.pg.aliyun.com"
}

This is env cluster1-hangzhou !
Config file is {
  database-address: "hangzhou-db.pg.aliyun.com"
}

This is env cluster1-hangzhou !
Config file is {
  database-address: "hangzhou-db.pg.aliyun.com"
}

This is env cluster1-hangzhou !
Config file is {
  database-address: "hangzhou-db.pg.aliyun.com"
}

summary

This paper focuses on the multi cluster application distribution function of ACK One, which can help enterprises manage the multi cluster environment. Through the unified application distribution entrance provided by the multi cluster master example, the distribution strategies of application multi cluster distribution, differentiated configuration, workflow management and so on are realized. Combined with the global traffic management of GTM, the application disaster recovery system for managing the two places and three centers is quickly established.

In addition to multi cluster application distribution, ACK One supports the connection and management of Kubernetes clusters in any region and any infrastructure, provides consistent management and community compatible API s, and supports unified operation and maintenance control of computing, network, storage, security, monitoring, logs, jobs, applications, traffic, etc. Alibaba cloud distributed cloud container platform (hereinafter referred to as ACK One) is an enterprise class cloud native platform for hybrid cloud, multi cluster, distributed computing, disaster recovery and other scenarios. For more information, you can see the product introduction, distributed cloud container platform ACK One [8].

Programs Team

Two places and three centers disaster recovery scheme of ACK One application system

summary

Scheme architecture

prerequisite

Application deployment

Flow management

Deployment validation

summary

Related links

Hot Tags