SpringCloud-- service degradation (Hystrix, degradation, fusing, monitoring)

Series link:

service degradation

First, you need to understand the concept of service avalanche:

When calling between multiple microservices, suppose that microservice A calls microservice B and microservice C, and microservice B and microservice C call other microservices, which is called "fan out". If the call response time of A microservice on the fan out link is too long or unavailable, the call to microservice A will occupy more and more system resources, leading to the "avalanche effect" of the system.

Hystrix

Hystrix is an open source library used to handle the delay and fault tolerance of distributed systems. It can ensure that when a dependency fails, it will not lead to overall service failure and avoid cascading failures (avalanches).

Main role

Service degradation: for example, when a service is busy and cannot keep the client's request waiting, it should immediately return to the client an alternative solution.
Service fusing: when a service has a problem and is stuck, the user cannot wait all the time. It is necessary to close all access to the service and then call the service to downgrade.
Service flow restriction: for example, in the seckill scenario, users cannot access the server instantly, and only how many requests can be made at a time.
Near real time monitoring

Hystrix

service degradation

For example, when a service is busy and cannot keep the client's request waiting, it should immediately return to the client an alternative.

Degradation occurs:

Abnormal program operation
overtime
Service fusing occurs
Thread pool / semaphore full

Create pay module with degradation mechanism

Name: cloud provider hystrix payment8001

pom file

		<!-- hystrix-->
        <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-starter-netflix-hystrix</artifactId>
        </dependency>

Configuration file. To reduce resource consumption, the stand-alone version Eureka is used here

        server:
          port: 8001
        spring:
          application:
            name: cloud-provider-hystrix-payment

        eureka:
          client:
            register-with-eureka: true
            fetch-registry: true
            service-url:
              defaultZone: http://eureka7001.com:7001/eureka
        #      defaultZone: http://eureka7001.com:7001/eureka,http://eureka7002.com:7002/eureka

Main startup class

        @SpringBootApplication
        @EnableEurekaClient
        public class PaymentHystrixMain8001 {
            public static void main(String[] args){
                SpringApplication.run(PaymentHystrixMain8001.class,args);
            }
        }

service, saving time without writing interface

        @Service
        public class PaymentService {

            /*
            Correct method
             */
            public String paymentInfo_OK(Integer id){
                return "Thread pool:"+Thread.currentThread().getName()+" paymentInfo_OK,id: "+id+"\t"+"???";
            }

            /*
            How to make mistakes
             */
            public String paymentInfo_TimeOut(Integer id){
                int timeNumber = 5;
                try{
                    TimeUnit.SECONDS.sleep(timeNumber);
                } catch (InterruptedException e) {
                    e.printStackTrace();
                }
                return "Thread pool:"+Thread.currentThread().getName()+" paymentInfo_TimeOut,id: "+id+"\t"+"!!!";
            }
        }

controller

        @RestController
        @Slf4j
        @RequestMapping("/payment")
        public class PaymentController {
            @Resource
            private PaymentService paymentService;
            @Value("${server.port}")
            private String serverPort;

            @GetMapping("/hystrix/ok/{id}")
            public String paymentInfo_OK(@PathVariable("id") Integer id){
                String result = paymentService.paymentInfo_OK(id);
                log.info("****result: "+result);
                return result;
            }

            @GetMapping("/hystrix/timeout/{id}")
            public String paymentInfo_TimeOut(@PathVariable("id") Integer id){
                String result = paymentService.paymentInfo_TimeOut(id);
                log.info("****result: "+result);
                return result;
            }
        }

Start the project. It can be used when the concurrency is low. We use JMeter to concurrent 2w request TimeOut interfaces, and then access the OK interface to find that we also need to wait. This is because the method under stress test takes up most of the server's resources, causing other requests to slow down.

Create an order module with degradation

Name: cloud consumer feign hystrix order80

pom

		<!-- openfeign -->
        <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-starter-openfeign</artifactId>
        </dependency>
		<!-- hystrix-->
        <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-starter-netflix-hystrix</artifactId>
        </dependency>

configuration file

        server:
          port: 80

        eureka:
          client:
            register-with-eureka: false
            service-url:
              defaultZone: http://eureka7001.com:7001/eureka

Main startup class

        @SpringBootApplication
        @EnableFeignClients
        public class OrderHystrixMain80 {
            public static void main(String[] args){
                SpringApplication.run(OrderHystrixMain80.class,args);
            }
        }

Interface for remote calling pay module

        @FeignClient(value = "CLOUD-PROVIDER-HYSTRIX-PAYMENT")
        public interface PaymentHystrixService {

            @GetMapping("/payment/hystrix/ok/{id}")
            public String paymentInfo_OK(@PathVariable("id") Integer id);

            @GetMapping("/payment/hystrix/timeout/{id}")
            public String paymentInfo_TimeOut(@PathVariable("id") Integer id);
        }

controller:

        @RestController
        @Slf4j
        @RequestMapping("/consumer")
        public class OrderHystrixController {
            @Resource
            private PaymentHystrixService paymentHystrixService;

            @GetMapping("/payment/hystrix/ok/{id}")
            public String paymentInfo_OK(@PathVariable("id") Integer id){
                String result = paymentHystrixService.paymentInfo_OK(id);
                return result;
            }

            @GetMapping("/payment/hystrix/timeout/{id}")
            public String paymentInfo_TimeOut(@PathVariable("id") Integer id){
                String result = paymentHystrixService.paymentInfo_TimeOut(id);
                return result;
            }
        }

Test, start the order module, and test the timeout interface of the 20000 concurrent pay module again. It is found that the order access is also slow. If the Ribbon timeout parameter is not configured, the call will also report an error.

reason

The reason for this phenomenon is that the worker thread of the thread pool in the tomcat container of 8001 is occupied by the timeout interface. At this time, 80 calls to the ok interface in 8001 cannot be processed.

solve

8001 timeout / downtime: 80 can not wait all the time, and service degradation is required
8001 no problem: if the service provider fails or needs to wait less than the service provider, it needs to handle the degradation itself

Configure service degradation

Modify pay module

Add @HystrixCommand annotation for the method that the service will delay

        /*
        How to make mistakes
         */
        @HystrixCommand(fallbackMethod = "paymentInfo_TimeOutHandler",commandProperties = {
            @HystrixProperty(name = "execution.isolation.thread.timeoutInMilliseconds",value = "3000")
    	})//Set the peak value of the call timeout time, and the set paymentinfo will be called in case of timeout_ Timeouthandler method
        public String paymentInfo_TimeOut(Integer id){
            int timeNumber = 5;
            //int age = 10/0;
            //Other exceptions can also trigger service degradation
            try{
                TimeUnit.SECONDS.sleep(timeNumber);
            } catch (InterruptedException e) {
                e.printStackTrace();
            }
            return "Thread pool:"+Thread.currentThread().getName()+" paymentInfo_TimeOut,id: "+id+"\t"+"!!!";
        }
        public String paymentInfo_TimeOutHandler(Integer id){
            return "Thread pool:"+Thread.currentThread().getName()+" The system is busy, please try again later,id: "+id+"\t"+". . . ";
        }

On the main startup class, add the annotation @enableclircuitbreaker to activate hystrix

        @SpringBootApplication
        @EnableEurekaClient
        @EnableCircuitBreaker
        public class PaymentHystrixMain8001 {
            public static void main(String[] args){
                SpringApplication.run(PaymentHystrixMain8001.class,args);
            }
        }

Access the timeout interface of 8001, and you can see that degradation is triggered

Modify the order module

Generally, the degraded service is placed on the client. In addition, for modifications in @HystrixCommand, it is recommended to restart the microservice.

Modify the configuration file. feign enables the hystrix support

        feign:
          hystrix:
            enabled: true

Add @EnableHystrix to the main startup class and enable hystrix

        @SpringBootApplication
        @EnableFeignClients
        @EnableHystrix
        public class OrderHystrixMain80 {
            public static void main(String[] args){
                SpringApplication.run(OrderHystrixMain80.class,args);
            }
        }

Modify the controller [pay module timeout is changed to 3s, with a limit of 5s]

        @GetMapping("/payment/hystrix/timeout/{id}")
        @HystrixCommand(fallbackMethod = "paymentInfo_TimeOutHandler",commandProperties = {
                @HystrixProperty(name = "execution.isolation.thread.timeoutInMilliseconds",value = "1500")
        })
        public String paymentInfo_TimeOut(@PathVariable("id") Integer id){
            String result = paymentHystrixService.paymentInfo_TimeOut(id);
            return result;
        }
        public String paymentInfo_TimeOutHandler(Integer id){
            return "The 80 system is busy, please try again later,id: "+id+"\t"+". . . ";
        }

In the test, it takes 3s for the order to access pay, but it only runs for 1.5s, so it will be degraded in the order
Problems with the current two degraded modules
- The degradation method is written together with the business method, with a high degree of coupling
- Each business method has a downgrade method, and there are many duplicate codes

Duplicate code problem

Solution:

Configure a global degradation method. All methods can use this degradation method. For some special creation, create methods separately.

Create a global method

        //Global degradation method
        public String paymentInfo_Global_FallbackMethod(){
            return "Global Exception handling information, please try again later,...";
        }

Use the @DefaultProperties annotation to specify it as the global demotion method (default demotion method)

        @RestController
        @Slf4j
        @RequestMapping("/consumer")
        @DefaultProperties(defaultFallback = "paymentInfo_Global_FallbackMethod")
        public class OrderHystrixController {
            ...
        }

If the business method does not specify a specific degradation method, the default degradation method will be used

        @HystrixCommand
        public String paymentInfo_TimeOut(@PathVariable("id") Integer id){
            String result = paymentHystrixService.paymentInfo_TimeOut(id);
            return result;
        }

test

The problem of code coupling

Solution:

Modify the order module. From here, the pay module will not degrade the service. The service degradation can be written in the order module.

The PaymentHystrixService interface is used to remotely call the pay module. Here, we create a class implementation interface to perform unified degradation processing.

        @Component
        public class PaymentFallbackService implements PaymentHystrixService{
            @Override
            public String paymentInfo_OK(@PathVariable("id") Integer id) {
                return "----PaymentFallbackService fall back-paymentInfo_OK,...";
            }

            @Override
            public String paymentInfo_TimeOut(@PathVariable("id") Integer id) {
                return "----PaymentFallbackService fall back-paymentInfo_TimeOut,...";
            }
        }

Ensure that the festrix support of feign is enabled in the configuration file. Finally, make the implementation class of payment hystrix service effective, and modify the @FeignClient annotation of the interface
```
@FeignClient(value = "CLOUD-PROVIDER-HYSTRIX-PAYMENT",fallback = PaymentFallbackService.class)
```
Start the test, start the order and pay to access the OK interface normally

If the pay service is shut down, the order accesses the OK interface again (there is no other degradation method)

Solve the problems after coupling:

The In this way, the problem of code coupling is solved, but there are too many duplicate codes. Each method has a degradation method

Service fusing

When a service has a problem:

You need to close all access [circuit breakers] to this service, and then call service degradation.
When it is detected that the service response is normal, the call link is resumed.

Continue to modify pay module

Modify the Payservice interface and add methods related to service fusing

        //Service fusing
        @HystrixCommand(fallbackMethod = "paymentCircuitBreaker_fallback",commandProperties = {
                @HystrixProperty(name = "circuitBreaker.enable",value = "true"),//Whether the circuit breaker is enabled
                @HystrixProperty(name = "circuitBreaker.requestVolumeThreshold",value = "10"),//Number of requests
                @HystrixProperty(name = "circuitBreaker.sleepWindowInMilliseconds",value = "10000"),//Time window period
                @HystrixProperty(name = "circuitBreaker.errorThresholdPercentage",value = "60")//Trip after failure rate reaches
        })
        public String paymentCircuitBreaker(@PathVariable("id") Integer id){
            if(id<0)
                throw new RuntimeException("****id Cannot be negative");
            String serialNumber = IdUtil.simpleUUID();//hutool tool
            return Thread.currentThread().getName()+"\t"+"Call succeeded, serial number:"+serialNumber;
        }
        public String paymentCircuitBreaker_fallback(Integer id){
            return "id: "+id+"，Cannot be negative. Please try again later,...";
        }

Not to mention configuring the service degradation method. For the configured parameters:

If there are more than 10 concurrent operations or 6 of the 10 concurrent operations fail, the circuit breaker will be opened
The request will be attempted within 10 seconds of the time window period, and the circuit breaker will be closed if the request is successful

Modify the controller and add a test method

        //Service fusing
        @GetMapping("/circuit/{id}")
        public String paymentCircuitBreaker(@PathVariable("id") Integer id){
            String result = paymentService.paymentCircuitBreaker(id);
            log.info("****result: "+result);
            return result;
        }

Test:
- Multiple visits with error rate over 60%:
- At this time, the service is blown, and an error will be reported even if the access is correct:
- When the waiting window period has expired, cancel the fusing

Summary and supplement

All configurable properties of Hystrix

In addition, all parameter configurations can view the member variables in the HystrixCommandProperties class.

Overall process of fusing

When a request comes in, first query the cache. If there is a cache, return it directly; If the cache does not exist, go to 2
Check whether the circuit breaker is turned on. If it is turned on, Hystrix will directly forward the request to the degraded return, and then return; If the circuit breaker is closed, go to 3.
Judge whether the thread pool and other resources are full. If they are full, the downgrade method will be used. If the resources are not full, it will go to 4.
Determine what type of Hystrix we use, decide whether to call the constructor or the run method, and then process the request
Then, Hystrix will report the result information of this request to the circuit breaker. The circuit breaker receives the information and judges whether the conditions for opening or closing the circuit breaker are met.
If this request fails to be processed, it will enter the degradation method again; If the processing is successful, judge whether the processing has timed out. If it has timed out, enter the degradation method. If there is no timeout, the request is processed successfully and the result is returned to the controller.

Service monitoring

HystrixDashboard

The HystrixDashboard is a quasi real-time call monitoring provided by Hystrix. It records the execution information of all requests initiated by Hystrix and displays them in a graphical form.

Use of HystrixDashboard

Create a project named cloud consumer hystrix dashboard9001

pom file

		<dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-starter-netflix-hystrix-dashboard</artifactId>
        </dependency>

configuration file
```
        server:
          port: 9001
```

Main startup class

        @SpringBootApplication
        @EnableHystrixDashboard
        public class HystrixDashboarMain9001 {
            public static void main(String[] args){
                SpringApplication.run(HystrixDashboarMain9001.class,args);
            }
        }

Modify all pay modules (800180028003...) and add a pom dependency. We have configured it before

		<dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-actuator</artifactId>
        </dependency>

Start 9001 and access: http://localhost:9001/hystrix

At this time, you can only access the HystrixDashboard, which does not mean that 80018002 has been monitored. If you want to monitor, you also need to configure (8001 as an example):

Main startup class addition of 8001

	/**
     * This configuration is for service monitoring and has nothing to do with service fault tolerance. The upgrade of springcloud
     * ServletRegistrationBean Because the default path of SpringBoot is not "/hystrix.stream“
     * Just configure the upper and lower servlet s in your project
     */
    @Bean
    public ServletRegistrationBean getServlet() {
        HystrixMetricsStreamServlet streamServlet = new HystrixMetricsStreamServlet() ;
        ServletRegistrationBean registrationBean = new ServletRegistrationBean(streamServlet);
        registrationBean.setLoadOnStartup(1);
        registrationBean.addUrlMappings("/hystrix.stream");
        registrationBean.setName("HystrixMetricsStreamServlet");
        return  registrationBean;
    }