Distributed micro service e-commerce project

Recently, it took nearly two months to complete a distributed microservice project. This blog will summarize the key technologies involved in the project.


Some middleware and service software used in the project are installed and configured using docker. If you do not use docker to install and configure these software, you can refer to the following blog post:
Docker common software installation

Technologies used in the project:

  • Microservice framework: SpringBoot (2.2.2), SpringCloud (Hoxton.SR1), SpringCloudAlibaba (2.2.0.RELEASE)
  • Database: mysql (8.0.17), redis
  • Persistence layer framework: mybatis plus (3.2.0)
  • Retrieval middleware: elasticsearch (7.4.2)
  • Distributed cache: SpringCache
  • Distributed lock: Redisson (3.12.5)
  • Message queue: AMQP rabbitmq
  • Dynamic and static separation: Nginx
  • Scheduled task: Spring Schedule
  • Share Session:spring Session
  • Template engine: thymeleaf

Some of the gateways, service monitoring, service fusing, degradation and current limiting use technologies in SpringBoot and SpringCloudAlibaba, so I won't list them one by one.

--------------------------------------Split line--------------------------------------

Next, I will explain the key technologies used in each part of the process of a customer entering the shopping website, from logging in, retrieving goods, adding to the shopping cart, placing orders, going to pay, and killing goods. Each process will be attached with a corresponding business logic diagram, so as to understand the process of the entire e-commerce project.

1, Login

There are two important points for user login:

  1. Users log in with social accounts (Weibo, wechat, QQ, etc.)
  2. The user login information can be displayed everywhere (sub domain name) at one login (parent domain name) in the website

1. social account login

In the project, the blogger uses the social account of Weibo to log in. For details, please refer to the Weibo OAuth2.0 document:
Weibo OAuth2.0 login - use the interface for in-depth development, suitable for back-end developers Before using, you need to create an application project to obtain App Key and App Secret.

The general process of social login is as follows:

Use the Access Token to obtain user information through the microblog API: Weibo API

2.SpringSession shared login information

After a user logs in, no matter which page of the website the user visits, each micro service should know the login information of the current user. In the project, bloggers use SpringSession to solve the problem of Session sharing.

However, there is another problem. The user login information is stored in the Session. The underlying layer of the Session also obtains data from the server through cookies. However, the domain names of different services of the website are different. To solve this problem, it is necessary to expand the domain name range of the Cookie that stores the user information to the parent domain, so as to ensure that each service can obtain cookies.

Using SpringSession is the same as Session. Get HttpSession and call Session SetAttribute () method stores data, but SpringSession stores data in Redis, ensuring that distributed services can share sessions.

SpringSession configuration class:

public class RedisSessionConfig {

     * cooike Serializer: Custom cookie scope
    public CookieSerializer cookieSerializer() {
        DefaultCookieSerializer cookieSerializer = new DefaultCookieSerializer();

        // Set the scope of the cookie as the parent domain of the item. All domains can be accessed

        return cookieSerializer;

     * Redis Serializer: set the serialization mechanism stored in Redis
    public RedisSerializer<Object> redisSerializer() {
        return new GenericJackson2JsonRedisSerializer();

Login simple business sorting:

2, Retrieve items

Commodity retrieval mainly refers to the use of ElasticSearch. There are two points in the project:

  1. Commodity on the shelf: you need to create an index in advance according to the fields of commodity information and the mapping relationship of each field, and save the entered commodity information;
  2. Commodity retrieval: dynamically build DSL statements according to the retrieval parameters, send retrieval requests to the ES server according to the DSL statements, and get the retrieval response and package it as the specified data return.

Simple business sorting of retrieving commodities:

3, Add to cart

There is a concern for adding items to the shopping cart:

  • Temporary users and login users: temporary users can also add products to the shopping cart. After closing the page, they can view the previously added products next time.

To solve this business problem, two shopping cart data structures are used to store the temporary shopping cart and the user's shopping cart in Redis. When the user logs in to view the shopping cart, the goods in the temporary shopping cart will be merged into the user's shopping cart.

Solution: create an interceptor. For each user who is not logged in, when using the shopping cart service, a user key will be assigned and saved in the Cookie. Set the Cookie to be saved for 1 month. In this way, temporary users can view the products they have added within one month.

CartInterceptor interceptor class:

public class CartInterceptor implements HandlerInterceptor {

    public static ThreadLocal<UserInfoTo> threadLocal = new ThreadLocal<>();

     * Before business execution: check the user login status. If it is a temporary user, assign a user key
    public boolean preHandle(HttpServletRequest request, HttpServletResponse response, Object handler) throws Exception {
        UserInfoTo userInfoTo = new UserInfoTo();

        HttpSession session = request.getSession();
        MemberRespVo member = (MemberRespVo) session.getAttribute(AuthServerConstant.LOGIN_USER);
        // 1. user login:
        if (member != null) {
        // 2. temporary user: if it already exists, get the user key from the browser
        Cookie[] cookies = request.getCookies();
        if (cookies != null && cookies.length > 0) {
            for (Cookie cookie : cookies) {
                if (cookie.getName().equals(CartConstant.TEMP_USER_COOKIE_NAME)) {
                    userInfoTo.setTempUser(true); // The tag is already a temporary user, so it is not necessary to save the cookie

        // 3. temporary user: when using the shopping cart for the first time, a user key will be automatically generated, and the user key will be saved in the browser cookie in the postHandle
        if (StringUtils.isEmpty(userInfoTo.getUserKey())) {
            String userKey = UUID.randomUUID().toString();

        // Before the method is executed, the user's login status is stored in ThreadLocal to facilitate the later controller's execution
        return true;

     * After business execution:
     *  If you visit the shopping cart for the first time, save the temporary user's user key into the browser cookie to ensure that this user key will be carried with each subsequent visit
    public void postHandle(HttpServletRequest request, HttpServletResponse response, Object handler, ModelAndView modelAndView) throws Exception {
        UserInfoTo userInfoTo = threadLocal.get();
        // If the shopping cart is used for the first time, a user key will be saved in the browser cookie
        if (!userInfoTo.isTempUser()) {
            Cookie cookie = new Cookie(CartConstant.TEMP_USER_COOKIE_NAME, userInfoTo.getUserKey());
            cookie.setMaxAge(CartConstant.TEMP_USER_COOKIE_TIMEOUT); // Set the expiration time of 1 month

When Redis saves the commodity information of each shopping cart, it uses the hash type. Using the hash type can make it very fast to view the shopping cart items, settlement and other business logic.

Simple business sorting of shopping cart:

4, Place an order

Order service involves multi service distributed transactions. How to ensure order creation, inventory deduction, order rollback, and inventory rollback are the key issues for order placement:

  1. SpringCloudAlibabaSeata: a framework to ensure distributed service transactions. However, due to the heavy lock, it is not conducive to the order service with large user traffic, but to the back-end goods on the shelves and other businesses;
  2. Distributed transaction (flexible transaction + reliable message + final consistency): RabbitMQ is used for message communication between services to ensure the final consistency between orders and inventory. The advantage of using message queue is that it can process business quickly.

Bloggers use RabbitMQ to ensure the consistency of transactions between order services and inventory services when placing orders and reducing inventory in the project.

Insert a diagram here to briefly understand the workflow of RabbitMQ:

In the message queue, you can use the dead letter queue to control the effective time of the order. If the order fails to complete the payment within the effective time, the dead letter queue will send a message to the inventory service to unlock the inventory, and the inventory quantity will be rolled back manually in the inventory service.

Workflow diagram of message queue between order service and inventory service:

5, Payment

Because it is only a simple project, I use the third-party Alipay sandbox environment for payment test. If I don't know about Alipay sandbox environment, I can refer to: sandbox environment

During the payment process, Alipay uses RSA asymmetric encryption algorithm to ensure the security of the payment process. The RSA encryption algorithm can be simply understood through the following figure:

Therefore, when using the payment interface, we need to upload a merchant public key to Alipay (the merchant's private key is safeguarded by itself), and Alipay will give us a Alipay public key. During the payment process, we use the merchant's private key to encrypt the data. Alipay performs signature verification and decryption through the merchant's public key, and sends the response to us again through Alipay private key encryption. We can obtain the payment response results only after we use Alipay public key to decrypt and decrypt.

Call the template class of Alipay payment:

@ConfigurationProperties(prefix = "alipay")
public class AlipayTemplate {

    //id of the application created on Alipay
    private String app_id = "2016102600763438";

    // Merchant private key, your RSA2 private key in PKCS8 format
    private String merchant_private_key = xxx;
    // Alipay public key, viewing address: https://openhome.alipay.com/platform/keyManage.htm Corresponding to Alipay public key under APPID.
    private String alipay_public_key = xxx;
    // The server [asynchronous notification] page path requires a full path in the format of http:// and cannot be added? User defined parameters such as id=123 must be accessible through the Internet
    // Alipay will quietly send us a request to tell us the information of successful payment
    private String notify_url;

    // The page path of page Jump synchronization notification requires a full path in the format of http:// and cannot be added? User defined parameters such as id=123 must be accessible through the Internet
    //Synchronization notification, payment successful, generally jump to the success page
    private String return_url;

    // Signature method
    private String sign_type = "RSA2";

    // Character encoding format
    private String charset = "utf-8";

    // Order timeout
    private String timeout = "30m";

    // Alipay gateway; https://openapi.alipaydev.com/gateway.do
    private String gatewayUrl = "https://openapi.alipaydev.com/gateway.do";

    public String pay(PayVo vo) throws AlipayApiException {

        //AlipayClient alipayClient = new DefaultAlipayClient(AlipayTemplate.gatewayUrl, AlipayTemplate.app_id, AlipayTemplate.merchant_private_key, "json", AlipayTemplate.charset, AlipayTemplate.alipay_public_key, AlipayTemplate.sign_type);
        //1. Generate a payment client according to the configuration of Alipay
        AlipayClient alipayClient = new DefaultAlipayClient(gatewayUrl,
                app_id, merchant_private_key, "json",
                charset, alipay_public_key, sign_type);

        //2. Create a payment request. / / set the request parameters
        AlipayTradePagePayRequest alipayRequest = new AlipayTradePagePayRequest();

        //Merchant order number, the only order number in the merchant website order system, required
        String out_trade_no = vo.getOut_trade_no();
        //Payment amount, required
        String total_amount = vo.getTotal_amount();
        //Order name, required
        String subject = vo.getSubject();
        //Product description, can be blank
        String body = vo.getBody();

        alipayRequest.setBizContent("{\"out_trade_no\":\"" + out_trade_no + "\","
                + "\"total_amount\":\"" + total_amount + "\","
                + "\"subject\":\"" + subject + "\","
                + "\"body\":\"" + body + "\","
                + "\"timeout_express\":\"" + timeout + "\","
                + "\"product_code\":\"FAST_INSTANT_TRADE_PAY\"}");

        String result = alipayClient.pageExecute(alipayRequest).getBody();

        //You will receive a response from Alipay. The response is a page. As long as the browser displays this page, it will automatically go to the cashier page of Alipay
        System.out.println("Response from Alipay:" + result);

        return result;


If the payment is completed, Alipay will have two returns (direct return and asynchronous callback). The direct return jumps to the user's order list. The asynchronous callback can perform subsequent processing on the order (unlocking inventory, modifying order status, etc.), and the asynchronous callback is faster and faster than the direct return. To use the asynchronous callback, the Alipay server must first return a success, otherwise the Alipay server will always send the asynchronous request (for details: Alipay asynchronous callback).

Handling asynchronous callback methods:

     * Alipay asynchronous callback notification: https://opendocs.alipay.com/open/270/105902
    public String handleAilpayed(HttpServletRequest request, PayAsyncVo vo) throws AlipayApiException, UnsupportedEncodingException {
        System.out.println("Enter payment asynchronous callback...");
        // As long as we receive the asynchronous notification from Alipay, telling us that the order has been successfully paid and returning success, Alipay will not notify us
        // 1. signature verification
        // Get feedback from Alipay POST
        Map<String,String> params = new HashMap<String,String>();
        Map<String,String[]> requestParams = request.getParameterMap();
        for (Iterator<String> iter = requestParams.keySet().iterator(); iter.hasNext();) {
            String name = (String) iter.next();
            String[] values = (String[]) requestParams.get(name);
            String valueStr = "";
            for (int i = 0; i < values.length; i++) {
                valueStr = (i == values.length - 1) ? valueStr + values[i]
                        : valueStr + values[i] + ",";
            // Garbled code solution, this code is used in case of garbled code
//            valueStr = new String(valueStr.getBytes("ISO-8859-1"), "utf-8");
            params.put(name, valueStr);

        boolean signVerified = AlipaySignature.rsaCheckV1(params, alipayTemplate.getAlipay_public_key(), alipayTemplate.getCharset(), alipayTemplate.getSign_type()); //Call SDK to verify signature

        // 2. the signature is verified successfully and the order is processed
        if (signVerified) {
            String result = orderService.handlePayResult(vo);
            System.out.println("Signature verification succeeded..");
            return result;
        return "error";

Sorting of payment business:

6, Second kill

Commodity seckill is a classic high concurrency scenario. It is very important to make your application withstand millions of levels of concurrency. Here are some issues that should be paid attention to in high concurrency scenarios:

  1. Single service responsibility + independent deployment: ensure that the seckill service is deployed separately. Even if it is suspended, it will not affect the operation of other services;
  2. Seckill link encryption: to prevent malicious attacks, ensure that seckill links can only be accessed at the moment when seckill starts, and prevent seckill of goods in advance;
  3. Inventory preheating + quick deduction: before the start of the second kill, store the goods that need to be killed in Redis in advance. The deduction of inventory can be controlled by semaphores;
  4. Dynamic and static separation: ensure that dynamic requests for seckill and product details pages can enter the background service cluster, while some static resources can access Nginx;
  5. Malicious request interception: identify and intercept illegal attack requests, usually at the gateway layer;
  6. Flow peak staggering: use various means to share the instantaneous flow into a time period, such as entering the verification code during the second kill, adding to the shopping cart, etc;
  7. Current limiting & fusing & degradation: limit the number of times, limit the total amount, fast failure degradation, fuse isolation to prevent avalanche;
  8. Queue peak shaving: you can enter the successful seckill request into the queue, while the order service listens to the queue, slowly creates the queue, and then reduces the inventory.

With the above methods and the deployment of some clusters, it is not impossible to handle millions of concurrency. Bloggers also try to meet some of the above conditions in the project:

  1. Separate deployment of secsha service;
  2. A random code is added to each seckill commodity, which can only be obtained at the seckill time, and the seckill commodity must have a random code to perform seckill, ensuring seckill link encryption;
  3. Nearly three days' seckill commodities are stored in Redis, and the semaphore of distributed lock Redisson is used to deduct the commodity inventory;
  4. Use Nginx to ensure dynamic and static separation;
  5. SpringCloud Gateway service will intercept malicious requests;
  6. Use SpringCloudAlibaba Sentinel to limit, fuse and degrade the service;
  7. RabbitMQ is used to complete the queue peak shaving. After the goods are killed successfully, a message will be sent to the order service. The order service listens to the message and completes the subsequent order business logic.

Another thing to note is that for the launch of seckill products, bloggers here use the timing task annotation @enablesscheduling @scheduled (cron = "003 * *?") provided by Spring, The scheduled task carries out the second kill of commodities on the shelves at 3:00 every day, and the second kill field data of the last 3 days on the shelves.

Secsha business sorting:

In the seckill process, these operations have no database operations, service calls, etc. each step is executed very quickly, which improves the processing throughput of seckill business.

7, Other technical supplements

1. The message header is lost in the remote call of openfeign

Cause of the problem: when Feign is used for remote calls, a proxy object will be created to re encapsulate a RequestTemplate, but the Request does not carry the Cookie information of the original Request header, resulting in the empty content of the remote call access Session, and the user information stored in the Session cannot be obtained between services.

Solution: a RequestInterceptor interceptor will be provided during the process of building the RequestTemplate by Feign's proxy object. We can use this mechanism to modify the RequestTemplate and add the Cookie information in the original Request to the new RequestTemplate.

Feign's configuration class:

public class MyFeignConfig {
    public RequestInterceptor requestInterceptor() {
        return template -> {
            // 1. use RequestContextHolder to get the original request
            ServletRequestAttributes attributes = (ServletRequestAttributes) RequestContextHolder.getRequestAttributes();

            if (attributes != null) {
                HttpServletRequest request = attributes.getRequest();

                // 2. add the Cookie information in the original request to the new RequestTemplate
                String cookie = request.getHeader("Cookie");
                template.header("Cookie", cookie);

2. asynchronous orchestration, unable to get the main thread Request

In order to solve the problem of message header loss in OpenFeign remote call, we can get the original Request from RequestAttributes, set the Cookie of the original Request to Feign RequestTemplate, but call requestcontextholder in the Feign interceptor of asynchronous thread When getrequestattributes (), it is only the RequestAttributes of the new thread. The RequestAttributes of the new thread do not store the Request request of the main thread, so the context will be lost when the asynchronous task Feign is called.

main -> ThreadLocal (RequestAttributes(main) -> "/toTrade"Requested information)
          main("/toTrade") --- confirmOrder() ------------------------
                               thread1 -> ThreadLocal (RequestAttributes(thread1) -> null)
                               thread1:addressFuture ----------
                               thread2 -> ThreadLocal (RequestAttributes(thread2) -> null)
                               thread2:cartItemsFuture --------

Solution: before starting the asynchronous task, store the RequestAttributes(main) of the main thread in the RequestAttributes of the asynchronous thread for inter thread sharing, so that other threads can obtain the request context information through the RequestAttributes of the main thread.

// Get the RequestAttributes of the main front thread
RequestAttributes requestAttributes = RequestContextHolder.getRequestAttributes();

// 1. remotely query all receiving lists
CompletableFuture<Void> addressFuture = CompletableFuture.runAsync(() -> {
    // Set the RequestAttributes of the main thread to its own ThreadLocal
    List<MemberAddressVo> address = memberFeignService.getAddress(memberId);
}, executor);

8, Summary

It is very meaningful to spend two months to complete a distributed project. I have learned a lot from this project and consolidated what I have learned before. In general, it is very helpful to me. Continue!

Project GitHub warehouse address:

Tags: Java Docker Distribution

Posted by ppgpilot on Tue, 31 May 2022 20:15:05 +0530