ELKF - build and deploy ELKF log system from scratch

Introduction to ELKF

The log system is a very important system. A well-structured log system can help developers understand the status of the server and system security status more clearly, so as to ensure the stable operation of the server. At present, in a large-scale back-end architecture, a standard data acquisition solution is usually called ELK, namely ElasticSearch, Logstash and Kibana. For more lightweight data collection, Filebeat is also referenced.
Therefore, we chose the Elasticsearch + Logstash + filebeat + Kibana architecture for demonstration this time.

Elasticsearch: Distributed search engine. It has the characteristics of high scalability, high reliability, and easy management. It can be used for full-text retrieval, structured retrieval and analysis, and can combine the three. Elasticsearch is developed based on Lucene using Java and is one of the most widely used open source search engines. Wikipedia, StackOverflow, Github, etc. are all built on it.

Logstash: Data collection and processing engine. It supports dynamic collection of data from various data sources, and performs operations such as filtering, analyzing, enriching, and unifying the format on the data, and then stores them for subsequent use.

Kibana: Visualization platform. It can search and display index data stored in Elasticsearch. Use it to easily display and analyze data with charts, tables, and maps.

Filebeat: A lightweight data collection engine. Compared with the system resources occupied by Logstash, the system resources occupied by Filebeat are almost negligible. It is based on the source code of the original Logstash-fowarder. In other words: Filebeat is the new version of Logstash-fowarder, and it will also be the first choice of ELK Stack in Agent.

  1. Install Elasticsearch

    docker pulls the image, using version 7.10.2.

    docker pull elasticsearch:7.10.2
    

    Create a mount directory and authorize the created folder.

    mkdir -p /worker/elk/elasticsearch
    #Folder authorization
    sudo chown -R 777 /worker/elk/elasticsearch
    #Note here: If 777 permissions are not enough, give 1000: 1000 permissions
    

    Execute the docker run script

    docker run -d -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" --name es mirror image ID
    

    Copy the configuration inside the elasticsearch container to the mount directory

    docker cp es:/usr/share/elasticsearch/config/elasticsearch.yml /worker/elk/elasticsearch/
    docker cp es:/usr/share/elasticsearch/data /worker/elk/elasticsearch/
    

    Modify elasticsearch.yml

    cluster.name: "es-master"
    network.host: 0.0.0.0
    # 0.0.0.0 can indicate listening to all IP s
    http.host: 0.0.0.0
    # across requests
    http.cors.enabled: true
    http.cors.allow-origin: "*"
    

    delete container

    docker rm -f es
    

    Execute the dokcer script again to start elasticsearch and set the mount directory

    docker run -d -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" --name es \
         --restart=always \
         -e ES_JAVA_OPTS="-Xms128m -Xmx512m" \
         -h elasticsearch \
         -v /worker/elk/elasticsearch/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml \
         -v /worker/elk/elasticsearch/data:/usr/share/elasticsearch/data \
         -e LANG=C.UTF-8 \
         -e LC_ALL=C.UTF-8 \
         mirror image ID
        
        
    Parameter Description:
    -p 9200:9200: Map port 9200 of the container to port 9200 of the host
    -p 9300:9300: Map port 9300 of the container to port 9300 of the host, so that the clusters can communicate with each other
    --restart=always: docker restart autostart
    -e "discovery.type=single-node": singleton pattern
    -e ES_JAVA_OPTS="-Xms128m -Xmx512m": Configure memory size,the smallest/maximum memory
    -v mount directory
    
  2. Kibana

    docker pulls the image, using version 7.10.2

    docker pull kibana:7.10.2
    

    Create a mount directory and authorize the created folder (similar to the creation process of elasticsearch).

    mkdir -p /worker/elk/kibana
    #Folder authorization
    sudo chown -R 777 /worker/elk/kibana 
    

    Execute the docker run script

    sudo docker run -d --name kibana -p 5601:5601 mirror image ID
    

    Copy the configuration inside the kibana container to the mount directory

    docker cp kibana:/usr/share/kibana/config /worker/elk/kibana/
    docker cp kibana:/usr/share/kibana/data /worker/elk/kibana/   
    

    Modify the kibana.yml configuration file

    server.name: kibana
    server.host: "0.0.0.0"
    elasticsearch.hosts: [ "http://elasticsearch:9200" ]
    # show login page
    monitoring.ui.container.elasticsearch.enabled: true
    # language
    i18n.locale: "zh-CN"
    

    delete container

    docker rm -f kibana
    

    Execute the dokcer script again to start kibana, and set the mount directory

    docker run -d --name kibana -p 5601:5601 \
          --restart=always \
          --link es:elasticsearch \
          -h kibana \
          -v /worker/elk/kibana/config/kibana.yml:/usr/share/kibana/config/kibana.yml \
          -v /worker/elk/kibana/data:/usr/share/kibana/data \
          mirror image ID
          
    Parameter Description:
    --link: useā€“link parameter, docker Will automatically share the variables of the two containers. when visiting,no longer through IP way to access,Instead, it is accessed by the container name. access above es container, which can be directly elasticsearch:9200 can access es container.
    
  3. Logstash

    docker pulls the image, using version 7.10.2

    docker pull logstash:7.10.2
    

    Create a mount directory and authorize the created folder.

    mkdir -p /worker/elk/logstash
    #Folder authorization
    sudo chown -R 777 /worker/elk/logstash
    

    create container

    docker run -p 4560:4560 --name logstash -d mirror image ID
    

    Copy the configuration inside the logstash container to the mount directory

    docker container cp logstash:/usr/share/logstash/config /worker/elk/logstash 
    docker container cp logstash:/usr/share/logstash/pipeline /worker/elk/logstash
    docker container cp logstash:/usr/share/logstash/data /worker/elk/logstash
    

    Modify the default configuration of the logstash.yml file, the path of the configuration file (/worker/elk/logstash/config)

    http.host: "0.0.0.0"
    #allow monitoring
    xpack.monitoring.enabled: true
    xpack.monitoring.elasticsearch.hosts: [ "http://elasticsearch:9200" ]
    

    Modify the file configuration of logstash.conf, the path of the configuration file (/worker/elk/logstash/pipeline), configure according to the specific situation

    # Listen to port 5044 as an input source
    input {
      beats {
        port => 5044
      }
    }
    
    # filter
    filter {
      grok {
        match => { "message" => "%{COMBINEDAPACHELOG}" }
      }
      date {
        match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
      }
    }
    
    # output to es
    output {
      elasticsearch {
        hosts => ["elasticsearch:9200"]
        # Index name, get the parameters set by filebeat here
        index => "%{[fields][service]}-%{+YYYY.MM.dd}"
      }
    
      stdout {
        codec => rubydebug
      }
    }
    
    
    

    stop and delete the container

    docker rm -f logstash
    

    Restart the container and set the mount directory

    docker run -d \
        --name logstash \
        -p 5044:5044 \
        --restart=always \
        -h logstash \
        --link es:elasticsearch \
        -v /worker/elk/logstash/config:/usr/share/logstash/config \
        -v /worker/elk/logstash/pipeline:/usr/share/logstash/pipeline \
        mirror image ID
    

    Enter the logstash container and install the plugin

    bin/logstash-plugin install logstash-codec-json_lines
    
  4. filebeat

    docker pull docker.elastic.co/beats/filebeat:7.10.2
    

    Then, execute the docker start script.

    docker run  -d --name filebeat -p 9000:9000 mirror image ID
    

    Next, create a filebeat mount directory and authorize the created file.

    mkdir -p /worker/elk/filebeat
    sudo chown -R 777 /worker/elk/filebeat
    
    # Copy the files inside the container to the host.
    docker cp filebeat:/usr/share/filebeat/filebeat.yml /worker/elk/filebeat
    

    Modify the configuration file filebeat.yml

    filebeat.config:
      modules:
        path: ${path.config}/modules.d/*.yml
        reload.enabled: false
    
    processors:
      - add_cloud_metadata: ~
      - add_docker_metadata: ~
    
    filebeat.inputs:
    - type: tcp
      enabled: true
      # Setting parameters
      fields:
        service: estate
      max_message_size: 10MiB
      host: "0.0.0.0:9000"
    
    # output to logstash
    output.logstash:
      hosts: ["logstash:5044"]
    
    

    delete container

    docker rm -f filebeat
    

    Next, modify the docker startup script and add the -v mount directory.

    docker run  -d \
    	--name filebeat \
    	-p 9000:9000 \
    	-h filebeat \
    	-v /worker/elk/filebeat/filebeat.yml:/usr/share/filebeat/filebeat.yml \
    	--restart=always \
    	--link logstash:logstash \
    	-e TZ=Asia/Shanghai \
    	-e output.logstash.hosts=["logstash:5044"] \
        mirror image ID
    
  5. connect filebeat

    The springboot project connects to filebeat for log output, and the address is 192.168.189.128:9000 (according to your own ip). For details, please refer to other articles.

  6. configure kibana

    Enter the kibana page, the address is http://192.168.189.128:5601/ (according to your own ip).

    ![Insert picture description here](https://img-blog.csdnimg.cn/9075b03633ad43b48ae15a4dbeb5be6f.jpeg#pic_center Enter the index name of the logstash configuration and click Next.


See that there is already log output.

Tags: Operation & Maintenance Big Data Docker ELK ElasticSearch

Posted by the_last_tamurai on Fri, 16 Dec 2022 20:47:43 +0530