Introduction to ELKF

The log system is a very important system. A well-structured log system can help developers understand the status of the server and system security status more clearly, so as to ensure the stable operation of the server. At present, in a large-scale back-end architecture, a standard data acquisition solution is usually called ELK, namely ElasticSearch, Logstash and Kibana. For more lightweight data collection, Filebeat is also referenced.
Therefore, we chose the Elasticsearch + Logstash + filebeat + Kibana architecture for demonstration this time.

Elasticsearch: Distributed search engine. It has the characteristics of high scalability, high reliability, and easy management. It can be used for full-text retrieval, structured retrieval and analysis, and can combine the three. Elasticsearch is developed based on Lucene using Java and is one of the most widely used open source search engines. Wikipedia, StackOverflow, Github, etc. are all built on it.

Logstash: Data collection and processing engine. It supports dynamic collection of data from various data sources, and performs operations such as filtering, analyzing, enriching, and unifying the format on the data, and then stores them for subsequent use.

Kibana: Visualization platform. It can search and display index data stored in Elasticsearch. Use it to easily display and analyze data with charts, tables, and maps.

Filebeat: A lightweight data collection engine. Compared with the system resources occupied by Logstash, the system resources occupied by Filebeat are almost negligible. It is based on the source code of the original Logstash-fowarder. In other words: Filebeat is the new version of Logstash-fowarder, and it will also be the first choice of ELK Stack in Agent.

1. Install Elasticsearch

   docker pulls the image, using version 7.10.2.

   docker pull elasticsearch:7.10.2
   

   Create a mount directory and authorize the created folder.

   mkdir -p /worker/elk/elasticsearch
   #Folder authorization
   sudo chown -R 777 /worker/elk/elasticsearch
   #Note here: If 777 permissions are not enough, give 1000: 1000 permissions
   

   Execute the docker run script

   docker run -d -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" --name es mirror image ID
   

   Copy the configuration inside the elasticsearch container to the mount directory

   docker cp es:/usr/share/elasticsearch/config/elasticsearch.yml /worker/elk/elasticsearch/
   docker cp es:/usr/share/elasticsearch/data /worker/elk/elasticsearch/
   

   Modify elasticsearch.yml

   cluster.name: "es-master"
   network.host: 0.0.0.0
   # 0.0.0.0 can indicate listening to all IP s
   http.host: 0.0.0.0
   # across requests
   http.cors.enabled: true
   http.cors.allow-origin: "*"
   

   delete container

   docker rm -f es
   

   Execute the dokcer script again to start elasticsearch and set the mount directory

   docker run -d -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" --name es \
        --restart=always \
        -e ES_JAVA_OPTS="-Xms128m -Xmx512m" \
        -h elasticsearch \
        -v /worker/elk/elasticsearch/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml \
        -v /worker/elk/elasticsearch/data:/usr/share/elasticsearch/data \
        -e LANG=C.UTF-8 \
        -e LC_ALL=C.UTF-8 \
        mirror image ID
       
       
   Parameter Description:
   -p 9200:9200: Map port 9200 of the container to port 9200 of the host
   -p 9300:9300: Map port 9300 of the container to port 9300 of the host, so that the clusters can communicate with each other
   --restart=always: docker restart autostart
   -e "discovery.type=single-node": singleton pattern
   -e ES_JAVA_OPTS="-Xms128m -Xmx512m": Configure memory size,the smallest/maximum memory
   -v mount directory
   

2. Kibana

   docker pulls the image, using version 7.10.2

   docker pull kibana:7.10.2
   

   Create a mount directory and authorize the created folder (similar to the creation process of elasticsearch).

   mkdir -p /worker/elk/kibana
   #Folder authorization
   sudo chown -R 777 /worker/elk/kibana 
   

   Execute the docker run script

   sudo docker run -d --name kibana -p 5601:5601 mirror image ID
   

   Copy the configuration inside the kibana container to the mount directory

   docker cp kibana:/usr/share/kibana/config /worker/elk/kibana/
   docker cp kibana:/usr/share/kibana/data /worker/elk/kibana/   
   

   Modify the kibana.yml configuration file

   server.name: kibana
   server.host: "0.0.0.0"
   elasticsearch.hosts: [ "http://elasticsearch:9200" ]
   # show login page
   monitoring.ui.container.elasticsearch.enabled: true
   # language
   i18n.locale: "zh-CN"
   

   delete container

   docker rm -f kibana
   

   Execute the dokcer script again to start kibana, and set the mount directory

   docker run -d --name kibana -p 5601:5601 \
         --restart=always \
         --link es:elasticsearch \
         -h kibana \
         -v /worker/elk/kibana/config/kibana.yml:/usr/share/kibana/config/kibana.yml \
         -v /worker/elk/kibana/data:/usr/share/kibana/data \
         mirror image ID
         
   Parameter Description:
   --link: use–link parameter, docker Will automatically share the variables of the two containers. when visiting,no longer through IP way to access,Instead, it is accessed by the container name. access above es container, which can be directly elasticsearch:9200 can access es container.
   

3. Logstash

   docker pulls the image, using version 7.10.2

   docker pull logstash:7.10.2
   

   Create a mount directory and authorize the created folder.

   mkdir -p /worker/elk/logstash
   #Folder authorization
   sudo chown -R 777 /worker/elk/logstash
   

   create container

   docker run -p 4560:4560 --name logstash -d mirror image ID
   

   Copy the configuration inside the logstash container to the mount directory

   docker container cp logstash:/usr/share/logstash/config /worker/elk/logstash 
   docker container cp logstash:/usr/share/logstash/pipeline /worker/elk/logstash
   docker container cp logstash:/usr/share/logstash/data /worker/elk/logstash
   

   Modify the default configuration of the logstash.yml file, the path of the configuration file (/worker/elk/logstash/config)

   http.host: "0.0.0.0"
   #allow monitoring
   xpack.monitoring.enabled: true
   xpack.monitoring.elasticsearch.hosts: [ "http://elasticsearch:9200" ]
   

   Modify the file configuration of logstash.conf, the path of the configuration file (/worker/elk/logstash/pipeline), configure according to the specific situation

   # Listen to port 5044 as an input source
   input {
     beats {
       port => 5044
     }
   }
   
   # filter
   filter {
     grok {
       match => { "message" => "%{COMBINEDAPACHELOG}" }
     }
     date {
       match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
     }
   }
   
   # output to es
   output {
     elasticsearch {
       hosts => ["elasticsearch:9200"]
       # Index name, get the parameters set by filebeat here
       index => "%{[fields][service]}-%{+YYYY.MM.dd}"
     }
   
     stdout {
       codec => rubydebug
     }
   }
   
   
   

   stop and delete the container

   docker rm -f logstash
   

   Restart the container and set the mount directory

   docker run -d \
       --name logstash \
       -p 5044:5044 \
       --restart=always \
       -h logstash \
       --link es:elasticsearch \
       -v /worker/elk/logstash/config:/usr/share/logstash/config \
       -v /worker/elk/logstash/pipeline:/usr/share/logstash/pipeline \
       mirror image ID
   

   Enter the logstash container and install the plugin

   bin/logstash-plugin install logstash-codec-json_lines
   

4. filebeat

   docker pull docker.elastic.co/beats/filebeat:7.10.2
   

   Then, execute the docker start script.

   docker run  -d --name filebeat -p 9000:9000 mirror image ID
   

   Next, create a filebeat mount directory and authorize the created file.

   mkdir -p /worker/elk/filebeat
   sudo chown -R 777 /worker/elk/filebeat
   
   # Copy the files inside the container to the host.
   docker cp filebeat:/usr/share/filebeat/filebeat.yml /worker/elk/filebeat
   

   Modify the configuration file filebeat.yml

   filebeat.config:
     modules:
       path: ${path.config}/modules.d/*.yml
       reload.enabled: false
   
   processors:
     - add_cloud_metadata: ~
     - add_docker_metadata: ~
   
   filebeat.inputs:
   - type: tcp
     enabled: true
     # Setting parameters
     fields:
       service: estate
     max_message_size: 10MiB
     host: "0.0.0.0:9000"
   
   # output to logstash
   output.logstash:
     hosts: ["logstash:5044"]
   
   

   delete container

   docker rm -f filebeat
   

   Next, modify the docker startup script and add the -v mount directory.

   docker run  -d \
   	--name filebeat \
   	-p 9000:9000 \
   	-h filebeat \
   	-v /worker/elk/filebeat/filebeat.yml:/usr/share/filebeat/filebeat.yml \
   	--restart=always \
   	--link logstash:logstash \
   	-e TZ=Asia/Shanghai \
   	-e output.logstash.hosts=["logstash:5044"] \
       mirror image ID
   

5. connect filebeat

   The springboot project connects to filebeat for log output, and the address is 192.168.189.128:9000 (according to your own ip). For details, please refer to other articles.

6. configure kibana

   Enter the kibana page, the address is http://192.168.189.128:5601/ (according to your own ip).
   

   ![Insert picture description here](https://img-blog.csdnimg.cn/9075b03633ad43b48ae15a4dbeb5be6f.jpeg#pic_center Enter the index name of the logstash configuration and click Next.
   
   

See that there is already log output.