These Shell analysis server log commands are collected and collected~

My small website runs on the Linux server of the 3A platform. I occasionally analyze my website server logs to see the traffic of the website. Let's see if there is any sabotage by Hei Kuo! So I collected and sorted out some server log analysis commands. You can try!

1. See how many IP accesses there are:

awk '{print $1}' log_file|sort|uniq|wc -l

2. View the number of times a page has been accessed:

grep "/index.php" log_file | wc -l

3. Check how many pages are accessed by each IP:

awk '{++S[$1]} END {for (a in S) print a,S[a]}' log_file > log.txt

sort -n -t ' ' -k 2 log.txt coordination sort Further sort

4. Sort the number of pages accessed by each IP from small to large:

awk '{++S[$1]} END {for (a in S) print S[a],a}' log_file | sort -n

5. To view which pages are accessed by an IP:

grep ^111.111.111.111 log_file| awk '{print $1,$7}'

6. Remove the page of search engine statistics:

awk '{print $12,$1}' log_file | grep ^\"Mozilla | awk '{print $2}' |sort | uniq | wc -l

7. Check the number of IP accesses in one hour at 14:00 on August 16, 2015:

awk '{print $4,$1}' log_file | grep 16/Aug/2015:14 | awk '{print $2}'| sort | uniq | wc -l

8. View the top ten ip addresses accessed

awk '{print $1}' |sort|uniq -c|sort -nr |head -10 access_log

uniq -c is equivalent to grouping statistics and putting statistics first

cat access.log|awk '{print $1}'|sort|uniq -c|sort -nr|head -10
cat access.log|awk '{counts[$(11)]+=1}; END {for(url in counts) print counts[url], url}

9. Top 10 files or pages visited

cat log_file|awk '{print $11}'|sort|uniq -c|sort -nr | head -10

cat log_file|awk '{print $11}'|sort|uniq -c|sort -nr|head -20

awk '{print $1}' log_file |sort -n -r |uniq -c | sort -n -r | head -20

Top 20 IPS with the largest access

10. The number of visits through the subdomain name is calculated according to the referer, which is slightly inaccurate

cat access.log | awk '{print $11}' | sed -e ' s/http:\/\///' -e ' s/\/.*//' | sort | uniq -c | sort -rn | head -20

11. List the files with the largest transfer size

cat www.access.log |awk '($7~/\.php/){print $10 " " $1 " " $4 " " $7}'|sort -nr|head -100

12. List the pages with output greater than 200000 bytes (about 200 KB) and the occurrence times of corresponding pages

cat www.access.log |awk '($10 > 200000 && $7~/\.php/){print $7}'|sort -n|uniq -c|sort -nr|head -100

13. If the last column of the log records the transfer time of the page file, the most time-consuming page to the client is listed

cat www.access.log |awk '($7~/\.php/){print $NF " " $1 " " $4 " " $7}'|sort -nr|head -100

14. List the most time-consuming pages (more than 60 seconds) and the corresponding page occurrence times

cat www.access.log |awk '($NF > 60 && $7~/\.php/){print $7}'|sort -n|uniq -c|sort -nr|head -100

15. List files whose transfer time exceeds 30 seconds

cat www.access.log |awk '($NF > 30){print $7}'|sort -n|uniq -c|sort -nr|head -20

16. List the number of each process running on the current server, in reverse order

ps -ef | awk -F ' ' '{print $8 " " $9}' |sort | uniq -c |sort -nr |head -20

17. View the current concurrent accesses of apache

What is the difference between the numbers of MaxClients in httpd.conf

netstat -an | grep ESTABLISHED | wc -l

18. You can view data using the following parameters:

ps -ef|grep httpd|wc -l
1388

Count the number of httpd processes. One process will be started for each request. It is used in the Apache server.

Indicates that Apache can handle 1388 concurrent requests. This value can be automatically adjusted by Apache according to the load situation

netstat -nat|grep -i "80"|wc -l

4341

netstat -an will print the current network link status of the system, while grep -i "80" is used to extract the connections related to port 80, and wc -l will count the number of connections.
The final number returned is the total number of requests from all 80 ports

netstat -na|grep ESTABLISHED|wc -l

376

netstat -an will print the current network link status of the system, and grep ESTABLISHED will extract the information of the established connection. Then wc -l statistics
The final number returned is the total number of established connections of all 80 ports.

netstat -nat||grep ESTABLISHED|wc

You can view the detailed records of all established connections

19. Output the number of connections per ip and the total number of connections in each state

netstat -n | awk '/^tcp/ {n=split($(NF-1),array,":");if(n<=2)++S[array[(1)]];else++S[array[(4)]];++s[$NF];++N} END {for(a in S){printf("%-20s %s\n", a, S[a]);++I}printf("%-20s %s\n","TOTAL_IP",I);for(a in s) printf("%-20s %s\n",a, s[a]);printf("%-20s %s\n","TOTAL_LINK",N);}'

20. Other collection

Analyze the top 20 URL s of the page under the log file on May 4, 2012 and sort them

cat access.log |grep '04/May/2012'| awk '{print $11}'|sort|uniq -c|sort -nr|head -20

Query the URL address of the visited page, which contains www.abc The IP address of the com web site

cat access_log | awk '($11~/\www.abc.com/){print $1}'|sort|uniq -c|sort -nr

Get the 10 IP addresses with the highest access, and also query by time

cat linewow-access.log|awk '{print $1}'|sort|uniq -c|sort -nr|head -10

Time period: query the log time period

cat log_file | egrep '15/Aug/2015|16/Aug/2015' |awk '{print $1}'|sort|uniq -c|sort -nr|head -10

Analyze visits from August 15, 2015 to August 16, 2015 "/ index.php? G = member & M = Public & A = sendvalidcode "

cat log_file | egrep '15/Aug/2015|16/Aug/2015' | awk '{if($7 == "/index.php?g=Member&m=Public&a=sendValidCode") print $1,$7}'|sort|uniq -c|sort -nr

($7~/.php/) contains the output of. PHP. This sentence means the most time-consuming 100 PHP pages

cat log_file |awk '($7~/\.php/){print $NF " " $1 " " $4 " " $7}'|sort -nr|head -100

List the most time-consuming pages (more than 60 seconds) and the corresponding page occurrence times

cat access.log |awk '($NF > 60 && $7~/\.php/){print $7}'|sort -n|uniq -c|sort -nr|head -100

Statistics website traffic

cat access.log |awk '{sum+=$10} END {print sum/1024/1024/1024}'

Statistics 404 connections

awk '($9 ~/404/)' access.log | awk '{print $9,$7}' | sort

Statistics: http status

cat access.log |awk '{counts[$(9)]+=1}; END {for(code in counts) print code, counts[code]}'
cat access.log |awk '{print $9}'|sort|uniq -c|sort -rn

Concurrent per second

watch "awk '{if($9~/200|30|404/)COUNT[$4]++}END{for( a in COUNT) print a,COUNT[a]}' log_file|sort -k 2 -nr|head -n10"

Bandwidth statistics

cat apache.log |awk '{if($7~/GET/) count++}END{print "client_request="count}'

Find the 10 IP S that have the most access times on a day

cat /tmp/access.log | grep "20/Mar/2011" |awk '{print $3}'|sort |uniq -c|sort -nr|head

What are the IPS with the highest number of ip connections doing that day

cat access.log | grep "10.0.21.17" | awk '{print $8}' | sort | uniq -c | sort -nr | head -n 10

The 10 periods with the highest number of ip connections in an hour unit

awk -vFS="[:]" '{gsub("-.*","",$1);num[$2" "$1]++}END{for(i in num)print i,num[i]}' log_file | sort -n -k 3 -r | head -10

Find the most visited minutes

awk '{print $1}' access.log | grep "20/Mar/2011" |cut -c 14-18|sort|uniq -c|sort -nr|head

Take 5 minutes log

if [ $DATE_MINUTE != $DATE_END_MINUTE ] ;then #
It is determined whether the start timestamp and the end timestamp are equal

START_LINE=sed -n "/$DATE_MINUTE/=" $APACHE_LOG|head -n1 # if not, the line number of the start timestamp and the line number of the end timestamp are taken out

View the link status of tcp

netstat -nat |awk '{print $6}'|sort|uniq -c|sort -rn 

netstat -n | awk '/^tcp/ {++S[$NF]};END {for(a in S) print a, S[a]}' 

netstat -n | awk '/^tcp/ {++state[$NF]}; END {for(key in state) print key,"\t",state[key]}' 

netstat -n | awk '/^tcp/ {++arr[$NF]};END {for(k in arr) print k,"\t",arr[k]}' 

netstat -n |awk '/^tcp/ {print $NF}'|sort|uniq -c|sort -rn 

netstat -ant | awk '{print $NF}' | grep -v '[a-z]' | sort | uniq -c
netstat -ant|awk '/ip:80/{split($5,ip,":");++S[ip[1]]}END{for (a in S) print S[a],a}' |sort -n 

netstat -ant|awk '/:80/{split($5,ip,":");++S[ip[1]]}END{for (a in S) print S[a],a}' |sort -rn|head -n 10 

awk 'BEGIN{printf ("http_code\tcount_num\n")}{COUNT[$10]++}END{for (a in COUNT) printf a"\t\t"COUNT[a]"\n"}'

Find the top 20 IP addresses of the number of requests (commonly used to find the attack source):

netstat -anlp|grep 80|grep tcp|awk '{print $5}'|awk -F: '{print $1}'|sort|uniq -c|sort -nr|head -n20 
netstat -ant |awk '/:80/{split($5,ip,":");++A[ip[1]]}END{for(i in A) print A[i],i}' |sort -rn|head -n20

Sniff the access of port 80 with tcpdump to see who is the highest

tcpdump -i eth0 -tnn dst port 80 -c 1000 | awk -F"." '{print $1"."$2"."$3"."$4}' | sort | uniq -c | sort -nr |head -20

Find more time_wait connection

netstat -n|grep TIME_WAIT|awk '{print $5}'|sort|uniq -c|sort -rn|head -n20

Find more SYN connections

netstat -an | grep SYN | awk '{print $5}' | awk -F: '{print $1}' | sort | uniq -c | sort -nr | more

Process by port column

netstat -ntlp | grep 80 | awk '{print $7}' | cut -d/ -f1

Checked the number of connections and the current number of connections

netstat -ant | grep $ip:80 | wc -l 
netstat -ant | grep $ip:80 | grep EST | wc -l

View IP access times

netstat -nat|grep ":80"|awk '{print $5}' |awk -F: '{print $1}' | sort| uniq -c|sort -n

The Linux command analyzes the current link status

netstat -n | awk '/^tcp/ {++S[$NF]} END {for(a in S) print a, S[a]}'

watch "netstat -n | awk '/^tcp/ {++S[$NF]} END {for(a in S) print a, S[a]}'" # you can always monitor through watch

LAST_ACK 5 # closing a TCP connection needs to be closed in two directions. Both parties send fin to indicate the closing of unidirectional data. When the communication parties send the last fin, the sender is at last_ In the Ack state, when the sender receives the acknowledgement of the other party (the Ack acknowledgement of fin), the whole TCP connection is really closed;

SYN_RECV 30 # indicates the number of requests waiting to be processed;

ESTABLISHED 1597 # indicates the normal data transmission state;

FIN_WAIT1 51 # indicates that the server side actively requests to close the tcp connection;

FIN_WAIT2 504 # indicates that the client is disconnected;

TIME_WAIT 1057 # indicates the number of requests that have been processed and wait for the timeout to end;