Table of contents
Chapter 3 Getting Started with Shell Scripting
2. The first Shell script: helloworld
3. Second Shell Script: Multi-Command Processing
Chapter 4 Variables in the Shell
Chapter 6 Conditional Judgment
(1) Comparison between two integers
(2) Judge according to file permissions
Chapter 7 Process Control (Key Points)
2. Case Practice: Adding from 1 to 100
Chapter 8 read reads console input
Chapter 10 Shell Tools (Key Points)
2. Option parameter description
2. Option parameter description
3. Command function description
2. Option parameter description [There are many parameters, only two are listed]
Chapter 11 Real Enterprise Interview Questions (Key Points)
Chapter 1 Shell Overview
Why should big data programmers learn Shell?
1) It is necessary to understand the Shell program written by the operation and maintenance personnel.
2) Occasionally write some simple Shell programs to manage the cluster and improve development efficiency.
Chapter 2 The Shell Parser
(1) The Shell parsers provided by Linux are:
[root@hadoop102 ~]# cat /etc/shells /bin/sh /bin/bash /usr/bin/sh /usr/bin/bash /bin/tcsh /bin/csh
(2) The relationship between bash and sh [soft link]
[root@hadoop102 bin]# ll |grep bash -rwxr-xr-x. 1 root root 964536 4 March 1 2020 bash lrwxrwxrwx. 1 root root 10 3 April 4 2022 bashbug -> bashbug-64 -rwxr-xr-x. 1 root root 6964 4 March 1 2020 bashbug-64 lrwxrwxrwx. 1 root root 4 3 April 4 2022 sh -> bash
(3) The default parser of Centos is bash
[root@hadoop102 bin]# echo $SHELL /bin/bash
Chapter 3 Getting Started with Shell Scripting
1. script format
The script starts with #!/bin/bash (specify the parser) [# is not a comment, it is an executed command]
2. The first Shell script: helloworld
(1) Requirements: Create a Shell script to output helloworld
(2) Case practice:
[root@hadoop102 ~]# mkdir datas [root@hadoop102 ~]# cd datas/ [root@hadoop102 datas]# pwd /root/datas [root@hadoop102 datas]# touch helloworld.sh [root@hadoop102 datas]# vim helloworld.sh
Enter the following in helloworld.sh
#!/bin/bash echo "helloworld dahaige"
(3) Common execution methods of scripts
The first one: use the relative path or absolute path of the bash or sh+ script (no need to give the script +x permission)
relative paths to sh+ scripts
[root@hadoop102 datas]# sh helloworld.sh --------------------- relative path of sh+ script helloworld dahaige [root@hadoop102 datas]# sh /root/datas/helloworld.sh -----------------sh+ script absolute path helloworld dahaige [root@hadoop102 datas]# bash helloworld.sh ----------------------- relative path of bash+ script helloworld dahaige [root@hadoop102 datas]# bash /root/datas/helloworld.sh ------------- absolute path of bash+ script helloworld dahaige
The second type: Execute the script using the absolute path or relative path of the input script (must have executable permission +x)
[root@hadoop102 datas]# ./helloworld.sh -bash: ./helloworld.sh: Insufficient authority [root@hadoop102 datas]# ll -rw-r--r--. 1 root root 39 12 month 4 16:23 helloworld.sh [root@hadoop102 datas]# chmod +x helloworld.sh ------ (a) First of all, you must give +x permission to the helloworld.sh script [root@hadoop102 datas]# ll -rwxr-xr-x. 1 root root 39 12 month 4 16:23 helloworld.sh [root@hadoop102 datas]# ./helloworld.sh --------------- (b) execute script - relative path helloworld dahaige [root@hadoop102 datas]# /root/datas/helloworld.sh ----(b) execute script - absolute path helloworld dahaige
Notice:
-
- The first execution method is essentially that the bash parser helps you execute the script, so the script itself does not need execution permission.
- The second execution method is essentially that the script needs to be executed by itself, so execution permission is required.
3. Second Shell Script: Multi-Command Processing
(1) Requirements:
Create a banzhang.txt in the /root/ directory, and add "I love cls" to the banzhang.txt file.
(2) Case practice:
[root@hadoop102 datas]# touch batch.sh [root@hadoop102 datas]# vim batch.sh #!/bin/bash cd /root/ touch banzhang.txt echo "I Love cls">>banzhang.txt [root@hadoop102 datas]# ll -rw-r--r--. 1 root root 74 12 month 4 16:42 batch.sh -rwxr-xr-x. 1 root root 39 12 month 4 16:23 helloworld.sh [root@hadoop102 datas]# bash batch.sh batch.sh: line 3:cd: /root/temp/: No such file or directory [root@hadoop102 datas]# mkdir /root/temp/ [root@hadoop102 datas]# bash batch.sh [root@hadoop102 datas]# ls /root/temp/ banzhang.txt [root@hadoop102 datas]# cat /root/temp/banzhang.txt I Love cls
Chapter 4 Variables in the Shell
4.1 System variables
1. Common system variables
$HOME, $PWD, $SHELL, $USER, etc.
2. Case Practice
(1) View the value of the system variable
[root@hadoop102 datas]# echo $HOME /root -------------current user home directory [root@hadoop102 datas]# echo $PWD /root/datas [root@hadoop102 datas]# echo $SHELL /bin/bash [root@hadoop102 datas]# echo $USER root
(2) Display all variables in the current Shell: set
4.2 Custom variables
1. basic grammar
(1) Define variables: variable = value
(2) Undo variable: unset variable
(3) Declare static variables: readonly variables, note: cannot unset
2. Variable Definition Rules
(1) The variable name can be composed of letters, numbers and underscores, but cannot start with a number. The environment variable name is recommended to be capitalized.
(2) There can be no spaces on both sides of the equal sign
(3) In bash, the default type of variables is a string type, which cannot be directly used for numerical operations.
(4) If the value of the variable has spaces, it needs to be enclosed in double quotes or single quotes.
3. Case Practice
(1) Define variable A
[root@hadoop102 datas]# A=1 --------Define variable A [root@hadoop102 datas]# echo $A 1
(2) Re-assign variable A
[root@hadoop102 datas]# A = 2 ------------ There can be no spaces on both sides of "=" bash: A: command not found... [root@hadoop102 datas]# A= 2 bash: 2: command not found... [root@hadoop102 datas]# A=2 -------------Reassign variable A [root@hadoop102 datas]# echo $A 2
(3) Undo variable A
[root@hadoop102 datas]# unset A [root@hadoop102 datas]# echo $A [root@hadoop102 datas]#
(4) Declare static variable B=2, cannot unset
[root@hadoop102 datas]# readonly B=2 [root@hadoop102 datas]# echo $B 2 [root@hadoop102 datas]# readonly B=3 ------------ read only can not be changed -bash: B: read-only variable [root@hadoop102 datas]# unset B ------------ read-only cannot be canceled until the virtual machine is restarted -bash: unset: B: Unable to unset: read only variable
(5) In bash, the default types of variables are all string types, and numerical operations cannot be performed directly
[root@hadoop102 datas]# C=1+1 ---------The default is string type [root@hadoop102 datas]# echo $C 1+1
(6) If the value of the variable has spaces, it needs to be enclosed in double quotes or single quotes
[root@hadoop102 datas]# D=banzhang love mm bash: love: command not found... [root@hadoop102 datas]# D="banzhang love mm" ---------- Long spaces need to be enclosed in double quotation marks [root@hadoop102 datas]# echo $D banzhang love mm
(7) Variables can be promoted to global environment variables, which can be used by other Shell programs
export variable name
[root@hadoop102 datas]# vim helloworld.sh
#!/bin/bash
echo "helloworld dahaige"
echo $D -------------newly added code
[root@hadoop102 datas]# echo $D
banzhang love mm
[root@hadoop102 datas]# ./helloworld.sh
helloworld dahaige
----------Found and did not print out the variable $D value, the actual missing output is empty, indicating that the variable is not globally effective
[root@hadoop102 datas]# export D ----------- promoted to a global environment variable
[root@hadoop102 datas]# ./helloworld.sh
helloworld dahaige
banzhang love mm - ----Global variables are already in effect
4.3 Special variables: $n
1. basic grammar
$n (function description: n is a number, $0 represents the name of the script, $1-$9 represents the first to ninth parameters, and more than ten parameters need to be enclosed in braces, such as )
2. Case Practice
(1) Output the script file name, the values of input parameter 1 and input parameter 2
[root@hadoop102 datas]# touch parameter.sh [root@hadoop102 datas]# vim parameter.sh #!/bin/bash echo "$0 $1 $2 $3" [root@hadoop102 datas]# bash parameter.sh ---$0 is the script name parameter.sh [root@hadoop102 datas]# bash parameter.sh banzhang --- $1 is the first parameter parameter.sh banzhang [root@hadoop102 datas]# bash parameter.sh banzhang love --- $ display three parameters parameter.sh banzhang love [root@hadoop102 datas]# bash parameter.sh banzhang love mm --- display four parameters parameter.sh banzhang love mm [root@hadoop102 datas]# bash parameter.sh banzhang love mm 1 --- more parameters are not displayed parameter.sh banzhang love mm
4.4 Special variables: $#
1. basic grammar
$# (Function description: Get the number of all input parameters, often used in loops).
2. Case Practice
(1) Get the number of input parameters
[root@hadoop102 datas]# vim parameter.sh [root@hadoop102 datas]# cat parameter.sh #!/bin/bash echo "$0 $1 $2 $3" echo $# ---------------------------- Newly added code [root@hadoop102 datas]# bash parameter.sh parameter.sh 0 [root@hadoop102 datas]# bash parameter.sh banzhang parameter.sh banzhang 1 [root@hadoop102 datas]# bash parameter.sh banzhang love mm parameter.sh banzhang love mm 3 [root@hadoop102 datas]# bash parameter.sh banzhang love mm 111 222 parameter.sh banzhang love mm 5
4.5 Special variables: $*, $@
1. basic grammar
$* (Function description: this variable represents all parameters in the command line, $* treats all parameters as a whole)
$@ (Function description: this variable also represents all parameters in the command line, $@ treats each parameter differently)
2. Case Practice
(1) Print all parameters entered
[root@hadoop102 datas]# vim parameter.sh #!/bin/bash echo "$0 $1 $2 $3" echo $# echo $* echo $@ [root@hadoop102 datas]# bash parameter.sh banzhang love mm parameter.sh banzhang love mm 3 banzhang love mm --------$*Represents all the parameters in the command line, $*Treat all parameters as a whole banzhang love mm --------$@Represents all the parameters in the command line, $@Treat each parameter separately
4.6 Special variables: $?
1. basic grammar
$? (Function description: the return status of the last executed command. If the value of this variable is 0, it proves that the previous command was executed correctly; if the value of this variable is not 0 (the specific number is determined by the command itself), then It proves that the previous command was executed incorrectly.)
2. Case Practice
(1) Determine whether the helloworld.sh script is executed correctly
[root@hadoop102 datas]# ./helloworld.sh helloworld dahaige banzhang love mm [root@hadoop102 datas]# $? --------command error bash: 0: command not found... [root@hadoop102 datas]# echo $? -----The previous command was not executed correctly, so it is not 0 127 [root@hadoop102 datas]# ./helloworld.sh helloworld dahaige banzhang love mm [root@hadoop102 datas]# echo $? ---------- The last command is executed correctly, and returns 0 0
Chapter 5 Operators
1. basic grammar
(1) "$((operation expression))" or "$[operation expression]"
(2) expr + , - , \*, /, % Add, subtract, multiply, divide, take remainder
Note: There must be spaces between expr operators
2. Case practice:
(1)Calculate 3+2 the value of [root@hadoop102 ~]# expr 3+2 3+2 [root@hadoop102 ~]# expr 3 + 2 5 [root@hadoop102 ~]# expr 3 +2 expr: Grammatical errors (2)Calculate 3-2 the value of [root@hadoop102 ~]# expr 3 - 2 1 (3)Calculate (2+3)X4 the value of (a)expr Calculate in one step [root@hadoop102 ~]# expr ( 3 + 2 ) \* 4 -bash: unexpected symbol `3' syntax error near [root@hadoop102 ~]# expr (3 + 2) \* 4 -bash: unexpected symbol `3' syntax error near [root@hadoop102 ~]# expr `expr 3 + 2 `\* 4 expr: Grammatical errors [root@hadoop102 ~]# expr `expr 3 + 2` \* Must be strictly implemented, even if a space is not in the right place. (emphasis `, the same key as ~) 20 (b)use $[Expression]Way [root@hadoop102 ~]# s=$[(2+3)*4] [root@hadoop102 ~]# echo $s 20
Chapter 6 Conditional Judgment
1. basic grammar
[ condition ] (note that there must be spaces before and after condition)
Note: The condition is true if it is not empty, [ atguigu ] returns true, and [] returns false.
2. Common Judgment Conditions
(1) Comparison between two integers
= string comparison
-lt less than (less than) -le less than or equal to (less equal)
-eq equal to (equal) -gt greater than (greater than)
-ge greater than or equal to (greater equal) -ne not equal to (Not equal)
(2) Judge according to file permissions
-r has read permission (read) -w has write permission (write)
-x has permission to execute (execute)
(3) Judging by file type
-f file exists and is a regular file (file)
-e file exists (existence) -d file exists and is a directory (directory)
3. Case Practice
(1) Whether 23 is greater than or equal to 22
(1)23 Is it greater than or equal to 22 [root@hadoop102 ~]# [23 -ge 22] bash: [23: command not found... [root@hadoop102 ~]# [ 23 -ge 22 ] [root@hadoop102 ~]# echo $? -------$? Judge whether the last command is executed correctly, the correct value is 0, and the incorrect value is other 0 [root@hadoop102 ~]# [ 23 -le 22 ] [root@hadoop102 ~]# echo $? 1
(2) Whether helloworld.sh has write permission
[root@hadoop102 datas]# [ -w helloworld.sh ] [root@hadoop102 datas]# echo $? 0 [root@hadoop102 datas]# ll -rw-r--r--. 1 root root 11 12 month 4 16:44 banzhang.txt -rwxr-xr-x. 1 root root 48 12 month 4 17:19 helloworld.sh [root@hadoop102 datas]# [ -x banzhang.txt ] [root@hadoop102 datas]# echo $? 1
(3) Whether the file in the /home/atguigu/cls.txt directory exists
[root@hadoop102 datas]# [ -e /root/datas/cls.txt ] [root@hadoop102 datas]# echo $? 1 [root@hadoop102 datas]# [ -e /root/datas/banzhang.txt ] [root@hadoop102 datas]# echo $? 0
(4) Multi-condition judgment (&& indicates that the next command will be executed only when the previous command is successfully executed, and || indicates that the next command will be executed only after the previous command fails to execute)
[root@hadoop102 datas]# [ condition ] && echo OK ||echo notok OK [root@hadoop102 datas]# [ condition ] && [ ] ||echo notok notok
Chapter 7 Process Control (Key Points)
7.1 if judgment
1. basic grammar
if [ conditional judgment ];then [Brackets[]There must be spaces on the left and right]
program
fi
or
if [ conditional judgment ]
then
program
fiPrecautions:
(1) [Conditional judgment formula], there must be a space between the square brackets and the conditional judgment formula
(2) There must be a space after if
2. Case Practice
(1) Input a number, if it is 1, then output banzhang zhen shuai, if it is 2, then output cls zhen mei, if it is other, output nothing.
[root@hadoop102 datas]# vim if.sh
#!/bin/bash
if [ $1 -eq 1 ];then
echo "banzhang zhen shuai"
elif [ $1 -eq 2 ]
then
echo "cls zhen mei"
fi
[root@hadoop102 datas]# bash if.sh
if.sh: line 3:[: -eq: expecting a unary expression
if.sh: line 5:[: -eq: expecting a unary expression
[root@hadoop102 datas]# bash if.sh 1
banzhang zhen shuai
[root@hadoop102 datas]# bash if.sh 2
cls zhen mei
[root@hadoop102 datas]# bash if.sh 3
[root@hadoop102 datas]# 
7.2 case statement
1. basic grammar
case $variable name in
"value 1")
If the value of the variable is equal to the value 1, execute program 1
;;
"value 2")
If the value of the variable is equal to value 2, execute program 2
;;
...Omit other branches...
*)
If none of the variables have values above, execute this program
;;
esacPrecautions:
1) The end of the case line must be the word "in", and each pattern match must end with a right bracket ")".
2) The double semicolon ";;" indicates the end of the command sequence, which is equivalent to break in java.
3) The last "*)" indicates the default mode, which is equivalent to default in java.
2. Case Practice
(1) Input a number, if it is 1, output banzhang, if it is 2, output cls, if it is other, output renyao.
[root@hadoop102 datas]# vim case.sh
#!/bin/bash
case $1 in
1)
echo "banzhang"
;;
2)
echo "cls"
;;
*)
echo "renyao"
;;
esac
[root@hadoop102 datas]# bash case.sh 1
banzhang
[root@hadoop102 datas]# bash case.sh 2
cls
[root@hadoop102 datas]# bash case.sh 3
renyao
7.3 for loop
1. Basic Grammar 1
for (( initial value;loop control condition;variable change ))
do
program
done2. Case Practice: Adding from 1 to 100
[root@hadoop102 datas]# vim for.sh
#!/bin/bash
s=0
for((i=1;i<100;i++))
do
s=$[$s+$i]
done
echo $s
[root@hadoop102 datas]# bash for.sh
5050
3. Basic Grammar 2
for variable in Value 1 Value 2 Value 3...
do
program
done4. Case Practice
(1) Print all input parameters
[root@hadoop102 datas]# vim for2.sh
#!/bin/bash
for i in $*
do
echo "banzhang xihuan $i"
done
[root@hadoop102 datas]# bash for2.sh mm
banzhang xihuan mm
[root@hadoop102 datas]# bash for2.sh mm cls
banzhang xihuan mm
banzhang xihuan cls
[root@hadoop102 datas]# bash for2.sh mm cls xiaoze
banzhang xihuan mm
banzhang xihuan cls
banzhang xihuan xiaoze(2) Compare the difference between $* and $@
(a) Both $* and $@ represent all parameters passed to the function or script, and when they are not included by double quotes "", all parameters are output in the form of $1 $2 ... $n.
[root@hadoop102 datas]# vim for2.sh
#!/bin/bash
for i in $* --------------$*It is output separately without double quotes
do
echo "banzhang xihuan $i"
done
for j in $@ --------------separate output
do
echo "banzhang xihuan $j"
done
[root@hadoop102 datas]# bash for2.sh mm cls xiaoze
banzhang xihuan mm
banzhang xihuan cls
banzhang xihuan xiaoze
banzhang xihuan mm
banzhang xihuan cls
banzhang xihuan xiaoze(b) When they are enclosed by double quotes "", "$*" will take all parameters as a whole and output all parameters in the form of "$1 $2 ...$n"; "$@" will separate each parameter, Output all arguments in the form "$1" "$2"..."$n".
[root@hadoop102 datas]# vim for2.sh
#!/bin/bash
#All parameters in $* are regarded as a whole, so this for loop will only loop once
for i in "$*" --------------after double quotes $*as a whole
do
echo "banzhang xihuan $i"
done
#Each parameter in $@ is regarded as independent, so if there are several parameters in "$@", it will cycle several times
for j in "$@" --------------separate output
do
echo "banzhang xihuan $j"
done
[root@hadoop102 datas]# bash for2.sh mm cls xiaoze
banzhang xihuan mm cls xiaoze --------------------one cycle
banzhang xihuan mm --------------------multiple loops
banzhang xihuan cls
banzhang xihuan xiaoze
7.4 while loop
1. basic grammar
while [ conditional judgment ] -----------Notice [ ] spaces, both sides and while []There must be spaces in between
do
program
done2. Case Practice
[Conditional Judgment] does not support the >= operator, where for is because of the (()) form
(1) Add from 1 to 100
[root@hadoop102 datas]# vim while.sh
#!/bin/bash
s=0
i=1
while [ $i -le 100 ]
do
s=$[$s + $i]
i=$[$i + 1]
done
echo $s
[root@hadoop102 datas]# bash while.sh
5050
Chapter 8 read reads console input
1. basic grammar
read(options)(parameter)
options:
-p: Specify the prompt when reading the value; -------------console input value
-t: Specify the time to wait (in seconds) when reading a value
parameter
Variable: Specify the variable name to read the value from2. Case Practice
(1) Within 7 seconds of the prompt, read the name entered by the console
[root@hadoop102 datas]# vim read.sh #!/bin/bash read -t 7 -p "input your name " NAME echo $NAME [root@hadoop102 datas]# sh read.sh input your name ----------------------------Automatically end after waiting 7 seconds [root@hadoop102 datas]# sh read.sh input your name banzhang banzhang -------------------print the entered characters
Chapter 9 Functions
9.1 System functions
1. basename basic syntax
basename [string / pathname] [suffix]
Function description: basename command will remove all prefixes including the last one ('/')characters, and then display the string.
#is to intercept the file name
options:
suffix as a suffix, if suffix was designated, basename will pathname or string middle suffix remove.2. Case Practice
(1) Intercept the file name of the /root/banzhang.txt path
[root@hadoop102 datas]# basename /root/datas/banzhang.txt banzhang.txt [root@hadoop102 datas]# basename /root/datas/banzhang.txt .txt banzhang
3. Basic syntax of dirname
dirname+file absolute path Function description: Remove the file name (non-directory part) from the given file name containing the absolute path, and then return the remaining path (directory part))
4. Case Practice
(1) Obtain the path of the banzhang.txt file
[root@hadoop102 datas]# dirname /root/datas/banzhang.txt /root/datas
9.2 Custom functions
1. basic grammar
[ function ] funname[()]
{
Action;
[return int;]
}
funname2. Skills
(1) The function must be declared before calling the function, and the shell script is run line by line. Does not compile first like other languages.
(2) The return value of the function can only be obtained through the $? system variable, which can be displayed and added: return return, if not added, the result of the last command will be used as the return value. return followed by the value n(0-255)
3. Case Practice
(1) Calculate the sum of the two input parameters
[root@hadoop102 datas]# vim sum.sh
#!/bin/bash
function sum() --------------------The function must be defined first, because it is executed from top to bottom, and it cannot be seen in the definition below
{
s=0
s=$[$1+$2]
echo $s
}
read -p "input your paratemer1:" p1
read -p "input your paratemer2:" p2
sum $p1 $p2
[root@hadoop102 datas]# bash sum.sh
input your paratemer1:1
input your paratemer2:2
3
Chapter 10 Shell Tools (Key Points)
10.1 cut
The job of cut is to "cut", specifically, it is responsible for cutting data in the file. [Do not change the original file, cut it and print it to the console]
The cut command cuts bytes, characters, and fields from each line of a file and outputs them.
1. Basic usage
cut [option parameter] filename Description: The default delimiter is tab
2. Option parameter description
| option parameter | Function |
| -f | Column number, which column to extract |
| -d | Delimiter, split columns according to the specified delimiter |
3. Case Practice
(0) Data preparation
[root@hadoop102 datas]# vim cut.txt dong shen guan zhen wo wo lai lai le le
(1) Cut the first column of cut.txt
[root@hadoop102 datas]# cut -d " " -f 1 cut.txt --- Which file should be cut first, and the middle is the command parameter dong guan wo lai le
(2) Cut the second and third columns of cut.txt
[root@hadoop102 datas]# cut -d " " -f 2,3 cut.txt shen zhen wo lai le
(3) Cut out guan in the cut.txt file
[root@hadoop102 datas]# cat cut.txt |grep guan guan zhen [root@hadoop102 datas]# cat cut.txt |grep guan|cut -d " " -f 1 guan
(4) Select the system PATH variable value, all paths after the second ":" start:
[root@hadoop102 datas]# echo $PATH /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/opt/module/jdk1.8.0_211/bin: [root@hadoop102 datas]# echo $PATH |cut -d : -f 3- -----not added - is to take the third column, added - is to take all after the third column /usr/sbin:/usr/bin:/opt/module/jdk1.8.0_211/bin
(5) IP address printed after cutting ifconfig
[root@hadoop102 datas]# ifconfig ens33 |grep "inet 192"
inet 192.168.10.102 netmask 255.255.255.0 broadcast 192.168.10.255
[root@hadoop102 datas]# ifconfig ens33 |grep "inet 192" |cut -d " " -f 10
192.168.10.10210.2 sed
sed is a stream editor that processes content one line at a time. When processing, the currently processed line is stored in a temporary buffer, which is called "pattern space", and then the content in the buffer is processed with the sed command. After the processing is completed, the content of the buffer is sent to the screen. Then process the next line, and repeat until the end of the file. The file contents are not changed unless you use redirected storage output.
1. Basic usage
sed [option parameter] 'command' filename
2. Option parameter description
| option parameter | Function |
| -e | Directly perform sed action editing in command line mode [used for multiple commands, not needed for a single command] |
3. Command function description
| Order | Functional description |
| a | Added, a string can be connected after a, and it will appear on the next line |
| d | delete |
| s | find and replace |
4. Case Practice
(0) Data preparation
[root@hadoop102 datas]# vim sed.txt dong shen guan zhen wo wo lai lai le le
(1) Insert the word "mei nv" into the second line of sed.txt and print it.
[root@hadoop102 datas]# cat sed.txt dong shen guan zhen wo wo lai lai le le [root@hadoop102 datas]# sed "2a mei nv" sed.txt ---2 is the second line, a is the parameter a, which means adding [the original file will not change] dong shen guan zhen mei nv --------------------------------------display directly wo wo lai lai le le
Note: the file has not changed
(2) Delete all lines containing wo in the sed.txt file
[root@hadoop102 datas]# sed "/wo/d" sed.txt ------------- parameter d is behind dong shen guan zhen --------------------------------Include"wo"directly delete a line of lai lai le le
(3) Replace wo with ni in the sed.txt file
[root@hadoop102 datas]# sed "s/wo/ni/g" sed.txt ---- parameter s is replacement, replace "wo" with "ni", parameter g is global replacement dong shen guan zhen ni ni lai lai le le
Note: 'g' means global, replace all
(4) Delete the second line in the sed.txt file and replace wo with ni
[root@hadoop102 datas]# cat sed.txt dong shen guan zhen wo wo lai lai le le [root@hadoop102 datas]# sed -e "2d" -e "s/wo/ni/g" sed.txt ----Multiple commands use parameter -e dong shen ni ni lai lai le le
10.3 awk
A powerful text analysis tool that reads files line by line, slices each line with spaces as the default delimiter, and then analyzes and processes the cut parts.
1. Basic usage
awk [option parameter] 'pattern1{action1} pattern2{action2}...' filename 'match this expression{perform this operation}'
pattern: express AWK What to look for in the data, is the matching pattern
action: A sequence of commands to execute when a match is found2. Option parameter description [There are many parameters, only two are listed]
| option parameter | Function |
| -F | Specify input file fold delimiter |
| -v | Assign a value to a user-defined variable |
3. Case Practice
(0) Data preparation
[root@hadoop102 datas]# cp /etc/passwd ./
(1) Search for all lines starting with the root keyword in the passwd file, and output column 7 of the line.
[root@hadoop102 datas]# awk -F : '/^root/ {print $7}' passwd ------ "-F" input specifies the delimiter
/bin/bash(2) Search for all lines starting with the root keyword in the passwd file, and output the first and seventh columns of the line, separated by "," in the middle.
[root@hadoop102 datas]# awk -F : '/^root/ {print $1","$7}' passwd
root,/bin/bashNote: only the lines that match the pattern will execute the action
(3) Only display the first and seventh columns of /etc/passwd, separated by commas, and add the column name user, shell in front of all lines;
Add "dahaige,/bin/zuishuai" to the last line.
[root@hadoop102 datas]# awk -F :
'BEGIN{print "user,shell"} {print $1","$7} END{print "dahaige,bin/zuishuai"}' passwd
user,shell [as long as it is printed{print xx}]
root,/bin/bash
...
dahaige,bin/zuishuaiNote: BEGIN is executed before all data read rows; END is executed after all data is executed.
(4) Increase the user id in the passwd file by 1 and output
[root@hadoop102 datas]# awk -F : -v i=1 '{print $3+i}' passwd ---"-F" is to cut, "-v" is to define a variable
1
...4. awk's built-in variables
| variable | illustrate |
| FILENAME | file name |
| NR | Number of records read |
| NF | The number of domains in the browse record (after cutting, the number of columns) |
5. Case Practice
(1) Statistics passwd file name, line number of each line, number of columns of each line
[root@hadoop102 datas]# awk -F : '{print FILENAME "," NR "," NF}' passwd -- file name, line number, column
passwd,1,7
passwd,2,7
passwd,3,7
passwd,4,7
passwd,5,7
passwd,6,7(2) Cutting IP
[atguigu@hadoop102 datas]$ ifconfig eth0 | grep "inet addr" |
awk -F: '{print $2}' | awk -F " " '{print $1}'
192.168.1.102(3) Query the line number of the blank line in sed.txt [interview question]
[root@hadoop102 datas]# cat sed.txt
dong shen
guan zhen
wo wo
lai lai
le le
[root@hadoop102 datas]# awk '/^$/ {print NR}' sed.txt ----- empty line, that is /^$/
510.4 sort
The sort command is very useful in Linux, it sorts files and outputs the sorted results to standard output.
1. Basic Grammar
sort( option)(argument)
| options | illustrate |
| -n | Sort by numerical value |
| -r | to sort in reverse order [reverse order] |
| -t | Set the separator used for sorting [Cut] |
| -k | Specify the column to be sorted [by which column] |
Parameters: Specify the list of files to be sorted
2. Case study
(0) Data preparation
[root@hadoop102 datas]# vim sort.sh bb:40:5.4 bd:20:4.2 xz:50:2.3 cls:10:3.5 ss:30:1.6
(1) Sort in reverse order according to the third column separated by ":".
[root@hadoop102 datas]# sort -t : -nrk 2 sort.sh "-n" sorts in positive order, "-r" in reverse order, xz:50:2.3 "-k"specifies the column,"-t"by delimiter bb:40:5.4 ss:30:1.6 bd:20:4.2 cls:10:3.5
Chapter 11 Real Enterprise Interview Questions (Key Points)
11.1 JD.com
Question 1: Use Linux commands to query the line number of the blank line in file1 [such as 404, 500, ... to filter abnormal data - data cleaning]
[root@hadoop102 datas]# awk '/^$/{print NR}' sed.txt
5Question 2: The content of the file chengji.txt is as follows:
Zhang San 40
Li Si 50
Wang Wu 60
Use Linux commands to calculate the sum of the second column and output
[root@hadoop102 datas]# vim chengji.txt
Zhang San 40
Li Si 50
Wang Wu 60
[root@hadoop102 datas]# cat chengji.txt |awk -F " " '{sum+=$2} END{print sum}'
15011.2 Sohu & Hexun
Question 1: How to check whether a file exists in Shell script? What to do if it doesn't exist?
[root@hadoop102 datas]# vim exist.sh
#!/bin/bash
if [ -f file.txt ];then
echo "file exists"
else
echo "file does not exist"
fi
[root@hadoop102 datas]# ./exist.sh
file does not exist
[root@hadoop102 datas]# touch file.txt
[root@hadoop102 datas]# ./exist.sh
file exists
11.3 Sina
Question 1: Write a script in the shell to sort an unordered column of numbers in the text
[root@hadoop102 datas]# vim test.txt
9
8
7
6
5
4
3
2
10
1
[root@hadoop102 datas]# sort -n test.txt |awk '{a+=$0;print $0}END{print "sum="a}'
1
2
3
4
5
6
7
8
9
10
sum=5511.4 Jinhe Network
Question 1: Please use a shell script to write out the file names that contain the character "shen" in the content of all text files under the current folder (/root)
[root@hadoop102 datas]# grep -r "shen" /root/|cut -d : -f 1 /root/datas/cut.txt /root/datas/sed.txt