1. modify the Linux host name
hostnamectl set-hostname dhf1
Or modify the configuration file
vim /etc/sysconfig/network NETWORKING=yes HOSTNAME=dhf1
2. modify IP
vim /etc/sysconfig/network-scripts/ifcfg-eth0 systemctl restart network
3. modify the mapping relationship between host name and IP
vim /etc/hosts 192.xxx.xxx.227 dhf1 192.xxx.xxx.228 dhf2 192.xxx.xxx.229 dhf3 192.xxx.xxx.230 dhf4 192.xxx.xxx.231 dhf5 192.xxx.xxx.232 dhf6 192.xxx.xxx.233 dhf7
4. turn off the firewall
systemctl status firewalld systemctl stop firewalld systemctl disable firewalld
5.ssh login free
ssh-keygen -t rsa
(four carriage returns) after executing this command, two file IDS will be generated_ RSA (private key), id_rsa.pub (public key) copy the public key to the machine (including this machine) to be login free:
ssh-copy-id dhf1
| Machines that need to generate public keys | Machines to copy to |
|---|---|
| dhf1 | dhf1,dhf2,dhf3,dhf4,dhf5,dhf6,dhf7 |
| dhf2 | dhf1,dhf2 |
| dhf3 | dhf3,dhf4,dhf5,dhf6,dhf7 |
6. install JDK, configure environment variables, etc
export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.272.b10-1.el7_9.x86_64 export JRE_HOME=$JAVA_HOME/jre export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:$JRE_HOME/lib source /etc/profile
7. restart the machine
Reboot
8. cluster planning
| host name | IP | Installed software | Running processes |
|---|---|---|---|
| dhf1 | 192.xxx.xxx.227 | jdk,hadoop | NameNode,DFSZKFailoverController(zkfc) |
| dhf2 | 192.xxx.xxx.228 | jdk,hadoop | NameNode,DFSZKFailoverController(zkfc) |
| dhf3 | 192.xxx.xxx.229 | jdk,hadoop | ResourceManager |
| dhf4 | 192.xxx.xxx.230 | jdk,hadoop | ResourceManager |
| dhf5 | 192.xxx.xxx.231 | jdk,hadoop,zookeeper | DataNode,NodeManager,JournalNode,QuorumPeerMain |
| dhf6 | 192.xxx.xxx.232 | jdk,hadoop,zookeeper | DataNode,NodeManager,JournalNode,QuorumPeerMain |
| dhf7 | 192.xxx.xxx.233 | jdk,hadoop,zookeeper | DataNode,NodeManager,JournalNode,QuorumPeerMain |
Note: in hadoop2.0, there are usually two namenodes, one in active state and the other in standby state. The active namenode provides external services, while the Standby NameNode does not provide external services. It only synchronizes the status of the active namenode, so that it can quickly switch when it fails. Hadoop officially provides two HDFS HA solutions, NFS and QJM. Here we use the simple QJM. In this scheme, the primary and standby namenodes synchronize metadata information through a group of journalnodes. As long as a piece of data is successfully written to most journalnodes, it is considered as successful. An odd number of journalnodes are usually configured. A zookeeper cluster is also configured here for ZKFC (DFSZKFailoverController) failover. When the active namenode hangs, it will automatically switch the Standby NameNode to standby state. There are two resourcemanagers, one is active and the other is standby. The status is coordinated by zookeeper. Namenode and ResourceManager are separated because of performance problems. Because they both occupy a large amount of resources, they are separated and started on different machines.
9. install zookeeper
9.1. Installing and configuring the zooekeeper cluster
(operate on dhf5)
cd /cdc/apache-zookeeper-3.5.8-bin/conf/ cp zoo_sample.cfg zoo.cfg
Modify: zoo cfg
vim zoo.cfg dataDir=/cdc/apache-zookeeper-3.5.8-bin/tmp server.1=dhf5:2888:3888 server.2=dhf6:2888:3888 server.3=dhf7:2888:3888
Save exit
Then create a tmp folder
mkdir /cdc/apache-zookeeper-3.5.8-bin/tmp
Create another empty file
touch /cdc/apache-zookeeper-3.5.8-bin/tmp/myid
Last write ID to this file
echo 1 > /cdc/apache-zookeeper-3.5.8-bin/tmp/myid
9.2 copy the configured zookeeper to other nodes
scp -r /cdc/apache-zookeeper-3.5.8-bin/ dhf6:/cdc/ scp -r /cdc/apache-zookeeper-3.5.8-bin/ dhf7:/cdc/
Note: modify the contents of /cdc/apache-zookeeper-3.5.8-bin/tmp/myid corresponding to dhf6 and dhf7
dhf6:
echo 2 > /cdc/apache-zookeeper-3.5.8-bin/tmp/myid
dhf7:
echo 3 > /cdc/apache-zookeeper-3.5.8-bin/tmp/myid
10. install hadoop
10.1 installing and configuring hadoop cluster
(operate on dhf1)
10.1.1 add hadoop to the environment variable
vim /etc/profile export HADOOP_HOME=/cdc/hadoop-3.3.0 export PATH=$PATH:$JAVA_HOME/bin:$JRE_HOME/bin:$HADOOP_HOME/bin export HDFS_NAMENODE_USER=root export HDFS_DATANODE_USER=root export HDFS_SECONDARYNAMENODE_USER=root export YARN_RESOURCEMANAGER_USER=root export YARN_NODEMANAGER_USER=root export HDFS_JOURNALNODE_USER=root export HDFS_ZKFC_USER=root
10.1.2 configuring HDFS
(all Hadoop configuration files are in the $HADOOP_HOME/etc/hadoop directory)
First, obtain Hadoop through the hadoop classpath command_ Classpath, as follows:
/cdc/hadoop-3.3.0/etc/hadoop:/cdc/hadoop-3.3.0/share/hadoop/common/lib/*:/cdc/hadoop-3.3.0/share/hadoop/common/*:/cdc/hadoop-3.3.0/share/hadoop/hdfs:/cdc/hadoop-3.3.0/share/hadoop/hdfs/lib/*:/cdc/hadoop-3.3.0/share/hadoop/hdfs/*:/cdc/hadoop-3.3.0/share/hadoop/mapreduce/*:/cdc/hadoop-3.3.0/share/hadoop/yarn:/cdc/hadoop-3.3.0/share/hadoop/yarn/lib/*:/cdc/hadoop-3.3.0/share/hadoop/yarn/*
10.1.2.1 modify Hadoop env sh
export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.272.b10-1.el7_9.x86_64 export HADOOP_CLASSPATH=/cdc/hadoop-3.3.0/etc/hadoop:/cdc/hadoop-3.3.0/share/hadoop/common/lib/*:/cdc/hadoop-3.3.0/share/hadoop/common/*:/cdc/hadoop-3.3.0/share/hadoop/hdfs:/cdc/hadoop-3.3.0/share/hadoop/hdfs/lib/*:/cdc/hadoop-3.3.0/share/hadoop/hdfs/*:/cdc/hadoop-3.3.0/share/hadoop/mapreduce/*:/cdc/hadoop-3.3.0/share/hadoop/yarn:/cdc/hadoop-3.3.0/share/hadoop/yarn/lib/*:/cdc/hadoop-3.3.0/share/hadoop/yarn/*
10.1.2.2 modify the core site xml
<configuration> <!-- appoint hdfs of nameservice by ns1 --> <property> <name>fs.defaultFS</name> <value>hdfs://ns1</value> </property> <!-- appoint hadoop Temporary directory --> <property> <name>hadoop.tmp.dir</name> <value>/cdc/hadoop-3.3.0/tmp</value> </property> <!-- appoint zookeeper address --> <property> <name>ha.zookeeper.quorum</name> <value>dhf5:2181,dhf6:2181,dhf7:2181</value> </property> <property> <name>hadoop.proxyuser.root.hosts</name> <value>*</value> </property> <property> <name>hadoop.proxyuser.root.groups</name> <value>*</value> </property> </configuration>
10.1.2.3 modify HDFS site xml
<configuration> <!--appoint hdfs of nameservice by ns1，Need and core-site.xml Consistent in --> <property> <name>dfs.nameservices</name> <value>ns1</value> </property> <!-- ns1 There are two below NameNode，namely nn1，nn2 --> <property> <name>dfs.ha.namenodes.ns1</name> <value>nn1,nn2</value> </property> <!-- nn1 of RPC mailing address --> <property> <name>dfs.namenode.rpc-address.ns1.nn1</name> <value>dhf1:9000</value> </property> <!-- nn1 of http mailing address --> <property> <name>dfs.namenode.http-address.ns1.nn1</name> <value>dhf1:50070</value> </property> <!-- nn2 of RPC mailing address --> <property> <name>dfs.namenode.rpc-address.ns1.nn2</name> <value>dhf2:9000</value> </property> <!-- nn2 of http mailing address --> <property> <name>dfs.namenode.http-address.ns1.nn2</name> <value>dhf2:50070</value> </property> <!-- appoint NameNode Metadata for JournalNode Storage location on --> <property> <name>dfs.namenode.shared.edits.dir</name> <value>qjournal://dhf5:8485;dhf6:8485;dhf7:8485/ns1</value> </property> <!-- appoint JournalNode Location of data on local disk --> <property> <name>dfs.journalnode.edits.dir</name> <value>/cdc/hadoop-3.3.0/journal</value> </property> <!-- open NameNode Fail auto switch --> <property> <name>dfs.ha.automatic-failover.enabled</name> <value>true</value> </property> <!-- Configuration failure automatic switching implementation mode --> <property> <name>dfs.client.failover.proxy.provider.ns1</name> <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value> </property> <!-- Configure the isolation mechanism method. Multiple mechanisms are separated by line feed, that is, each mechanism temporarily uses one line--> <property> <name>dfs.ha.fencing.methods</name> <value> • sshfence • shell(/bin/true) </value> </property> <!-- use sshfence Required for isolation mechanism ssh No login --> <property> <name>dfs.ha.fencing.ssh.private-key-files</name> <value>/root/.ssh/id_rsa</value> </property> <!-- to configure sshfence Isolation mechanism timeout --> <property> <name>dfs.ha.fencing.ssh.connect-timeout</name> <value>30000</value> </property> </configuration>
10.1.2.4 modify mapred site xml
<configuration> <!-- appoint mr Frame is yarn mode --> <property> <name>mapreduce.framework.name</name> <value>yarn</value> </property> <property> <name>yarn.app.mapreduce.am.env</name> <value>HADOOP_MAPRED_HOME=$HADOOP_HOME</value> </property> <property> <name>mapreduce.map.env</name> <value>HADOOP_MAPRED_HOME=$HADOOP_HOME</value> </property> <property> <name>mapreduce.reduce.env</name> <value>HADOOP_MAPRED_HOME=$HADOOP_HOME</value> </property> </configuration>
10.1.2.5 modify yarn site xml
<configuration> <!-- open RM High reliability --> <property> <name>yarn.resourcemanager.ha.enabled</name> <value>true</value> </property> <!-- appoint RM of cluster id --> <property> <name>yarn.resourcemanager.cluster-id</name> <value>yrc</value> </property> <!-- appoint RM Name of --> <property> <name>yarn.resourcemanager.ha.rm-ids</name> <value>rm1,rm2</value> </property> <!-- Specify separately RM Address of --> <property> <name>yarn.resourcemanager.hostname.rm1</name> <value>dhf3</value> </property> <property> <name>yarn.resourcemanager.hostname.rm2</name> <value>dhf4</value> </property> <property> <name>yarn.resourcemanager.webapp.address.rm1</name> <value>dhf3:8088</value> </property> <property> <name>yarn.resourcemanager.webapp.address.rm2</name> <value>dhf4:8088</value> </property> <!-- appoint zk Cluster address --> <property> <name>yarn.resourcemanager.zk-address</name> <value>dhf5:2181,dhf6:2181,dhf7:2181</value> </property> <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property> <property> <name>yarn.application.classpath</name> <value>/cdc/hadoop-3.3.0/etc/hadoop:/cdc/hadoop-3.3.0/share/hadoop/common/lib/*:/cdc/hadoop-3.3.0/share/hadoop/common/*:/cdc/hadoop-3.3.0/share/hadoop/hdfs:/cdc/hadoop-3.3.0/share/hadoop/hdfs/lib/*:/cdc/hadoop-3.3.0/share/hadoop/hdfs/*:/cdc/hadoop-3.3.0/share/hadoop/mapreduce/*:/cdc/hadoop-3.3.0/share/hadoop/yarn:/cdc/hadoop-3.3.0/share/hadoop/yarn/lib/*:/cdc/hadoop-3.3.0/share/hadoop/yarn/*</value> </property> XML </configuration>
10.1.2.6 modify workers (workers)
(workers refers to the location of the child node. Since HDFS is to be started on dhf1 and yarn is to be started on dhf3, the workers file on dhf1 specifies the location of datanode, and the workers file on dhf3 specifies the location of nodemanager.)
vim workers dhf5 dhf6 dhf7
10.2 copy the configured hadoop to other nodes
scp -r /cdc/hadoop-3.3.0/ root@dhf2:/cdc/ scp -r /cdc/hadoop-3.3.0/ root@dhf3:/cdc/ scp -r /cdc/hadoop-3.3.0/ root@dhf4:/cdc/ scp -r /cdc/hadoop-3.3.0/ root@dhf5:/cdc/ scp -r /cdc/hadoop-3.3.0/ root@dhf6:/cdc/ scp -r /cdc/hadoop-3.3.0/ root@dhf7:/cdc/
11. start service
11.1 start zookeeper cluster
(start zk on dhf5, dhf6 and dhf7 respectively) (QuorumPeerMain)
cd /cdc/apache-zookeeper-3.5.8-bin/bin/ ./zkServer.sh start
Viewing status: one leader and two follower s
./zkServer.sh status
11.2 start journalnode
cd /cdc/hadoop-3.3.0/;rm -rf journal/ns1/;rm -rf logs/; rm -rf tmp/;
(execute on dhf5, dhf6 and tcast07 respectively)
cd /cdc/hadoop-3.3.0/sbin/ ./hadoop-daemon.sh start journalnode
Run the jps command to verify that there are more JournalNode processes on dhf5, dhf6, and dhf7
11.3 format HDFS
(execute command on dhf1)
hdfs namenode -format
After formatting, the core site Hadoop tmp. Dir configuration generates a file. Here I configure /cdc/hadoop-3.3.0/tmp, and then copy /cdc/hadoop-3.3.0/tmp to /cdc/hadoop-3.3.0/ of dhf2.
scp -r /cdc/hadoop-3.3.0/tmp/ root@dhf2:/cdc/hadoop-3.3.0/
11.4 format ZK
(execute on dhf1)
hdfs zkfc -formatZK
11.5 start HDFS
(execute on dhf1)
cd /cdc/hadoop-3.3.0/sbin/ ./start-dfs.sh
11.6 start YARN
(execute on dhf3)
cd /cdc/hadoop-3.3.0/sbin/ ./start-yarn.sh
12. verification
NameNode 'dhf2:9000' (active)
NameNode 'dhf1:9000' (standby)
View the status of datanode nodes all online
First upload a file to hdfs
hadoop fs -mkdir /dhf hadoop fs -put /test.txt /dhf hadoop fs -ls /dhf
Then kill the active NameNode(dhf2)
kill -9 16950
Access via browser: http://192.xxx.xxx.227:50070
NameNode 'dhf1:9000' (active)
At this time, the NameNode on dhf1 becomes active
hadoop fs -ls /dhf
The file just uploaded still exists
Manually start the suspended NameNode
./hadoop-daemon.sh start namenode
Access via browser: http://192.xxx.xxx.228:50070
NameNode 'dhf2:9000' (standby)
Follow the official account, add the author's wechat, and discuss more together.
