hadoop cluster rapid deployment

1. modify the Linux host name

hostnamectl set-hostname dhf1

Or modify the configuration file

vim /etc/sysconfig/network

2. modify IP

vim /etc/sysconfig/network-scripts/ifcfg-eth0
systemctl restart network

3. modify the mapping relationship between host name and IP

vim /etc/hosts
192.xxx.xxx.227 dhf1
192.xxx.xxx.228 dhf2
192.xxx.xxx.229 dhf3
192.xxx.xxx.230 dhf4
192.xxx.xxx.231 dhf5
192.xxx.xxx.232 dhf6
192.xxx.xxx.233 dhf7

4. turn off the firewall

systemctl status firewalld
systemctl stop firewalld
systemctl disable firewalld

5.ssh login free

ssh-keygen -t rsa

(four carriage returns) after executing this command, two file IDS will be generated_ RSA (private key), id_rsa.pub (public key) copy the public key to the machine (including this machine) to be login free:

ssh-copy-id dhf1
Machines that need to generate public keys Machines to copy to
dhf1 dhf1,dhf2,dhf3,dhf4,dhf5,dhf6,dhf7
dhf2 dhf1,dhf2
dhf3 dhf3,dhf4,dhf5,dhf6,dhf7

6. install JDK, configure environment variables, etc

export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-
export JRE_HOME=$JAVA_HOME/jre
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:$JRE_HOME/lib
source /etc/profile

7. restart the machine


8. cluster planning

host name IP Installed software Running processes
dhf1 192.xxx.xxx.227 jdk,hadoop NameNode,DFSZKFailoverController(zkfc)
dhf2 192.xxx.xxx.228 jdk,hadoop NameNode,DFSZKFailoverController(zkfc)
dhf3 192.xxx.xxx.229 jdk,hadoop ResourceManager
dhf4 192.xxx.xxx.230 jdk,hadoop ResourceManager
dhf5 192.xxx.xxx.231 jdk,hadoop,zookeeper DataNode,NodeManager,JournalNode,QuorumPeerMain
dhf6 192.xxx.xxx.232 jdk,hadoop,zookeeper DataNode,NodeManager,JournalNode,QuorumPeerMain
dhf7 192.xxx.xxx.233 jdk,hadoop,zookeeper DataNode,NodeManager,JournalNode,QuorumPeerMain

Note: in hadoop2.0, there are usually two namenodes, one in active state and the other in standby state. The active namenode provides external services, while the Standby NameNode does not provide external services. It only synchronizes the status of the active namenode, so that it can quickly switch when it fails. Hadoop officially provides two HDFS HA solutions, NFS and QJM. Here we use the simple QJM. In this scheme, the primary and standby namenodes synchronize metadata information through a group of journalnodes. As long as a piece of data is successfully written to most journalnodes, it is considered as successful. An odd number of journalnodes are usually configured. A zookeeper cluster is also configured here for ZKFC (DFSZKFailoverController) failover. When the active namenode hangs, it will automatically switch the Standby NameNode to standby state. There are two resourcemanagers, one is active and the other is standby. The status is coordinated by zookeeper. Namenode and ResourceManager are separated because of performance problems. Because they both occupy a large amount of resources, they are separated and started on different machines.

9. install zookeeper

9.1. Installing and configuring the zooekeeper cluster

(operate on dhf5)

cd /cdc/apache-zookeeper-3.5.8-bin/conf/
cp zoo_sample.cfg zoo.cfg

Modify: zoo cfg

vim zoo.cfg

Save exit

Then create a tmp folder

mkdir /cdc/apache-zookeeper-3.5.8-bin/tmp

Create another empty file

touch /cdc/apache-zookeeper-3.5.8-bin/tmp/myid

Last write ID to this file

echo 1 > /cdc/apache-zookeeper-3.5.8-bin/tmp/myid

9.2 copy the configured zookeeper to other nodes

scp -r /cdc/apache-zookeeper-3.5.8-bin/ dhf6:/cdc/
scp -r /cdc/apache-zookeeper-3.5.8-bin/ dhf7:/cdc/

Note: modify the contents of /cdc/apache-zookeeper-3.5.8-bin/tmp/myid corresponding to dhf6 and dhf7


echo 2 > /cdc/apache-zookeeper-3.5.8-bin/tmp/myid


echo 3 > /cdc/apache-zookeeper-3.5.8-bin/tmp/myid

10. install hadoop

10.1 installing and configuring hadoop cluster

(operate on dhf1)

10.1.1 add hadoop to the environment variable

vim /etc/profile
export HADOOP_HOME=/cdc/hadoop-3.3.0
export HDFS_ZKFC_USER=root

10.1.2 configuring HDFS

(all Hadoop configuration files are in the $HADOOP_HOME/etc/hadoop directory)

First, obtain Hadoop through the hadoop classpath command_ Classpath, as follows:

/cdc/hadoop-3.3.0/etc/hadoop:/cdc/hadoop-3.3.0/share/hadoop/common/lib/*:/cdc/hadoop-3.3.0/share/hadoop/common/*:/cdc/hadoop-3.3.0/share/hadoop/hdfs:/cdc/hadoop-3.3.0/share/hadoop/hdfs/lib/*:/cdc/hadoop-3.3.0/share/hadoop/hdfs/*:/cdc/hadoop-3.3.0/share/hadoop/mapreduce/*:/cdc/hadoop-3.3.0/share/hadoop/yarn:/cdc/hadoop-3.3.0/share/hadoop/yarn/lib/*:/cdc/hadoop-3.3.0/share/hadoop/yarn/* modify Hadoop env sh

export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-
export HADOOP_CLASSPATH=/cdc/hadoop-3.3.0/etc/hadoop:/cdc/hadoop-3.3.0/share/hadoop/common/lib/*:/cdc/hadoop-3.3.0/share/hadoop/common/*:/cdc/hadoop-3.3.0/share/hadoop/hdfs:/cdc/hadoop-3.3.0/share/hadoop/hdfs/lib/*:/cdc/hadoop-3.3.0/share/hadoop/hdfs/*:/cdc/hadoop-3.3.0/share/hadoop/mapreduce/*:/cdc/hadoop-3.3.0/share/hadoop/yarn:/cdc/hadoop-3.3.0/share/hadoop/yarn/lib/*:/cdc/hadoop-3.3.0/share/hadoop/yarn/* modify the core site xml

<!-- appoint hdfs of nameservice by ns1 -->
<!-- appoint hadoop Temporary directory -->
<!-- appoint zookeeper address -->
</configuration> modify HDFS site xml

<!--appoint hdfs of nameservice by ns1,Need and core-site.xml Consistent in -->
<!-- ns1 There are two below NameNode,namely nn1,nn2 -->
<!-- nn1 of RPC mailing address -->
<!-- nn1 of http mailing address -->
<!-- nn2 of RPC mailing address -->
<!-- nn2 of http mailing address -->
<!-- appoint NameNode Metadata for JournalNode Storage location on -->
<!-- appoint JournalNode Location of data on local disk -->
<!-- open NameNode Fail auto switch -->
<!-- Configuration failure automatic switching implementation mode -->
<!-- Configure the isolation mechanism method. Multiple mechanisms are separated by line feed, that is, each mechanism temporarily uses one line-->
• sshfence
• shell(/bin/true)
<!-- use sshfence Required for isolation mechanism ssh No login -->
<!-- to configure sshfence Isolation mechanism timeout -->
</configuration> modify mapred site xml

<!-- appoint mr Frame is yarn mode -->
</configuration> modify yarn site xml

<!-- open RM High reliability -->
<!-- appoint RM of cluster id -->
<!-- appoint RM Name of -->
<!-- Specify separately RM Address of -->
<!-- appoint zk Cluster address -->
</property> XML
</configuration> modify workers (workers)

(workers refers to the location of the child node. Since HDFS is to be started on dhf1 and yarn is to be started on dhf3, the workers file on dhf1 specifies the location of datanode, and the workers file on dhf3 specifies the location of nodemanager.)

vim workers

10.2 copy the configured hadoop to other nodes

scp -r /cdc/hadoop-3.3.0/ root@dhf2:/cdc/
scp -r /cdc/hadoop-3.3.0/ root@dhf3:/cdc/
scp -r /cdc/hadoop-3.3.0/ root@dhf4:/cdc/
scp -r /cdc/hadoop-3.3.0/ root@dhf5:/cdc/
scp -r /cdc/hadoop-3.3.0/ root@dhf6:/cdc/
scp -r /cdc/hadoop-3.3.0/ root@dhf7:/cdc/

11. start service

11.1 start zookeeper cluster

(start zk on dhf5, dhf6 and dhf7 respectively) (QuorumPeerMain)

cd /cdc/apache-zookeeper-3.5.8-bin/bin/
./zkServer.sh start

Viewing status: one leader and two follower s

./zkServer.sh status

11.2 start journalnode

cd /cdc/hadoop-3.3.0/;rm -rf journal/ns1/;rm -rf logs/; rm -rf tmp/;

(execute on dhf5, dhf6 and tcast07 respectively)

cd /cdc/hadoop-3.3.0/sbin/
./hadoop-daemon.sh start journalnode

Run the jps command to verify that there are more JournalNode processes on dhf5, dhf6, and dhf7

11.3 format HDFS

(execute command on dhf1)

hdfs namenode -format

After formatting, the core site Hadoop tmp. Dir configuration generates a file. Here I configure /cdc/hadoop-3.3.0/tmp, and then copy /cdc/hadoop-3.3.0/tmp to /cdc/hadoop-3.3.0/ of dhf2.

scp -r /cdc/hadoop-3.3.0/tmp/ root@dhf2:/cdc/hadoop-3.3.0/

11.4 format ZK

(execute on dhf1)

hdfs zkfc -formatZK

11.5 start HDFS

(execute on dhf1)

cd /cdc/hadoop-3.3.0/sbin/

11.6 start YARN

(execute on dhf3)

cd /cdc/hadoop-3.3.0/sbin/

12. verification


NameNode 'dhf2:9000' (active)


NameNode 'dhf1:9000' (standby)

View the status of datanode nodes all online

First upload a file to hdfs

hadoop fs -mkdir /dhf
hadoop fs -put /test.txt /dhf
hadoop fs -ls /dhf

Then kill the active NameNode(dhf2)

kill -9 16950

Access via browser: http://192.xxx.xxx.227:50070

NameNode 'dhf1:9000' (active)

At this time, the NameNode on dhf1 becomes active

hadoop fs -ls /dhf

The file just uploaded still exists

Manually start the suspended NameNode

./hadoop-daemon.sh start namenode

Access via browser: http://192.xxx.xxx.228:50070

NameNode 'dhf2:9000' (standby)

Follow the official account, add the author's wechat, and discuss more together.

Tags: flink Hadoop cluster

Posted by FireyIce01 on Thu, 02 Jun 2022 12:52:25 +0530