1, Zookeeper and Replicated LevelDB cluster principle
Interview question: how to ensure high availability after introducing message queue?
ActiveMQ cluster is built based on Zookeeper and LevelDB. The cluster provides high availability cluster function in the active and standby mode to avoid single point of failure.
ActiveMQ official website master-slave introduction: http://activemq.apache.org/masterslave.html
As you can see, three methods are supported here: shared file system master-slave, JDBC master-slave, and replicable LevelDB storage.
LevelDB is a persistence engine launched after ActiveMQ version 5.6. It uses a custom index to replace the common BTree index. Its persistence performance is higher than that of KahaDB. Although the default mode is KahaDB, LevelDB may become a trend in the future.
ActiveMQ version 5.9 also provides a LevelDB and Zookeeper based data replication mode, which is the preferred data replication scheme for the master slave mode.
Shared file system master-slave: http://activemq.apache.org/shared-file-system-master-slave
JDBC master-slave: http://activemq.apache.org/jdbc-master-slave
Replicable LevelDB storage: http://activemq.apache.org/replicated-leveldb-store.html
The main thing is replicable LevelDB storage.
Principle description:
Use the Zookeeper cluster to register all ActiveMQ brokers. However, as long as one Broker can provide services, it is regarded as a Master. Other brokers in standby status are regarded as Slave.
If the master fails to provide services due to a fault, Zookeeper will select a Broker from the Slave to act as the master. During operation, the Slave synchronizes their storage status by connecting to the master. The Slave does not accept client connections. All storage operations will be copied to the Slave connected to the master. If the Master goes down, the Slave with the latest update will become the master. After recovery, the failed node rejoins the group as a Slave connection master.
All message operations that need to be synchronized can only be completed after the storage state is copied to other legal nodes.
If replicas=3 is configured, the legal nodes are (3/2) +1=2. The Master will store and update, and then wait until (2-1) =1 Slave is stored and updated before reporting success.
2, Zookeeper and Replicated LevelDB cluster deployment planning description
Zookeeper Download address: https://zookeeper.apache.org/releases.html Click Download and select the image address to Download. Here I Download Zookeeper-3.4.6.
Set up a single pseudo cluster. Of course, you can also set up a real cluster. Why is it called pseudo cluster? Because three zookeepers are built on one virtual machine, which is distinguished by port number. In a real cluster, three zookeepers are located on three machines. Let's start.
After downloading zookeeper, use tar -zxvf apache-zookeeper-3.4.6 Tar GZ is extracted into a directory. I created a /zk in the root directory_ Cluster directory, execute mv apache-zookeeper-3.4.6 zookeeper01 to modify the directory name, and create a zoo Cfg file, create the data folder and dataLog folder in zookeeper01. The two copies of zookeeper01 are zookeeper02 and zookeeper03. At this time, in the zookeeper folder, there are three zookeeper01, zookeeper02 and zookeeper03. Let's review the zoos of these three zookeeper01, zookeeper02 and zookeeper03 CFG.
| Zookeeper port | ActiveMQWeb port | ActiveMQ protocol port |
| 2181 | 8161 | 61616 |
| 2182 | 8162 | 61617 |
| 2183 | 8163 | 61618 |
zookeeper01 of zoo.cfg tickTime=2000 initLimit=10 syncLimit=5 clientPort=2181 dataDir=/usr/zookeeper/zookeeper01/data dataLogDir=/usr/zookeeper/zookeeper01/datalogs server.1=192.168.0.123:2881:3881 server.2=192.168.0.123:2882:3882 server.3=192.168.0.123:2883:3883 zookeeper02 of zoo.cfg tickTime=2000 initLimit=10 syncLimit=5 clientPort=2182 dataDir=/usr/zookeeper/zookeeper02/data dataLogDir=/usr/zookeeper/zookeeper02/datalogs server.1=192.168.0.123:2881:3881 server.2=192.168.0.123:2882:3882 server.3=192.168.0.123:2883:3883 zookeeper03 of zoo.cfg tickTime=2000 initLimit=10 syncLimit=5 clientPort=2183 dataDir=/usr/zookeeper/zookeeper03/data dataLogDir=/usr/zookeeper/zookeeper03/datalogs server.1=192.168.0.123:2881:3881 server.2=192.168.0.123:2882:3882 server.3=192.168.0.123:2883:3883
Specify myid, enter the dataDir directory specified above, create a new myid file, and write the id values, which are 1, 2, and 3 respectively, that is, server The following values are equivalent to numbering zookeeper.
Then you can start the zookeeper. Execute in the bin directory of the three files respectively/ Zkserver The SH start command starts three zookeepers.
Another command is provided:/ Zkserver SH start forest can directly print out the error information to facilitate troubleshooting.
Here is a case. When a cluster is configured and a single machine is started, the status will be checked with status, and the error contacting service will be prompted It is probably not running This error occurs when all three servers are started up. Finally, it is normal to use status to check the status. It may be that other servers in the configuration file cannot be read when one server is started up.
Then write three scripts, directly start the three zookeepers through one command, close the three zookeepers, and check the status of the three zookeepers. There is no need to switch directory operations back and forth to reduce the trouble.
Start (create a file using the command vim start-zookeeper-all.sh)
#!/bin/sh cd /usr/zookeeper/zookeeper01/bin ./zkServer.sh start cd /usr/zookeeper/zookeeper02/bin ./zkServer.sh start cd /usr/zookeeper/zookeeper03/bin ./zkServer.sh start
Close (create the file using the command vim stop-zookeeper-all.sh)
#!/bin/sh cd /usr/zookeeper/zookeeper01/bin ./zkServer.sh stop cd /usr/zookeeper/zookeeper02/bin ./zkServer.sh stop cd /usr/zookeeper/zookeeper03/bin ./zkServer.sh stop
View status (create a file using the command vim status-zookeeper-all.sh)
#!/bin/sh cd /usr/zookeeper/zookeeper01/bin ./zkServer.sh status cd /usr/zookeeper/zookeeper02/bin ./zkServer.sh status cd /usr/zookeeper/zookeeper03/bin ./zkServer.sh status
Then use the command Chmod u+x start zookeeper all SH, Chmod u+x stop zookeeper all SH, Chmod u+x status zookeeper all SH adds run permissions to these three files, and then just execute start-zookeeper-all SH or stop zookeeper all SH or status zookeeper all SH can complete the function.
So far, the pseudo cluster deployment has been completed, and the three zookeeper s have started to work normally. You can see which is the master and which is the follower through the status.
3, Zookeeper and Replicated LevelDB cluster deployment configuration
Create /mq_cluster directory, use cp -r apache-activemq-5.15.11 activemq01 to copy one copy, and then repeat it twice to copy activemq02 and activemq03 respectively.
At this point, the three folders are located in the activemq directory side by side.
Modify the port of the management console, enter the conf directory, and find jetty XML file. Modify the port attribute of the jettyPort node. Change the port attribute in activemq02 to 8162. Similarly, change the port attribute in activemq03 to 8163.
<bean id="jettyPort" class="org.apache.activemq.web.WebConsolePort" init-method="start">
<!-- the default port number for the web console -->
<property name="host" value="0.0.0.0"/>
<property name="port" value="8161"/>
</bean>Host name mapping vim /etc/host is called activemqCluster here
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 10.5.96.57 activeCluster
For the configuration of MQ cluster, add activemq The brokerName in the XML configuration file is changed to the same
<broker xmlns="http://activemq.apache.org/schema/core" brokerName="activemqCluster" dataDirectory="${activemq.data}">Configure the persistence of the cluster, enter the conf directory of the three files respectively, and edit activemq XML.
Replace the persistenceAdapter node with the following content. You need to modify the configurations in the three files. Note that the ports in bind should not be repeated. Here, 636316326333 are used respectively.
<replicatedLevelDB directory="${activemq.data}/leveldb"
replicas="3"
bind="tcp://0.0.0.0:63631"
zkAddress="10.5.96.48:2181,10.5.96.48:2182,10.5.96.48:2183"
hostname="activemqCluster"
sync="local_disk"
zkPath="/activemq/leveldb-stores"/>
</persistenceAdapter>The following describes the meaning of the following parameter attributes:
Directory: the directory where the data is stored. It is stored in leveldb under the data directory.
replicas: the number of nodes is 3.
bind: the address of the binding.
zkAddress: the address and port number of the zookeeper cluster are the addresses of the zookeeper cluster built previously.
hostname:ip address.
sync: synchronizes on the local disk.
Zkpathh: if activemq is managed by zookeeper, leveldb storage will be written under this path.
Start the zookeeper cluster first, and then the activemq cluster.
Refer to VIM start zookeeper all SH, write a start ActiveMQ all SH and stop ActiveMQ all SH and status ActiveMQ all The SH script is easy to run.
ps -ef | grep zookeeper and ps -ef | grep activemq can be used to verify whether the startup is successful. If zookeeper has 3 process numbers and activemq has 3 process numbers, the startup is successful.
Enter the bin directory of any zookeeper and execute/ Zkcli SH -server 127.0.0.1:2181 connect a zookeeper to check whether ActiveMQ is successfully registered.
Then use the ls / command to view the services registered with zookeeper. You can find that there is an ActiveMQ corresponding to activemq If zkpathh in XML is other content, other content will be displayed here. Of course, you can also use the command ls /activemq to view the content under ActiveMQ. According to the above configuration, you can know that there is a folder of leveldb stores under ActiveMQ. Using ls / ActiveMQ / leveldb stores, you can see three data, which are the three nodes of ActiveMQ.
Screenshot of my cluster
Check which ActiveMQ service is the master? Continue to operate in the session just now. Use the command: get /activemq/leveldb stores/00000000006, modify the last 06 to 07 and 08, and view the output respectively. You will see the following contents. Pay attention to the value of the first item, and find that it is not null. Then this node is the master of ActiveMQ, and the rest is slave.
4, Replicated LevelDB cluster failover and validation
ActiveMQ clients can only access the Broker of the Master, but other brokers in the Slave cannot. Therefore, the Broker connected to the client should use the failover protocol (failover protocol).
Use the command lsof -i: port number to check the operation of ActiveMQ. Because the port numbers we configured earlier are 816181628163, the port numbers here are also these. It is found that one command returns results, and the other two do not. Then use 10.5.96.48: port number /admin to access. Here, I use 0.5.96.48:8163/admin to access the management page. Try ports 8161 and 8162 and find that they cannot be accessed. At this time, in ActiveMQ cluster, only the master under port 8163 can be accessed, and the rest cannot be accessed, but the data remains synchronized.
When an ActiveMQ node is down or a Zookeeper node is down, the ActiveMQ service is still running normally. If there is only one ActiveMQ node left, ActiveMQ cannot run normally because the master cannot be selected.
Similarly, if only one Zookeeper node is active, ActiveMQ cannot provide services normally regardless of whether the ActiveMQ node is alive or not, because the high availability of the ActiveMQ cluster depends on the high availability of the Zookeeper cluster.
Modify the code of Producer and Consumer, and set ActiveMQ_ The URL attribute is changed to failover protocol, and the code is as follows
producer
public class JmsProduce {
public static final String ACTIVEMQ_URL = "failover:(tcp://10.5.96.48:61616,tcp://10.5.96.48:61617,tcp://10.5.96.48:61618)?randomize=false";
public static final String QUEUE_NAME = "active-cluster";
public static void main(String[]args) throws JMSException {
//1. create a connection factory
ActiveMQConnectionFactory activeMQConnectionFactory = new ActiveMQConnectionFactory(ACTIVEMQ_URL);
//2. through the connection factory, obtain the connection and start the access
Connection connection = activeMQConnectionFactory.createConnection();
connection.start();
//3. create a session
//Two eucalyptus trees, the first is called transaction, and the second is called receipt
Session session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
//4 Create destination (specifically queue or topic)
Queue queue = session.createQueue(QUEUE_NAME);
//5. create the producer of the message
MessageProducer messageProducer = session.createProducer(queue);
//Persistence settings for messages (now non persistent)
messageProducer.setDeliveryMode(DeliveryMode.PERSISTENT);
//6. use messageProducer to produce 3 messages and send them to the queue
for (int i = 0; i <3 ; i++) {
// 7. create message
TextMessage textMessage = session.createTextMessage("Msg---" + i);
//8. message producer sends to mq
messageProducer.send(textMessage);
}
//9. turn off the resource and turn it on in reverse
messageProducer.close();
session.close();
connection.close();
System.out.println("**** Publish message to MQ Over ****");
}
}consumer
public class JmsConsumer {
public static final String ACTIVEMQ_URL = "failover:(tcp://10.5.96.48:61616,tcp://10.5.96.48:61617,tcp://10.5.96.48:61618)?randomize=false";
public static final String QUEUE_NAME = "active-cluster";
public static void main(String[]args) throws JMSException, IOException {
System.out.println("I am consumer 2");
//1. create a connection factory
ActiveMQConnectionFactory activeMQConnectionFactory = new ActiveMQConnectionFactory(ACTIVEMQ_URL);
//2. through the connection factory, obtain the connection and start the access
Connection connection = activeMQConnectionFactory.createConnection();
connection.start();
//3. create a session
//Two parameters, the first is called transaction and the second is called sign in
Session session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
//4. create a destination (specifically queue or Topic)
Queue queue = session.createQueue(QUEUE_NAME);
//Create consumer
MessageConsumer messageConsumer = session.createConsumer(queue);
while(true){
TextMessage textMessage = (TextMessage) messageConsumer.receive(4000);
if( null !=textMessage ){
System.out.println("***Consumer receives message:" + textMessage.getText());
}else{
break;
}
}
messageConsumer.close();
session.close();
connection.close();
}
}Then start the producer to send the message. Go to the management page to check. You can find that a message has been sent successfully. Start the consumer again, and then the message is consumed by the consumer. Everything is normal at this time.
Next, we try to kill an ActiveMQ to simulate the failure of ActiveMQ. Use the command kill -9 to kill the master process of ActiveMQ. Assuming that 8163 is killed here, one of 8161 or 8162 will become a master node after the election of zookeeper. The access to 8163 just now proves that the third process is a master. Therefore, kill the master port, and then zookeeper will select a master, as shown in the figure below. Access to 8162 proves that the cluster is successful
Use start ActiveMQ all When the ActiveMQ cluster is started, the ActiveMQ cluster will fail to start. I don't know why. Sometimes, you need to retry several times to start it.
In addition, you also need to check whether the address in the activemq node in Zookeeper is null. If the address of the master is null, you will not be able to access it. Check the configuration carefully. If not, start over.
reference
https://blog.csdn.net/qq_36059561/article/details/103915642
https://blog.csdn.net/qq_36059561/article/details/103916980
https://blog.csdn.net/qq_36059561/article/details/103965376