Transferred from:
java command --jstack tool
ps -mp 1 -o THREAD,tid,time | sort -rn
jstack 1|grep D29 -A 30
printf "%x\n" 10426
jstack -l 10424 | grep 28ba
jstack pid
jstack 1|grep 28ba
1, Introduction
Jstack is a stack trace tool that comes with the Java virtual machine. Jstack is used to print out the Java stack information of a given java process ID or core file or remote debugging service. If it is on a 64 bit machine, you need to specify the option "-J-d64". The jstack usage of Windows only supports the following methods:
jstack [-l] pid
It is mainly divided into two functions:
a. Do local or remote thread dump for the living process; The
b. Do thread dump for the core file.
Jstack is used to generate a thread snapshot of the Java virtual machine at the current time. Thread snapshot is a collection of method stacks being executed by each thread in the current Java virtual machine. The main purpose of generating thread snapshot is to locate the causes of long-time pauses of threads, such as deadlock between threads, dead loop, long-time waiting caused by requesting external resources, etc. When a thread pauses, you can view the call stack of each thread through jstack to know what the unresponsive thread is doing in the background or waiting for. If a java program crashes and generates a core file, the jstack tool can be used to obtain the java stack and native stack information of the core file, so that you can easily know how the Java program crashes and where the problem occurs in the program. In addition, the jstack tool can also be attached to the running Java program to see the java stack and native stack information of the running Java program at that time. If the running Java program presents a hung state, jstack is very useful.
The So,jstack command is mainly used to view the call stack of Java threads, and can be used to analyze thread problems (such as deadlocks).
Thread status
If you want to analyze the status of a thread through the jstack command, you must first know the status of the thread. The following statuses are some of the thread statuses that you may see when you use the jstack command to view the thread stack information:
NEW, not started. Does not appear in Dump.
RUNNABLE, executed in the virtual machine. In the running state, the word locked may also be seen in it, indicating that it has obtained a lock.
BLOCKED, BLOCKED and waiting for the monitor lock. It is BLOCKED by some synchronizers.
Waiting, waiting indefinitely for another thread to perform a specific operation. Wait for a condition or monitor to occur, and generally stay in the statements such as park(), wait(), sleep(),join().
Timed_ WAITING, a time limited wait for a specific operation of another thread. The difference between wait and WAITING is that wait() and other statements add a time limit wait(timeout).
TERMINATED, exited.
Monitor
In multithreaded Java programs, to realize the synchronization between threads, we need to talk about monitor. Monitor is the main means to realize mutual exclusion and cooperation between threads in Java. It can be regarded as a lock of an object or Class. Each object has a monitor. The following figure describes the relationship between threads and monitor, as well as the state transition diagram of threads:
Entrt set: indicates that the thread requests to obtain the lock of the object through synchronized. If the object is not locked, enter the owner; Otherwise, wait in the entry area. Once the object lock is released by other threads, it immediately participates in the competition.
The owner: indicates that a thread successfully competes for an object lock.
Wait set: indicates that the thread releases the lock of the object through the wait method of the object and waits to be awakened in the wait set.
As can be seen from the figure, a Monitor can only be owned by one thread at a certain time. This thread is "Active Thread", while other threads are "waiting threads", waiting in two queues "Entry Set" and "Wait Set" respectively. The thread state waiting in the "Entry Set" is "Waiting for monitor entry", while the thread state waiting in the "Wait Set" is "in Object.wait()". First look at the threads in the "Entry Set". We call the code segment protected by synchronized a critical region. When a thread requests to enter the critical area, it enters the "Entry Set" queue. The corresponding code is like:
synchronized(obj) { ......... }
Call decoration
Indicates that the thread performs additional important operations during method invocation. Important information about thread Dump analysis. Decorate the method call above.
Locked < address > target: the owner of the monitor successfully applies for an object lock using synchronized.
Waiting to lock < address > Objective: failed to apply for an object lock using synchronized. Wait in the entry area.
Waiting on < address > Objective: after using synchronized to successfully apply for an object lock, release the lock and wait in the waiting area.
Parking to wait for
locked
at oracle.jdbc.driver.PhysicalConnection.prepareStatement - locked <0x00002aab63bf7f58> (a oracle.jdbc.driver.T4CConnection) at oracle.jdbc.driver.PhysicalConnection.prepareStatement - locked <0x00002aab63bf7f58> (a oracle.jdbc.driver.T4CConnection) at com.jiuqi.dna.core.internal.db.datasource.PooledConnection.prepareStatement
Through the synchronized keyword, you can successfully obtain the lock of the object, become the owner of the monitor, and operate in the critical area. Object locks are thread reentrant.
waiting to lock
at com.jiuqi.dna.core.impl.CacheHolder.isVisibleIn(CacheHolder.java:165) - waiting to lock <0x0000000097ba9aa8> (a CacheHolder) at com.jiuqi.dna.core.impl.CacheGroup$Index.findHolder at com.jiuqi.dna.core.impl.ContextImpl.find at com.jiuqi.dna.bap.basedata.common.util.BaseDataCenter.findInfo
Through the synchronized keyword, the lock of the object is not obtained, and the thread waits in the entry area of the monitor. At the top of the call stack, the thread status is Blocked.
waiting on
at java.lang.Object.wait(Native Method) - waiting on <0x00000000da2defb0> (a WorkingThread) at com.jiuqi.dna.core.impl.WorkingManager.getWorkToDo - locked <0x00000000da2defb0> (a WorkingThread) at com.jiuqi.dna.core.impl.WorkingThread.run
After successfully obtaining the lock of the object through the synchronized keyword, the wait method is called to enter the WAITING area of the object and wait. At the top of the call stack, the thread status is WAITING or TIMED_WATING.
parking to wait for
park is a basic thread blocking primitive that does not block objects through the monitor. The new mechanism that will appear with the concurrent package is different from the synchronized system.
Thread action
Causes of thread status
RUNNABLE: the status is generally RUNNABLE.
In object Wait(): WAITING in the WAITING area. The status is WAITING or TIMED_WAITING.
waiting for monitor entry: enter the zone to wait. The status is BLOCKED.
waiting on condition: waiting in the waiting area and being park.
sleeping: a dormant thread called thread Sleep().
Wait on condition this state occurs when a thread waits for a condition to occur. The specific reasons can be analyzed in combination with stacktrace. The most common case is that the thread is in sleep state and waiting to be awakened. Another common situation is waiting for network IO: before java introduced nio, for each network connection, there was a corresponding thread to handle network read-write operations. Even if there is no read-write data, the thread is still blocked in the read-write operations. This may cause a waste of resources and pressure on the thread scheduling of the operating system. A new mechanism is adopted in NewIO to improve the performance and scalability of the server program. Waiting for the network to read and write. This may be a symptom of a network bottleneck. The thread could not execute because of a network block. One is that the network is very busy, almost consuming all the bandwidth, and there is still a large amount of data waiting for the network to read and write; Another situation may be that the network is idle, but the packets cannot arrive normally due to routing problems. Therefore, it is necessary to make a comprehensive analysis in combination with some performance observation tools of the system. For example, netstat counts the number of packets sent per unit time. If it obviously exceeds the bandwidth limit of the network; Observe the CPU utilization. If the CPU time in the system state is higher than that in the user state, the CPU time ratio is higher; If the program is running on the Solaris 10 platform, you can use the dtrace tool to view the system calls. If you observe that the number of read/write system calls or the running time is far ahead; These all point to the network bottleneck caused by the limited network bandwidth. (from http://www.blogjava.net/jzone/articles/303979.html)
2, Command format
jstack [ option ] pid
jstack [ option ] executable core
jstack [ option ] [server-id@]remote-hostname-or-IP
Description of common parameters
1)options:
Executable Java executable from which the core dump was produced (it may be a Java executable that generates core dump)
core dump file of the information to be printed
Remote hostname or ip hostname or ip of the remote debug service
Server id unique id, if there are multiple remote debug services on a host
2) Basic parameters:
-F force to print stack information when 'jstack [-l] pid' has no corresponding response. If the direct jstack has no response, it is used to force jstack). Generally, it is not necessary to use
-L long list Print additional information about the lock, for example, belonging to java Util The concurrent ownable synchronizers list will cause the JVM to pause for a long time (it may be many times worse. For example, an ordinary jstag may be the same as a GC for a few milliseconds. Adding -l means nearly a second). It is not recommended to use -l. Not required in general
-m print all stack information of java and native c/c++ frameworks You can print the JVM stack and display the native stack frame on the JVM, which is not required for general application troubleshooting
-h | -help print help information
pid needs to print the java process id of configuration information, which can be queried with jps
Analysis tool for thread dump:
- IBM Thread and Monitor Dump Analyze for Java A compact Jar package can be easily sorted and browsed by status, thread name and function of thread stay.
- http://spotify.github.io/threaddump-analyzer Spotify provides a Web based online analysis tool that aggregates threads associated with locks or conditions.
3, Use case
1,jstack pid
~$ jps -ml org.apache.catalina.startup.Bootstrap ~$ jstack 5661 2013-04-16 21:09:27 Full thread dump Java HotSpot(TM) Server VM (20.10-b01 mixed mode): "Attach Listener" daemon prio=10 tid=0x70e95400 nid=0x2265 waiting on condition [0x00000000] java.lang.Thread.State: RUNNABLE "http-bio-8080-exec-20" daemon prio=10 tid=0x08a35800 nid=0x1d42 waiting on condition [0x70997000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x766a27b8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:104) at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:32) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) at java.lang.Thread.run(Thread.java:662) ........
\jstack -l 4089 >1 Txt, view 1 Txt contents are as follows:
2014-03-14 10:47:04 Full thread dump Java HotSpot(TM) Client VM (20.45-b01 mixed mode, sharing): "Attach Listener" daemon prio=10 tid=0x08251400 nid=0x11bd runnable [0x00000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "DestroyJavaVM" prio=10 tid=0xb3a0a800 nid=0xffa waiting on condition [0x00000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "Query Listener" prio=10 tid=0xb3a09800 nid=0x1023 runnable [0xb3b72000] java.lang.Thread.State: RUNNABLE at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408) - locked <0x70a84430> (a java.net.SocksSocketImpl) at java.net.ServerSocket.implAccept(ServerSocket.java:462) at java.net.ServerSocket.accept(ServerSocket.java:430) at com.sun.tools.hat.internal.server.QueryListener.waitForRequests(QueryListener.java:76) at com.sun.tools.hat.internal.server.QueryListener.run(QueryListener.java:65) at java.lang.Thread.run(Thread.java:662) Locked ownable synchronizers: - None "Low Memory Detector" daemon prio=10 tid=0x08220400 nid=0x1000 runnable [0x00000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "C1 CompilerThread0" daemon prio=10 tid=0x08214c00 nid=0xfff waiting on condition [0x00000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "Signal Dispatcher" daemon prio=10 tid=0x08213000 nid=0xffe runnable [0x00000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "Finalizer" daemon prio=10 tid=0x0820bc00 nid=0xffd in Object.wait() [0xb5075000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x7a2b6f50> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118) - locked <0x7a2b6f50> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134) at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:171) Locked ownable synchronizers: - None "Reference Handler" daemon prio=10 tid=0x0820a400 nid=0xffc in Object.wait() [0xb50c7000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x7a2b6fe0> (a java.lang.ref.Reference$Lock) at java.lang.Object.wait(Object.java:485) at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116) - locked <0x7a2b6fe0> (a java.lang.ref.Reference$Lock) Locked ownable synchronizers: - None "VM Thread" prio=10 tid=0x08200000 nid=0xffb runnable "VM Periodic Task Thread" prio=10 tid=0x08222400 nid=0x1001 waiting on condition JNI global references: 1317
In general, the thread information output through jstack mainly includes jvm threads, user threads, etc. The jvm thread will exist when the jvm is started. For user threads, they are generated only when the user accesses them.
2. jstack view what the threads are doing. You can see which threads occupy CPU for a long time. Locate and solve the problem as soon as possible
http://www.iteye.com/topic/1114219
1.top Find out which process consumes cpu High. implement top Command, the default is the process view, where PID Is the process number21125 co_ad2 18 0 1817m 776m 9712 S 3.3 4.9 12:03.24 java5284 co_ad 21 0 3028m 2.5g 9432 S 1.0 16.3 6629:44 jaHere we analyze 21125 java process2.top in shift+h Or“ H"Find out which thread consumes cpu high Input first top,Then press shift+h Or“ H",The thread view opens, pid Is the thread number 21233 co_ad2 15 0 1807m 630m 9492 S 1.3 4.0 0:05.12 java 20503 co_ad2_s 15 0 1360m 560m 9176 S 0.3 3.6 0:46.72 java Here we analyze the 21233 thread and note that this thread belongs to the 21125 process. 3.apply jstack The command outputs the thread stack at this time and saves it to a file named jstack.log. Note: output thread stack and save top Command snapshots should be taken at the same time as possible.
Because jstack.log File logged thread ID It is hexadecimal and needs to top The thread number displayed by the command is converted to hexadecimal.
4. jstack Find information about this thread jstack [process]|grep -A 10 [Thread hex] Namely: jstack 21125|grep -A 10 52f1 -A 10 Indicates that the last 10 rows of the row are found. 21233 convert to hexadecimal 52 with calculator f1,Note that the letters are lowercase. Results: "http-8081-11" daemon prio=10 tid=0x00002aab049a1800 nid=0x52bb in Object.wait() [0x0000000042c75000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:485) at org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:416)
Find 52 in results f1,You can see what the current thread is doing.
3. Code example
Run code:
/** * @author hollis */ public class JStackDemo1 { public static void main(String[] args) { while (true) { //Do Nothing } } }
First, there is jps to view the process number:
hollis@hos:~$ jps 29788 JStackDemo1 29834 Jps 22385 org.eclipse.equinox.launcher_1.3.0.v20130327-1440.jar
Then use jstack to view the stack information:
hollis@hos:~$ jstack 29788 2015-04-17 23:47:31 ...Several contents are omitted here... "main" prio=10 tid=0x00007f197800a000 nid=0x7462 runnable [0x00007f197f7e1000] java.lang.Thread.State: RUNNABLE at javaCommand.JStackDemo1.main(JStackDemo1.java:7)
What can we see from this stack information? We can see that there is currently one user level thread. The thread is in the runnable state and executes to jstackdemo1 The seventh line of Java. Look at the following code:
/** * @author hollis */ public class JStackDemo1 { public static void main(String[] args) { Thread thread = new Thread(new Thread1()); thread.start(); } } class Thread1 implements Runnable{ @Override public void run() { while(true){ System.out.println(1); } } }
The thread stack information is as follows:
"Reference Handler" daemon prio=10 tid=0x00007fbbcc06e000 nid=0x286c in Object.wait() [0x00007fbbc8dfc000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x0000000783e066e0> (a java.lang.ref.Reference$Lock) at java.lang.Object.wait(Object.java:503) at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133) - locked <0x0000000783e066e0> (a java.lang.ref.Reference$Lock)
We can see:
Thread status: calling stack of WAITING thread resources currently locked by the thread: <0x0000000783e066e0> resources currently WAITING by the thread: <0x0000000783e066e0>
Why wait for the same resource locked at the same time:
During the execution of the thread, the Monitor of this object is obtained first (corresponding to locked <0x0000000783e066e0>). When obj Wait (), the thread gives up the ownership of the Monitor and enters the "wait set" queue (corresponding to waiting on <0x0000000783e066e0>).
4, How to analyze
1. Analysis of thread Dump
principle
Reasoning combined with code reading. Mutual derivation and verification of thread Dump and source code are required.
The root cause of bugs is often not directly reflected on the call stack. You must pay special attention to all calls before the current call of the thread.
Starting point
Waiting in the entry area
"d&a-3588" daemon waiting for monitor entry [0x000000006e5d5000] java.lang.Thread.State: BLOCKED (on object monitor) at com.jiuqi.dna.bap.authority.service.UserService$LoginHandler.handle() - waiting to lock <0x0000000602f38e90> (a java.lang.Object) at com.jiuqi.dna.bap.authority.service.UserService$LoginHandler.handle()
Thread status BLOCKED, thread action wait on monitor entry, calling decoration waiting to lock always appear together. Indicates that a conflicting call already exists at the code level. Code that is bound to have problems needs to be minimized.
Synchronization block blocking
One thread locks an object, and a large number of other threads wait on the object.
"blocker" runnable java.lang.Thread.State: RUNNABLE at com.jiuqi.hcl.javadump.Blocker$1.run(Blocker.java:23) - locked <0x00000000eb8eff68> (a java.lang.Object) "blockee-11" waiting for monitor entry java.lang.Thread.State: BLOCKED (on object monitor) at com.jiuqi.hcl.javadump.Blocker$2.run(Blocker.java:41) - waiting to lock <0x00000000eb8eff68> (a java.lang.Object) "blockee-86" waiting for monitor entry java.lang.Thread.State: BLOCKED (on object monitor) at com.jiuqi.hcl.javadump.Blocker$2.run(Blocker.java:41) - waiting to lock <0x00000000eb8eff68> (a java.lang.Object)
Continuous IO operations can be blocked in the RUNNABLE state. For example: database deadlock, network read / write. Pay special attention to the analysis of the real state of IO threads. Generally speaking, IO calls that are captured as RUNNABLE are problematic.
The following stack shows that the thread status is RUNNABLE. The call stack is on SocketInputStream or SocketImpl, socketRead0 and other methods. The call stack contains jdbc related packages. Most likely, a database deadlock has occurred
"d&a-614" daemon prio=6 tid=0x0000000022f1f000 nid=0x37c8 runnable [0x0000000027cbd000] java.lang.Thread.State: RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(Unknown Source) at oracle.net.ns.Packet.receive(Packet.java:240) at oracle.net.ns.DataPacket.receive(DataPacket.java:92) at oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:172) at oracle.net.ns.NetInputStream.read(NetInputStream.java:117) at oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1034) at oracle.jdbc.driver.T4C8Oall.receive(T4C8Oall.java:588)
Thread divided scheduling sleep
Normal thread pool wait
"d&a-131" in Object.wait() java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) at com.jiuqi.dna.core.impl.WorkingManager.getWorkToDo(WorkingManager.java:322) - locked <0x0000000313f656f8> (a com.jiuqi.dna.core.impl.WorkingThread) at com.jiuqi.dna.core.impl.WorkingThread.run(WorkingThread.java:40)
Suspicious thread wait
"d&a-121" in Object.wait() java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:485) at com.jiuqi.dna.core.impl.AcquirableAccessor.exclusive() - locked <0x00000003011678d8> (a com.jiuqi.dna.core.impl.CacheGroup) at com.jiuqi.dna.core.impl.Transaction.lock()
Starting point summary
wait on monitor entry: there must be a problem
runnable: note the IO thread
In object Wait(): pay attention to the non thread pool waiting
2. Deadlock analysis
After learning how to use the jstack command, we can see how to use jstack to analyze deadlocks, which is something we must master. What is a deadlock? So called deadlock : refers to a blocking phenomenon caused by two or more processes competing for resources or communicating with each other during the execution process. If there is no external force, they will not be able to move forward. At this time, the system is in a deadlock state or the system has a deadlock. These processes that are always waiting for each other are called deadlock processes. To put it bluntly, I now want to eat eggs and cakes. There are eggs and cakes on the table, but my friend and I picked up eggs and diseases at the same time. I have eggs in my hand, but I need the cakes in his hand. He had a cake in his hand, but he wanted the egg in my hand. In this way, if we can't get eggs and cakes at the same time, we can't continue to do the following work (making eggs and cakes). Therefore, this creates a deadlock. Look at a deadlock program:
package javaCommand; /** * @author hollis */ public class JStackDemo { public static void main(String[] args) { Thread t1 = new Thread(new DeadLockclass(true));//Create a thread Thread t2 = new Thread(new DeadLockclass(false));//Create another thread t1.start();//Start a thread t2.start();//Start another thread } } class DeadLockclass implements Runnable { public boolean falg;// Control thread DeadLockclass(boolean falg) { this.falg = falg; } public void run() { /** * If the value of false is true, the t1 thread is called */ if (falg) { while (true) { synchronized (Suo.o1) { System.out.println("o1 " + Thread.currentThread().getName()); synchronized (Suo.o2) { System.out.println("o2 " + Thread.currentThread().getName()); } } } } /** * If the value of false is false, the t2 thread is called */ else { while (true) { synchronized (Suo.o2) { System.out.println("o2 " + Thread.currentThread().getName()); synchronized (Suo.o1) { System.out.println("o1 " + Thread.currentThread().getName()); } } } } } } class Suo { static Object o1 = new Object(); static Object o2 = new Object(); }
When I start the program, let's look at the console:
We found that the program only outputs two lines, and then the program does not print anything else, but the program does not stop. This creates a deadlock. When thread 1 uses synchronized to lock o1, thread 2 also uses synchronized to lock o2. When both threads finish the first print task, thread 1 wants to lock o2 and thread 2 wants to lock o1. However, thread 1 currently locks o1 and thread 2 locks o2. Therefore, the two thought that Chengdu could not continue the implementation, resulting in a deadlock.
Then, let's take a look at the thread stack information using jstack:
Found one Java-level deadlock: ============================= "Thread-1": waiting to lock monitor 0x00007f0134003ae8 (object 0x00000007d6aa2c98, a java.lang.Object), which is held by "Thread-0" "Thread-0": waiting to lock monitor 0x00007f0134006168 (object 0x00000007d6aa2ca8, a java.lang.Object), which is held by "Thread-1" Java stack information for the threads listed above: =================================================== "Thread-1": at javaCommand.DeadLockclass.run(JStackDemo.java:40) - waiting to lock <0x00000007d6aa2c98> (a java.lang.Object) - locked <0x00000007d6aa2ca8> (a java.lang.Object) at java.lang.Thread.run(Thread.java:745) "Thread-0": at javaCommand.DeadLockclass.run(JStackDemo.java:27) - waiting to lock <0x00000007d6aa2ca8> (a java.lang.Object) - locked <0x00000007d6aa2c98> (a java.lang.Object) at java.lang.Thread.run(Thread.java:745) Found 1 deadlock.
Haha, the stack is clearly written. It tells us found one Java level deadlock, and then points out the contents of the two threads causing the deadlock. Then, more detailed deadlock information is displayed through Java stack information for the threads listed above. He said
When Thread-1 wants to execute line 40, it currently locks the resource <0x00000007d6aa2ca8>, but it is waiting for the resource <0x00000007d6aa2c98>thread-0. When it wants to execute line 27, it currently locks the resource <0x00000007d6aa2c98>, but it is waiting for the resource < 0x00000007d6aa2ca8>. Because both threads hold resources and need each other's resources, it creates a deadlock. Once we find the cause, we can analyze the specific problems and solve the deadlock.
other
When the virtual machine executes Full GC, all user threads will be blocked. Therefore, the thread that obtains the synchronization lock may also be blocked. When viewing the thread Dump, first check the memory usage.
For the stack of ThreadDump made by jstack, the following information can be reflected( Derived from):
- If the same call stack often occurs, we have more than 80% reasons to determine that the code has a performance problem (except for the part reading the network);
- If the same call stack appears on the same thread (tid), we have great reason to believe that there may be many loops or dead loops in this code;
- If a call stack often appears with a lock in it, please check the cause of the lock. The global lock may cause a performance problem;
- In a less stressful cluster (w<2), we seldom get stacks with business codes, and generally in a complete stack, there are only 1-2 business code stacks at most,
- If it happens frequently, be sure to check the code for performance problems.
- If you suspect that there is a dead lock problem, please find out all the lock IDs to see if there are duplicate lock IDs.
jstack -m will print the JVM stack information, involving C and c++ code, which may need to be coordinated gdb command To analyze.
Frequent GC problems or memory overflow problems
1, Viewing thread ID S using jps
2, Use jstat -gc 333125020 to view the GC. Generally, pay more attention to the PERM area to view the growth of GC.
3, Use jstat -gcpause: extra output of last GC reason
4, Use jmap -dump:format=b,file=heapDump 3331 to generate a heap dump file
5, Use jhat or visualization tools (Eclipse Memory Analyzer, IBM heapananalyzer) to analyze heap conditions.
6, Combined with code to solve memory overflow or leakage problems.
Deadlock problem
1, Viewing thread ID S using jps
2, Using jstack 3331: viewing threads
Main references:
Java command Learning Series (II) -- Jstack
java jstack dump thread introduction and explanation