Java Performance – Part 3 Application Code Performance
© 2011 IBM Corporation
Java Concepts and Troubleshooting Agenda
■
■
■
■
■
■
2
Session 1 (October 17th, 11:00AM Eastern) – Introduction to IBM Java Chris Bailey, Java Service Architect – Java Memory Management Peter Shipton, JVM Technology Lead – Garbage Collection Peter Shipton, JVM Technology Lead Session 2 (October 19th, 11:00AM Eastern) – Java Tooling Overview Venkat L, Java Service Technical Lead – Java Performance Part 1 Michael Tunnicliffe, Java Service Engineer Session 3: (October 21st, 11:00AM Eastern) – Java Performance Part 2 Karl Taylor,
Java VM Developer
Session 4: (October 24th, 11:00AM Eastern) – Java Performance Part 3 Ian Partridge, Session 5: (October 26th, 11:00AM Eastern) – OutOfMemoryErrors Matthew Kilner,
Java Service Engineer
Java Service Team Lead
Feedback! – Please provide any feedback you have to
[email protected] or
[email protected]
© 2011 IBM Corporation
Important Disclaimers
THE INFORMATION CONTAINED IN THIS PRESENTATION IS PROVIDED FOR INFORMATIONAL PURPOSES ONLY. WHILST EFFORTS WERE MADE TO VERIFY THE COMPLETENESS AND ACCURACY OF THE INFORMATION CONTAINED IN THIS PRESENTATION, IT IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED. ALL PERFORMANCE DATA INCLUDED IN THIS PRESENTATION HAVE BEEN GATHERED IN A CONTROLLED ENVIRONMENT. YOUR OWN TEST RESULTS MAY VARY BASED ON HARDWARE, SOFTWARE OR INFRASTRUCTURE DIFFERENCES. ALL DATA INCLUDED IN THIS PRESENTATION ARE MEANT TO BE USED ONLY AS A GUIDE. IN ADDITION, THE INFORMATION CONTAINED IN THIS PRESENTATION IS BASED ON IBM’S CURRENT PRODUCT PLANS AND STRATEGY, WHICH ARE SUBJECT TO CHANGE BY IBM, WITHOUT NOTICE. IBM AND ITS AFFILIATED COMPANIES SHALL NOT BE RESPONSIBLE FOR ANY DAMAGES ARISING OUT OF THE USE OF, OR OTHERWISE RELATED TO, THIS PRESENTATION OR ANY OTHER DOCUMENTATION. NOTHING CONTAINED IN THIS PRESENTATION IS INTENDED TO, OR SHALL HAVE THE EFFECT OF: - CREATING ANY WARRANT OR REPRESENTATION FROM IBM, ITS AFFILIATED COMPANIES OR ITS OR THEIR SUPPLIERS AND/OR LICENSORS
3
3
© 2011 IBM Corporation
What is Java Performance?
■
Java performance problems are grouped into 4 main scenarios: – System resource contention/starvation – Garbage Collection performance – Application code performance – External delays
■
First stage is to understand which problem you have
4
© 2011 IBM Corporation
Application Code Performance
■
5
Three particular problem areas: – High CPU usage – Synchronized Resource Contention – External Delays
© 2011 IBM Corporation
High CPU Usage of Application
■
Could be the cause of a hang: – Single thread “spinning” and taking all CPU time
■
Could be inefficient code
■
High CPU will be seen in topas and perfmon performance counters as seen previously
■
First step is to understand whether CPU cost is specific to any given thread, or collection of threads
■
And then to understand what those threads are doing
6
© 2011 IBM Corporation
Profiling CPU Usage: AIX
7
■
Use of: tprof –skex sleep 60:
■
Shows percentage of CPU in each process and used by each thread – “Shared” is shared libraries, “Other” is the JVM interpreter and JITed code – TID is in decimal and needs to be convered to hex, eg. 688211 = 0xA8053
© 2011 IBM Corporation
Mapping to Thread Stack: AIX (javacore output)
8
© 2011 IBM Corporation
Profiling CPU Usage: Windows
■
Use of Thread “Processor Time” and “ID Thread” counters in perfmon
■
Use report option rather than graph to see ID Thread value:
■
“ID Thread” is in decimal and needs to be converted to hex – ie, Thread 10 has “ID Thread” 9652, which is 0x25B4
9
© 2011 IBM Corporation
Mapping to Thread Stack: Windows
10
© 2011 IBM Corporation
Profiling the Methods
■
A series of javacore.txt files provides a rough idea of what the threads are doing – For a single thread using 100% CPU, the stack track should be sufficient – For a group of threads, it shows some of the methods being executed
■
Thread and Monitor Dump Analyzer helps with multi-javacore analysis
■
However method profiling provides a much more accurate picture – Shows complete set of methods being executed – Provides call graphs
■
Recommendation: Health Center
11
© 2011 IBM Corporation
Health Center
■
Live monitoring tool with very low overhead – Goal of 0 Tracing is enabled if count > 0
© 2011 IBM Corporation
Triggering Examples
■
Following format used to specify triggers in method events: – trigger=method{method spec,entry action,exit action,delay count,match count} – On entering any method that matches the method spec indicated, the entry action is executed. – When exiting the method, the exit action is performed. – If the delay count is specified, the entry and exit actions are only carried out when entry and exit have occurred more times than the delay count. – If the match count is specified, the actions are only carried out a maximum of that many times.
26
■
Examples:
■
-Xtrace:trigger=method{java/lang/StackOverflowError*, sysdump} – create a system dump on the first (and only the first) instance of a StackOverflowError method being called - which is the method.
■
-Xtrace:resumecount=1 -Xtrace:trigger=method{HelloWorld.main,resume,suspend} – trace all threads once HelloWorld.main() is called and stop tracing when HelloWorld.main() returns. © 2011 IBM Corporation
Triggering and Method Trace in Action
■
-Xtrace: iprint=mt, methods={myapp/MyTime*}, resumecount=1, trigger=method{myapp/MyTime.main,resume,suspend}
21:05:47.992*0x806cb00 static method 21:05:47.994 0x806cb00 21:05:47.994 0x806cb00 809baec 21:05:47.994 0x806cb00 arguments: () 21:05:47.994 0x806cb00 21:05:47.994 0x806cb00 809baf0 21:05:47.994 0x806cb00 arguments: () 21:05:48.079 0x806cb00 21:05:48.079 0x806cb00 static method ■
27
mt.3 mt.19 mt.0 mt.18 mt.6 mt.0 mt.18 mt.6 mt.9
> myapp/MyTime.main([Ljava/lang/String;)V Bytecode - Static method arguments: ([L@55D8CB98) > myapp/MyTime.()V Bytecode method, This = - Instance method receiver: myapp/MyTime@55D8CBA8 < myapp/MyTime.()V Bytecode method > myapp/MyTime.test()V Bytecode method, This = - Instance method receiver: myapp/MyTime@55D8CBA8 < myapp/MyTime.test()V Bytecode method < myapp/MyTime.main([Ljava/lang/String;)V Bytecode
Only real time (79ms) is in the call to MyTime.test()
© 2011 IBM Corporation
Triggering and Method Trace in Action
■
Could now drill down into MyTime.test(): – Extend scope of methods traced, and reduce scope of tracing into MyTime.test()
■
-Xtrace: iprint=mt, methods={myapp/*}, resumecount=1, trigger=method{myapp/MyTime.test,resume,suspend} 21:07:14.968*0x806cb00 21:07:14.970 0x806cb00 arguments: () 21:07:15.067 0x806cb00 21:07:15.067 0x806cb00 21:07:15.067 0x806cb00 21:07:15.069 0x806cb00
28
mt.0 mt.18
> myapp/MyTime.test()V Bytecode method, This = 809baf0 - Instance method receiver: myapp/MyTime@55D8CBA8
mt.3 mt.19 mt.9 mt.6
> myapp/MyTimer.getTime()V Bytecode static method - Static method arguments: () < myapp/MyTimer.getTime()V Bytecode static method < myapp/MyTime.test()V Bytecode method
© 2011 IBM Corporation
Profiling Thread Waits
■
Delayed method is com.ibm.as400.access.AS400ThreadedServer.receive
■
Can profile calls to this method via method trace with the following:
■
-Xtrace:maximal=mt,output=mtrace#.out,10m,10,V – Trace methods to 10 files named mtrace#.out each sized at 10MB
■
Vmethods={com/ibm/as400/access/AS400ThreadedServer.receive} – Trace entry and exit of AS400ThreadedServer.receive
29
© 2011 IBM Corporation
Summary
■
Lots of different performance problem causes: – System level resources – Java and middleware tuning – Application code – External resources
■
A number of capabilities provided at each level to help diagnose: – Operating System tools – Java debug capabilities: Verbose GC, Javacore.txt, Method Trace – Java tools: Health Center, GC and Memory Analyzer, Thread and Monitor Dump Analyzer
30
© 2011 IBM Corporation
Questions?
31
© 2011 IBM Corporation
References
■
Get Products and Technologies: – IBM Developer Kits for Java: • https://www.ibm.com/developerworks/java/jdk/ – IBM Monitoring and Diagnostic Tools for Java: • https://www.ibm.com/developerworks/java/jdk/tools/
■
Learn: – IBM Java InfoCenters: • https://www.ibm.com/developerworks/java/jdk/docs.html
■
Discuss: – IBM Java Runtimes and SDKs Forum: • http://www.ibm.com/developerworks/forums/forum.jspa?forumID=367&start=0 – IBM Java Technology Community: • https://www.ibm.com/developerworks/mydeveloperworks/groups/service/html/communityview? communityUuid=738b7897-cd38-4f24-9f05-48dd69116837 – IBM on Troubleshooting Java Applications Blog: • https://www.ibm.com/developerworks/mydeveloperworks/blogs/troubleshootingjava/
32
32
© 2011 IBM Corporation
Copyright and Trademarks
© IBM Corporation 2011. All Rights Reserved.
IBM, the IBM logo, and ibm.com are trademarks or registered trademarks of International Business Machines Corp., and registered in many jurisdictions worldwide.
Other product and service names might be trademarks of IBM or other companies.
A current list of IBM trademarks is available on the Web – see the IBM “Copyright and trademark information” page at URL: www.ibm.com/legal/copytrade.shtml
33
33
© 2011 IBM Corporation
