AIX5.3 CPU占用高问题核查步骤
1、topas查看占用cpu占用最高的进程的PID
2、执行: ps -mp PID -o THREAD 以查找相应进程下所有正在占用 CPU 的线程的TID
您应当查看“CP”列(表示 CPU 占用率),看其中哪些线程的此项值比较高并从中挑选一个线程。
3、通过执行以下命令对服务器进行 Thread Dump:
kill -3 PID
4、将步骤2中找到的占用CPU高的线程TID转换成16进制,用对应的16进制数值在3步骤中生成的javacore文件中查找对应的线程堆栈(native ID为该16进制值)
分析实例:
1、topas查看占用CPU最高的进程PID:528494
2、ps -mp 528494 -o THREAD
显示内容如下:
web2:/app1/opentfs2$ ps -mp 528494 -o THREAD
USER PID PPID TID ST CP PRI SC WCHAN F TT BND COMMAND
opentfs2 528494 627044 - A 390 60 142 * 242001 - - /usr/java5/bin/java -Xms1024m -Xmx2048m -da -Dplatform
- - - 507987 S 0 82 1 f100070f10007c40 8410404 - - -
- - - 688141 S 0 82 1 f100070f1000a840 8410404 - - -
- - - 974999 S 0 82 1 f100060026c018c8 400404 - - -
- - - 1102075 R 98 176 0 - 400000 - - -
- - - 1167493 S 0 82 1 f100070f10011d40 8410404 - - -
- - - 1200235 S 0 82 1 f100070f10012540 8410404 - - -
......
3、kill -3 528494 手动在domain的目录下生成 528494 进程的 Thread Dump(javacore文件):javacore.20130819.150717.528494.0002.txt
4、根据步骤2中TID "1102075" 在 CP 列中有一个高值(它达到“98”,而其它 TID 几乎为 0),将1102075转换成16进制为:10D0FB,在javacore.20130819.150717.528494.0002.txt中查找10D0FB对应的线程堆栈信息:
"[STUCK] ExecuteThread: '94' for queue: 'weblogic.kernel.Default (self-tuning)'" (TID:0x37187A00, sys_thread_t:0x3713F324, state:CW, native ID:0x0010D0FB) prio=1
at java/util/zip/Deflater.deflateBytes(Native Method)
at java/util/zip/Deflater.deflate(Deflater.java:332(Compiled Code))
at java/util/zip/DeflaterOutputStream.deflate(DeflaterOutputStream.java(Compiled Code))
at java/util/zip/DeflaterOutputStream.write(DeflaterOutputStream.java:138(Compiled Code))
at java/util/zip/GZIPOutputStream.write(GZIPOutputStream.java:95(Compiled Code))
at com/linkage/webframework/util/compress/CompressStream.write(CompressStream.java:41(Compiled Code))
at org/apache/tapestry/request/ResponseOutputStream.innerWrite(ResponseOutputStream.java:246(Compiled Code))
at org/apache/tapestry/request/ResponseOutputStream.write(ResponseOutputStream.java:271(Compiled Code))
at sun/nio/cs/StreamEncoder$ConverterSE.implWrite(StreamEncoder.java:276(Compiled Code))
at sun/nio/cs/StreamEncoder.write(StreamEncoder.java:186(Compiled Code))
at java/io/OutputStreamWriter.write(OutputStreamWriter.java:214(Compiled Code))
at java/io/BufferedWriter.write(BufferedWriter.java:191(Compiled Code))
at java/io/PrintWriter.write(PrintWriter.java:375(Compiled Code))
at org/apache/tapestry/AbstractMarkupWriter.printRaw(AbstractMarkupWriter.java:749(Compiled Code))
at org/apache/tapestry/html/NestedHTMLWriter.close(NestedHTMLWriter.java:61(Compiled Code))
at org/apache/tapestry/html/Body.renderComponent(Body.java:407(Compiled Code))
at org/apache/tapestry/AbstractComponent.render(AbstractComponent.java:859(Compiled Code))
at org/apache/tapestry/BaseComponent.renderComponent(BaseComponent.java:118(Compiled Code))
at org/apache/tapestry/AbstractComponent.render(AbstractComponent.java:859(Compiled Code))
at org/apache/tapestry/AbstractPage.renderPage(AbstractPage.java:300(Compiled Code))
at org/apache/tapestry/engine/RequestCycle.renderPage(RequestCycle.java:368(Compiled Code))
at org/apache/tapestry/engine/AbstractEngine.renderResponse(AbstractEngine.java:749(Compiled Code))
at org/apache/tapestry/engine/PageService.service(PageService.java:77(Compiled Code))
at org/apache/tapestry/engine/AbstractEngine.service(AbstractEngine.java:889(Compiled Code))
at org/apache/tapestry/ApplicationServlet.doService(ApplicationServlet.java:200(Compiled Code))
at org/apache/tapestry/ApplicationServlet.doGet(ApplicationServlet.java:161(Compiled Code))
at javax/servlet/http/HttpServlet.service(HttpServlet.java:743(Compiled Code))
at javax/servlet/http/HttpServlet.service(HttpServlet.java:856(Compiled Code))
at weblogic/servlet/internal/StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227(Compiled Code))
at weblogic/servlet/internal/StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125(Compiled Code))
at weblogic/servlet/internal/ServletStubImpl.execute(ServletStubImpl.java:283(Compiled Code))
at weblogic/servlet/internal/TailFilter.doFilter(TailFilter.java:26(Compiled Code))
at weblogic/servlet/internal/FilterChainImpl.doFilter(FilterChainImpl.java:42(Compiled Code))
at com/linkage/component/AppCompress.doFilter(AppCompress.java:95(Compiled Code))
at weblogic/servlet/internal/FilterChainImpl.doFilter(FilterChainImpl.java:42(Compiled Code))
at weblogic/servlet/internal/WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3242(Compiled Code))
at weblogic/security/acl/internal/AuthenticatedSubject.doAs(AuthenticatedSubject.java:321(Compiled Code))
at weblogic/security/service/SecurityManager.runAs(SecurityManager.java:121(Compiled Code))
at weblogic/servlet/internal/WebAppServletContext.securedExecute(WebAppServletContext.java:1950(Compiled Code))
at weblogic/servlet/internal/WebAppServletContext.execute(WebAppServletContext.java:1916(Compiled Code))
at weblogic/servlet/internal/ServletRequestImpl.run(ServletRequestImpl.java:1371(Compiled Code))
at weblogic/work/ExecuteThread.execute(ExecuteThread.java:209(Compiled Code))
at weblogic/work/ExecuteThread.run(ExecuteThread.java:181)
根据该堆栈,即可定位问题所在。