最近一个运行了4年的javaee web项目，经常接到客户反馈系统无法打开。登录服务器查看服务，发现是tomcat自动关闭了。基本是3到4天发生一次。
运维人员开始以为是其他服务杀死了tomcat服务，没放在心上，解决方法就是直接重启tomcat。
最终捅了篓子，运维人员被客户投诉，扣了一个月的绩效。
解决这个bug兜兜转转来到了我这里。既然接到任务，那就开干，没有解决不了的bug。
系统的运行环境如下：
tomcat6.0
32位jdk7.0
window server2003 32位，32G内存。
查看日志，如果tomcat闪崩，都会在tomcat的bin目录下生成以"hs_err"开头的日志文件。打开最新的日志文件，首先看到的是下面一段话：
# There is insufficient memory for the Java Runtime Environment to continue. # Native memory allocation (malloc) failed to allocate 32756 bytes for ChunkPool::allocate # Possible reasons: # The system is out of physical RAM or swap space # In 32 bit mode, the process size limit was hit # Possible solutions: # Reduce memory load on the system # Increase physical memory or swap space # Check if swap backing store is full # Use 64 bit Java on a 64 bit OS # Decrease Java heap size (-Xmx/-Xms) # Decrease number of Java threads # Decrease Java thread stack sizes (-Xss) # Set larger code cache with -XX:ReservedCodeCacheSize= # This output file may be truncated or incomplete. # # Out of Memory Error (allocation.cpp:211), pid=7864, tid=6556 # # JRE version: Java(TM) SE Runtime Environment (7.0_79-b15) (build 1.7.0_79-b15) # Java VM: Java HotSpot(TM) Server VM (24.79-b02 mixed mode windows-x86 ) # Failed to write core dump.大概意思就是内存不够了，无法分配32756字节的空间。同时给出几个解决方法：
1、减少系统内存负载；
2、增加物理内存或者交换空间；
3、在64位操作系统上使用64位jdk；
4、减少java heap大小；
5、减少java线程数量；
6、减少java线程堆栈大小。
通过上面的内容可以得出，jvm无法分配32756 bytes的内存空间。
从接到任务开始，我一直以为是jvm配置出错，导致内存不够用，只需调整下新生代、老年代的配置即可。
继续往下看日志文件，找到"GC Heap History (10 events):"这一行，这个记录jvm最后10次垃圾回收时堆的变化情况。
GC Heap History (10 events): Event: 572312.299 GC heap before {Heap before GC invocations=5046 (full 357): PSYoungGen total 201472K, used 200685K [0x573c0000, 0x63bc0000, 0x63bc0000) eden space 198144K, 100% used [0x573c0000,0x63540000,0x63540000) from space 3328K, 76% used [0x63540000,0x637bb528,0x63880000) to space 3328K, 0% used [0x63880000,0x63880000,0x63bc0000) ParOldGen total 843776K, used 422602K [0x23bc0000, 0x573c0000, 0x573c0000) object space 843776K, 50% used [0x23bc0000,0x3d872b18,0x573c0000) PSPermGen total 262144K, used 51848K [0x03bc0000, 0x13bc0000, 0x23bc0000) object space 262144K, 19% used [0x03bc0000,0x06e62138,0x13bc0000) Event: 572312.305 GC heap after Heap after GC invocations=5046 (full 357): PSYoungGen total 201472K, used 1103K [0x573c0000, 0x63bc0000, 0x63bc0000) eden space 198144K, 0% used [0x573c0000,0x573c0000,0x63540000) from space 3328K, 33% used [0x63880000,0x63993c90,0x63bc0000) to space 3328K, 0% used [0x63540000,0x63540000,0x63880000) ParOldGen total 843776K, used 423618K [0x23bc0000, 0x573c0000, 0x573c0000) object space 843776K, 50% used [0x23bc0000,0x3d970b18,0x573c0000) PSPermGen total 262144K, used 51848K [0x03bc0000, 0x13bc0000, 0x23bc0000) object space 262144K, 19% used [0x03bc0000,0x06e62138,0x13bc0000) } Event: 572351.132 GC heap before {Heap before GC invocations=5047 (full 357): PSYoungGen total 201472K, used 199247K [0x573c0000, 0x63bc0000, 0x63bc0000) eden space 198144K, 100% used [0x573c0000,0x63540000,0x63540000) from space 3328K, 33% used [0x63880000,0x63993c90,0x63bc0000) to space 3328K, 0% used [0x63540000,0x63540000,0x63880000) ParOldGen total 843776K, used 423618K [0x23bc0000, 0x573c0000, 0x573c0000) object space 843776K, 50% used [0x23bc0000,0x3d970b18,0x573c0000) PSPermGen total 262144K, used 51848K [0x03bc0000, 0x13bc0000, 0x23bc0000) object space 262144K, 19% used [0x03bc0000,0x06e62138,0x13bc0000) Event: 572351.137 GC heap after Heap after GC invocations=5047 (full 357): PSYoungGen total 201472K, used 1615K [0x573c0000, 0x63bc0000, 0x63bc0000) eden space 198144K, 0% used [0x573c0000,0x573c0000,0x63540000) from space 3328K, 48% used [0x63540000,0x636d3ec8,0x63880000) to space 3328K, 0% used [0x63880000,0x63880000,0x63bc0000) ParOldGen total 843776K, used 423674K [0x23bc0000, 0x573c0000, 0x573c0000) object space 843776K, 50% used [0x23bc0000,0x3d97eb18,0x573c0000) PSPermGen total 262144K, used 51848K [0x03bc0000, 0x13bc0000, 0x23bc0000) object space 262144K, 19% used [0x03bc0000,0x06e62138,0x13bc0000) } Event: 572398.649 GC heap before {Heap before GC invocations=5048 (full 357): PSYoungGen total 201472K, used 199759K [0x573c0000, 0x63bc0000, 0x63bc0000) eden space 198144K, 100% used [0x573c0000,0x63540000,0x63540000) from space 3328K, 48% used [0x63540000,0x636d3ec8,0x63880000) to space 3328K, 0% used [0x63880000,0x63880000,0x63bc0000) ParOldGen total 843776K, used 423674K [0x23bc0000, 0x573c0000, 0x573c0000) object space 843776K, 50% used [0x23bc0000,0x3d97eb18,0x573c0000) PSPermGen total 262144K, used 51848K [0x03bc0000, 0x13bc0000, 0x23bc0000) object space 262144K, 19% used [0x03bc0000,0x06e62138,0x13bc0000) Event: 572398.655 GC heap after Heap after GC invocations=5048 (full 357): PSYoungGen total 201472K, used 1998K [0x573c0000, 0x63bc0000, 0x63bc0000) eden space 198144K, 0% used [0x573c0000,0x573c0000,0x63540000) from space 3328K, 60% used [0x63880000,0x63a73830,0x63bc0000) to space 3328K, 0% used [0x63540000,0x63540000,0x63880000) ParOldGen total 843776K, used 423703K [0x23bc0000, 0x573c0000, 0x573c0000) object space 843776K, 50% used [0x23bc0000,0x3d985cc0,0x573c0000) PSPermGen total 262144K, used 51848K [0x03bc0000, 0x13bc0000, 0x23bc0000) object space 262144K, 19% used [0x03bc0000,0x06e62138,0x13bc0000) } Event: 576881.689 GC heap before {Heap before GC invocations=5049 (full 357): PSYoungGen total 201472K, used 200142K [0x573c0000, 0x63bc0000, 0x63bc0000) eden space 198144K, 100% used [0x573c0000,0x63540000,0x63540000) from space 3328K, 60% used [0x63880000,0x63a73830,0x63bc0000) to space 3328K, 0% used [0x63540000,0x63540000,0x63880000) ParOldGen total 843776K, used 423703K [0x23bc0000, 0x573c0000, 0x573c0000) object space 843776K, 50% used [0x23bc0000,0x3d985cc0,0x573c0000) PSPermGen total 262144K, used 51850K [0x03bc0000, 0x13bc0000, 0x23bc0000) object space 262144K, 19% used [0x03bc0000,0x06e62850,0x13bc0000) Event: 576881.696 GC heap after Heap after GC invocations=5049 (full 357): PSYoungGen total 201472K, used 3155K [0x573c0000, 0x63bc0000, 0x63bc0000) eden space 198144K, 0% used [0x573c0000,0x573c0000,0x63540000) from space 3328K, 94% used [0x63540000,0x63854cb0,0x63880000) to space 3328K, 0% used [0x63880000,0x63880000,0x63bc0000) ParOldGen total 843776K, used 423703K [0x23bc0000, 0x573c0000, 0x573c0000) object space 843776K, 50% used [0x23bc0000,0x3d985cc0,0x573c0000) PSPermGen total 262144K, used 51850K [0x03bc0000, 0x13bc0000, 0x23bc0000) object space 262144K, 19% used [0x03bc0000,0x06e62850,0x13bc0000) } Event: 580535.452 GC heap before {Heap before GC invocations=5050 (full 357): PSYoungGen total 201472K, used 201299K [0x573c0000, 0x63bc0000, 0x63bc0000) eden space 198144K, 100% used [0x573c0000,0x63540000,0x63540000) from space 3328K, 94% used [0x63540000,0x63854cb0,0x63880000) to space 3328K, 0% used [0x63880000,0x63880000,0x63bc0000) ParOldGen total 843776K, used 423703K [0x23bc0000, 0x573c0000, 0x573c0000) object space 843776K, 50% used [0x23bc0000,0x3d985cc0,0x573c0000) PSPermGen total 262144K, used 51856K [0x03bc0000, 0x13bc0000, 0x23bc0000) object space 262144K, 19% used [0x03bc0000,0x06e64228,0x13bc0000) Event: 580535.459 GC heap after Heap after GC invocations=5050 (full 357): PSYoungGen total 200960K, used 1858K [0x573c0000, 0x63bc0000, 0x63bc0000) eden space 197632K, 0% used [0x573c0000,0x573c0000,0x634c0000) from space 3328K, 55% used [0x63880000,0x63a50be0,0x63bc0000) to space 3584K, 0% used [0x634c0000,0x634c0000,0x63840000) ParOldGen total 843776K, used 423703K [0x23bc0000, 0x573c0000, 0x573c0000) object space 843776K, 50% used [0x23bc0000,0x3d985cc0,0x573c0000) PSPermGen total 262144K, used 51856K [0x03bc0000, 0x13bc0000, 0x23bc0000) object space 262144K, 19% used [0x03bc0000,0x06e64228,0x13bc0000) }