实时跟踪和调整忙碌的Tomcat服务器

ApacheTomcat enjoys the reputation of being a small but very formidableservlet container capable of handling heavy production loads. Its mostcommon role probably is as the application server for lightweight J2EEapplications (servlets, plain old Java objects, and no EJBs). In such arole, it often handles significant production loads, for which thesetypes of lightweight applications are best suited.Tomcat's defaultinstallation, though, is configured to handle medium loads. Running anapplication on Tomcat in high-load environments requires furthertuning. I recently worked on a project that presented this exactproblem. We had to configure Tomcat servers to handle significanttraffic (maximum sustained load was 1,450 requests per second generatedby 150 or more unique, concurrent visitors, divided across 2–4 servers)and determine how appropriate Tomcat's configuration is for changingload demands such as traffic spikes and continuously increasing loads.
Thekey finding we discovered was that the configuration and size ofTomcat's resource pools had a significant effect on the overallscalability and performance of the server. Tomcat with properlyconfigured resource pools is capable of handling heavy Web traffic withsustained performance. However, how do you know exactly what theappropriate size of these resource pools would be, as well as how totrack their usage in real time? A pool that is too small obviouslywould be a bottleneck that directly affects the end user experience. Apool that is too large would consume vital system resources such as CPUand/or memory and potentially threaten the stability of the platform.
Whiletrying to determine those crucial parameters and ratios, I developed anapproach that helped us not only track and understand the mostappropriate server capacity-related settings but also open the door toother similar real-time tracking and monitoring approaches. Thisarticle reviews the merits of my proposed technique and discusses theexact steps for implementing something similar in your applications.
Tomcat Resource PoolsFirst,a review of the types and roles of Tomcat component pools is in order.A resource pool is a pool of reusable objects that are vital forapplication processing yet expensive to instantiate on demand. Tomcat'sconnector thread pool and database connection pool are two such pools.They have a direct impact on the overall throughput of the server aswell as on the individual applications.
Connector Thread PoolTheconnector pool is a pool of threads that accepts connections fromstandard Tomcat connector ports (e.g., 8080 for HTTP, 8009 for AJP) andhands them off to the processing components. This pool plays anessential role in Tomcat's processing throughput capacity. Every singlerequest directed into Tomcat goes through the connector pool. If thepool is too small and too many requests get backlogged, the requestgets denied.
If this pool has too few activerequest-processing objects and the traffic increases dramatically, adelay in request processing results due to pool componentinstantiation. If the pool is too large (i.e., the number of readythreads is too high), CPU cycles and memory may become overused.
Therefore,knowing how to set the proper size and other parameters of this pool iscrucial for the Web application to function properly under heavy loads.By default, the connector pool is preconfigured with the followingsettings:
Minimum number of spare threads (Number of threads that are always in the pool) – 4
Maximum number of threads in the pool – 200
Database Connection PoolThedatabase connection pool is an essential component to all J2EEapplications that utilize pooled JDBC connections, which in myexperience is the majority of J2EE applications. The databaseconnection pool, as provided by Tomcat's implementation of Jakarta'sDBCP, follows a similar configuration pattern as the connector pool. Bydefault, the database pool always holds three idle connections and amaximum of 15 active connections. If the connections are exhausted, theapplication will not be able to get a connection to the database,causing run time errors that are potentially fatal for the application.
Resource Pools Under Heavy LoadsBothof these pools, under heavy loads, can quickly become major performancebottlenecks—a hard limiting factor in application scalability. At thesame time, with the proper knowledge of real-time resource utilization,you can achieve amazing results in application responsiveness andscalability. The challenge is performing appropriate datacollection and monitoring of pooled resources under heavy loads in realtime and with low overhead. The solution is a low-overhead,near-real-time monitoring and data collection component, which givesyou the best picture of Tomcat's pooled resources utilization. Withthat information, you can configure Tomcat's pools to handle anyprojected demands with relative ease.
Before diving into theproposed solution, get to know the Tomcat components that will play keyroles in the custom performance-monitoring component.
The Value of Tomcat ValvesValveis a core architectural component of the Tomcat server. Its mainpurpose is to provide a standard, flexible, and extensible way fortying the server or user-defined events to requests coming into aTomcat server on a per-request basis. In that sense, Valves are similarto filters in Java Web applications.You can chain them together to perform some action upon each request.This per-request nature is the key ingredient for the customperformance monitoring mechanism.
In a Web environment,it is far more meaningful to track server parameters on a per-requestbasis than on some pre-determined time interval. Due to the erratic,spike-prone nature of the Web, the server may be idle one moment andthen flooded with requests the next. For that reason, Valve providesfar better and more pertinent information than regular intervaltracking, in terms of how Tomcat's resources are utilized on arequest-by-request basis.
Using Tomcat MBeans for Resource MonitoringTomcat server ships with a rich set of MBean objects (see JMX standard)that provide unprecedented access to the server's control andmanagement functions, as well as to server-managed resources anddeployed applications. While JMX-based server control andmanagement functions are powerful and comprehensive, using JMX formanaging and monitoring production servers can be controversial.Typical JMX-based management requires client software and remoteconnectivity components. These components, such as JConsole andRMI-based JMX servers, can add overhead to already busy productionmachines. In addition, monitoring may require relaxation of stringentsecurity rules. Because of these reasons, the solution that thisarticle proposes does not require complete JMX support. (Working withTomcat's JMX features is beyond the scope of this article. See thepreviously published DevX article on best practices for monitoring Tomcat using MBeans.)
Insteadof accessing MBeans through the external console, monitoring Valveaccesses the local MBeans directly on the server without incurring anyoverhead or interfering with the security rules of the system. Thissolution is concerned only with the MBeanServer object and basicattribute retrieval of resource pool MBeans. For the JDBC Pool, it usesthe MBean identified in Table 1.
[table][tr]DataSource MBean[/tr][tr]PropertyValueDescription[/tr][tr][td]name[/td][td]"jdbc/testDB"[/td][td]Must match the name of the JDBC datasource in context.xml file[/td][/tr][tr][td]class[/td][td]"javax.sql.DataSource"[/td][td]Constant value[/td][/tr][tr][td]host[/td][td]"localhost"[/td][td]Name of the host server as defined in server.xml file[/td][/tr][tr][td]path[/td][td]"/Test"[/td][td]Path of the Web application to which the JDBC source is attached[/td][/tr][tr][td]type[/td][td]"DataSource"[/td][td]Constant value[/td][/tr][/table][table][tr][td]Table 1. MBean Properties for JDBC Pool in Resource Monitoring Solution[/td][/tr][/table]For the Connector Threads Pool, it uses the MBean identified in Table 2.
[table][tr]Connector MBean[/tr][tr]PropertyValueDescription[/tr][tr][td]name[/td][td]"jk-8009"[/td][td]Thisis the connector type that you want to track. Two common connectors arejk-8009 for Tomcat AJP connector and http-8080 for HTTP connector. Usejk-8009 if Tomcat is attached to the Web server via mod_jk, otherwiseuse http-8080.[/td][/tr][tr][td]type[/td][td]"ThreadPool"[/td][td]Constant value[/td][/tr][/table][table][tr][td]Table 2. MBean Properties for Connector Threads Pool in Resource Monitoring Solution[/td][/tr][/table]For the JDBC DataSource MBean, the solution reads out the following MBean attributes:
numActive – number of currently active JDBC connections for the pool maxActive – maximum number of available connections For the connector thread pool MBean, the solution reads out the following:
currentThreadsBusy – number of threads currently busy processing incoming requests currentThreadCount– total number of loaded threads ready to process requests (Thesethreads are available to get immediately busy with spikes of traffic.)
How It Works
Aseach request comes into Tomcat connector, the custom,performance-tracking Valve gets invoked. This Valve gathers informationfrom the MBeans about the utilization of the pooled resources and thenprints that information into a log file, along with the timestamp andother potentially useful information about the nature of the request.This log file is available for monitoring in real time, as well as forlater analysis of the server's performance and capacity. For mysolution, I chose to modify the AccessLogValve supplied with the Tomcatsource code. AccessLogValve produces a log file formatted according toWeb logging standards, so its output looks identical to the standardApache logs. Since these log statements contain very useful informationby themselves, I preferred to keep them and just append extra resourcepool state information to the log output. Of course, you can choosewhatever information you want logged.
The following are the specific implementation steps for the solution:
Implementa custom resource-tracking Valve. Tomcat supplies a predefinedBaseValve that you can extend for this purpose. It is located in the catalina.jar file. Add this jar file to your classpath and have your custom Valve extend it as follows:
public class ResourceTrackingAccessLogValve
extends ValveBase
implements Lifecycle {
(Note:I also implemented the Lifecycle interface. That is not important. Itjust adds a marks object with extra manageability features.)
When extending the base Valve, override the method as follows:
public void invoke(Request request, Response response)
throws IOException, ServletException {
This contains the code that will query MBeans for resource utilization and log that information to the log files.
Inside the invoke method, access the resource pools MBeans to read the performance attributes as follows (See the accompanying source code and its comments for details):
//Lazy instantiate MBeanServer using Tomcat's core
// MbeanUtility class
if ( this.mbeanServer == null ) {
this.mbeanServer = MBeanUtils.createServer ();
}//end if
//Get the instance of the db pool MBean object
// by fetching it through the ObjectName (connectionPoolMBeanID)
if ( this.dbPoolObjectName == null ) {
this.connectionPoolMBeanID.put( "name","\"" + this.getJdbcName() + "\"" );
this.connectionPoolMBeanID.put( "class", "javax.sql.DataSource" );
this.connectionPoolMBeanID.put ( "host", this.getHost() );
this.connectionPoolMBeanID.put( "path", this.getApplicationPath() );
this.connectionPoolMBeanID.put( "type", "DataSource" );
this.dbPoolObjectName = ObjectName.getInstance( this.domain ,
this.connectionPoolMBeanID );
}
MBeanInfo mBeanInfo = mbeanServer.getMBeanInfo ( this.dbPoolObjectName );
MBeanAttributeInfo attributeInfo[] = mBeanInfo.getAttributes();
for (int i = 0; i
Log out the performance information each time the Valve is accessed, as follows:
result.append( " | busy threads: " + this.threadsBusyCount );
result.append( " active threads: " + this.threadsCurrentCount );
result.append( " active db connections: " + this.dbConnectionCount );
result.append ( " max db connections : " + this.dbMaxSize );
These log statements will produce output in the log file that looks similar to this:
127.0.0.1 [29/Aug/2006:12:58:05:265 -0400] 12 200 896 /Test/ GET - - - - | busy
threads: 2 active connector threads: 51 active db connections: 8 max db connections : 30
Register your Valve. Insert the following statement into Tomcat's server.xml file, right before the closing tag:
directory="logs" prefix="localhost_resource_log." suffix=".txt"
c
domain="Catalina"
applicati
pattern="%a %t %D %s %b %U %q %m" resolveHosts="false"/>
Thisdirective instructs Tomcat to enable your custom Valve, and itconfigures the Valve to produce standard Web server log output into a localhost_resource_log.txt file.
Installthe Valve. I chose to deploy the custom Valve as a jar file and placeit inside the Tomcat's server/lib directory. Putting this jar into theserver's classpath provides it access to all of the Tomcat internalclasses that the custom monitoring process utilizes.
Parseand interpret the logs. As you run load and stress tests against yourapplication, ResourceTrackingAccessLogValve will fill the localhost_resource_log.txt file with invaluable information related to the resource consumption and pooled resources utilization at run time.
Thisinformation will be of tremendous value for your productionconfiguration efforts. Observe the maximum number of threads, and takenote of the ratio between busy connector threads and active databaseconnections. This ratio will tell you a lot about the health of yourapplication, not only in terms of what your application consumes undervarious loads, but also how truly scalable your application is.Scalable applications tend to stay conservative on database connectionusage as the number of Web connections grows.
Further FunctionsThetechnique described in this article provides a practical understandingof Tomcat's resource utilization under heavy loads. In my experience,understanding key configuration parameters such as resource pools andunderstanding the real-time needs of production applications areessential to running scalable and responsive applications. However, youcan use my proposed approach for more than just tracking resourceutilization.
If you examine the Valve included with theaccompanying source code and spend some time learning the access methodfor MBeans, you will probably find it useful for some other needs suchas security tracking and pure performance monitoring.
I alsostrongly encourage you to use tools such as JConsole to explore therichness of Tomcat's MBeans API and learn more about the internals ofthe server as well as the runtime behavior of the application.