JVM Application and Thread Profiling

The JVM from Sun has a number of builtin functions that work across all the platforms supported and that includes Linux. One very useful utility within the JVM to monitor performance is the HProf profiler tool. It tracks the calls made within the system and can dump a summary of threads and call stack ordered by some statistic (usually CPU time) when desired.

To get help on this utility, just run:

% java -Xrunhprof:help

at the command line to get help. The runhprof utility is perhaps the lightest weight and least intrusive of profiling tools for the developer and often can pinpoint the most offending call in the Java program. Usage is fairly simple. Modify the command line startup of your application by inserting the following:

% java -Xrunhprof:cpu=samples,file=hprof.txt  [rest of your commandline]

To get an output of statistics at a certain point, just do a process status (ps -ax) to get the PID of the Java process. Then send a kill -3 signal to the PID and then view the hprof.txt output.

Linux System Performance

The JVM is like any other application on Linux and it's impact on system performance can be monitored. A lot of folks rely on /proc tools like top which is quite the handy tool. By default, top provides some default information on the JVM process like Resident Memory Set Size, Total Memory Size, Shared Segments, percent CPU, Runtime Status, etc.

System monitoring can basically show the impact of what running the JVM has on resources, such as cpu and memory. There are other utilities like the vmstat, mpstat, iostat and netstat that can also show system, network and I/O utilization by the application running on the system and many times that can help point to certain problems that can then lead to possible solutions.

Is it the Java or the Operating System?

To start assessing if a performance problem is due to Linux or Java, one starts by looking at the percentage of cputime and how it's split. Perhaps folks have been wondering what system versus user time meant in the top and vmstat/mpstat outputs. In general, system time is the part that spent in executing system calls, while the time spent in application code is called user time.

There are several hundred library calls in Linux and other UNIX-like OSes that provide system level services. These might be things that handle low level read(2) or write(2) calls for I/O. And if an inordinate amount of time is spent in servicing system calls, the system time can represent a very high fraction of performance. In many cases, if system time is more than 15% of total CPU, there's a good chance that some kernel or library tuning can significantly improve performance. If, however, one notices that the user time is more than 85%, then the place to look is more likely at inefficiencies either within the JVM itself or the Java code.

These are generalities and they are sometimes mistaken. In fact, if your application does a lot of I/O, it may in fact make a lot of system calls and the statistics will point to a high system time. But that may be normal for an application that does a lot of I/O.

-- JamesCLiu - 22 Jan 2004