Resin Documentation

health report

Resin Professional includes a PDF report of your system's health.

test

The quickest method to get a health report is to use the command-line "pdf-report". The pdf-report will ask the server to generate a report immediately.

unix> resinctl pdf-report

generated /var/resin/log/default-Summary-20110921T1218.pdf

<resin xmlns="http://caucho.com/ns/resin"
          xmlns:resin="urn:java:com.caucho.resin">
<cluster id="">

  <resin:import path="${__DIR__}/health.xml"/>
  
  <health:PdfReport>
     <path>${resin.root}/doc/admin/pdf-gen.php</path>
     <report>Summary</report>
     <snapshot/>
     <mailto>user@example.com</mailto>
     <profile-time>60s</profile-time>
     
     <health:IfCron value="0 0 * * 0"/>
   </health:PdfReport>

   ...
</cluster>   
</resin>   

<resin xmlns="http://caucho.com/ns/resin"
          xmlns:resin="urn:java:com.caucho.resin">
<cluster id="">

  <resin:import path="${__DIR__}/health.xml"/>
  
  <health:PdfReport>
     <path>${resin.root}/doc/admin/pdf-gen.php</path>
     <title>Restart</title>
     <watchdog/>
     <mailto>user@example.com</mailto>
     
     <health:OnRestart/>
   </health:PdfReport>

   ...
</cluster>   
</resin>   

• Summary: The key JVM, OS and Resin versions.
• Meter Graphs: Graphs of statistics over the reported time.
• Heap Dump: The top heap memory usage.
• CPU Profile: A CPU profile of the system, showing the most active locations.
• Thread Dump: All the threads in Resin at the time the snapshot was taken.
• Log Report: The most recent "warning" level messages from java.util.logging.
• JMX Dump: Data from all the JMX mbeans at the time the snapshot was taken.

The reports are designed be used in several situations:

• bug report/support: for Resin's own support, it's much easier to debug a problem when we have a full snapshot to work from.
• restart/crach analysis: if the JVM crashes for some reason, the watchdog system report can help track down the cause.
• CPU and performance: if the JVM shows unexpected CPU use or sluggish performance, the report can help show the main causes.
• thread spikes and locking: the thread dump and CPU can be used to track down locked code.

The heap dump gives a quick overview of the memory allocation of the system. It's generally useful as a check for any unusual allocations.

The heap dump is sorted by "self+desc", which is the objects own size plus its descendant size. A java.lang.String for example would include the char[] as part of its "self+desc".

The following is an example of a basic idle Resin heap dump.

 Class Name                                self+desc  self  count

 byte[]                                     22.23M  22.23M   14741
 com.caucho.db.block.Block                  20.28M  134.9K    2410
 char[]                                     13.89M  13.89M  122606
 com.caucho.util.LruCache                    7.52M   30.4K     317
 java.lang.String                            7.15M   1.97M   61426
 ...

The first items, the byte[] and Block are primarily Resin's internal proxy cache and distributed cache database. Notice that the "self" for the Block is much smaller than its. "self+desc", because each Block has a large byte[] buffer.

Similarly, the String "self+desc" is much large than its "self" because it includes the char[] buffer.

For CPU problems and stuck threads, the Thread Dump will show what each thread is doing in the system. You can use the report to see if many threads are piled up in an unusual location, like and unexpected lock, or track down a spinning thread.

The thread dump report merges threads which share the same stack trace. The merged threads will all be listed together, followed by their stack trace.

Blocked threads and the lock's owning thread are grouped together, so it's easier to see which thread is preventing many threads from continuing.

The following example shows a normal blocking situation. The JDK's SSL implementation only allows one thread to accept a connection at a time. All other threads will wait for the first thread. In this case, the thread named "http://*:8444-17" owns the SocksSocketImpl. Three threads are waiting in line for the lock: "http://*:8444-1", "http://*:8444-1", and "http://*:8444-1".

http://*:8444-17
    java.net.PlainSocketImpl.socketAccept
    -- locked java.net.SocksSocketImpl@1199747469
    java.net.PlainSocketImpl.accept
    java.net.ServerSocket.implAccept
    ...
    com.caucho.env.thread.ResinThread.runTasks
    com.caucho.env.thread.ResinThread.run
  
http://*:8444-1
  waiting on java.net.SocksSocketImpl@4782b18d owned by [126] http://*:8444-17
http://*:8444-10
  waiting on java.net.SocksSocketImpl@4782b18d owned by [126] http://*:8444-17
http://*:8444-11
  waiting on java.net.SocksSocketImpl@4782b18d owned by [126] http://*:8444-17
    java.net.PlainSocketImpl.accept
    java.net.ServerSocket.implAccept
    com.sun.net.ssl.internal.ssl.SSLServerSocketImpl.accept
    ...
    com.caucho.env.thread.ResinThread.runTasks
    com.caucho.env.thread.ResinThread.run

...

Because the CPU profile is calculated by repeated thread dumps, it's possible for a single stack trace to have more than 100% of the time when multiple threads are waiting at the same place.

You may need to skip the first set of waiting threads to see the profile traces you're interested in.

CPU Profile

Time:    60.1s
GC-Time: 0.303s
Ticks:   601
Sample-Period: 100
End: 2011-09-21 16:33

5000.00% 3005.00s  com.caucho.vfs.JniSocketImpl.nativeAccept()  RUNNABLE (JNI)
                   com.caucho.vfs.JniSocketImpl.nativeAccept()
                   com.caucho.vfs.JniSocketImpl.accept()
                   com.caucho.vfs.JniServerSocketImpl.accept()
                   com.caucho.network.listen.TcpSocketLinkListener.accept()
                   com.caucho.network.listen.TcpSocketLink.accept()
                   com.caucho.network.listen.TcpSocketLink.handleAcceptTask()

3564.23% 2142.10s  com.caucho.env.thread.ResinThread.waitForTask()   WAITING
                   sun.misc.Unsafe.park()
                   java.util.concurrent.locks.LockSupport.park()
                   com.caucho.env.thread.ResinThread.waitForTask()
                   com.caucho.env.thread.ResinThread.runTasks()
                   com.caucho.env.thread.ResinThread.run()

...

The most recent warning logs are reported as part of the heap dump.

Log(Warning)

2011-09-21 11:06:07 warning  WarningService: Resin restarting due to
                                    configuration change

The JMX dump includes the full report of all the JMX MBeans in the system along with their values. Using this part of the report is somewhat specialized, either checking configured values against expectations or looking at statistics that aren't graphed as part of the metering system.

The JMX beans are sorted alphabetically.

JMX Dump

JMImplementation:type=MBeanServerDelegate
   ImplementationName     Resin-JMX
   ImplementationVendor   Caucho Technology
   ImplementationVersion  Resin-4.0.s110921
   MBeanServerId          Resin-JMX
   SpecificationName      Java Management Extensions
   SpecificationVendor    Sun Microsystems
   SpecificationVersion  1.4
    
com.sun.management:type=HotSpotDiagnostic
  ...