Using jstat to report custom JVM metric sets

I've always been missing possibility to configure custom headers in JStat. Of course there are a lot of predefined data sets, but it'll be nicer if we could create our own data set. And as you probably already devised I'm writing this post because such functionality is of course available :) Unfortunately I haven't found it in any documentation so now I'll try to fill this gap.

First thing we have to do is to provide custom descriptor with possible JStat options. This descriptor is just a text file containing something we'll call "jstat specification language". To make this custom file available to JStat we should place it in the following path: $HOME/.jvmstat/jstat_options
If you want to view the bundled options please refer to file in OpenJDK repository.

The specification language is pretty similar to json files, and it contains the group of option elements. Each option should be threaten as a set of columns that can be shown in single jstat execution. Just to name some of the predefined options: gcnew, gccause or printcompilation.

Each option element consists of several column segments. I think it's quite obvious what column means :) And whats the most important in this descriptor is just a column specification.

Each column must contain at least two nodes: header and data. Header is used to describe the column and can be aligned using special char ^ which I'll call "the grip". Grip means that it sticks the header with particular side of the column, so:

• ^Name will be aligned to the left,
• ^Name^ will be centered,
• Name^ will be aligned to the right.

The next important node is the data column. It uses PerfCounter metrics and is able to make some basic arithmetic operations - like add, minus, divide and multiply as well as use parenthesis to group operations . You can also group If you want to see all metrics that are available via this mechanism just can invoke
$jcmd <PID> PerfCounter.print
and see the output values.

Sample minimum file content can be like that:

option customgc {
  column {
    header "Tenuring"
    data sun.gc.policy.tenuringThreshold
  }
}

When we'll invoke it using
$jstat -customgc <PID> 1s 3
we'll see something like:

Tenuring
6
4
5

We can also use the operations to show for example joint young generation usage:

option customgc {
  column {
    header "YoungC"
    data sun.gc.generation.0.space.0.used + sun.gc.generation.0.space.1.used + sun.gc.generation.0.space.2.used
  }
}

There are also four additional columns that are used to setup layout for our column.

1. First is for alignments setting. We can choose if we want to align our data to left, center or right by setting align element to one of above values. 
2. In the case of number metrics we can specify string used as DecimalFormat input by entering it in the format node.
3. We're also able to specify the size of the column by adding width element with particular length.
4. Last but not least is a scaling functionality. Because most of the metrics contain just raw output from JVM we need to transform it a little bit to make if useful for human eye. This can be done with the use of scale attribute set to one of the below values (token column).

   token	factor	desc
   raw	1	no scaling
   percent	1/100	convert to percentage
   K	1024	kilo
   M	1024*1024	mega
   G	1024*1024*1024	giga
   n	10^-9	nano
   u	10^-6	micro
   m	10^-3	milli
   us	10^-6	microseconds
   ms	10^-3	milliseconds
   s	1	seconds
   min	1/60	minutes
   h	1/3600	hour

Now let's see the polished example that will show how we can use additional properties:

option customgc {
  column {
    header "YoungC^"
    data sun.gc.generation.0.space.0.used + sun.gc.generation.0.space.1.used + sun.gc.generation.0.space.2.used
    align right
    scale M
    width 7
    format "0.0"
  }
  column {
    header "OldC^"
    data sun.gc.generation.1.space.0.used
    align right
    scale M
    width 7
    format "0.0"
  }
}

Which produces

 YoungC    OldC
   67.7   161.0
   37.8   165.4
   92.2   182.8

End of topic :) Good luck!