For a while I’ve been looking for a way to do some synthetic benchmarks for multiple Java JVM versions and vendors on my ARM SBCs since I run some services that depend on it, mostly ElasticSearch but also a Minecraft server.

I did some simple benchmarks a while back comparing the versions of Go on ARM to check that version 1.11 brought improvements. The results can be seen on this GIST.

Since I’m in the process of updating my ElasticSearch stack images (and will attend Elastic Engineer training soon) I wanted to check which JVM would bring better results. According to Elastic Support Matrix, the latest version (6.5.4), is supported on JDK 1.8 and JDK 11.

After lots of googling and failures to find a Java application that I could successfully build and run on multiple JVMs, I found DaCapo Benchmark Suite, a suite packed in a .jar file that makes everyone's life easier.

DaCapo provides multiple synthetic benchmarks and while I didn't delve too much to see what each focus, I elected some to run on my server.

The SBC used is a Firefly RK3399, with a Rockchip 3399 processor that contains two A72 cores and four A53 cores with 4GB of RAM. Below some system info on it:

All tests were done on the same board with most of its services stopped. This is in no way a scientific or official benchmark, just a measure of performance for my use using the same board for multiple JVMs.

The Java JVMs I chose to test were:

  • Oracle Java 1.8.0_172 — docker pull carlosedp/debian-oraclejava:8–172
  • OpenJDK Java 1.8.0_181 — docker pull openjdk:8u181-jdk-alpine
  • OpenJDK Java 1.8.0_191 — docker pull openjdk:8u191-jdk-alpine
  • OpenJDK Java 11.0.1 — docker pull openjdk:11.0.1-jdk-stretch

The tests were run under Docker containers for each of the JVMs where the OpenJDK ones were the official images from DockerHub and the Oracle Java image is the one I built for my ElasticSearch stack.

For each JDK, I pulled the image and mounted my local dir where the DaCapo suite was downloaded. I then run the suite 5 times and recorded all times printed by the suite. Below are the commands used to run the tests:

Below is the info on each JVM:

Oracle Java 1.8.0_172

OpenJDK Java 1.8.0_181

OpenJDK Java 1.8.0_191

OpenJDK Java 11.0.1

The output of each benchmark is similar to:

Here is the raw data I collected into a spreadsheet and a graph showing the results.

Times in ms, less is better

I skipped a couple tests on some JVMs since the results were already obvious. Also I saw some crashes from some tests, like the eclipse one on JDK 11.

In the end, I saw that both versions of the JDK 1.8 were almost on par followed by the JDK 11 with still some gap on performance.

I chose to migrate my images from the custom Oracle Java 1.8_172 to the OpenJDK 1.8_181 due to better numbers and wait until the JDK 11 is performant enough to use it.

Send me your feedback ou suggestions on improved tests or results seen in your own labs in https://twitter.com/carlosedp.

Writing everything cloud and all the tech behind it. If you like my projects and would like to support me, check my Patreon on https://www.patreon.com/carlosedp