Java Benchmarks on ARM64

For a while I’ve been looking for a way to do some synthetic benchmarks for multiple Java JVM versions and vendors on my ARM SBCs since I run some services that depend on it, mostly ElasticSearch but also a Minecraft server.

I did some simple benchmarks a while back comparing the versions of Go on ARM to check that version 1.11 brought improvements. The results can be seen on this GIST.

Since I’m in the process of updating my ElasticSearch stack images (and will attend Elastic Engineer training soon) I wanted to check which JVM would bring better results. According to Elastic Support Matrix, the latest version (6.5.4), is supported on JDK 1.8 and JDK 11.

After lots of googling and failures to find a Java application that I could successfully build and run on multiple JVMs, I found DaCapo Benchmark Suite, a suite packed in a .jar file that makes everyone's life easier.

DaCapo provides multiple synthetic benchmarks and while I didn't delve too much to see what each focus, I elected some to run on my server.

The SBC used is a Firefly RK3399, with a Rockchip 3399 processor that contains two A72 cores and four A53 cores with 4GB of RAM. Below some system info on it:

All tests were done on the same board with most of its services stopped. This is in no way a scientific or official benchmark, just a measure of performance for my use using the same board for multiple JVMs.

The Java JVMs I chose to test were:

• Oracle Java 1.8.0_172 — docker pull carlosedp/debian-oraclejava:8–172
• OpenJDK Java 1.8.0_181 — docker pull openjdk:8u181-jdk-alpine
• OpenJDK Java 1.8.0_191 — docker pull openjdk:8u191-jdk-alpine
• OpenJDK Java 11.0.1 — docker pull openjdk:11.0.1-jdk-stretch

The tests were run under Docker containers for each of the JVMs where the OpenJDK ones were the official images from DockerHub and the Oracle Java image is the one I built for my ElasticSearch stack.

The Execution

For each JDK, I pulled the image and mounted my local dir where the DaCapo suite was downloaded. I then run the suite 5 times and recorded all times printed by the suite. Below are the commands used to run the tests:

$ docker run -it --rm -v $(pwd):/test openjdk:8u181-jdk-alpine sh
$ cd /test
for X in `seq 1 5`; do java -jar dacapo.jar lusearch; done 2>&1 >/dev/null |grep PASSED
for X in `seq 1 5`; do java -jar dacapo.jar pmd; done 2>&1 >/dev/null |grep PASSED
for X in `seq 1 5`; do java -jar dacapo.jar sunflow; done 2>&1 >/dev/null |grep PASSED
for X in `seq 1 5`; do java -jar dacapo.jar avrora; done 2>&1 >/dev/null |grep PASSED
for X in `seq 1 5`; do java -jar dacapo.jar xalan; done 2>&1 >/dev/null |grep PASSED
for X in `seq 1 5`; do java -jar dacapo.jar jython; done 2>&1 >/dev/null |grep PASSED
for X in `seq 1 5`; do java -jar dacapo.jar eclipse; done 2>&1 >/dev/null |grep PASSED

Below is the info on each JVM:

Oracle Java 1.8.0_172

$ docker run -it --rm -v $(pwd):/test carlosedp/debian-oraclejava:8-172 sh
java version "1.8.0_172"
Java(TM) SE Runtime Environment (build 1.8.0_172-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.172-b11, mixed mode)

OpenJDK Java 1.8.0_181

$ docker run -it --rm -v $(pwd):/test openjdk:8u181-jdk-alpine sh
openjdk version "1.8.0_181"
OpenJDK Runtime Environment (IcedTea 3.9.0) (Alpine 8.181.13-r0)
OpenJDK 64-Bit Server VM (build 25.181-b13, mixed mode)

OpenJDK Java 1.8.0_191

$ docker run -it --rm -v $(pwd):/test openjdk:8u191-jdk-alpine sh
openjdk version "1.8.0_191"
OpenJDK Runtime Environment (IcedTea 3.10.0) (Alpine 8.191.12-r0)
OpenJDK 64-Bit Server VM (build 25.191-b12, mixed mode)

OpenJDK Java 11.0.1

$ docker run -it --rm -v $(pwd):/test openjdk:11.0.1-jdk-stretch sh
openjdk version "11.0.1" 2018-10-16
OpenJDK Runtime Environment (build 11.0.1+13-Debian-2bpo91)
OpenJDK 64-Bit Server VM (build 11.0.1+13-Debian-2bpo91, mixed mode)

The output of each benchmark is similar to:

===== DaCapo 9.12-MR1 xalan PASSED in 10012 msec =====
===== DaCapo 9.12-MR1 xalan PASSED in 8958 msec =====
===== DaCapo 9.12-MR1 xalan PASSED in 11354 msec =====
===== DaCapo 9.12-MR1 xalan PASSED in 9307 msec =====
===== DaCapo 9.12-MR1 xalan PASSED in 10983 msec =====

On to the results

Here is the raw data I collected into a spreadsheet and a graph showing the results.

Times in ms, less is better

I skipped a couple tests on some JVMs since the results were already obvious. Also I saw some crashes from some tests, like the eclipse one on JDK 11.

Conclusion

In the end, I saw that both versions of the JDK 1.8 were almost on par followed by the JDK 11 with still some gap on performance.

I chose to migrate my images from the custom Oracle Java 1.8_172 to the OpenJDK 1.8_181 due to better numbers and wait until the JDK 11 is performant enough to use it.

Send me your feedback ou suggestions on improved tests or results seen in your own labs in https://twitter.com/carlosedp.