Part 2: Native microservice in GraalVM

Part 2: Native microservice in GraalVM

Last week I posted Part 1 of this series of blogposts about GraalVM.

We looked at GraalVM and what it can do. We dove into the native-image command and transformed a simple HelloWorld application into a native application (running in Docker).

This time I want to go beyond “Hello World” and build something useful (despite all the limitations listed below). A CRUD microservice with REST API and database access.

TLDR; If you’re just intersted in the end result, go right to the results

Limitations

So what are some of the limitations that GraalVM currently has? (using version 1.0.0 RC6 at time of writing)

Well, the Java VM is a very complete and dynamic system able to handle a lot of dynamic changes. This is something a native application can’t do as easily. GraalVM needs to analyse your application up front and discover all these things. Therefor, things like reflection and dynamic classloading are very hard to do (but not impossible).

To ‘emulate’ a dynamic running JVM the GraalVM project is shipped with Substrate VM.

Substrate VM is a framework that allows ahead-of-time (AOT) compilation of Java applications under closed-world assumption into executable images or shared objects (ELF-64 or 64-bit Mach-O).

The Java VM for example has a garbage collector, when you eliminate the JVM, you’ll still need to free your objects. This is something Substrate VM (written in Java!) does for you.

To read about the limitations of Substrate VM, look at this LIMITATIONS.md.

Spring Boot

We want to do more than just ‘HelloWorld’, how about an entire microservice?

If you talk about microservices in Java, most people will immediately say: Spring Boot. While you can certainly discuss the micro-part, it is by far the most popular way of writing (enterprise) Java applications at the moment. The problem is that most Spring Boot applications quickly grow, having Docker images of 600+ mb and runtime memory usage of 250+ mb is to be expected.

So it seems this is a perfect candidate to turn into a native application.

But there is some instant bad news: It won’t work

At least, at this moment in time. The developers of Spring Boot are working very hard with the developers of GraalVM to fix all the problems. A lot of work has already been done, you can check out the progress in this Spring issue.

Micronaut.io

A framework that also covers the entire spectrum and does work with GraalVM is micronaut.io. If you want something that works out of the box, check out their entire stack.

But I’d like to do it enirely myself, find the pitfalls and learn about the limitations of GraalVM at the moment. This is very useful to understand what you can and can’t do. I’m going to build my own GraalVM stack!

Web alternative: SparkJava

Instead of turning to Spring Boot or micronaut, let’s keep our application really micro and use something else. Spark Framework is a small web framework. And it works like a charm using native-image.

First I created a simple Maven project and to use the native-image command I needed all the Maven dependencies as a JAR file on the class path after compilation. To have this I added the following plugin to the pom.xml:

<!-- Make all the JAR files go into the output target/lib -->
<plugin>
   <groupId>org.apache.maven.plugins</groupId>
   <artifactId>maven-dependency-plugin</artifactId>
   <executions>
      <execution>
         <id>copy-dependencies</id>
         <phase>prepare-package</phase>
         <goals>
            <goal>copy-dependencies</goal>
         </goals>
         <configuration>
            <outputDirectory>${project.build.directory}/lib</outputDirectory>
            <overWriteReleases>false</overWriteReleases>
            <overWriteSnapshots>false</overWriteSnapshots>
            <overWriteIfNewer>true</overWriteIfNewer>
            </configuration>
         </execution>
    </executions>
</plugin>

Next I added the following dependency:

<dependency>
   <groupId>com.sparkjava</groupId>
   <artifactId>spark-core</artifactId>
   <version>2.7.2</version>
</dependency>

Now we can write our code and run the HelloWorld web application:

import static spark.Spark.*;

public class HelloWorld {
    public static void main(String[] args) {
        get("/hello", (req, res) -> "Hello World");
    }
}

Building everything is the same as in Part 1.

There are two differences though. The base image can no longer be FROM scratch because we need access to networking. Second, we need to expose the port of the web application to the outside using EXPOSE 4567.

Also we need to add the following option to native-image:

-H:+ReportUnsupportedElementsAtRuntime

This option eliminates some problems during the analysis phase out of the way.

Running this Dockerfile results in a “Hello World” in the browser at http://localhost:4567/hello. Using just 4 mb of runtime memory (!).

Dependency Injection: Google Guice

Another big problem at the moment using GraalVM is trying to use dependency injection. First I tried to use Google Guice. It is marketed as a ‘low overhead’ dependency injection framework. When I fired up the native-image command I got the following exception (amongst many others):

Exception in thread "main" java.lang.AssertionError: java.lang.NoSuchMethodException: java.lang.Integer.parseInt(java.lang.String)
  at java.lang.Throwable.<init>(Throwable.java:265)
  at java.lang.Error.<init>(Error.java:70)
  at java.lang.AssertionError.<init>(AssertionError.java:58)
  at java.lang.AssertionError.<init>(AssertionError.java:74)
  at com.google.inject.internal.TypeConverterBindingProcessor.convertToPrimitiveType(TypeConverterBindingProcessor.java:130)
  at com.google.inject.internal.TypeConverterBindingProcessor.prepareBuiltInConverters(TypeConverterBindingProcessor.java:46)
  at com.google.inject.internal.InjectorShell$Builder.build(InjectorShell.java:152)
  at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:104)
  at com.google.inject.Guice.createInjector(Guice.java:96)
  at com.google.inject.Guice.createInjector(Guice.java:73)
  at com.google.inject.Guice.createInjector(Guice.java:62)
  at com.royvanrijn.graal.Main.<init>(Main.java:39)
  at com.royvanrijn.graal.Main.main(Main.java:52)
  at com.oracle.svm.core.JavaMainWrapper.run(JavaMainWrapper.java:163)

It seems that Google Guice internally uses a way to call Integer.parseInt using reflection, but GraalVM doesn’t understand this. But luckely, we can help GraalVM a bit.

To fix this first problem I added the following file to my project:

[
  {
    "name" : "java.lang.Class",
    "allDeclaredConstructors" : true,
    "allPublicConstructors" : true,
    "allDeclaredMethods" : true,
    "allPublicMethods" : true
  },
  {
    "name" : "java.lang.Integer",
    "methods" : [
      { "name" : "parseInt", "parameterTypes" : ["java.lang.String"]}
    ]
  },
  {
    "name" : "java.lang.Long",
    "methods" : [
      { "name" : "parseLong", "parameterTypes" : ["java.lang.String"]}
    ]
  }
  ... etc ...

And during the build of native-image I pass the following option:

-H:ReflectionConfigurationFiles=src/main/resources/graal_config.json

Now we have instructed Substrate VM/GraalVM that our application will do a reflection lookup to Integer.parseInt (amongst other calls). It now understands this and loads everything.

For some reason though, for each dependency I kept getting the following exception:

Could not find a suitable constructor in com.royvanrijn.graal.domain.UserRepositoryImpl. Classes must have either one (and only one) constructor annotated with @Inject or a zero-argument constructor that is not private.
  at com.royvanrijn.graal.domain.UserRepositoryImpl.class(Unknown Source)
  at java.lang.Throwable.<init>(Throwable.java:250)

The application works great running it from java but not after using native-image. Time for something different!

Dependency Injection: Dagger 2

Instead of Google Guice or Spring I decided to go with another framework: Dagger 2

This framework has one big advantage over all the others: It works compile-time.

Sure, setting it up takes a little bit more time. You’ll need to include a Maven plugin that does all the magic during compilation. But it is a perfect solution (currently) for GraalVM’s native-image. All the injection-magic is done during compilation, so when running the application everything is already nicely wired up and static.

Database access (Hibernate/Oracle)

Finally, to complete my CRUD application I tried to access our Oracle database. GraalVM and the database are both created and maintained by Oracle so I hoped this would work out of the box…. but (spoiler): It didn’t.

The main problem here is the code in Oracle’s JDBC driver, this turned out to be a very hard thing to get working, this took about an entire day!

First off, there are a lot of static initializer blocks and some of those are starting Threads. This is something the SubstrateVM’s analyzer can’t handle (again: see LIMITATIONS.md).

It was throwing errors like:

Error: com.oracle.graal.pointsto.constraints.UnsupportedFeatureException: Detected a started Thread in the image heap. Threads running in the image generator are no longer running at image run time. The object was probably created by a class initializer and is reachable from a static field. By default, all class initialization is done during native image building.You can manually delay class initialization to image run time by using the option --delay-class-initialization-to-runtime=<class-name>. Or you can write your own initialization methods and call them explicitly from your main entry point.
Trace: 
  at parsing oracle.jdbc.driver.OracleTimeoutThreadPerVM.stopWatchdog(OracleTimeoutThreadPerVM.java:64)
Call path from entry point to oracle.jdbc.driver.OracleTimeoutThreadPerVM.stopWatchdog(): 
  at oracle.jdbc.driver.OracleTimeoutThreadPerVM.stopWatchdog(OracleTimeoutThreadPerVM.java:64)
  at oracle.jdbc.driver.OracleDriver.deregister(OracleDriver.java:517)
  at oracle.jdbc.driver.OracleDriver$$Lambda$443/1007825518.deregister(Unknown Source)
  at java.sql.DriverManager.deregisterDriver(DriverManager.java:414)

Again, like before, the exception itself does provide a solution. The issue here are static initializer blocks starting threads during analysis, but this can be countered by delaying the class initialization. This isn’t trivial here because I don’t have access to the code in Oracle’s JDBC driver. But in the end I managed to get it working by adding the following parameters to the native-image command:

--delay-class-initialization-to-runtime=oracle.jdbc.driver.OracleDriver,java.sql.DriverManager,org.hibernate.jpa.HibernatePersistenceProvider

The next problem was getting the persistence.xml file to load. Hibernate is using ClassLoader.getResources() for this during runtime and for some reason I couldn’t get this to work. I knew there was a way to add resources into the native image but I struggled to get it working, the flag is called -H:IncludeResources= and you can add a regex here.

It wasn’t until I browsed the Substrate VM source code and extracted the parsing from ResourcesFeature.java. Running this code locally showed me everything I tried was wrong.

Things I tried that didn’t work:

-H:IncludeResources=target/classes/*
-H:IncludeResources=target/classes/.*
-H:IncludeResources=target/classes/META-INF/persistence.xml
-H:IncludeResources=META-INF/persistence.xml
-H:IncludeResources=*/persistence.xml

... and more and more ...

This finally worked (including all XSD’s, properties and everything in META-INF):

-H:IncludeResources=.*.properties|.*META-INF/persistence.xml|.*.xsd

It turns out the listing goes through all the JAR files and directories and matches them against a relative path which in our case is:

  • /logging.properties
  • /META-INF/persistence.xml
  • etc

So having META-INF/persistence.xml or logging.properties isn’t enough. This cost me way more time than it should have. A nice feature to add to GraalVM would be to list all the resources being added because at some point I was convinced it should just work and the problem was in my code somewhere.

Next problem: Xerces

This library gave me nightmares before as a Java developer, but luckily this time the problems could be fixed easily by adding more reflection exceptions to our -H:ReflectionConfigurationFiles=reflection.json.

  ...
  {
    "name" : "com.sun.org.apache.xerces.internal.impl.dv.xs.SchemaDVFactoryImpl",
    "methods" : [
      {
        "name" : "<init>", "parameterTypes" : []
      }
    ]
  },
  {
    "name" : "com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderFactoryImpl",
    "methods" : [
      {
        "name" : "<init>", "parameterTypes" : []
      }
    ]
  },
  (and some more, see GitHub for details)
  ...

Also xerces needed a resource bundle to load:

-H:IncludeResourceBundles=com.sun.org.apache.xerces.internal.impl.xpath.regex.message

Sigh, okay, still making progress.

Now again a problem with Hibernate and resources. I got the following StringIndexOutOfBoundsException running the application:

Caused by: java.lang.StringIndexOutOfBoundsException: String index out of range: -1
  at java.lang.Throwable.<init>(Throwable.java:265)
  at java.lang.Exception.<init>(Exception.java:66)
  at java.lang.RuntimeException.<init>(RuntimeException.java:62)
  at java.lang.IndexOutOfBoundsException.<init>(IndexOutOfBoundsException.java:56)
  at java.lang.StringIndexOutOfBoundsException.<init>(StringIndexOutOfBoundsException.java:69)
  at java.lang.String.substring(String.java:1967)
  at org.hibernate.boot.archive.internal.ArchiveHelper.getJarURLFromURLEntry(ArchiveHelper.java:45)
  at org.hibernate.jpa.boot.internal.PersistenceXmlParser.parsePersistenceXml(PersistenceXmlParser.java:256)
  at org.hibernate.jpa.boot.internal.PersistenceXmlParser.parsePersistenceXml(PersistenceXmlParser.java:240)

It turns out due to the way GraalVM labels its resources we get to a point where Hibernate is confused. It calls the following code with the wrong parameters:

Input:

  • url: “META-INF/persistence.xml”
  • entry: “/META-INF/persistence.xml”
public static URL getJarURLFromURLEntry(URL url, String entry) throws IllegalArgumentException {
   ...
   file = file.substring( 0, file.length() - entry.length());
   ...

With this input entry.length()=25 is bigger than url.length()=24 resulting in file.substring(0, -1), sigh.

To fix this I’ve created a so called shadowed class. This is a class with the exact same signature that I am compiling and adding to the classpath. Because my version of the class is loaded before the Hibernate version of the class, my version is used, I’m overriding the Hibernate version. This is obviously very ugly, but it does the job surprisingly well!

I used Math.max(0, file.length() - entry.length()) to fix getting a ‘-1’ in the substring:

public static URL getJarURLFromURLEntry(URL url, String entry) throws IllegalArgumentException {
   ...
   file = file.substring( 0, Math.max(0, file.length() - entry.length()));
   ...

And of course, a new problem pops up. Again with the resources, GraalVM seems to have put resource as a protocol of all the loaded resources. Opening the resource using an java.lang.URL caused more problems in ArchiveHelper because GraalVM doesn’t recognise ‘resource’ as a valid protocol (huh?). This meant I needed to make another small patch in the shadowed ArchiveHelper:

public static URL getJarURLFromURLEntry(URL url, String entry) throws IllegalArgumentException {
   ...
   } else if ( "resource".equals(protocol)) {
      // FIX: Added for GraalVM, just return the URL
      return url;
   }
   ...

The next big problem was getting oracle.jdbc.driver.OracleDriver to accept the fact that GraalVM doesn’t support JMX (and might never support it). The driver tried to load MXBeans and MBeans from a static initializer block, this caused major headaches…. but in the end I managed to solve this again by shadowing another class:

package oracle.as.jmx.framework;

import javax.management.MBeanServer;

/**
 * Shadowing class: oracle.as.jmx.framework.PortableMBeanFactory
 */
public class PortableMBeanFactory {

    /** Just return null, GraalVM doesn't support MBeans/JMX */
    public MBeanServer getMBeanServer() {
        return null; // Do nothing.
    }
}

Still the initial static initializer block in oracle.jdbc.driver.OracleDriver wouldn’t load and broke the native compilation. Browsing the decompiled code I noticed the following lines which might cause a problem:

...
try {
   new OraclePKIProvider();
} catch (Throwable var3) {
   ;
}
...

This class isn’t on the classpath oddly enough, so I decided to create a dummy/shadow class again, just in case:

package oracle.security.pki;

/** Another shadowing class... */
public class OraclePKIProvider {
   // Placeholder for GraalVM.
}

The next problem is Hibernate’s dynamic runtime proxies. This is something GraalVM can’t handle so we need to make sure Hibernate’s magic is done before we start our application. Luckily there is a Maven plugin which does just that:

<plugin>
   <groupId>org.hibernate.orm.tooling</groupId>
   <artifactId>hibernate-enhance-maven-plugin</artifactId>
   <version>5.3.6.Final</version>
   <executions>
      <execution>
         <configuration>
            <failOnError>true</failOnError>
            <enableLazyInitialization>true</enableLazyInitialization>
            <enableDirtyTracking>true</enableDirtyTracking>
            <enableAssociationManagement>true</enableAssociationManagement>
            <enableExtendedEnhancement>false</enableExtendedEnhancement>
         </configuration>
         <goals>
             <goal>enhance</goal>
         </goals>
      </execution>
   </executions>
</plugin>

Now we have everything in place and we can use the EntityManager to access our database and execute queries…. right?

Well it turns out, the application does start, and it comes a long way. But there is one thing I wasn’t able to fix, loading the definitions.

Hibernate has two ways of loading/mapping the actual class to the database:

  • Hbm files
  • Annotations

First I tried to use annotations, but this failed because at runtime the native (pre-loaded) classes don’t have any knowledge left of the annotations they once had.

The second method is using an HBM xml file, but this too failed. Reading the XML file again needs support for annotations and JAXB failed on me.

So we’ll have to stop here. The JDBC driver was working, so probably plain old SQL would work perfectly. Hibernate for now eludes me.

Update: Previously I mentioned Hibernate was working, this was a mistake on my part!

Docker: Multi-stage build

After Part 1 some people suggested my Dockerfiles could be made cleaner with a so called ‘multistage’ build. More information can be found here: Docker multistage

After some changes I now have a single Dockerfile with two FROM sections in it. The first section is the builder-part, the second part is the host-part. My docker build command now uses that first image to build, passes everything to the second image and builds our resulting Docker container. Really nice, I’m learning new techniques every day.

Compilation speed

One thing I noticed during this entire process, the analysis time used by native-image grew exponentially. The HelloWorld application from Part 1 took just a couple of seconds, but look at the following output:

[app:9]    classlist:  10,398.63 ms
[app:9]        (cap):   1,181.85 ms
[app:9]        setup:   2,329.11 ms
...
[app:9]   (typeflow): 477,904.73 ms
[app:9]    (objects): 2,641,498.60 ms
[app:9]   (features):  26,826.28 ms
[app:9]     analysis: 3,158,033.10 ms
[app:9]     universe: 221,809.62 ms
[app:9]      (parse):  29,723.95 ms
[app:9]     (inline):  49,021.78 ms
[app:9]    (compile): 104,756.90 ms
[app:9]      compile: 193,699.62 ms
[app:9]        image:  31,052.63 ms
[app:9]        write:   6,717.73 ms
[app:9]      [total]: 3,625,666.08 ms

It now took a whopping 60 minutes to compile to native! And during the creation of this project I needed to add a single line the reflecion.json and restart the entire build a lot of times. Transforming a project to work with native-image right now is a really time-consuming endeavour.

When compiling on my MacBook for MacOS, the compilation time is much shorter, just a couple of minutes. The problem here seems to be Docker and building for Ubuntu.

Final result: a working native microservice

I’m proud to say I now have a simple CRUD native microservice, written in Java, using Java libraries…. that is almost working. With a bit more work on the GraalVM/SubstrateVM side I’m pretty sure this could work in the near future.

It uses:

  • SparkJava
  • GSON
  • Dagger 2
  • SLF4J + java.util.logging
  • Hibernate/Oracle (almost working…)

This allows me to serve REST/JSON objects from the database with some Java code in between, what most microservices do.

All the sources are on GitHub: check it out

Check out all the code on GitHub and try it out for yourself. To get it up and running all you need it to set up a database and put the connection information in persistence.xml.

Start up time

The microservice has a very fast startup time:

Sep 25, 2018 7:22:54 PM org.eclipse.jetty.server.AbstractConnector doStart
INFO: Started ServerConnector@3650e607{HTTP/1.1,[http/1.1]}{0.0.0.0:4567}
Sep 25, 2018 7:22:54 PM org.eclipse.jetty.server.Server doStart
INFO: Started @486ms

Compare this to the Java 8 version (same code):

Sep 25, 2018 9:12:09 PM org.eclipse.jetty.server.AbstractConnector doStart
INFO: Started ServerConnector@6d5ff62e{HTTP/1.1,[http/1.1]}{0.0.0.0:4567}
Sep 25, 2018 9:12:09 PM org.eclipse.jetty.server.Server doStart
INFO: Started @3406ms

With 486ms compared to 3406ms the native version starts 7x faster.

Memory consumption

The Java version consumes 267mb of memory, while the native version takes just 20.7mb, so it is 13x smaller.

Conclusion

At the moment GraalVM is still in its infancy stage, there are still a lot of areas that can use some improvement. Most problems I’ve encountered have to do with resource loading or reflection. The build/analysis-cycle becomes pretty long when you add more and more classes and if you need to add reflection-configuration exceptions each build, this process is quite cumbersome.

Frameworks and libraries will start to notice GraalVM (once it gains more traction) and they will change their code to work better with Graal. For example the team behind Spring Boot is already actively working together with the GraalVM team to get their framework working.

Now some people are shouting to their laptops:

Why not just use Go/Rust/some other language that compiles to native by default!?

That is a very good point. If you want to use Go or Rust, go ahead!

Java is the most popular programming language in the world (according to the tiobe index). It is the language with most libraries and frameworks (except for Javascript probably). Changing entire development teams to learn Go and/or Rust, changing companies to a new language is very hard to do. Using native-image might be a more accessible way of transitioning to native backends IMO.

I’ll for sure be keeping track of GraalVM, not just because of the native-image capabilities, but also because the amazing speed of their VM.

Did you know the Oracle Database has GraalVM support? You can create queries which use Javascript functions or Java methods!

Did you know there is a Graal AOT compiler inside your JDK right now? (see: JEP-295)

Sources: All the code from this blogpost can be found here on GitHub.


Part 1: Java to native using GraalVM

Part 1: Java to native using GraalVM

One of the most amazing projects I’ve learned about this year is GraalVM.

I’ve learned about this project during Devoxx Poland (a Polish developer conference) at a talk by Oleg Šelajev. If you’re curious about everything GraalVM has to offer, not just the native Java compilation, please watch his video.

GraalVM is a universal/polyglot virtual machine. This means GraalVM can run programs written in:

  • Javascript
  • Ruby
  • Python 3
  • R
  • JVM-based languages (such as Java, Scala, Kotlin)
  • LLVM-based languages (such as C, C++).

In short: Graal is very powerful.

There is also the possibility to mix-and-match languages using Graal, do you want to make a nice graph in R from your Java code? No problem. Do you want to call some fast C code from Python, go ahead.

Installing GraalVM

In this blogpost though we’ll look at another powerful thing Graal can do: native-image compilation

Instead of explaining what it is, let’s just go ahead, install GraalVM and try it out.

To install GraalVM, download and unpack, update PATH parameters and you’re ready to go. When you look in the /bin directory of Graal you’ll see the following programs:

Here we recognise some usual commands, such as ‘javac’ and ‘java’. And if everything is setup correctly you’ll see:

$ java -version
openjdk version "1.8.0_172"
OpenJDK Runtime Environment (build 1.8.0_172-20180626105433.graaluser.jdk8u-src-tar-g-b11)
GraalVM 1.0.0-rc6 (build 25.71-b01-internal-jvmci-0.48, mixed mode)

Hello World with native-image

Next up, let’s create a “Hello World” application in Java:

public class HelloWorld {
   public static void main(String... args) {
      System.out.println("Hello World");
   }
}

And just like your normal JDK, we can compile and run this code in the Graal virtual machine:

$ javac HelloWorld.java
$ java HelloWorld
Hello World

But the real power of Graal becomes clear when we use a third command: native-image

This command takes your Java class(es) and turns them into an actual program, a standalone binary executable, without any virtual machine! The commands you pass to native-image very similar to what you would pass to java. In this case we have the classpath and the Main class:

$ native-image -cp . HelloWorld
Build on Server(pid: 63941, port: 60051)*
[helloworld:63941]    classlist:   1,236.06 ms
[helloworld:63941]        (cap):   1,885.61 ms
[helloworld:63941]        setup:   2,758.47 ms
[helloworld:63941]   (typeflow):   3,031.39 ms
[helloworld:63941]    (objects):   2,136.63 ms
[helloworld:63941]   (features):      46.04 ms
[helloworld:63941]     analysis:   5,304.17 ms
[helloworld:63941]     universe:     205.46 ms
[helloworld:63941]      (parse):     640.12 ms
[helloworld:63941]     (inline):   1,155.06 ms
[helloworld:63941]    (compile):   3,436.76 ms
[helloworld:63941]      compile:   5,594.76 ms
[helloworld:63941]        image:     749.82 ms
[helloworld:63941]        write:     653.29 ms
[helloworld:63941]      [total]:  16,753.87 ms
$ ls -ltr
-rw-r--r--  1 royvanrijn  wheel  119 Sep 20 09:36 HelloWorld.java
-rw-r--r--  1 royvanrijn  wheel  425 Sep 20 09:38 HelloWorld.class
-rwxr-xr-x  1 royvanrijn  wheel  5596400 Sep 20 09:41 helloworld
$ ./helloworld 
Hello World

Now we have an executable that prints “Hello World”, without any JVM in between, just 5.6mb. Sure, for this example 5mb isn’t that small, but it is much smaller than having to package and install an entire JVM (400+mb)!

Docker and native-image

So what else can we do? Well, because the resulting program is a binary, we can put it into a Docker image without ANY overhead. To do this we’ll need two different Dockerfile’s, the first is used to compile the program against Linux (instead of MacOS or Windows), the second image is the ‘host’ Dockerfile, used to host our program.

Here is the first Dockerfile:

FROM ubuntu

RUN apt-get update && \
    apt-get -y install gcc libc6-dev zlib1g-dev curl bash && \
    rm -rf /var/lib/apt/lists/*

# Latest version of GraalVM (at the time of writing)
ENV GRAAL_VERSION 1.0.0-rc6
ENV GRAAL_FILENAME graalvm-ce-${GRAAL_VERSION}-linux-amd64.tar.gz

# Download GraalVM
RUN curl -4 -L https://github.com/oracle/graal/releases/download/vm-${GRAAL_VERSION}/${GRAAL_FILENAME} -o /tmp/${GRAAL_FILENAME}

# Untar and move the files we need:
RUN tar -zxvf /tmp/${GRAAL_FILENAME} -C /tmp \
    && mv /tmp/graalvm-ce-${GRAAL_VERSION} /usr/lib/graalvm

RUN rm -rf /tmp/*

# Create a volume to which we can mount to build:
VOLUME /project
WORKDIR /project

# And finally, run native-image
ENTRYPOINT ["/usr/lib/graalvm/bin/native-image"]

This image can be created as follows:

$ docker build -t royvanrijn/graal-native-image:latest .

Using this image we can create a different kind of executable. Let’s create our application using the just created docker image:

$ docker run -it \
  -v /Projects/graal-example/helloworld/:/project --rm \
  royvanrijn/graal-native-image:latest \
  --static -cp . HelloWorld -H:Name=app

Build on Server(pid: 11, port: 40905)*
[app:11]    classlist:   3,244.85 ms
[app:11]        (cap):   1,023.94 ms
[app:11]        setup:   1,986.81 ms
[app:11]   (typeflow):   4,285.18 ms
[app:11]    (objects):   2,008.19 ms
[app:11]   (features):      57.07 ms
[app:11]     analysis:   6,446.49 ms
[app:11]     universe:     255.45 ms
[app:11]      (parse):     926.85 ms
[app:11]     (inline):   1,496.69 ms
[app:11]    (compile):   4,953.85 ms
[app:11]      compile:   7,689.47 ms
[app:11]        image:     806.53 ms
[app:11]        write:     573.77 ms
[app:11]      [total]:  21,160.90 ms
$ ls -ltr app
-rwxr-xr-x  1 royvanrijn  wheel  6766144 Sep 20 10:11 app
$ ./app
-bash: ./app: cannot execute binary file

This results in an executable ‘app’, but this is one I can’t start on my MacBook, because it is a statically linked Ubuntu executable. So what do all these commands mean? We’ll let’s break it down:

The first part is just running Docker:
     docker run -it

Next we map my directory containing the class files to the volume /project in the Docker image:
     -v /Projects/graal-example/helloworld/:/project --rm

This is the Docker image we want to run, the one we just created:
     royvanrijn/graal-native-image:latest

And finally we have the commands we pass to native-image inside the Docker image
We start with --static, this causes the created binary to be a statically linked executable
     --static

We have the class path and Main class:
     -cp . HelloWorld

And finally we tell native-image to name the resulting executable 'app'
     -H:Name=app

But we can do something cool with it using the following, surprisingly empty, Dockerfile:

FROM scratch
COPY app /app
CMD ["/app"]

We start with the most empty Docker image you can have, scratch and we copy in our app executable and finally we run it. Now we can build our helloworld image:

$ docker build -t royvanrijn/graal-helloworld:latest .
Sending build context to Docker daemon  34.11MB
Step 1/3 : FROM scratch
 ---> 
Step 2/3 : COPY app /app
 ---> f0894b299e8f
Removing intermediate container 37182de1ef68
 ---> 49ff43413c7a
Step 3/3 : CMD ["/app"]
 ---> Running in ea69a913d243
Removing intermediate container ea69a913d243
 ---> ab33b4d59de3
Successfully built ab33b4d59de3
Successfully tagged royvanrijn/graal-helloworld:latest

$ docker images
REPOSITORY                                                  TAG                         IMAGE ID            CREATED             SIZE
royvanrijn/graal-helloworld                                 latest                      ab33b4d59de3        5 seconds ago       6.77MB

We’ve now turned our Java application into a very small Docker image with a size of just 6.77MB!

In the next blogpost Part 2 we’ll take a look at Java applications larger than just HelloWorld. How will GraalVM’s native-image handle those applications, and what are the limitations we’ll run into?


Part 2: OpenJ9 versus HotSpot

Part 2: OpenJ9 versus HotSpot

Intro

Yesterday I compared different JDK versions and OpenJ9 versus HotSpot on memory and speed. The memory part of the test was realistic if you ask me, an actual working Spring Boot application that served REST objects.

The speed/CPU test however was… lacking. Sorting some random arrays, just one specific test.

Today I decided to test OpenJ9 and HotSpot a bit more using an actual benchmark: SPECjvm2008.

SPECjvm2008

SPEC (Standard Performance Evaluation Corporation) has a couple of well defined benchmarks and tests, including an old JVM benchmark called SPECjvm2008. This is an elaborate benchmark testing things like compression, compiling, XML parsing and much more. I decided to download this and give it a spin versus OpenJ9 and HotSpot. This should be a much fairer comparison.

Initially I encountered some issues, some of the tests didn’t work against Java 8 and the tests wouldn’t even start against Java 9+. But eventually I got it working by excluding a couple of benchmarks with the following parameters:

java -jar SPECjvm2008.jar startup.helloworld startup.compiler.compiler  startup.compress startup.crypto.aes startup.crypto.rsa startup.crypto.signverify startup.mpegaudio startup.scimark.fft startup.scimark.lu startup.scimark.monte_carlo startup.scimark.sor startup.scimark.sparse startup.serial startup.sunflow startup.xml.validation compiler.compiler compress crypto.aes crypto.rsa crypto.signverify derby mpegaudio scimark.fft.large scimark.lu.large scimark.sor.large scimark.sparse.large scimark.fft.small scimark.lu.small scimark.sor.small scimark.sparse.small scimark.monte_carlo serial sunflow xml.validation

Testing

The Docker images used in these tests are both Java 8 with OpenJDK8, but one with HotSpot underneath, the other with OpenJ9:

  • adoptopenjdk/openjdk8
  • adoptopenjdk/openjdk8-openj9

Again I started the Docker image with a directory linked to the host containing the SPEC benchmark:

  • Start Docker:
docker run -it -v /Projects/SPECjvm2008:/app/SPECjvm2008 adoptopenjdk/openjdk8-openj9 /bin/bash
  • Go to the correct directory:
cd /app/SPECjvm2008
  • Run the (working) tests:
java -Xmx600m -jar SPECjvm2008.jar startup.helloworld startup.compiler.compiler  startup.compress startup.crypto.aes startup.crypto.rsa startup.crypto.signverify startup.mpegaudio startup.scimark.fft startup.scimark.lu startup.scimark.monte_carlo startup.scimark.sor startup.scimark.sparse startup.serial startup.sunflow startup.xml.validation compiler.compiler compress crypto.aes crypto.rsa crypto.signverify derby mpegaudio scimark.fft.large scimark.lu.large scimark.sor.large scimark.sparse.large scimark.fft.small scimark.lu.small scimark.sor.small scimark.sparse.small scimark.monte_carlo serial sunflow xml.validation

Results

After waiting a long time for the benchmark to finish, I’ve got the following results:

Chart with SPECjvm2008 results

The graph is measured in ops/m, higher is better. Results may vary of course depending on hardware.

In most cases HotSpot is faster than OpenJ9, and in two cases HotSpot is much faster, crypto and derby. It appears this is a case where HotSpot is doing something special that J9 isn’t doing (yet?). This might be important to know if you’re working on applications that do a lot of cryptology, for example high performance secured endpoints.

One place where OpenJ9 came out on top is XML validation. Parsing/validation is also an important part in most modern applications, so this could be a case where J9 makes up some lost ground in actual production code.

Conclusion

Is there a real conclusion from this? I don’t think so.

The real lesson here is: Experiment, measure and you’ll know. Never decided anything based on some online benchmark.

If there is anything else you’d love me to test, send me a tweet: royvanrijn


Part 1: OpenJ9 versus HotSpot

Part 1: OpenJ9 versus HotSpot

TLDR;

OpenJ9 and IBM J9 are a different JVM implementation from the default Oracle HotSpot JVM. With the modern adoptopenjdk pre-made Docker images it is easy to swap and test different combinations and pick the right JVM for you.

The rumours seem to be true, OpenJ9 seems to blow HotSpot away on memory usage. HotSpot seems to have the edge CPU-wise.

OpenJ9

In the Java world most people are familiar with OpenJDK. This is a complete JDK implementation including the HotSpot JVM engine. Not a lot of developers know or try alternatives to HotSpot. Asking around some colleagues remembered the name JRockit, nobody mentioned IBM J9 and/or Eclipse OpenJ9.

I’ve read that OpenJ9 is very good with memory management and is tailered for usage in the cloud/in containers. OpenJ9 is an independent implementation of the JVM. It’s origins are IBM’s Java SDK/IBM J9 which can trace its history back to OTI Technologies Envy Smalltalk (thanks Dan Heidinga!).

With the current rise in microservice usage (and most services are not so micro in Java). I recon this could become a hot topic again!

Testing

Before the Docker-era it was relatively hard to compare different JVMs, versions. You needed to download, install, script and run everything. But now a lot of pre-made images are available online.

Here is my idea on how to test the JVMs:

  1. Create a simple Spring Boot application
  2. Start the application in various Docker Images
  3. Measure memory usage after startup and GC
  4. Measure the time it takes to run a small CPU-intensive test

This is by no means a thorough test or benchmark, but it should give us a basic idea of what we can expect from the virtual machines.

Spring Boot application

The Spring Boot application I created contains the following endpoints:

  1. A REST endpoint that calls the GC (trying to make it fair)
  2. A REST endpoint that creates 1000 large random arrays and sorts them, returns the runtime (in ms)

Here is the listing of the CPU-test:

@RestController
public class LoadTestController {

    @RequestMapping("/loadtest")
    public LoadTestResult loadtest() {

        long before = System.currentTimeMillis();

        Random random = new Random();

        for(int i = 0; i < 1000; i++) {
            long[] data = new long[1000000];
            for(int l = 0; l < data.length; l++) {
                data[l] = random.nextLong();
            }
            Arrays.sort(data);
        }

        return new LoadTestResult(System.currentTimeMillis() - before);
    }
}

Again, we can argue endlessly about if this test makes sense and is even remotely relevant… but still it should give us some basic idea of what kind of performance we can expect. If the rumoured memory improvements are true, might there be a performance hit? Is there a performance trade-off?

JVM images

I’ve decided to test the following images.

First we have the (slim) openjdk images for 8/9/10/11:

  • openjdk:8-slim
  • openjdk:9-slim
  • openjdk:10-slim
  • openjdk:11-slim

Next there are the adoptopenjdk images for 8/9/10:

  • adoptopenjdk/openjdk8
  • adoptopenjdk/openjdk9
  • adoptopenjdk/openjdk10

Then we have OpenJ9, again provided by adoptopenjdk for 8, 9 and a nightly build of 9 (see my previous blogpost):

  • adoptopenjdk/openjdk8-openj9
  • adoptopenjdk/openjdk9-openj9
  • adoptopenjdk/openjdk9-openj9:nightly

And I decided to include IBM’s own J9 image as well:

  • ibmcom/ibmjava:8-jre

Testing with Docker

After building my Spring Boot application I launched each Docker image using the following command:

docker run -it -v /Projects/temp/spring-boot-example:/app/spring-boot-example -p 8080:8080 IMAGE_NAME /bin/bash

I’m mapping my “spring-boot-example” project folder to “/apps/spring-boot-example” so I can start the JAR file inside the container. Also I’m forwarding port 8080 back to my host so I can call the endpoints.

Next, inside the container, I launch the Spring Boot application:

java -jar /app/spring-boot-example/target/spring-boot-example-0.0.1-SNAPSHOT.jar

After waiting a bit, calling the endpoints a couple of times and performing a GC I measured the memory usage.

After that I called the “/loadtest” endpoint containing the array-sorting test and waited for the results.

Memory benchmark

Here are the results of the memory used by the simple Spring Boot application:

Chart with memory usage per Docker Image

At first you can see that the memory usage for Java 8 is much higher than for Java 9 and 10, good!

But the biggest shock is how much less memory OpenJ9 and J9 are using, almost 4x less memory if you compare Java 8 with OpenJ9. I’m amazed, how does this even work? Now we can almost call our Spring Boot service micro!

I’ve also experimented with running some production Spring Boot code (not just simple examples) and here I’ve seen improvements up to 40-50% decrease in memory usage.

CPU benchmark

Online I’ve read that OpenJ9 isn’t as good as HotSpot if you look at CPU intensive tasks. That is why I created a small test for this as well.

1000 arrays with 1000000 random long values being sorted. This takes around 100 seconds, this should give the JVM enough time to adjust and optimize. I’ve called the benchmark twice for each tested image. I’ve recorded the second time trying to eliminate warmup times.

Chart with CPU usage per Docker Image

In the chart we can see that indeed the J9 and OpenJ9 images are slower, not by much max 18%. It seems for this particular testcase Java 8 beats most Java 9 implementations (except coupled with OpenJ9).

What’s next?

My current project has a lot more memory issues than CPU issues on production (frequently running out of memory while having 1-2% CPU usage). We are definitely thinking about switching to OpenJ9 in the near future!

We did already encounter some issues during testing:

  1. Hessian: (binary protocol) has a build-in assumption that System.identityHashCode always returns a positive number. For HotSpot this is true but OpenJ9/J9 can also return negative numbers. This is an open issue and the Hessian project hasn’t fixed this in a couple of years, seems to be dead? Our solution is to move away from Hessian altogether
  2. Instana: We love our monitoring tool Instana, but it had some problems connecting their agent to OpenJ9/J9. Luckily the people at Instana helped us identify a bug and a fix should be published today (and is automatically updated, w00t!)

Open questions I haven’t looked in to:

  1. Can you still get/use jmap/hprof information etc in OpenJ9?
  2. How will it hold up during longer production runs?
  3. Will we find other weird bugs? It feels tricky…

Have you tried OpenJ9/J9? Let me know in the comments.

Is there anything else you’d love to see tested? The best way to contact me is to send me a tweet royvanrijn.


Java and Docker, the limitations

Java and Docker, the limitations

TLDR;

Java and Docker aren’t friends out of the box. Docker can set memory and CPU limitations that Java can’t automatically detect. Using either Java Xmx flags (cumbersome/duplicated) or the new experimental JVM flags we can solve this issue.

Docker love for Java is in its way in newer versions of both OpenJ9 and OpenJDK 10!

Mismatch in virtualization

The combination of Java and Docker isn’t a match made in heaven, initially it was far from it. For starters, the whole premise of the JVM, Java Virtual Machine, was that having a Virtual Machine makes the underlying hardware irrelevant from the program’s point of view.

So what do we gain by packaging our Java application inside a JVM (Virtual Machine) inside a Docker container? Not a lot, for the most part you are duplicating JVMs and Linux containers, which kills memory usage. This just sounds silly.

It does make it easy to bundle together your program, the settings, a specific JDK, Linux settings and (if needed) an application server and other tools as one ‘thing’. This complete container has a better level of encapsulation from a devops/cloud point of view.

Problem 1: Memory

Most applications in production today are still using Java 8 (or older) and this might give you problems. Java 8 (before update 131) doesn’t play nice with Docker. The problem is that the amount of memory and CPUs available to the JVM isn’t the total amount of memory and CPU of your machine, it is what Docker is allowing you to use (duh).

For example if you limit your Docker container to get only 100MB of memory, this isn’t something ‘old’ Java was aware of. Java doesn’t see this limit. The JVM will claim more and more memory and go over this limit. Docker will then take action into its own hands and kill the process inside the container if too much memory is used! The Java process is ‘Killed’. This is not what we want…

To fix this you will also need to specify to Java there is a maximum memory limit. In older Java versions (before 8u131) you needed to specify this inside your container by setting -Xmx flags to limit the heap size. This feels wrong, you’d rather not want to define these limits twice, nor do you want to define this ‘inside’ your container.

Luckily there are better ways to fix this now. From Java 9 onwards (and from 8u131+ onwards, backported) there are flags added to the JVM:

-XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap

These flags will force the JVM to look at the Linux cgroup configuration. This is where Docker containers specify their maximum memory settings. Now, if your application reaches the limit set by Docker (500MB), the JVM will see this limit. It’ll try to GC. If it still runs out of memory the JVM will do what it is supposed to do, throw an OutOfMemoryException. Basically this allows the JVM to ‘see’ the limit that has been set by Docker.

From Java 10 onwards (see test below) these experimental flags are the new default and are enabled using the -XX:+UseContainerSupport flag (you can disable this behaviour by providing -XX:-UseContainerSupport).

Problem 2: CPU

The second problem is similar, but it has to do with the CPU. In short, the JVM will look at the hardware and detect the amount of CPU’s there are. It’ll optimize your runtime to use those CPU’s. But again, Docker might not allow you to use all these CPU’s, there is another mismatch here. Sadly this isn’t fixed in Java 8 or Java 9, but was tackled in Java 10.

From Java 10 onwards the available CPUs will be calculated in a different way (by default) fixing this problem (also with UseContainerSupport).

Testing Java and Docker memory handling

As a fun exercise, lets verify and test how Docker handles out of memory using a couple of different JVM versions/flags and even a different JVM.

First we create a test application, one that simply ‘eats’ memory and doesn’t free it.

import java.util.ArrayList;
import java.util.List;

public class MemEat {
    public static void main(String[] args) {
        List l = new ArrayList<>();
        while (true) {
            byte b[] = new byte[1048576];
            l.add(b);
            Runtime rt = Runtime.getRuntime();
            System.out.println( "free memory: " + rt.freeMemory() );
        }
    }
}

We can start Docker containers and run this application to see what will happen.

Test 1: Java 8u111

First we’ll start with a container that has an older version of Java 8 (update 111).

docker run -m 100m -it java:openjdk-8u111 /bin/bash

We compile and run the MemEat.java file:

javac MemEat.java

java MemEat
...
free memory: 67194416
free memory: 66145824
free memory: 65097232
Killed

As expected, Docker has killed the our Java process. Not what we want (!). Also you can see the output, Java thinks it still has a lot of memory left to allocate.

We can fix this by providing Java with a maximum memory using the -Xmx flag:

javac MemEat.java

java -Xmx100m MemEat
...
free memory: 1155664
free memory: 1679936
free memory: 2204208
free memory: 1315752
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
	at MemEat.main(MemEat.java:8)

After providing our own memory limits, the process is halted correctly, the JVM understands the limits it is operating under. The problem is however that you are now setting these memory limits twice, for Docker AND for the JVM.

Test 2: Java 8u144

As mentioned, with the new flags this has been fixed, the JVM will now follow the settings provided by Docker. We can test this using a newer JVM.

docker run -m 100m -it adoptopenjdk/openjdk8 /bin/bash

(this OpenJDK Java image currently contains, at the time of writing, Java 8u144)

Next we compile and run the MemEat.java file again without any flags:

javac MemEat.java

java MemEat
...
free memory: 67194416
free memory: 66145824
free memory: 65097232
Killed

The same problem exists. But we can now supply the experimental flags mentioned above:

javac MemEat.java
java -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap MemEat
...
free memory: 1679936
free memory: 2204208
free memory: 1155616
free memory: 1155600
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
	at MemEat.main(MemEat.java:8)

This time we didn’t set any limits on the JVM by telling it what the limits are, we just told the JVM to look at the correct settings! Much better.

Test 3: Java 10u23

Some people in the comments and on Reddit mentioned that Java 10 solves everything by making the experimental flags the new default. This behaviour can be turned off by disabling this flag: -XX:-UseContainerSupport.

When I tested this it initially didn’t work. At the time of writing the AdoptAJDK OpenJDK10 image is packaged with jdk-10+23. This JVM apparently doesn’t understand the ‘UseContainerSupport’ flag (yet) and the process was still killed by Docker.

docker run -m 100m -it adoptopenjdk/openjdk10 /bin/bash

Testing the code (and even providing the flag manually):

javac MemEat.java

java MemEat
...
free memory: 96262112
free memory: 94164960
free memory: 92067808
free memory: 89970656
Killed

java -XX:+UseContainerSupport MemEat

Unrecognized VM option 'UseContainerSupport'
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.

Test 4: Java 10u46 (Nightly)

I decided to try the latest ‘nightly’ build of AdoptAJDK OpenJDK 10. Instead of Java 10+23 it includes 10+46.

docker run -m 100m -it adoptopenjdk/openjdk10:nightly /bin/bash

There is a problem in this nightly build though, the exported PATH points to the old Java 10+23 directory, not to 10+46, we need to fix this.

export PATH=$PATH:/opt/java/openjdk/jdk-10+46/bin/

javac MemEat.java

java MemEat
...
free memory: 3566824
free memory: 2796008
free memory: 1480320
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
  at MemEat.main(MemEat.java:8)

Succes! Without providing any flags Java 10 correctly detected Dockers memory limits.

Test 5: OpenJ9

I’ve also been experimenting with OpenJ9 recently, this free alternative JVM has been open sourced from IBMs J9 and is now maintained by Eclipse.

Read more about OpenJ9 in my next blogpost.

It is fast and is very good with memory management, mindblowlingly good, often using up to 30-50% less memory for our microservices. This almost makes it possible to classify Spring Boot apps as ‘micro’ with a 100-200mb runtime nstead of 300mb+. I’m planning on doing a write-up about this very soon.

To my surprise however, OpenJ9 doesn’t yet have an option similar to the flags currently (backported) in Java 8/9/10+ for cgroup memory limits. For example if we apply the previous testcase to the latest AdoptAJDK OpenJDK 9 + OpenJ9 build:

docker run -m 100m -it adoptopenjdk/openjdk9-openj9 /bin/bash

And we add the OpenJDK flags (which are ignored by OpenJ9) we get:

java -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap MemEat
...
free memory: 83988984
free memory: 82940400
free memory: 81891816
Killed

Oops, the JVM is killed by Docker again.

I really hope a similar option will be added soon to OpenJ9, because I’d love to run this in production without having to specify the maximum memory twice. Eclipse/IBM is working on a fix for this, there are already issues and even pull requests for this issue.

UPDATE: (not recommended hack)

A slightly ugly/hacky way to fix this is using the following composed flag:

java -Xmx`cat /sys/fs/cgroup/memory/memory.limit_in_bytes` MemEat
...
free memory: 3171536
free memory: 2127048
free memory: 2397632
free memory: 1344952
JVMDUMP039I Processing dump event "systhrow", detail "java/lang/OutOfMemoryError" at 2018/05/15 14:04:26 - please wait.
JVMDUMP032I JVM requested System dump using '//core.20180515.140426.125.0001.dmp' in response to an event
JVMDUMP010I System dump written to //core.20180515.140426.125.0001.dmp
JVMDUMP032I JVM requested Heap dump using '//heapdump.20180515.140426.125.0002.phd' in response to an event
JVMDUMP010I Heap dump written to //heapdump.20180515.140426.125.0002.phd
JVMDUMP032I JVM requested Java dump using '//javacore.20180515.140426.125.0003.txt' in response to an event
JVMDUMP010I Java dump written to //javacore.20180515.140426.125.0003.txt
JVMDUMP032I JVM requested Snap dump using '//Snap.20180515.140426.125.0004.trc' in response to an event
JVMDUMP010I Snap dump written to //Snap.20180515.140426.125.0004.trc
JVMDUMP013I Processed dump event "systhrow", detail "java/lang/OutOfMemoryError".
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
  at MemEat.main(MemEat.java:8)

In this case the heap size is limited to the memory allocated to the Docker instance, this works for older JVMs and OpenJ9. This is of course wrong because the container itself and other parts of the JVM off the heap also use memory. But it seems to work, appearantly Docker is lenient in this case. Maybe some bash-guru will make a better version subtracting a portion from the bytes for other processes.

Anyway, don’t do this, it might not work.

Test 6: OpenJ9 (Nightly)

Someone suggested using the latest ‘nightly’ build for OpenJ9.

docker run -m 100m -it adoptopenjdk/openjdk9-openj9:nightly /bin/bash

This will get the latest nightly build of OpenJ9, and it has two things:

  1. Another broken PATH parameter, fix that.
  2. The JVM has support for the new flag UseContainerSupport (like Java 10 will)
export PATH=$PATH:/opt/java/openjdk/jdk-9.0.4+12/bin/

javac MemEat.java

java -XX:+UseContainerSupport MemEat
...
free memory: 5864464
free memory: 4815880
free memory: 3443712
free memory: 2391032
JVMDUMP039I Processing dump event "systhrow", detail "java/lang/OutOfMemoryError" at 2018/05/15 21:32:07 - please wait.
JVMDUMP032I JVM requested System dump using '//core.20180515.213207.62.0001.dmp' in response to an event
JVMDUMP010I System dump written to //core.20180515.213207.62.0001.dmp
JVMDUMP032I JVM requested Heap dump using '//heapdump.20180515.213207.62.0002.phd' in response to an event
JVMDUMP010I Heap dump written to //heapdump.20180515.213207.62.0002.phd
JVMDUMP032I JVM requested Java dump using '//javacore.20180515.213207.62.0003.txt' in response to an event
JVMDUMP010I Java dump written to //javacore.20180515.213207.62.0003.txt
JVMDUMP032I JVM requested Snap dump using '//Snap.20180515.213207.62.0004.trc' in response to an event
JVMDUMP010I Snap dump written to //Snap.20180515.213207.62.0004.trc
JVMDUMP013I Processed dump event "systhrow", detail "java/lang/OutOfMemoryError".
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space

TADAAA, a fix is on the way!

Oddly it seems this flag isn’t enabled by default in OpenJ9 like it is in Java 10 though. Again: Make sure you test this is you want to run Java inside a Docker container.

Conclusion

IN SHORT: Be aware of the mismatch, the limitations. Test your memory settings and JVM flags, don’t assume anything.

If you are running Java inside a Docker container, make sure that you have Docker memory limits AND limits in the JVM or a JVM that understands these limits.

If you’re not able to upgrade your Java version set your own limits using -Xmx.

For Java 8 and Java 9, update to the latest version and use:

-XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap

For Java 10, make sure it understands the ‘UseContainerSupport’ (update to latest) and just run it.

For OpenJ9 (which I highly recommend for bringing down your memory footprint in production) for now set the limits using -Xmx, but soon there will be a version that understands the ‘UseContainerSupport’ flag.


In a microservices landscape; When do you update?

In a microservices landscape; When do you update?

This week I’ve been stuggling with the following question:

When is the right time to upgrade?

By upgrading, I mean everything: libraries, tools, Java versions, application servers, MQ servers…

My current project uses a reactive upgrade policy, we upgrade for four reasons:

  1. Something is broken and fixed in a later version
  2. We need or want to use a new feature
  3. Support for a version we’re using is being dropped
  4. The old version we’re using has a known security issue/CVE

The first two reasons are entirely up to the programmers to decide. The third reason is up to the company that gives us support. For the fourth upgrade reason, security, we have some automation in place. We are using the OWASP dependency checker Maven plugin for our libraries.

But:

  • Is a reactive update policy good enough?
  • Are there any pro-active strategies?
  • Do you want to invest in keeping your microservices up-to-date?
  • Do you let your services deteriorate and dispose, replace them in the future with new technologies?

More on this in my (kind-of-weekly) vlog:


Fear of upgrading to Java 9, is it justified?

Fear of upgrading to Java 9, is it justified?

It seems that the adoptation of Java 9 is slow, very slow. In a Twitter poll this week I asked: “Which version of Java are you using in production?”

The poll got almost 300 replies and to my surprise just 3% of the respondents are using Java 9 at the moment. Most are using Java 8 and there are even more people using Java 6… So what is holding people back? Why is almost no-one using Java 9?

It turns out a lot of people are scared to upgrade to Java 9. They think it’ll be hard to do, libraries and tools will fail them. They are scared for another Jar-hell.

That is why I took one of my larger projects, that has been running in production for a long time, and try to upgrade it to Java 9.

Upgrading: Walk in the park or walk in jar-hell?

Here are my results and findings, in a new format: a vlog

If you like the vlog and follow my website, please subscribe to my channel on YouTube!


Inverse Conway's law

Inverse Conway's law

Today I got to present at Devoxx Poland on being agile and managing your architecture.

One of the points I made during the talk had to do with Conway’s Law. For those unfamiliar with it, here it is:

organizations which design systems … are constrained to produce designs which are copies of the communication structures of these organizations M. Conway

This basically means that the code you create is likely to reflect the way the people and teams in your company communicate. For example, banks are usually pretty strict and have a tendency to document things. The code written at banks in turn have the tendency to use SOAP for example, verbose and strict.

The project I’m currently working on has a very agile mindset, we change a lot and have open communication. This reflects in our codebase as well, all the code is easy to understand and easy to change (thankfully!).

But lately we’ve started to move away from our old monolith and we’ve began cutting it up into microservices. That’s when I noticed something odd, which I call the Inverse Conway’s Law (maybe it is Roy’s Law?):

organisations which radically change their system design should expect changes in communication structure Roy van Rijn

What does this mean?

We’ve started to move towards a microservice environment, where each team has a responsibility over a couple of services. They’ve developed it and they own it. This ownership completely changed the culture in our team. Up to now if the monolithic backend was broken or offline, everyone was hurting and everyone wanted to fix this as soon as possible. With microservices, this is gone. The first thing we now do is check which microservice failed. If it isn’t theirs, it is not their problem, it’s someone elses problem now!

This is just one example on how changing the design of our system changed the culture and team dynamic. It is one thing that sounds logical when you think about it, but I didn’t expect this before we started the transition.


JavaOne Rock Star 2016

JavaOne Rock Star 2016

Today I got an email with the subject: “Congratulations JavaOne 2016 Rock Star!”. It turns out the JavaOne 2016 Ignite session was voted enough to receive a JavaOne Rock Star Award!

I was so excited, elated even!

Rocking that talk

But I quickly found out that the JavaOne Rock Star lifestyle isn’t as much fun as it sounds.

  • Being 3 hours late for a meeting, even a Rock Star can’t do that.
  • Peeing in the corner of your office while holding a bottle of Jack Daniels, still not appreciated.
  • Wearing ripped jeans, a leather jacket and other than that just chest hair, is considered too casual.
  • The groupies? It is like answering random Stack Overflow questions, non-stop, face to face.
  • After each coding session, smashing your MacBook on the floor… quickly becomes expensive.

Drunk Wombat holding bottle of Jack Daniels


Vikipedia: A Java YouTube bot

Vikipedia: A Java YouTube bot

YouTube, the future of television. I’ve got a lot of subscriptions to YouTube channels that deliver quality content, and those shows are ‘cast’ (using my Chromecast) to my TV. Another thing I often do is look up information, for example I watch talks from programming conferences like Devoxx using YouTube.

This gave me an idea, what if I can take some free information (like Wikipedia, all creative commons) and use that to create content for YouTube? Maybe I’ll even get some views :-)

So this is what I’ve come up with, the following video is generated completely automatically:

Step 1: Wikipedia API

The first step in creating these videos was to get some actual freely available content. For this I chose to use Wikipedia. This can be accessed using JSON/REST, and to quickly hack this into place I decided to use a framework called ‘Unirest’. This makes the while prototype easy:

// API URI for the English Wikipedia
private final String ENGLISH_WIKIPEDIA = "https://en.wikipedia.org/w/api.php";

// Using a 'search query', name of the page, retrieve the content:
private Optional<String> getPageContent(final String query) throws UnirestException {
    //Example URI created for query 'Stack Overflow': https://en.wikipedia.org/w/api.php?format=json&action=query&prop=extracts&titles=Stack%20Overflow&redirects=true
    HttpResponse<JsonNode> response = Unirest.get(ENGLISH_WIKIPEDIA)
            .queryString("format", "json")
            .queryString("action", "query")
            .queryString("prop", "extracts")
            .queryString("titles", query)
            .queryString("redirects", "true")
            .asJson();

    JSONObject pagesObject = response
            .getBody()
            .getObject()
            .getJSONObject("query")
            .getJSONObject("pages");
    if(pagesObject.keys().hasNext()) {
        String pageKey = pagesObject.keys().next();
        String htmlContent = pagesObject.getJSONObject(pageKey).getString("extract");
        return Optional.of(htmlContent);
    }
    return Optional.empty();
}

The resulting JSON (see this example) contains the Wikipedia page in simple processed HTML. From this HTML I used a library called ‘Jsoup’ to extract the plain text. The first alinea of text contains the introduction of the topic, this is what I’ve decided to use as content for the video. Initially I had all the content of the page in the videos, but they quickly turned out to be 30+ minutes long and very boring.

This is the code I used to go from HTML to the first alinea of text:

/**
 * Clean the HTML, get inner text of first alinea and break into sentences.
 * @param htmlContent
 * @return
 */
private List<String> cleanAndBreakIntoSentences(final String htmlContent) {
    Document document = Jsoup.parse(htmlContent);

    List<String> fullText = new ArrayList<>();
    Elements elements = document.body().children();

    Iterator<Element> iterator = elements.iterator();
    while(iterator.hasNext()) {
        Element elem = iterator.next();

        // All the text is contained in paragraph elements:
        if(elem.is("p")) {

            String source = elem.text();

            // Break the text into sentences using the smart JDK 'BreakIterator':
            BreakIterator breakIterator = BreakIterator.getSentenceInstance(Locale.US);
            breakIterator.setText(source);
            int start = breakIterator.first();
            for (int end = breakIterator.next(); end != BreakIterator.DONE; start = end, end = breakIterator.next()) {
                fullText.add(source.substring(start,end));
            }
        }
        // If we reach a header, stop processing, we only need the first section/alinea:
        if(elem.is("h2")) {
            // We break on the first (sub-) title:
            break;
        }
    }
    return fullText;
}

Step 2: Stunning visuals with Java 2D

We have some text, now how do we turn this into a video? We need visuals! I decided to cut the content into seperate lines, and for each line I have a single ‘slide’. This is an (buffered) image created using Java’s 2D API and saved to file.

/**
 * For each sentence, create a buffered image and paint some text on this.
 * We need the text to be 'wrapped', for this I repurpose a JTextArea, on a JPanel.
 * 
 * Also increase the font size so the text fits on the screen.
 */
public void paintText(Graphics2D g2, String text) {
    JPanel panel = new JPanel();
    panel.setOpaque(true);
    panel.setSize(new Dimension(1920, 1080));
    panel.setVisible(true);

    JTextArea area = new JTextArea();
    area.setSize(new Dimension(1920,1080));
    area.setFocusable(false);
    area.setBorder(BorderFactory.createEmptyBorder(BORDER_SIZE, BORDER_SIZE, BORDER_SIZE, BORDER_SIZE));
    area.setWrapStyleWord(true);
    area.setLineWrap(true);
    area.setText(text);
    area.setOpaque(true);

    int fontSize = 10;
    // Increase the font size as long as it fits on screen (and don't go bigger than 100, looks ugly):
    while(area.getPreferredScrollableViewportSize().getHeight() < 1080 && fontSize <= 100) {
        fontSize += 10;
        area.setFont(new Font("Futura", Font.PLAIN, fontSize));
    }
    area.setFont(new Font("Futura", Font.PLAIN, fontSize - 10));
    panel.add(area);

    // Print this panel on the Graphics2D object (which is a BufferedImage)
    panel.printAll(g2);
}

Step 3: Text-To-Speech

Instead of using a Java Text-To-Speech (TTS) library, which aren’t very clear or good, I decided to use the build-in OS/X command ‘say’. On the command line you can execute: say Hello world!

The ‘say’ command supports different voices and also has the ability to store the output in a file. Which is perfect for us!

    // Execute the say command with a given filename, voice and sentence to speak:
    final Process p = Runtime.getRuntime().exec("say -o " + filename + " -v " + voice + " -r 210 " + line + "");

    new Thread(() -> {
        BufferedReader input = new BufferedReader(new InputStreamReader(p.getInputStream()));
        String line1 = null;

        try {
            while ((line1 = input.readLine()) != null)
                System.out.println(line1);
        } catch (IOException e) {
            e.printStackTrace();
        }
    }).start();

    // Wait for the say-command to finish
    p.waitFor();

Step 4: Putting it all together

What do we have now? A list of sentences from Wikipedia, for each sentence we have an image and we have an audio snippet created with TTS. Now it is time to put everything together and create a video. The easiest way to do this is to use ffmpeg, a powerful command line video tool. To make this easier I created a small method ‘executeCommandLine’ which does the same thing as the snippet with ‘Runtime.getRuntime().exec’ shown above.

Combine the image (png file) and the TTS audio (mp4 file) into a series of MPG files:

    for(VikipediaSentence sentence : sentences) {
        String filename = sentence.getFilename();
        //Combine audio and image into a short video:
        executeCommandLine("ffmpeg -loop 1 -y -i " + filename + ".png -i " + filename + ".mp4 -tune stillimage -shortest " + filename + ".mpg");
    }

We now have a series of small MPG files. They need to be combined into a single video. For this we need a text file which contains a list of all the MPG files. This is stored on disc and next we execute the following command:

//merge all:
executeCommandLine("ffmpeg -f concat -safe 0 -i output/snippet_list.txt -c copy output/output.mpg");

Next we convert the video from one large MPG to MP4 file (which is a nicer, compressed format):

//convert to mp4:
executeCommandLine("ffmpeg -i output/output.mpg -c:v libx264 -c:a aac -strict experimental -b:a 192k -pix_fmt yuv420p -shortest output/output.mp4");

And to add a little more ‘production value’ I’ve gone online and found some soothing (free, creative commons) background music. This music is looped into a large audio file of 50 minutes (enough for all videos). Again using ffmpeg the audio is mixed in:

// add background music:
executeCommandLine("ffmpeg -i output/output.mp4 -i input/background_loop.mp3 -filter_complex amerge -c:v copy -c:a aac -strict experimental output/finaloutput.mp4");

There it is, we have a complete video with text audio, text visuals and background music! Only one thing left to do, uploading to YouTube!

Step 5: Uploading everything to YouTube

To automate the process of uploading, I first needed a place to put the videos, so I created a new YouTube channel named: Vikipedia

Next we can use the Google YouTube Data API. It turns out this is very easy and well documented. I don’t even need to share the code I wrote here because I used the example from Google’s website itself! Check out this amazing documentation. Very detailed and does exactly what I needed it to do.

The example code takes a video file and uploads it. The only things I changed were setting the correct description, I was ready to generate all the content!

Not implemented (yet?)

Things I wanted to implement (but didn’t and probably never will):

  • Add a little color to the videos (black and white is boring)
  • Add waveform from ffmpeg (just to keep it visually interesing)
  • Download (did this) and use (didn’t do this) the images from the Wikipedia page and add those to the video

Final thoughts

  • Yes, it is pretty easy to hack together a bot to create YouTube videos.
  • No, the quality isn’t that good… yet?
  • Yes, it can be made in a single afternoon!
  • No, I’m not going to automate the spidering of Wikipedia and upload a million videos… (but I could easily do that!)

Up to now all the videos created are based on a small list of ‘hot topics’ I found on Wikipedia and Twitter. I don’t want to automate the entire process and flood YouTube, the content just isn’t good enough for that. But it was a fun project for a lazy afternoon! I’ve learned a lot of new tricks, maybe you can benefit from that as well.

The next thing I want to do with the 400+ YouTube videos I’ve generated is to see how (and if) people are going to watch them. Do they show up in the search results? Which topics are searched and watched? Maybe more on that in a future post!

Do you have any good ideas on how to generate videos, let me know in the comments! I’ve created a bot that generates music some years, maybe I should improve that bot to upload its own YouTube music videos? ;-)


Three new JEPs

Three new JEPs

This morning Mark Reinhold submitted three brand new JEPs (JDK Enhancement Proposal).

  • JEP 300: Augment Use-Site Variance with Declaration-Site Defaults
  • JEP 301: Enhanced Enums
  • JEP 302: Lambda Leftovers

These proposals are enhancements to the JDK (Java Development Kit) and OpenJDK. A long term roadmap for the JDK projects, a look into what the future of Java might hold.

Let’s dive right in and take a quick look on what these proposals actually are!

JEP 300: Augment Use-Site Variance with Declaration-Site Defaults

When you currently use Java Generics you probably already know about wildcards. It is possible to set lower and upper bounds to generics using the keywords ? extends and ? super. There are two parts to using wildcards, declaration side and use side, this JEP focusses mainly on making the use side easier and more powerful.

Declaration side

It is possible to set a bound with a wildcard on the declaration side:

public class Shelter<X extends Animal> {
	...
}

In this example we’ve declared that the Shelter has to contain Animals. It is possible to create an Shelter<Cat> or Shelter<Dog> now, but not Shelter<Bike>.

Use side

The second way to use wildcards is on the use side. There we can speak of so called in and out-variable wildcards.

The following code uses the ‘extends’ keyword to create the in-variable:

    List<Animal> animals = new ArrayList<>();

    // Using extends we create an 'in' variable:
    List<? extends Animal> inAnimals = animals;

    // From the inAnimals we can read: Animal (not subclasses)
    Animal animal = inAnimals.get(0); // <- reading Animal is fine
    Cat cat = inAnimals.get(0); // <- compile error, we can't read Cat

    // We can write: nothing (!) making it almost read-only
    inAnimals.add(new Animal()); // <- compile error

There is also the ‘super’ keyword (not very common) to create an out-variable use side:

    List<Animal> animals = new ArrayList<>();

    // Using super we create an 'out' variable, setting an upper bound:
    List<? super Cat> outAnimals = animals;

    // From the outAnimals we can read: object (no superclasses!)
    Cat cat = outAnimals.get(0); // <- compile error, illegal
    Object o = outAnimals.get(0); // <- we can only read object

    // On the write side, we've set a bound to Cat:
    outAnimals.add(new Cat()); // <- correct!
    outAnimals.add(new Object()); // <- compile error
    outAnimals.add(new Animal()); // <- compile error

So what is JEP 300?

The motivation of the JEP states:

Since invariant uses of these type arguments are less flexible than their wildcard equivalents, while providing no extra power, a reasonable practice is to always use a wildcard when mentioning the type.

It is almost always more powerful and equivalent to use wildcards on the use side. The problem is that this is verbose and adds a lot of noise to your code. The proposal wants to make it possible to declare (at the declaration side) what the default wildcard strategy should be.

For example look at the following code:

    interface Function<contravariant T, covariant R> {
        R apply(T arg);
    
    }
    // other possible syntaxes are Function<-T, +R> or Function<in T, out R>

The compiler can now automatically treat every use of the type e.g., Function<String, Number> as if it had wildcards Function<? super String, ? extends Number>.

On the use side the proposal says:

Rather than making changes to subtyping, we preprocess the source code so that types like Function<String, Number> are implicitly expanded to Function<? super String, ? extends Number>.

This should make it more powerful and just as readable to using wildcards by default without even noticing them.

JEP 301: Enhanced Enums

The next proposal Mark has submitted is about using Enums with Generics. Look at the following example:

enum Pet {

    BELLA(new Dog("Bella")),
    BUDDY(new Dog("Buddy")),
    TIGGER(new Cat("Tigger"));

    Animal animal;

    Pet(Animal animal) {
        this.animal = animal;
    }

    public Animal get() {
        return animal;
    }
}

We have an enum with three pets. These pets have an instance of an animal inside them. And we have a way of retrieving them.

System.out.println( Pet.BELLA.get().getName() ); // prints: "Bella"

But one thing we can’t do it call the method “.bark()” on BELLA, because we don’t know this is a Dog.

The proposal JEP 301 wants to make it possible to correlate a specific type to an enum constant:

enum Pet<A extends Animal> {

    BELLA<Dog>(new Dog("Bella")),
    BUDDY<Dog>(new Dog("Buddy")),
    TIGGER<Cat>(new Cat("Tigger"));

    A animal;

    Pet(A animal) {
        this.animal = animal;
    }

    public A get() {
        return animal;
    }
}

Having coupled the Dog type to enum BELLA it should now we possible to call:

Pet.BELLA.get().bark(); // WOOF!
Pet.TIGGER.get().purr(); // Purrrrrrrrr.....

JEP 302: Lambda Leftovers

This is turning out to be a rather long blogpost but we’ve reached the final proposed JEP of the day: Lambda Leftovers

Accidental ambiguities

One of the problems people are having with the lambdas at the moment, and this JEP wants to improve, are accidental ambiguities.

For example:

private void apply(Predicate<Animal> filter) {
   ...
}

private void apply(Function<Animal, String> function) {
   ...
}

apply(input -> false); // <- compile error

/**
 * Ambiguous method call. Both:
 * apply (Predicate<Animal>)        in classname and
 * apply (Function<Animal, String>) in classname match
 */
    

Why are we having this problem? Both methods are possible candidates of the lambda. If we look further and see the return type ‘boolean’ it should be clear we wanted to call the Predicate method and not the Function (which should return a String). Right now this doesn’t work, but it is something the compiler could figure out for us.

Underscores

Up to Java 8 it was perfectly acceptable to use the underscore character ‘_’ as a variable name. But this has been changed, all leading up to JEP 302. In most languages with lambda’s it is possible to denote certain inputs as unused. This is done using the underscore. In Java this causes problems because it is right now a valid variable name.

Starting with Java 8 the use of an underscore as lambda argument caused a warning and from Java 9 the use of an underscore became an error. This allows future Java versions beyond 9 to be able to use the underscore for other purposes. And one of those purposes is to bring it back as a default variable name. Not causing collisions if used multiple times and it can’t be used as variable by calling _.toString() for example.

// Ignore the second parameter, just use i:
BiFunction<Integer, String, String> biss = (i, _) -> String.valueOf(i);

// Even possible, ignore both:
BiFunction<Integer, String, String> b2 = (_, _) -> "someString";
    

Shadowing of parameters

Another small change that is proposed is allowing to shadow parameters. For example look at the following code:

Map<String, Integer> map = ...
String key = computeSomeKey();

map.computeIfAbsent(key, key -> key.length()) // compile error, key already defined

map.computeIfAbsent(key, _ -> {
    String key = "";  // compile error, can't redefine key inside this scope
    return key.length();
});

In both cases ‘key’ is already defined, can’t be used as a parameter or local variable inside the lambda. The proposal wants to lift this restriction, allowing the use of ‘key’ as a parameter or as a local inside the lambda using so called ‘shadowing’.

One possible drawback is readability, and I too think that this might be a problem. For example:

Map<String, Integer> map = ...
String key = "theInitialKey";

map.computeIfAbsent(key, _ -> {
    String key = "theShadowKey"; // <- delete this
    return key.length();
});

The second variable “theShadowKey” is taking the place of the first “theInitialKey”. This can be quite confusing because if you delete the second declaration nothing would break. The other variable would come from the shadows and take its place. This feels dangerous and confusing to me, not very ‘Java’-like.

Conclusion

Another day, three new JEPs submitted. Java is moving forward and trying to improve the user experience with genetics and lambdas. These proposals look like good, small, low-impact improvements that can really benefit the common programmer.


Excuse me, sorry to interrupt...

Excuse me, sorry to interrupt...

Interruptions are a software developers worst nightmare. This is what you often hear. There is nothing worse than being interrupted while working hard on a problem, while you’re in the zone. You’ll lose your train of thought and the world collapses.

The idea is that in software development you are sometimes very deep into a problem, analysing code:

This orchestrator calls that service, it is managed by this class and talks to that queue. So this generator has these parameters and it all depends on… “hey I still need your time sheet

Rage

Poof… everything gone. Your mind is now filled with rage and lost the entire train of thought. This is the main reason programmers hate interruptions.

Root cause analysis

During the JavaOne 2016 conference I attended a talk by Holly Cummins called “Euphoria Despite the Despair”. In this talk she went in depth into what makes work fun and stressed the point that interruptions should be avoided at all cost.

She even talked about a team that has a dedicated duo that had ‘interruption duty’. A Slack-bot was created so you could ask which colleagues you could interrupt each day.

But… are we judging too quickly here? Are all interruptions bad? I don’t think they are. Time to dig a little bit deeper.

Sync versus async and good versus bad

Interruptions are synchronous and blocking. Someone walks up to you and stands near your desk trying to get your attention. You might have a few milliseconds to finish what you are doing but you’ll need to reply. Other forms of communication are asynchronous, for example mail or chat clients (Slack). Someone asks a question and when you have the time to reply, you reply.

Our team has evolved to use this to our advantage. Some problems ask for direct action, but most can be handled asynchronous. This is something we talk about and discuss.

For example: If my manager wants a time sheet? Ask me asynchronous, it can wait. If a colleague has a question but is able to do other things first? Ask asynchronous, it can wait. If a colleague is stuck and can’t continue to work? I’d be happy to drop what I’m doing and assist! This is what teams do, they work together and communicate!

Don’t throw all interruptions on one big pile and label them as bad, distinguish between the good and bad interruptions.

Mount the headphones, get in the zone

Sometimes, as a programmer, you just want to concentrate and solve some deep problem alone.

In our team this can be done in two ways:

  1. Work at home or in a small dedicated room in our office
  2. Put on your headphones, if they are on, you signal you don’t want to be disturbed

I do this too, mostly using my headphones. Most of the time I don’t even have music playing and I can hear everything the team says (this is important, more on this below!).

In my opinion though we need to rethink this. Is it really the best way to program when you are ‘in the zone’?

I think you are much more likely to solve problems when you openly share and collaborate. Pairing isn’t just for trivial work, instead: the trivial things are things you should do alone! Teams should have no secrets, at each point in time you should have a pretty good idea what each of your team buddies is working on. Actively ask questions, even trivial things.

Near-subliminal team updates

What? “Near-subliminal” team updates? Yes, I’m so sorry, I can’t think of a better name. Can you? Please tell me.

One thing that is very important in software development is the use of what I’ve just called near-subliminal team updates/mumblings. It is the main reason I have headphones on without any music, and why I don’t like working at different locations (at home or in cubicles).

Example

You are working on something and you feel the need to refactor a class. At this moment you say/mumble out loud:

“Hey team, I’m going to refactor classname because reason okay?”

This is what I call a near-subliminal team update, and they are very important. They can solve a lot of problems!

  1. Maybe you don’t agree with reason, this is the perfect time to talk about that. You don’t want to wait until a code review and tell them it was a bad idea later the same day.

  2. Maybe you are also working on something in classname and it might collide with your work. You don’t want to wait until you git pull and need to merge, tackle it right away!

  3. Maybe it isn’t relevant to you at all (which is most of the time)… Well, most likely you don’t even really notice the update. It didn’t contain any trigger words. Terms you are working on or have an opinion about. You aren’t really interrupted and can continue to work.

Conclusion

To summerize:

  1. Don’t treat all interruptions as evil, they are not…
  2. If you are annoyed about interrupts, talk about it
  3. Speak up, mumble, use those near-submliminal team updates!

Did I miss anything? How do you cope or deal with interruptions?

And please: If there is someone with a better name for these short near-subliminal status updates, let me know!


JavaOne 2016, the Future of Java EE

JavaOne 2016, the Future of Java EE

Lately there has been a lot of rumor going around about the future of Java EE. Oracle ex-employees Reza Rahman was one of the first to voice their concern about Java EE. It seemed that all development on the seperate JSRs (Java Specification Requests) that make up Java EE 8 ground to a halt and Oracle was thinking about stopping Java EE development all together.

Oracle finally gave some insight on their proposal of the future of Java EE during JavaOne 2016 (where I am right now).

What is Java EE?

First, lets take a step back and look at what Java EE actually is. If you download Java SDK/JRE you are able to compile classes and run them. In the language (java.* and javax.*) there are a couple of libraries that you can use. For example if you want to compute mathematical equations there are classes to use in the java.math.* package.

Oracle quickly realized it is hard to make larger enterprise applications using just these libraries. That is why they developed the following things:

  1. A set of interfaces (API) for Enterprise Applications (Java EE)
  2. A default server implemention for these API’s (Glassfish)
  3. A licensing structure for vendors that want to implement these API’s

So if you want to make an Java EE application you can use the Java EE API. For example you can define Servlets for web communication. Next when you deploy this application on an official licensed Java EE server it’ll make sure the methods are implemented and called when a URL is invoked.

Java EE at JavaOne 2016

During the keynote Oracle informed the crowd about Java EE 8. Rumor was that Oracle wanted to completely stop developing Java EE because the demand has been decreasing over the years. More and more ‘fat’-jar applications are being deployed on simple web servers that don’t implement the full Java EE specification. Oracle has to think about the return value, is the effort of developing a new Java EE 8 specification worth the effort? They clearly thought it wasn’t.

The problem for Oracle was the big backlash coming from the community. A couple of initiatives were started to create a voice and support for Java EE (for example the Java EE Guardians and EE Central). This put Oracle in a bad position. If they stopped developing Java EE the community would be very dissapointed. The initiatives would demand the rights to the API to keep developing it themselfs. But Oracle can’t do this because they are earning money from selling licenses.

The decision has been made to continue developing Java EE 8. And during the keynote they proposed a release of EE 8 in 2017. Going forward they’re talking about changing the focus to reflect ‘modern’ enterprise development. They want to add support for virtualization (Docker anyone?) and microservices, modernizing Java EE.

My thoughts

Parts of Java EE, for example CDI and JPA are in my opinion very succesful. Java EE looked at the options in the market and they’ve defined a new, clearer, general, evolved API that allows you to change the underlaying technology/vendor. This is very good! It is always valuable when vendors get together and define a shared API together based on the lessons learned from their own implementations.

The problem is that Java EE 8 as a whole is too large, it is a monolith. If I start a new project I’d love to pick some parts from Java EE and just use those. There is no need to have one huge Java EE certified server if instead you can pick one particular CDI and JPA implementation.

The future of Java EE isn’t to continue as one big bundled EE server. We need to break it up. All the parts can have their own lifecycle and seperate implementations. This makes it easier to get smaller certified implementations and doesn’t require large EE servers. This also allows for pruning, if certain parts aren’t relevant anymore, don’t use it and stop developing.

Vendors like Red Hat (owner of WildFly) are already breaking up their Java EE implementation using frameworks like WildFly Swarm. Swarm allows you to package and run just the parts of the specification you’re using. This is, what I think, the real future of enterprise Java.


Job titles when writing code

Job titles when writing code

The company I work for (JPoint) we don’t have work titles. Well, we do, but you’re free to pick one. Some people call themselfs ‘software developer’, some are having more luck as ‘software architect’, others label themselfs as ‘software craftsman’ and there might be a ‘software ninja/rockstar’ hanging around.

But I have a problem with that… all those terms don’t reflect on what we do. Currently I’m sitting in a session at JavaOne and I’m having the feeling people don’t realize what their job actually is.

#So, what does a software coder do?#

When creating software we instruct a computer what to do.

That is it.

We tell a machine how to respond. We don’t build things (develop), we don’t slay things (ninja!), we don’t rock, we just tell the computer what to do.

That is why a programming language is called… a language. It is a way we, as coders, communicate with the computer.

Is it that simple? Sadly no.

When writing code, we are not alone. There are also other coders that read/write/edit/analyse our code. It isn’t enough that just you and the machine understand, other coders have to understand as well.

We started out programming in binary, but this was way too complicated for humans. So we invented programming languages. First we created assembly. This made it easier for a human to tell the machine what to do. But assembly was still too complicated. Slowly the programming languages became closer and closer to human language.

#Alternatives#

My job is to write code, code that the machine understands. The code must also be pleasant to read and understand by other humans. Also, in this, I’m putting a lot of creativity.

Maybe you are a software author or software writer:

wri•ter (rahy-ter)

noun
1. a person engaged in writing (books, articles, stories, etc.) as an occupation or profession; an author or journalist.
2. a person who commits his or her thoughts, ideas, etc., to writing

This fits our job much better!

If you want something more exotic, and your code has a good flow and shows creativity, you might call yourself a: software poet:

po•et (ˈpoʊ ɪt)

noun
1. one who writes poetry.
2. one who displays imagination and sensitivity along with eloquent expression.

Saving the world with a code review

Saving the world with a code review

This morning I noticed the following tweet by fellow programmer (and runner) Arun Gupta:

The tweet contained the following cartoon by ‘Oppressive Silence’, check out their website for more laughs!

Oh no the robots (Source of the comic)

Solved by a code review?

The main question I’d like to ask:

Is this really something that you’ll find during a code review?

I think my answer below won’t surprise you, but the reasoning behind it might. First lets look at four reasons this bug should never happen to me (or any other experienced programmer).

1) Static variables/global state

The first obvious problem with the code is the usage of static state. The variable ‘isCrazyMurderingRobot’ is mutable and static, so anywhere in the code the value can be changed. This makes it very hard to reason about the variable and keep track of it. Any programmer can come along and change the value in some method. This is unacceptable, there is almost no reason to use mutable static variables.

Top tip: If you have variables, especially mutable variables, keep their scope as small as possible!

2) Final method arguments

If you solve the global state problem you’ll probably end up with something like this (translated to Java):

    private void interact_with_humans(boolean isCrazyMurderingRobot) {
        if(isCrazyMurderingRobot = true) {
            kill(humans);
        } else {
            be_nice_to(humans);
        }
    }

Whenever an argument is passed to a method I have the habit (and my IDE-settings enforce this for me) to make them final. Method arguments should never change inside a method. It is just strange if you need to do this when you think about it. It probably means the method should have its own (scoped) mutable variable.

In the code the following would happen:

    private void interact_with_humans(final boolean isCrazyMurderingRobot) {
        if(isCrazyMurderingRobot = true) {
        // Compilation error: 'Cannot assign a value to final variable 'isCrazyMurderingRobot'

The code won’t even compile before telling me what is wrong, the world is saved!

3) Static code analysis

Another reason this bug will never happen in my code is because we have static code analysis on all our projects. A program like FindBugs will immediately flag this as a major problem, assigning values inside an if-statement.

Look at this particular check in FindBugs:

QBA: Method assigns boolean literal in boolean expression (QBA_QUESTIONABLE_BOOLEAN_ASSIGNMENT)

This method assigns a literal boolean value (true or false) to a boolean variable inside an if or while expression. Most probably this was supposed to be a boolean comparison using ==, not an assignment using =.

Static code analysis is an essential tool in modern programming, it is just so convenient, never reads over certain bugs like a human would (spoiler: more on that in the final conclusion below).

4) Yoda conditions

The final reason this bug will never happen in my code is: Yoda conditions

A habit I learned a long time ago (in older languages) is writing my if-statements like Yoda. Good it is, mistakes you won’t make. The reason behind doing this is that it’ll prevent exactly the bug in the cartoon.

In the cartoon the if-statement would change from: ‘is crazy murdering robot true?’ to ‘true is crazy murdering robot?’.

If you do this, the following happens:

    private void interact_with_humans(boolean isCrazyMurderingRobot) {
        if(true = isCrazyMurderingRobot) {
        // Compilation error: 'Variable expected'

Conclusion / saving the world

So, do I think Arun is right? Will code reviews find the bug in the cartoon? Yes and no.

Humans are good at reasoning, thinking, being creative, finding bugs, but there is one thing we are horrible at: syntax parsing.

During a code review you are just as likely (maybe even more likely) to read over this typo and don’t notice it. A second pair of eyes won’t save the world here just yet.

But: when reviewing this piece of code, I will complain about the global state! If you start to refactor, encapsulating the logic and remove the mutable static variable the bug will be noticed and solved. So yes, if you do a code review, the code would have be been fixed and humans will be safe for just a little bit longer.