How to become a software architect

How to become a software architect

Yesterday, on the vJUG mailing list, Gilberto Santos asked the following question:

I’m working as Software Engineer and as intend to grow up to architect position, now on I was wondering how to best way to get there , should be :

Java certifications

Masters

Both ?

I replied to him over the email, but it might be something more developers are struggling with, time for a blog post!

Architects should code

There are two types of software architect:

  1. The one that draws Visio® diagrams
  2. The one that develops/codes

The first type of architect likes to talk, have meetings, draw diagrams and give orders. This, to me, is not good.

Architects should write code, they should be part of a team. In fact, everybody that writes code is effectively an architect and should have basic knowledge about architecture principles. For example every programmer should know about the SOLID principles.

Simon Brown has some good presentations on this topic on his website coding the architecture.

Mistakes

The best advice I can give a developer that wants to grow into a senior/architect position is:

Make mistakes, lots of them.

If you want to become a better architect the most important skill to master is the art of making mistakes, noticing them, and not repeating them in the future. Most ‘enterprise’ architects I’ve worked with set up a project and leave before the thing they had designed starts to rot, smell and fall apart. They never experience the flipside of what they set up, you can make everything look good on paper.

As a programmer and architect you need to experience this ‘project rot’. To learn you need to feel the mistakes that are in every system. Once you start seeing problems, admit the things you’ve made aren’t perfect, and start changing them. Try to fix the mistakes.

Matt Damon would say: “I’m going to have to refactor the shit out of this.”

This experience is what makes you a real architect. The best architects I’ve worked with knew instantly what needed to change and which solutions would work better than others. They develop an instinct for this. In Dutch we have a saying ‘they know where the shoe pinches’. Which means: just by looking at it they have a ‘feel’ for what hurts and will cause problems (in the future).

However….

This experience/instinct is what makes you a good architect and a good programmer. But sadly this doesn’t directly get you a better job or position within a company. Managers, recruiters and HR can’t easily test for this instinct. Sadly: the thing they look at most of the time is certifications.

For example, I’ve mentioned the SOLID principles, and they are very important. It takes a maximum of 5 minutes to learn the acronym and use it in job interviews or get a certification. To really understand it though you need to experience SOLID. You need to encounter situations where huge classes have a lot of responsibilities. This is the problem with most certifications: they encourage you to memorize rules and principles, not to understand them. I’d rather have someone in my team that naturely breaks up code into clean interfaces than someone who memorized that the ‘I’ in SOLID stands for the ‘Interface segregation principle’.

Once you developed this ‘architect instinct’, people will notice it. Colleagues will remember and this will help you get better jobs and positions in the future. You’ll need to prove yourself and grow your network. Try to get recommendations from people you’ve worked with and you will become a real senior programmer and architect.


The Prettiest Code

The Prettiest Code

This afternoon I started to wonder…

I’ve been a programmer now for 20+ years, but what is the best piece of code I’ve written in all these years?

The first thing that popped in my mind was this. It is a piece of code that can very quickly, in near linear time, generate de Bruijn sequences. I ‘invented’ it after reading a scientific paper that described how to quickly generate ‘Lyndon words’. I knew with Lyndon words you could easily generate the de Bruijn sequences. So I implemented it and adopted it for de Bruijn. Read more about this here.

Why is this my favourite piece of code? Well, it is small, compact and does something impressive. I couldn’t find a lot of implementations that had the fast runtime my code had. And it wasn’t trivial to implement, I had to do quite a lot of research and it felt like I myself did a little inventing. I probably wasn’t the first to write the algorithm though, but it felt like that for a while.

But is it…?

As a programmer I write code every day. And what I do even more is read code. If I have to guess, for each line I write I’ve read about ten times more. And after more than two decades of programming, I’ve seen my share of source code. This taught me something important. Is the piece of code I mentioned the best code I’ve written? I don’t think so.

What I thought the best code should have:

  • Looks smart
  • Amazes people
  • Jumps out
  • ‘I could have never created that!’
  • It is magical!

The code I picked sure had some of these properties. But is it really the prettiest code I’ve written? No..

When you are working in a large codebase the best (and prettiest) code has the following properties:

  • Doesn’t stand out
  • Looks trivial
  • You don’t notice it’s there
  • You hardly ever need to change it
  • ‘Gah, anyone could have written this’

The best code is code you don’t notice, code that doesn’t stand out, code that looks like anyone could have written it. It doesn’t contain smart things, it looks mundane. This is something we should always strive for… simple code.

So what is the best piece of code I’ve ever written? It is the code that will never be git-blamed, nobody will ask questions about it and nobody will even notice it exists.


Biometric passwords: No!

Biometric passwords: No!

Please sit down, we need to have a talk, programmer to programmer.

Over the last decade we’ve had a lot of problems with authentication. For example, we’ve stored plain text passwords in the database. We’ve learned from this and nobody is doing this anymore right? If you are, please deposit your programming-license in the nearest trash can.

Latest challenge: Biometrics

It is time to talk about the latest problem in IT: biometric data.

Some websites are using biometrics, such as your fingerprint, as your password. This sounds great, very hard to fake, unique to you. But there is a problem… what happens when there is a data leak?

If you store passwords in the database (hashed or not), and they get leaked, it is bad. You need to tell all the users to change their passwords immediately. But what happens when you store biometric data and it gets leaked?

The only way to change your fingerprint is this:

Fingerprint removal tool

Rather painful… and even worse, all devices and websites that use your fingerprint have the same password.

We don’t want to share passwords on multiple websites/devices!

Not a password

There is no real solution, as long as you ensist of using biometric data as a password. Even if you use a nice salted hash, it will eventually be leaked, with big consequences.

A better way to use biometrics in authentication is to treat it as a username. It is a great match, it identifies you. It is not your secret password, it is your username. That means you still need to provide a password, but having the added biometric username does increase security a lot. Of course if there is a database leak, your fingerprint can still be stolen, but that is the entire point. If you touch a glass door you’re also leaving your fingerprint. Using fingerprints as password is like dropping pieces of paper with your secret password all over the place.

Fingerprints (and other biometrics) are not secure, you can never change them once compromised, not suited as passwords. If you really want to use it, use them as usernames.


Deprecating Optional.get()

Deprecating Optional.get()

On the OpenJDK core mailinglist (and Twitter) there is a discussion about Java’s Optional. Before diving into that discussion, lets take a look at what Optional does and how you can use it.

Checking for null

What do you do when your code calls an external service or god forbid a microservice, and the result isn’t always available?

Most of the time the protocol you are using facilitates in the optional part, for example in REST you’ll get a 404 instead of JSON. Getting this 404 forces you to think about this scenario and do something when this happens.

But what do you do when you’re calling a framework (on the boundary of your code) and the value isn’t always known?

You either get the value or the result is a dreaded null. This causes a lot of null checks, or bugs where the code just crashes with a NullPointerException.

Example (old skool, Java 7):

    Order order = Database.readOrder(); //can be null
    if(order != null) {
        ProcessResult result = OrderEngine.process(order);
        if(result != null && result.succeeded()) {
            Database.storeResult(result);
        }
    }

This code is not very pleasant to read. But we Java programmers didn’t have or need anything better… until we started to adopt a more functional style of programming.

java.util.Optional

What happens when you are processing a stream and some values are null? You don’t want null checks inside a stream! This is where Java 8’s Optional comes in. If you’re not (yet!?) using Java 8, there are other implementations as well. For example Google Guava has an Optional as well.

Optional is a class that ‘might’ have a given value in it, or not, it is optional. So how exactly is this helpful? Instead of checking for null this wrapper class can handle some situations for you.

For example:

    Optional<Order> order = Database.readOrder();
    Optional<ProcessResult> result = order.map(OrderEngine::process);
    Optional<ProcessResult> filteredResult = result.filter(ProcessResult::succeeded);

    filteredResult.ifPresent(Database::storeResult);

Or the shorter ‘fluid’ version:

    Database.readOrder()
        .map(OrderEngine::process)
        .filter(ProcessResult::succeeded)
        .ifPresent(Database::storeResult);

Even if the Optional is empty in either reading the order or processing the order… nothing breaks. No NullPointerException, nothing, just no executed lambda storing the result in the end. We’ve eliminated the need for a null check.

As you can see Optional can really clean up your code. You don’t need to worry about null checks anymore.

So what is the problem with Optional.get()?

Optional.get() deprecation discussion

Pretty much out of the blue on the OpenJDK mailinglist an email arrived with a webrev (similar to a patch file) that contained the deprecation of Optional.get().

The get() method is too easy to find, and the name isn’t quite what you’d expect, and the webrev author claims there are a lot of cases online where people made the same mistakes.

Many programmers, when they first encounter Optional, don’t know what to do. They look in their IDE and the first thing that pops up is get().

It is just an easy method to call:

    Order order = Database.readOrder().get();

This works fine! Until there is a situation where the value is not available to the Optional. In that case it will throw an NoSuchElementException. How can we solve this? Well, we could do the following:

    Optional<Order> maybeOrder = Database.readOrder();
    if(maybeOrder.isPresent()) {
        Order order = maybeOrder.get();
    }

This is the ‘safe’ way, but it could just as well have been a null check now. There is likely a much cleaner way to process your Optional.

  • If you want to do something with the result, use filter, map, ifPresent (and many others).
  • If you need to return something, either return an Optional yourself, or get a default value by calling orElse, orElseGet or orElseThrow.

This is all you need, why have a get-method?

The proposal on the mailinglist is to deprecate the get() method and rename it to getWhenPresent(). This name change should warn people that it might not be present and they should check isPresent before calling get().

Instead of embracing this change some people on the mailinglist argue against deprecation, some of their reasons:

  • Renaming will break a LOT of code, well, not really break the code, it will cause deprecation warnings
  • getWhenPresent() instead of get() just adds noise to the code, it doesn’t solve anything
  • People should just read the JavaDoc, it clearly states what get() does and throws
  • Guava’s Optional also has the same get() method, they’ve never heard about the problem

The most honest and one of the more powerful replies in the discussion was from Brian Goetz himself:

As the person who chose the original (terrible) name, let me weigh in…

……

I’d like to see it fixed, and the sooner the better.

He is clearly in favor of deprecation… what is your opinion? Let me know in the comments!


The collapse of Java EE?

The collapse of Java EE?

Recently there has been a lot of discussion about the state of Java EE and Oracle’s stewardship. There seems to be happening a lot. There is the fact that a lot of evangelist are leaving Oracle. There have been (Twitter) ‘fights’ between developers from Pivotal and Reza Rahman. And there are the Java EE Guardians, a group formed by Reza after he left Oracle.

And during the last JCP ‘Executive Committee Meeting Minutes’ the London Java Community (LJC) openly expressed their worries:

Martijn said that while he recognizes Oracle’s absolute right to pursue a product strategy and allocate resources in ways that meet their business interests, the LJC is concerned that the lack of progress and the absence of any explanation from Oracle is doing significant harm to the Java community and ecosystem.

He explained that “splinter groups” are discussing taking over both the code work and thought leadership of Java EE, and that many companies are building proprietary frameworks such as microservices stacks, leading to even more fragmentation.

There are splinter groups forming, companies are building frameworks and stacks without following Java EE or contributing to future Java EE specs. People in the blogosphere/tweetosphere are complaining and worrying about it… but is it really a problem?

In my personal opinion: No.

There have always been companies experimenting, pioneering new technologies, without following Java EE specifications. This is for example how the Spring Framework got as big as it did. Remember however: Spring really shaped the future of Java EE, without it we might still be coding EntityBeans.

I think it might not even be a bad thing for Java EE to take a little break. There is a lot of unproven technology happening at the moment, for example there are the reactive frameworks and everything related to microservices. The landscape is changing quickly right now.

The worst thing that Java EE can do is to come up with their own new standards for these technologies while we, as developers, haven’t really worked out the quirks yet. Historically the best Java EE specs (IMHO) are the ones that came late to the party. But those are built on years of experimentation and crystallization. Those specs looked at everything the market had to offer, brought the relevant groups together and made it work.

So there is nothing wrong?

There is one big danger to Java EE right now. It is the fact people are complaining. If we don’t stop this, it might all become a selffulfilling prophecy.

Instead of worrying about Java EE, lets build tools and frameworks that are worth becoming an official spec. For example, look at the work Stephen Colebourne did with Joda Time. He was fed up with the horrible java.util.Date and decided to make something better. After years of programming and growing a huge fanbase it was finally turned into an excellent specification (JSR-310).

If you look at it, the most important thing Java EE might have done for us is bringing the relevant groups together, share ideas and distill the best practices (and writing those down as specifications). It is exactly the opposite of what is happening right now. I don’t mind splinter groups forming, if they get the right people together and work together towards forming solid specifications and implementations, why not?

I’m pretty sure Oracle (with Java EE) will take a look at the proposals and adopt them.

The most important thing is that we keep working together!

Update: Some people have warned me that I’m being too optimistic. But time will tell, maybe Oracle will kill off Java EE, maybe they won’t. Maybe everything will take a turn for the worse, maybe it won’t.

For now I’ll just do what the Dalai Lama suggests: Choose to be optimistic, it feels better.


Flipping the diamond - JEP 286

Flipping the diamond - JEP 286

After my blogpost yesterday Pros and cons of JEP 286 I’ve received a lot of feedback.

The more I’ve been thinking about var/val it seems that my biggest mental hurdle is the Java 7 diamond operator. The diamond operator is good, it eliminates typing, and I like it… but I have the feeling it could be so much better!

Instead of (or in addition to) adding var and val I’d love to see a solution where we could ‘flip’ the side of the diamond operator.

Look at the following example:

    // Java 7 with diamond operator:
    List<Integer> numbers = new ArrayList<>();

    // Proposed var: (saves 3 characters)
    var numbers = new ArrayList<Integer>();

    // Flipped diamond operator: (larger, but more readable IMHO)
    List<> numbers = new ArrayList<Integer>();

In this case adding var/val doesn’t improve much, we still need to specify our generics somewhere, and now it has moved.

But look at the following example:

    // Java 7 with diamond operator:
    Map<String, Integer> someMap = someMethod();

    // Proposed var:
    var someMap = someMethod();

    // Flipped diamond operator:
    Map<> someMap = someMethod();

The thing is: There is much more to win on the LHS with the diamond operator than we currently have with the RHS diamond. In most cases you’re going to call code that has already defined the typing, in all those cases you can skip it LHS.

If it is possible to add var and infer everything, it should also be technically possible to have a flipped side diamond operator right? Or am I missing something?


JavaMail API: Message in EML format

JavaMail API: Message in EML format

Our application was already using JavaMail (javax.mail.*) as a way to inform our users. But for logging purposes we wanted to store all the emails we send in our database (and make them downloadable using our GUI).

It turns out this is pretty easy to do!

Let’s start with some very basic email code we already had in place:

    // Some method to construct a MimeMessage:
    Message message = createMailMessage(input);

    Transport.send(message);

What we need to do now is to ‘render’ the entire email in a binary format, including all the possible attachements, multipart things, from and to headers etc.

It turns out there is a convinient method for doing just that: message.writeTo(OutputStream)

    Message message = createMailMessage(input); // some method to construct a MimeMessage

    // Retrieve the entire message as byte[]:
    ByteArrayOutputStream out = new ByteArrayOutputStream();
    message.writeTo(out);
    byte[] contents = out.toByteArray();

    // Next we store the byte[] in our database (JPA) entity:
    ArchivedMail archivedMail = new ArchivedMail(sender, contents, LocalDateTime.now());
    mailArchive.store(archivedMail);

    Transport.send(message);

Our own little POJO entity (ArchivedMail) is stored in the database with some additional information that allows us to search the messages. The final step is to make a download link and present the email in a readable format to the users.

We’re using Wicket and thus the following example is Wicket code, but you could just as easily create a Servlet to return the data:

    // Add the Wicket link-component to our page:
    add(new Link<ArchivedMail>("wicketLinkId", Model.of(somemail)) {

        @Override
        public void onClick() {
            IResourceStream resourceStream = new AbstractResourceStreamWriter() {
                @Override
                public void write(OutputStream output) throws IOException {
                    // When clicked output a stream which contains the raw byte[]:
                    output.write(getModelObject().getBytes());
                }

                @Override
                public String getContentType() {
                    // Add the content type for an EML file:
                    return "message/rfc822";
                }
            };

            // Schedule the handler to return our resource stream (with a fancy name):
            getRequestCycle().scheduleRequestHandlerAfterCurrent(
                new ResourceStreamRequestHandler(resourceStream).setFileName("mail.eml"));
        }
    });

Conclusion

Using JavaMail (javax.mail) it is very easy to get the ‘raw’ contents of an email when sending it.

This can be stored and downloaded in EML-format. It contains everything you need, mime, multipart, attachements and all the from/to headers.

Happy coding!


Pros and cons of JEP 286

Pros and cons of JEP 286

A couple of weeks ago a new JDK Enhancement Proposal (JEP) has been published: JEP 286

It proposes ‘var’ and possible also ‘val’ as a way to declare local variables. This means that for local variables you don’t need to specify the type of your variable when it can be safely infered.

Some examples:

    // Before:
    int ten = 10;
    List<String> list = new ArrayList<>();

    // After
    var ten = 10;    // infers int
    var list = new ArrayList<String>();  // infers ArrayList<String>

Personally I’m not convinced this is a good idea for Java, but OTOH some of my colleagues and co-workers are very happy with the proposal.

Let’s look at some of the pros and cons of this proposal.

Pro: Less typing!

There is one obvious pro: Less typing.

‘var’ is just three characters, while most other local variables type names are much longer.

Instead of typing int, List, Person or SpringObjectFactoryManagerTemplateProxyDelagate you just have var.

    // Instead of:
    List<Integer> myList = someMethod();

    // You can now have:
    var myList = someMethod();

    // We saved 10 keystrokes! (<- probably less if you know shortcuts in your IDE)

Con: Readability

The biggest advantage of Java over other languages is the readability. The language Java is a bit verbose, but this is actually a good thing when it comes to reading the code.

Code is read more than it is written

Consider the following:

    var myVariable = dependency.calculateSomething();

What is the type of myVariable? When you are writing the code, you probably have a good idea why you called dependency and what you receive as return value.

But when you are reading the code, there is no way of knowing what myVariable is… you probably need your IDE to tell you, or look at the code of the dependency.

I personally think this is a con regarding the JEP. I’d rather have a verbose language where the IDE helps me with autocomplete and hide things… than having a language that needs an IDE to help me makes sense of the code.

Pro: Adding var doesn’t break anything

Some peope think (and argue) that adding this feature breaks backwards compatibility (because of the new keyword).

But this is not true!

When ‘var’ gets added it won’t be a keyword, it’ll be a ‘reserved type name’. This means that the following code for example would be working just fine:

    // this compiles just fine:
    var var = "var";

Con: RHS versus LHS

This JEP focusses on the LHS (left-hand side) declaration by removing the need to specify a type. But recently, in Java 7, Java has introduced the diamond operator to eliminate verbosity in the RHS (right-hand side) declaration. With JEP 286, these two collide:

    // Before Java 7 'diamond operator':
    List<Integer> numbers = new ArrayList<Integer>();

    // With Java 7 'diamond operator', removing generic type on RHS:
    List<Integer> numbers = new ArrayList<>();

    // With JEP 286, we need to add the generic type again at the RHS...
    var numbers = new ArrayList<Integer>();

    // Java, make up your mind!

Pro and con: Refactoring

Some people have argued that, after JEP 286, refactoring can become easier. Look at the following, silly, example:

    public boolean hasClosed() {
        var list = getSomeList();
        for(var item : list) {
        	if(item.isClosed()) {
        		return true;
        	}
        }
        return false;
    }

No matter what getSomeList() returns, it should work as long as it has the method isClosed. I think this is a weird example, because normally you would define an interface with isClosed and every class that implements this interface can be replaced/refactored as well.

There is a counter argument that can be made, refactoring can also be dangerous with JEP 286, look at this (crafted) example:

    var answer = SomeCode.generate();
    System.out.println(answer + 42);

As long as the method generate returns a number the code works fine. But when someone changes the method to return an object or a String, it stops working without failing compilation. This argument however seems valid, but it would also break it you would have inlined the call to ‘System.out.println(SomeCode.generate() + 2);’.

This might make the problem a bit harder and more widespread. I believe there are more cases this can go wrong.

Try it out for yourself

The best way to get a feel for JEP 286 is just to try it out yourself!

There is a pre-compiled version of JDK-9 with JEP availabe for download at the website: iteratrlearning.

Conclusion

After looking at a lot of examples I’m still not convinced that JEP 286 is good nor bad. It can go either way. There are some good pros but also quite a lot of cons.

When discussing this JEP with co-workers and colleagues I often get the following reply:

The arguments you’re using have been used when C# adopted var/val, stop complaining, they did it.

But did you know most coding guidelines for C# warn you for using ‘var’?

Just read these guidelines from Microsoft:

  • Do not use var when the type is not apparent from the right side of the assignment.
  • Do not rely on the variable name to specify the type of the variable. It might not be correct.
  • Avoid the use of var in place of dynamic.
  • Use implicit typing to determine the type of the loop variable in for and foreach loops.

This, combined with readability, makes me lean towards a no for JEP 286 right now.

How about you? Leave a comment!


JShell API, design problems?

JShell API, design problems?

This weekend I continued working my project making a Java REPL adventure game that teaches the basics of programming (in Java). One main goal of starting this project was to get a better understanding of the JShell API, not just the tool. I’ve used the JDK 9 JShell command line tool before, and it worked great. But there is also a big API behind it you can programatically use.

Let me show you how this works:

        JShell shell = JShell.create();
        List<SnippetEvent> events = shell.eval("int i = 10;");

What have we done here? Well first we’ve created an instance of the JShell. This is just like starting the command line tool. Now we can call the eval() function to evaluate Java code. As a result you receive so called SnippetEvents, these are classes containing the information about what happened. Let’s see what the code above returns:

        for(SnippetEvent event : events) {
            System.out.println(event);
        }

        //Output: SnippetEvent(snippet=Snippet:VariableKey(i)#1-int i = 10;,previousStatus=NONEXISTENT,status=VALID,isSignatureChange=true,causeSnippetnullvalue=10)

There is quite some information in there, the event contains the actual ‘snippet’, the piece of code that was executed. Also there is a previous state, status and some other information. This is where the JShell API starts to feel weird to me. To get the Snippet from the SnippetEvent we can call the following method:

        SnippetEvent event = events.get(0);

        // To get the information, we call snippet (not getSnippet()):
        Snippet snippet = event.snippet();

        // Next we cast our snippet to the correct subclass:
        VarSnippet vSnippet = (VarSnippet) snippet;

        // Now we can retrieve information, for example the type we've created:
        System.out.println(vSnippet.typeName());

        //Output: int

        // We can also get the 'value' of the SnippetEvent:
        System.out.println(event.value());

        //Output: 10

There are a couple of points I don’t like about the current JShell API design here:

  • Why are the methods to get information called .snippet() and .typeName(), why not getSnippet() and getTypeName()?
  • Why is the result of vSnippet.typeName() a String?
  • Why is the result of event.value() a String? Didn’t we have an ‘int’?

First the question about the method names. In all of the JDK code they seem to be using the bean-specification, where you have getMethods() for getting and setMethod(…) for setting. I’m not sure why this isn’t the case for the JShell API? Is this a new way Oracle is designing their API’s? Do other new JDK API’s have the same syntax?

Next is the question regarding the usage of Strings. Why is the value() a String when we’ve clearly made something an int? Even worse, let’s create something that is not a primitive:

        List<SnippetEvent> events = shell.eval("jdk.jshell.JShell.create();");
        System.out.println(events.get(0).value());

        //Output: jdk.jshell.JShell@4cc0edeb

It is even worse, the only result we get from event.value() is a String and it is clearly just the .toString() of the actual backing object. For this API I would love to see it returning the actual object instead. I understand the JShell tool uses String’s and communicates the toString(), but shouldn’t the API be more general? Or am I missing something here?

Anyway, this is exactly why more people should try these API’s and share the feedback! Mail your feedback and suggestions to the mailing list.

What do you think? Please leave a comment below, or… even better, share your thoughts on the mailing list


Fantasy names using Java JShell

Fantasy names using Java JShell

I’ve been a big fan of the Java JShell from the early beginning. I was even the first one on the mailing list besides the people from Oracle. I’m working on a weekend/side project involving JavaFX and JShell API now for a couple of days and something was bothering me.

The project I’m working on is targetted at teaching kids to learn Java Programming in a fun way. But automaticly generated variable names called $1, $2, $3 etc are not ‘fun’ enough! Luckily I found the following method in the API:

JShell shell = JShell.builder()
                .tempVariableNameGenerator(new CustomNameGenerator())
                .build();

The name generator has to implement Supplier<String>, with a single method get(), very easy! The generator I’ve made generates semi-random fantasy character names. This is what I’ve ended up with:

-> 10 + 10
|  Expression value is: 20
|    assigned to temporary variable baiduth of type int

-> baiduth * 2
|  Expression value is: 40
|    assigned to temporary variable bleibriss of type int

-> System.out.println(bleibriss + baiduth)
60

-> 

Much better! If you’re interested in the project I’m working on, teaching kids Java with the JShell API and JavaFX, keep an eye out. When I have something worth showing I’ll push it to GitHub so everyone can enjoy it!


Getting to grips with Gatling: Custom Feeder

Getting to grips with Gatling: Custom Feeder

About a year ago someone on my current project did an initial setup of Gatling. This is a performance/stress test tool written in Scala, it is easy to use but very powerful. The basic idea is that you write the test scenario’s in code, the same way as you would create integration/unit tests. The advantage is that the resulting tests are easy to expand, maintain and you’ve got everything under version control.

One very powerful feature Gatling has is the Recorder. It is an application you can launch and it acts like a proxy server. Being a proxy server it can record and log all the calls you do to an application. Without writing any code it can automatically create a stress test!

But in my case someone has already created a scenario for me.

Basic Example

As a programmer it can be a bit daunting when you encounter a ‘foreign language’, in this case Scala. Fear not though, it is very easy to read, extend and write Gatling tests. This is the scenario I started with today:

class OurSimulation extends Simulation {

  // Define some basic HTTP protocol settings:
  val baseHttpProtocol = http
    .inferHtmlResources()
    .acceptHeader( """*/*""")
    .acceptEncodingHeader( """gzip,deflate""")
    .acceptLanguageHeader( """nl-NL,nl;q=0.8,en-US;q=0.6,en;q=0.4""")
    .contentTypeHeader( """text/xml""")
    .userAgentHeader( """Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.120 Safari/537.36""")

  // Define a variables that is inserted into the SOAP template:
  val incrementalId = new AtomicInteger

  // Define the scenario, read from CSV file, add additional variable to the session and post to OurEndpoint:
  val scn = scenario("Our scenario")
    .feed(csv("data.csv").circular)
    .exec(session => session.set("idField", incrementalId.getAndIncrement))
    .exec(OurEndpoint.post("SoapMessageTemplate.txt"))
}

object OurEndpoint {

  def post(soapMessage: String) = exec(http("RequestName")
      .post("""/ourapplication/webservices/OurSoapEndpoint""")
      .body(ELFileBody(soapMessage))
      .check(status.is(200))
    )
}

class OurSimulationLocal extends OurSimulation {

  // The instance we want to run, against localhost (we also have these for other servers)
  val httpProtocol = baseHttpProtocol.baseURL("http://localhost:1234")

  // Setup the test, ramp up 10 users over a period of 20 seconds
  setUp(scn.inject(rampUsers(10) over (20 seconds))).protocols(httpProtocol)
}

The SoapMessageTemplate.txt is just a plain text file containing one SOAP call, instead of data it contains tags like: ${field1}. There is also data.csv which is a CSV file with the field names in the first row, comma separated values in the following rows. The tags in the SOAP call get replaced by the values from the selected CSV row.

There is probably nothing more I need to explain, the code reads like it should. The main startingpoint is OurSimulationLocal, this class sets up the Simulation (using setUp()). The Simulation is contructed in the OurSimulation class, for each call it takes the ‘next’ value from the CSV, adds one additional field and posts the message to OurEndpoint, finally it checks for a 200 reply.

Custom Feeder

Instead of having a big list of different options in a CSV file, my task for today was to make a custom Feeder. It is the class that provides data as input for the SOAP messages. We want to have some random data that still makes sense to the application, based on some averages.

It looked pretty hard to do, not having programmed a lot of Scala. In the end it was a breeze! The only thing that bugs me is that there are so many different ways to write your code in Scala. I’m not experienced enough yet to know if it ‘reads’ like proper Scala code, I don’t yet know what ‘good’ and ‘bad’ code looks like.

object MyFeeder {

  val random = new Random()

  def apply():  Feeder[String] = {
    Iterator.continually(Map(
      ("name", Random.alphanumeric.take(5).mkString.toUpperCase),
      ("country", (if (random.nextDouble() <= percentageBuitenlands) "B" else "NL"))
    )
  }
}

class OurSimulation extends Simulation {

  //... snip ....

  // The same scenario, now using our own feeder:
  val feeder = MyFeeder()

  val scn = scenario("Our scenario")
    .feed(feeder)
    .exec(session => session.set("idField", incrementalId.getAndIncrement))
    .exec(OurEndpoint.post("SoapMessageTemplate.txt"))
}

Writing a custom feeder is pretty easy to do. The apply() function creates an instance of a Feeder[String] which in turn returns a Map containing [String, String] key value pairs. These are then used to fill the SOAP template!

Of course the real thing I build today is a bit larger and more complex, but the code is basically the same!


Keep the Force asleep

Keep the Force asleep

At the moment it is very hard for an online nerd on Twitter. Star Wars ep 7: The Force Awakens has been released into the wild and most of us haven’t seen it yet. But luckely there is a solution, I’ve created a handy bookmarklet.

A bookmarklet is a piece of Javascript code that you can add as a bookmark. When you are on a certain website (like Twitter) you can press the bookmarklet and the Javascript is executed. The bookmarklet code I’m using has been created using this great website: DeClutter Twitter and it removes most Star Wars related tweets from your timeline!

ForceFilter

Just drag this bookmarklet ForceFilter to the bookmark section of your browser. Next time you open Twitter, quickly press the bookmarklet before reading.

The code

Below is the (generated) code, again, made by DeClutter Twitter (all credits belong there):

javascript:
  $('body').bind('ajaxSuccess',declutter);
  
  void(declutter());

  function declutter() {
    // Remove tweets containing these terms:
    var rk=['star wars', 'the empire', 'the rebellion', 'han solo', 'theforceawakens', 'jar jar', 'kylo ren', 'rey', 'finn', 'poe dameron', 'xwing', 'deathstar', 'c-3po', 'r2d2', 'jedi', 'darth', 'sith', 'leia'];

    // Twitter timeline entries have class "stream-item" - loop through them, 
    // removing those that match blacklisted term
    $.each($('li.stream-item'), function() {
        var c = $(this).html(); 
        var p = $(this); 
        // Iterate through the keywords, removing the parent element 
        // where a match is found
        $.each(rk, function(i, v) { 
            // Lower-case the text so case issues don't trip us up
            if(c.toLowerCase().indexOf(v.toLowerCase()) > 0) {
                  p.remove();
            }
        });
    });
  }

Google Guava: Range with LocalDate

Google Guava: Range with LocalDate

This afternoon I was working on implementing a user story with (JodaTime) LocalDate ranges. The problem? I needed to represent a lot of periods, in days. The thing I needed to check: Is a certain LocalDate contained in this period.

First I started looking at JodaTime itself, it has support for Interval(s). But there is a ‘problem’, Interval is based around ReadableInstants, and LocalDate is a ReadablePartial, not a ReadableInstant! So to make my code work I had to do the following:

    // Use two LocalDate instances to create an Interval (the JodaTime way)
    private Interval createInterval(LocalDate fromDate, LocalDate toDate) {
        DateTime fromDateTime = fromDate.toDateTimeAtStartOfDay();
        DateTime toDateTime = toDate.toDateTimeAtStartOfDay();
        return new Interval(fromDateTime, toDateTime);
    }

    // Now to check an interval I needed to call:
    createdInterval.contains(inputDate.toDateTimeAtStartOfDay());

It works, sure, but the conversion to ReadableInstant with toDateTimeAtStartOfDay kind of bugged me. But luckely Google Guava has something to make the code much cleaner: Ranges

Using Range made the code much smaller and easier to read:

    // Use two LocalDate instances to create an Range (the Google Guava way)
    private Range<LocalDate> createRange(LocalDate fromDate, LocalDate toDate) {
        return Range.closedOpen(fromDate, toDate);
    }

    // To check:
    createRange.contains(inputDate);

I really love using Google Guava, it provides solutions for a lot of common programming challenges. Not only that, their API is very very clean and skillfully designed. The use of open/closed in Range.java is very handy and makes it easy to think about corner cases.


Makeblock, robots for kids (and adults)

Makeblock, robots for kids (and adults)

When I left the Port of Rotterdam (over a year ago) as consultant/software developer I got a lot of cool gifts. Amongst the gifts was a small blue box from Makeblock, called the ‘robot starter kit’. It was put away on my desk and sadly I’d forgotten about it. But recently a colleague of mine showed us the robot his daughter made. She got a similar robot from Makeblock for her birthday and it ‘became her new best friend’. He asked her if he could borrow the robot for one day and show it to us at work, she reluctantly said yes.

When I got home I suddenly remembered the blue box I had laying around in the corner of my desk. I decided to show it to my 3 year old daughter. She loved the idea of having her own little robot in her room. Searching online I couldn’t find a suggested appropriate age for these robots so I figured: if she wants to try, just go for it!

Starting the assembly of the robot (bananas not included)

She immediately jumped in and started handling the screwdriver like a pro. I was amazed to see the dedication, concentration and dexterity, when I was younger I had none of the three. In about half an hour we had the base constructed and she lost interest for a little while. I decided to continue alone for 10 minutes and the only thing left to do was assembling the tank tracks. So I called her over and we decided to hold a little race, 20 rubber pieces for me, 20 for her. It was a pain to assemble, but by introducing a small competition I only had one track to assemble, she did the other one all by herself.

The cool thing about these robots is their simplicity: The frame goes together quickly with a couple of screws and removable rivets. All the electronics are connected by RJ25 (telephone) jacks. The kit I got had an IR-receiver (+ remote) and ultrasonic sensor. The remote is great because she is too young for programming. The moment you put it together and place the batteries you can start driving the robot around. She loved it, look at her proud smug face:

I've build a robot, now what?

Next thing I’ll do it to teach it how to dance! Then I’ll teach it how to clean my room.

That evening, when she was vast asleep, I started working on the programming. The main board in this robot is compatible with an Arduino. To program the robot you connect it to your laptop using USB and fire up the Arduino IDE with the Makeblock library. Commands are very simple to use, for example reading the distance sensor is: ‘ultraSensor.distanceCm()’. That night, when my daughter was sleeping, I added an automatic drive-mode so the robot would drive around randomly avoiding all obstacles with help of the ultrasonic sensor. When she woke up I told her ‘go ahead and hit the 0 on the remote!’. She was blown away, her little robot had grown a mind of its own!

When she is a bit older we’re going to try out the mBlock/Scratch plugin. Scratch (for those unfamiliar) is an environment where you can visually program with graphical blocks. It is a drag and drop environment and works very well as a first programming tool for kids. It is also something we regularly use at Devoxx4Kids

Things I like about the Makeblock robot:

  • Easy to assemble (even for a 3yo)
  • Instructions had a lot of images
  • Fun to program

And of course, it isn’t perfect, things I didn’t like:

  • Should come with a CR2025 coin cell battery for the remote (I had to race to the store to get one)
  • Instructions only had images, nothing else (bit too brief)

Nontheless: Every parent should buy one!


Hacking a broken Roomba (with Raspberry Pi)

Hacking a broken Roomba (with Raspberry Pi)

roomba-googledag Every couple of weeks we organize a hack-day at work. Being a (Java) software consulting company we usually work on exploring new Java frameworks or we contribute to open source projects. But more and more we step out of just software and bring Raspberry Pi’s and other hardware. Another example where we hacked on hardware is playing with Software Defined Radio (SDR), my colleague Bert Jan did a talk about this at Devoxx 2015.

Broken Roomba

A couple of days before our last hack-day my wife was carrying our old ‘broken’ Roomba vacuuming robot. Most vacuuming parts were worn down and the front wheel keeps dropping out of its socket. She was going to throw a robot out! I can’t let this happen, I have to save the poor bastard.

So I started to do a little research. I’d heard about the Roomba Create before. It is a STEM-version of their vacuuming robot, for hacking purposes. It doesn’t have the cleaning parts, instead of has more connectors, made especially to be hacked/improved, it is the perfect base for your own little robot! I was very pleased to find that even my 500-series Roomba has a hidden mini-DIN connector that allows serial communication. iRobot provides a full listing of all the possible commands and outputs, which is a great thing!

Connecting a Raspberry Pi

Here is what my idea was: Take a mini-DIN cable, connect it to a Raspberry Pi and drive the Roomba myself! But I quickly run into two problems:

  • How do I connect the Raspberry Pi to a DIN cable?
  • How do I power the Raspberry Pi?

After some research I was able to solve both of the problems.

Mini-DIN to Raspberry Pi

After some research I found out some good and some bad news. First the good news, the Raspberry Pi has an UART TX and RX pin. These can be used to communicate with the Roomba without any effort.

The bad news? The Roomba has logic pins that have 0-5v and the Raspberry Pi has 0-3.3v. You can’t directly combine the two together or you’ll risk blowing a fuse on the Raspberry Pi. There is however a pretty simple solution, that is to use a Logic Level Converter. This is a small board that you give 3.3v + GND on one end, 5v + GND on the other end. Now it has four logic pins left that can be used to connect 3.3v logic to 5v logic bi-directionally.

So with a bit of creative freeform-soldering I ended up with the following extension to my Raspberry Pi:

freeform-soldering

The only thing left to do next is to connect the right DIN-wires to the Raspberry Pi. But luckely the documentation provided by iRobot had very clear images showing all the mini-DIN pins and what they do.

Powering the Raspberry Pi

The next challenge was actually power the Raspberry Pi. The Roomba has a battery pack, 16-17v, so that would be perfect. This seemed to be an easy challenge initially because the DIN-cable also provides access to the battery pack. But just before I hooked the Raspberry Pi to the (unregulated) battery wires of the DIN cable I found that there are actually some fuses between the DIN connector and the battery pack. The Raspberry Pi would draw too much current for these fuses!

Next I opened up the Roomba completely. Just below the ‘Dock’ button are two big pads directly connected to the battery pack. I soldered two wires to these pads. This gave me 16v (without the fuses) to power the Raspberry Pi. But how do I step this down to the 5v needed by the Raspberry Pi? I connected the wires to a UBEC and soldered a mini-USB cable to the output. Connected to the Raspberry Pi and… it worked like a charm!

Programming the Roomba

When we connect the Raspberry Pi to the Roomba using the DIN cable, it starts to play weird error-beeps. This is because by default the Raspberry Pi writes all console output to the serial connections (!). Please read this link to disable it.

Like I said, the serial commands needed to drive the Roomba are perfectly documented. Using the GNU RXTX library (RXTXcomm.jar) we can write the following Java code:

    // On the Raspberry Pi the Serial communication pins are connected to /dev/ttyAMA0
    CommPortIdentifier portIdentifier = CommPortIdentifier.getPortIdentifier("/dev/ttyAMA0");
    if( portIdentifier.isCurrentlyOwned() ) {
        System.out.println( "Error: Port is currently in use" );
    } else {
        CommPort commPort = portIdentifier.open(this.getClass().getName(), 2000);

        if( commPort instanceof SerialPort) {
            SerialPort serialPort = ( SerialPort )commPort;
            serialPort.setSerialPortParams(
                    115200,
                    SerialPort.DATABITS_8,
                    SerialPort.STOPBITS_1,
                    SerialPort.PARITY_NONE );

            InputStream in = serialPort.getInputStream();
            OutputStream out = serialPort.getOutputStream();

            ( new Thread( new SerialReader( in ) ) ).start();
            ( new Thread( new SerialWriter( out ) ) ).start();
        }
    }

Now we can send commands to the OutputStream, like documented. Using this I was able to drive the Roomba through SSH to the Raspberry Pi from my laptop!

For example, we can do the following:

    
    // Write imperial march as song '0':
    write(out, 140, 0, 9, 57, 30, 57, 30, 57, 30, 53, 20, 60, 10, 57, 30, 53, 20, 60, 10, 57, 45);
    // Play song '0':
    write(out, 141, 0);
    
    ... snip ...

    private void write(OutputStream out, int... data) throws IOException {
        // Sigh, unsigned Java:
        byte[] output = new byte[data.length];
        for(int i = 0; i < data.length; i++) {
            output[i] = (byte)(data[i]&0xFF);
        }
        out.write(output); 
    }

If you remove the DIN cable, it’ll still function perfectly as cleaning robot. So now I have a Star Wars bot to program during the day, and a cleaning robot at night.

Happy hacking!