royvanrijn

Introduction to Java Notebooks

Wed, 04 Jan 2023 23:35:00 +0100

Java Notebook is a new and exciting way to explore the Java programming language. It works a lot like Jupyter Notebooks.

Notebooks are a community standard for communicating and performing interactive computing. They are a document that blends computations, outputs, explanatory text, mathematics, images, and rich media representations of objects.

It allows you to interleave blocks of text (in Markdown), editable and executable Java code blocks and output blocks.

[1] [Run] [Edit]

System.out.println("Hello world!");

> Hello world!

As you can see, when we run the code block above, the output is immediately updated. It is similar to JShell, but in the browser.

Maven support

Notebooks are an ideal tool to teach programming languages and to display and explore libraries and frameworks. Notebooks also support the import of external libraries using Apache Maven. Here is an example using the load command to import Apache Commons.

[2] [Run] [Edit]

load org.apache.commons:commons-lang3;
import org.apache.commons.lang3.StringUtils;

System.out.println(StringUtils.reverse("Hello world!"));

> !dlrow olleH

Data processing with jandas

It is also possible to directly import resources from internet and parse them to Java objects and records using frameworks like jandas (loosely based on pandas).

Here is an interactive example:

[3] [Run] [Edit]

load jandas:jandas:0.24.0;
import org.jandas.*;

record Song(int rank, String songName, long amountStreams, String artist, Date releaseDate) {}

var songs = CsvReader.read("https://www.royvanrijn.com/examples/data/spotify_top_1000.csv", Song.class);

System.out.println("Lady Gaga currently has " +
    songs.stream().filter(s -> s.artist.equals("Lady Gaga")).count() + " top 1000 hits.");

System.out.println("Ed Sheeran has " +
    songs.stream().filter(s -> s.artist.equals("Ed Sheeran")).mapToLong(s -> s.amountStreams).sum() + " total streams");

System.out.println("Bohemian Rhapsody was released: " +
    songs.stream().filter(s -> s.songName.equals("Bohemian Rhapsody")).map(s -> s.releaseDate).findAny());

> Lady Gaga currently has 6 top 1000 hits.
> Ed Sheeran has 26513653 total streams.
> Bohemian Rhapsody was released: Fri Oct 31 00:00:00 CET 1975

Hosting your own Notebooks

It is very easy to host your own notebooks. Users can run notebooks locally (using Docker) or use one of the following free cloud services to run your Java Notebook:

If you build or maintain a framework or library, or if you have a website explaining Java programming: Please add these interactive notebooks to your website. This makes it incredibly easy for new developers to get acquainted with your code.

Wake up!

Sadly (as you might have noticed) the examples above do not work. It was all just a dream. There is no such thing as a (widely used) Java Notebook.

The reason I wrote this post is the famous TIOBE Index of programming languages.

Today is the first day since 2001 that Java isn’t in the top 3.

That is why we, as the Java community, should look at our competition. What did Python do right? Why is it topping the chart?

A major reason, I think, is the feedback and steep learning curve of Java. If you want to parse some data, would you rather download a Java Runtime, install an IDE, install Maven etc? Or would you rather open the browser, have documentation and tutorials inlined with running examples? This is SO much easier.

The same is true for learning frontend/Javascript, there are so many websites that allow you to jump right in and test your code, sites like jsfiddle or codepen.

My wish for 2023: Oracle, Microsoft, Google, Azul… PLEASE make this a reality.

They do exist…

Update 2023-05-01:

Almost forgot: There are some online Java Notebooks that provide much of the functionality given above, here are some options you can consider:

Apache Zeppelin (a Jupyter-like runtime that has Java bindings)
IJava (a Jupyter kernel for executing Java code)
And more, mostly single developer hacked/weekend projects

The main problem I have isn’t that these project don’t exist; we just don’t utilize them, at all. There is no widespread support for this. It would be such a powermove if Oracle launched a similar product and encouraged libraries to embed it (running on their cloud for example).

If you want to get into AI/deep learning for example, here are two examples, Java or Python, I know which I’d pick:

Java: Quickstart DeepLearning4J:

Twenty steps of installing Java, Maven, git, IntelliJ, check out a project…

Python: Tensorflow Quickstart for Beginners:

No setup at all; from the start you’re importing Tensorflow and running the examples.

It isn’t all bad, here is an example where it does work:

Java: DJL

Easy to overlook, but press the top button “Run this notebook online”. Using IJava and Jupyter, yay!

Perfect for: https:://dev.java

I’d love for dev.java to become more engaging or interactive, something like a notebook. Have a place online where you can “Try now” without having to install everything.

Currently the Getting Started goes as follows:

Open a plain text file, write your first class, without code formatting etc.
Now you could compile it and run TADAA!
But wait, you need to install a JDK first
Download JDK
Install JDK
Learn to use the compiler
Now run it.

Nobody got time for that anymore, sure, perhaps after you’ve decided to really go for it. But with notebooks you get instant results, you can tinker away, run stuff in the browser and see the results, perfect.

Anywaaaay, enough ranting for today ✌️

Solving the daily calendar puzzle

Thu, 07 Jul 2022 20:42:12 +0200

About a month ago I was watching a YouTube video where YouTuber Drew Gooden was struggling with a “weird” puzzle:

This is what the puzzle looks like:

It has 10 pieces that can be flipped/rotated and placed on a calendar. They claim there is a solution for each day/month/weekday combination. In the YouTube video Drew is struggling and says:

The problem is, I’ll never be able to prove that

But… what if we can?

As a programmer this sounds like a perfect little puzzle to solve.

Solving the puzzle

To solve this, I first encoded all the puzzle pieces, this is relatively easy to do because it’s just a 2D grid.

Next I wrote the code to generate all the possible rotations of a certain puzzle piece (and eliminated duplicates). Some pieces have more rotations than others because some are the same after a 180 degree rotation, others are not.

Next I blocked the target puzzle solutions on the board, for example “2nd June Monday”, placing three markers.

After that I just go through all possible piece/location combinations, placing the first puzzle piece in the top position and continuing down.

This was fast enough to solve the first puzzle after a couple of minutes.

But… not fast enough to generate all: 12 (months) x 31 (days) x 7 (weekdays) = 2.604 solutions

Example output: The image above has a solution for Thursday 8th of April. The code outputs this as:

Solution for: 8/4 [5]:
0 0 0 X 1 1 X 
2 2 0 0 4 1 X 
2 2 3 3 4 1 1 
X 2 3 4 4 5 5 
7 6 3 3 5 5 9 
7 6 6 6 6 8 9 
7 7 7 8 8 8 9 
X X X X X 8 9 

Each number 0-9 stands for a certain puzzle piece. The X’s are blocked off (solution and edge).

Improve solving speed

To improve the speed of the solver I decided to do a small flood fill after placing each puzzle piece. If we’ve created a spot where there are just 4 places left (a small island) I stopped looking further, because no puzzle piece will ever fill this spot.

This small algorithmic improvement was enough to make it fast enough to generate a puzzle solution for every single combination.

Links

Solutions: If you’re just looking for all the solutions: solutions.txt.

Code: If you want to take a look at the (Java) code, check out PuzzleADaySolver.java.

All The Music: the Megamix

Wed, 23 Mar 2022 22:23:03 +0100

Or a clickbait title:

How I became the world’s most prolific DJ, using code.

This week I stumbled across a cool project: All The Music.

Damien Riehl (programmer/copyright attorney) and Noah Rubin (programmer) decided to generate all possible songs with the basic 8 major notes (C4,D4,E4,F4,G4,A4,B4 and C5) with length 12. All these songs have been ‘freely’ released under the ‘Creative Commons’ license. Their goal is to stop copyright claims on melodies.

While watching their excellent TED talk and hearing about the challenges they had to generate these songs, my head instantly made some connections. They generated all songs of length n=12 with k=8 notes, this amounts to a staggering n^k = 8^12 = 68,719,476,736 unique songs.

All these songs are 12 notes long and have their own MIDI file which adds even more overhead. The size of this dataset is huge, 1.2TB compressed using GZIP.

Using a de Bruijn sequence

This is when I got an idea: perhaps we can use a de Bruijn sequence for this?

I’ve blogged about those sequences before, basically it is an optimal way to arrange these N elements into a single sequence so that each and every combination of K-length is present in the sequence.

For example if we have all combinations of 0,1,2 of length 4 the naïve way would be to do it:

Instead when creating a de Bruijn we have:

000012200210002212021211212222011221022211012101001011112001102111002012010202202[000]

Every possible 4-length combination/permutation is present in this single line (check them!).

What if we could remix every possible 12 note melody into one huge megamix!?

That would mean I’m mathematically the world’s best DJ, remixing almost all existing songs including EVERY song from the All The Music dataset into one song.

Setting to work

I already have some very efficient code to generate these sequences. What if I output the sequence as a single MIDI file?

Because a de Bruijn sequence usually wraps around, if we want to create all n=12 length melodies we’ll need to append the first n-1 notes to the end of the sequence (which I’ve done above as well). This means we’ll need just a single MIDI file with 68,719,476,747 unique notes in it.

This gave me a tiny problem: a MIDI file has a ‘LENGTH’ field stored in just 4-bytes. And 2^32 is only 4,294,967,295. So we’ve hit a technical problem, we can’t fit our remix into a single MIDI file.

To solve this I decided to cut the single song up into a collection of ‘smaller’ more managable songs. In the end I settled on 2052 unique songs that create one huge megamix album. On this album is every single song possible with notes C4,D4,E4,F4,G4,A4,B4 and C5 of length 12. The same as is contained in the ATM’s dataset.

When breaking up a de Bruijn sequence, each new song has to repeat the final n-1 notes of the previous song, that way each melody is contained in full. For example if we split the above sequence into two parts we’ll need to do:

Song 1: 000012200210002212021211212222011221022211012
Song 2: [012]101001011112001102111002012010202202[000]

This results in the following:

1 remix album: debruijn8-12.tar
Size: 16.735.957.504 bytes (16,75 GB on disk)

Contains:

2052 GZIP-ed MIDI songs
2051 songs with a 33,500,000 note melody
1 song with a 10,999,308 note melody

Really? Every song?

Let’s listen to some songs that are in the dataset (somewhere):

Example 1, Twinkle Twinkle:

Example 2, Jingle Bells:

Example 3, Can You Feel The Love Tonight.

All possible 12 note melodies are in the remix.

Is this better?

This got me thinking, why didn’t Damien and Noah go for this approach? It is much smaller and faster to generate (in a single morning).

So I turned to Twitter and asked Damien Riehl! And sure enough: his answer makes total sense:

We had initially considered a “de Bruijn” sequence. But if we were to use a single file, that would have down sides:

If someone infringes our work, it would only be a tiny percentage (0.0000000001%?) of the “work” — so someone would argue “fair use”

Same idea with others incorporating ATM works in theirs (“tiny percentage”)

So our technical/legal design is “One MIDI file per melody” — which I think is a legal feature, not a bug. 🙂

Of course I should have known there was a valid reason. He encouraged me to continue though and so I generated my own de Bruijn album. Now I can say I’ve officially remixed 68,719,476,736 songs. Is there like a Guiness Book of World Records entry for me now?

If you’re curious what this remix sounds like, here is a snippet:

It was a fun exercise! I really love de Bruijn sequences and learned a lot about streaming GZIP/File API’s to easily store everything (generating the sequence first isn’t an option).

The album is, for now, only stored on my hard disk, but I’m working with Damien to get the songs added to their ATM collection on the Internet Archive.

Oh, and you can’t have a remix album without a proper album cover:

Roadtrip to Ludolph van Ceulen

Mon, 14 Mar 2022 03:14:15 +0100

Today is March 14th (3/14), otherwise known as Pi Day!

Calculating Pi

To celebrate Pi Day people often try to come up with cool and interesting ways to calculate π, or to celebrate the number in weird ways.

Here is a great example by the excellent Matt Parker (from Stand-up Maths):

He’s using the “Isaac Newton”-way to calculate π by hand. This method was discovered by Isaac Newton and German mathematician Gottfried Wilhelm Leibniz in 1665. And it uses ‘infinite series’.

Before Newton

The method Newton used (infinite series) was mathematically superiour to all previous methods of calculating π.

That does raise a question: How did they calculate π before 1665?

First let’s take one step back, what exactly IS π? π is a mathematical constant: the ratio between a circle’s circumference and diameter. This means that if you have a circle with a diameter of 1, the circumference is π (~ 3.14159…).

Let’s try to come up with a very rough estimate what this ratio needs to be. What if we put our unit-circle with length 1 inside a square:

We can conclude that the circumference, and therefor π, is at most 4 x 1 = 4, this gives us the first upper bound.

Now lets look at the other case, where the square fits inside the circle. In this case we can see that we have an equilateral triangle, with two sides a that make the square. The length of a is easy to calculate using Pythagoras theorom, the long side has a diameter of 1, this means a needs to be a^2 + a^2 = 1. If we rearrange everything we’ll come to the conclusion that a = sqrt(0.5).

This means that the circumference of the smaller square is 4 x sqrt(0.5) = ~ 2.82...

We can conclude that π lies somewhere between 4 and 2.82 🎉

Archimedes method

But can we do even better? Sure! What if, instead of placing our circle between two squares, we use pentagons, hexagons, or shapes with even more sides?

As you can see, the shape get more and more rounded. So the upper and lower bounds converge towards our beloved π!

This method is known as Archimedes method. Archimedes calculated, using a shape with 128 sides, that π had to be between 3.141 and 3.142.

Ludolph van Ceulen

Now it’s time to introduce our hero of today: Ludolph van Ceulen

Ludolph spend his entire life to improve the calculated value of π. Instead of using a 128-sided shape like Archimedes, he calculated π using shapes with up to millions of sides. Let that sink in, a shape with a million sides, by hand. By the time of his death van Ceulen managed to calculate π up to 35 digits, an incredible feat.

Ludolph was so pleased with his life goal that he requested the upper and lower bounds to be carved into his tombstone. Sadly the original tombstone went missing not long after his death. However a new tombstone was added to the famous St. Peter’s Church in Leiden (The Netherlands) in the year 2000.

Visiting Leiden

About a month ago, my girlfriend and I, bored in lockdown, decided to take a roadtrip and visit the church in Leiden to see the tombstone for ourselves.

And there it is, here is a better picture of the plaque:

I’ve tried to translate the plaque from ‘old Dutch’ into English and it reads something like:

Buried here is Mr. Ludolph van Ceulen.
He was a Dutch professor of Mathematics at the university of this city (Leiden).
Born in Hildesheim in the year 1540 at 28th of January.
Passed away 31st of December 1610.
During his life, through hard labour, he found the circle's circumference compared to the diameter to be...

If the diameter is 1
The circumference is at least:

3141592653589793238462643383279502881
-------------------------------------
100000000000000000000000000000000000

And at most:

3141592653589793238462643383279502891
-------------------------------------
100000000000000000000000000000000000

And if the diameter is 100000000000000000000000000000000000

The circumference is at least 
314159265358979323846264338327950288

And at most:
314159265358979323846264338327950289

He spend his entire life to improve the upper and lower bounds, using more and more sided shapes, up to millions of sides. He was also the last person to improve the records using this labour intensive method. The method co-discovered by Isaac Newton (using infinite series) proved to be much easier and faster to calculate. After that calculating π by using complex shapes was a thing of the past.

If you’re ever in Leiden, be sure to visit the church and pay respects to the incredible diligence of Ludolph van Ceulen.

Happy Pi Day!

From Wordle to Nerdle

Thu, 03 Mar 2022 08:01:25 +0100

About two months ago I wrote a blogpost about what strategies you could use to solve Wordle in the most efficient way. In the end I wrote a program that looks ahead a single guess and finds the word that gives you the most information.

The main hype around Wordle (and all of the clones), seems to have settled down now… but I still play some of them.

Nerdle

One clone I really enjoyed playing the last couple of weeks has been Nerdle. In the game Nerdle all your guesses need to be a valid mathematical calculation. For example: 31 + 56 = 87.

In this game, you can’t just write down anything, each and every try needs to be a valid calculation.

Generating the guesses

Yesterday, when I had a bit of time with a colleague, we set about trying to adapt the Wordle-bot into a Nerdle-bot. In theory this would be extremely easy, the only hard part is getting a valid “wordlist”.

When I created the Wordle bot, I just downloaded a list of all valid Wordle words. But in this case I set myself the challenge to create all the valid guesses from scratch.

Patterns

First, let’s look at some of the rules. There are 8 characters you can fill, and the final part will always be:

=N with N is a single digit number between 0-9
=NN with NN being between 10 and 99; or
=NNN with NNN being between 100 and 999

It’s easy to generate all those options. However: the part before that is a little bit harder.

Calculations can, because of the limited size, be just one of two patterns (of various sizes): Value1 Operator Value2 or Value1 Operator Value2 Operator Value3

For example 12+34=46 or 10/2+1=6

These patterns are build up from left to right, starting first with a single digit number, appending the first operator, etc, until we have all valid 4, 5 or 6-character calculations.

Now obviously, the tricky part is that we need to evaluate all these calculations and check if the result fits in the amount of characters we have left, for example =NN! This code isn’t pretty… at all, even though I tried to make it generic. I won’t share it haha.

Bugs/problems

I quickly ran into the first big obstacle: How do you calculate the result of something like 1+10/2? You’ll first need to evaluate 10/2, store the result, and finally do 1+5 and check if =6 fits (which it does).

After writing some code and making sure we always evaluate in the correct order in case of Value1 Operator Value2 Operator Value3 I thought we were done. It showed me there are about 13k valid guesses. And this was wrong.

The first bug I encountered was silly. I falsely put in a rule that no result can be zero. But in some cases zero’s are allowed, for example ending with =0. After fixing this bug I found out I was still missing some valid guesses.

After diff-ing my list with a list I found online I noticed I was missing things like: 9/6*8=12

Because Nerdle doesn’t handle fractions, I thought that the divides should never have remainders. Thus I marked 9/6 invalid because it doesn’t evenly divide. However… we are multiplying afterwards (!) so the actual result (12) IS a valid guess for Nerdle.

Results

After fixing all the bugs I had a complete list of all valid Nerdle guesses, and here are some stats:

There are 17723 valid Nerdle answers

6839 of which contain a +
7810 of which contain a -
5574 of which contain a *
4024 of which contain a /
8607 of which contain two math operators
5890 end with =N
10515 end with =NN
1318 of which end with =NNN

And when I fired up the Wordle-checker, my code thinks the best start is: 49-37=12

Other great starters are: 39-24=15 34+27=61 9+8/2=13 7+8-13=2

After writing the bot I’ve tried it against the last three Nerdle challenges, and each game was solved after just three guesses!

For example yesterday the bot calculated the following guesses:

49-37=12 : ⬛⬛🟩🟪🟪🟪🟪⬛
31+56=87 : 🟪🟩⬛⬛⬛🟪⬛🟩

After just these two guesses there is only ONE valid guess left: 11-1-3=7: 🟩🟩🟩🟩🟩🟩🟩🟩

An algorithm for Wordle

Fri, 07 Jan 2022 08:01:25 +0100

If you’ve used Twitter during the begining of 2022 you’ll almost certainly have seen people posting tweets like this:

Wordle 202 3/6

⬜⬜⬜⬜🟨
🟨⬜⬜⬜⬜
🟩🟩🟩🟩🟩

This is the result of playing the latest viral game “Wordle”. The concept is very easy, you have to guess a 5-letter word each day. For each guess you get the basic “Mastermind” reply, green for the correct character, yellow if it’s in the wrong spot.

This game was fun, for a couple of days, but my curious mind started to wonder… what are the BEST words to play?

So I downloaded a huge list of 5-character words and started playing around with the dataset.

Update: I’ve updated the post using the ‘correct’ Wordle word-list, there are two lists, one with possible secret words, and one list with all words valid for guessing.

Eliminate by frequency

The first thing I tried was to try and eliminate the most frequently used characters first. With 5 guesses we can try and guess the most frequently present letters in the English language.

To get the frequency of the letters I used the following code:

        // Read all the words:
        List<String> words = Files.readAllLines(Path.of("wordle.txt"));

        // Count all the characters frequencies:
        final Map<Character, Long> frequency = words.stream()
                .flatMapToInt(w -> w.chars())
                .mapToObj(i -> (char)i)
                .collect(Collectors.groupingBy(
                        Function.identity(),
                        Collectors.counting()
                ));

        // Sort the characters by frequency and print:
        frequency.entrySet().stream()
                .sorted((e1, e2) -> Long.compare(e2.getValue(), e1.getValue()))
                .forEach(entry ->
            System.out.println(entry.getKey() + " " + entry.getValue())
        );

This results in the following distribution:

Using some simple code it’s easy to find pretty good words to start with, for example:

laser -> tonic -> dumpy

These three guesses will tell you information about the 15 most common characters in all of the 5-letter words. This is “a” strategy, but it’s far from perfect. We can do much better!

Most information per guess

Each time we guess a word, we get a reply back, it can be something like “⬜⬜⬜⬜⬜” or “🟨⬜⬜🟩🟨”. What we want to do is optimize for the information we’re getting in return. Each guess we do should eliminate most options.

If we go over the entire list of words, and compare all the replies we could get back against all other words, we get a distribution of the returned patterns.

I’ve decided to write some code to do this. For example if we pick the word smile, these patterns can be returned:

🟨⬜⬜⬜⬜: 1464 ⬜⬜⬜⬜⬜: 1352 ⬜⬜⬜⬜🟨: 1122 🟨⬜⬜⬜🟨:  967 ⬜⬜🟨⬜⬜:  690 ⬜⬜⬜🟨⬜:  610 
⬜⬜⬜⬜🟩:  504 🟨⬜🟨⬜⬜:  468 🟩⬜⬜⬜⬜:  467 ⬜🟨⬜⬜⬜:  417 🟨⬜⬜🟨⬜:  263 🟩⬜⬜⬜🟨:  252 
⬜⬜🟨⬜🟨:  251 ⬜⬜🟩⬜⬜:  196 🟨🟨⬜⬜⬜:  191 🟨⬜🟩⬜⬜:  187 ⬜⬜⬜🟨🟨:  183 ⬜⬜🟨🟨⬜:  154 
⬜🟨⬜⬜🟨:  154 🟨⬜🟨⬜🟨:  147 🟨⬜⬜⬜🟩:  136 ⬜⬜🟨⬜🟩:  136 ⬜⬜⬜🟩⬜:  128 🟨⬜⬜🟩⬜:  117 
⬜🟨🟨⬜⬜:  115 ⬜⬜⬜🟨🟩:  115 🟩⬜🟨⬜⬜:  112 🟩⬜⬜⬜🟩:  109 🟩⬜⬜🟨⬜:  107 ⬜🟨⬜🟨⬜:  106 
⬜🟨⬜⬜🟩:   99 🟩⬜🟩⬜⬜:   87 ⬜⬜🟩⬜🟩:   82 ⬜⬜⬜🟩🟩:   79 🟩🟨⬜⬜⬜:   63 ⬜⬜🟩🟨⬜:   63 
🟨⬜⬜🟨🟨:   62 ⬜⬜🟩⬜🟨:   58 🟨🟨⬜⬜🟨:   53 ⬜⬜⬜🟩🟨:   52 🟨⬜⬜🟩🟨:   49 ⬜⬜🟨🟩⬜:   46 
🟩⬜⬜🟩⬜:   43 🟨⬜🟩⬜🟨:   41 🟩⬜🟨⬜🟨:   41 🟨⬜🟨🟩⬜:   36 ⬜⬜🟩🟩⬜:   35 ⬜🟨🟩⬜⬜:   31 
🟩⬜🟩⬜🟩:   27 ⬜🟩⬜⬜🟨:   27 🟩⬜⬜🟨🟨:   27 🟨⬜🟩🟩⬜:   25 🟨⬜🟩🟨⬜:   25 ⬜🟨⬜🟩⬜:   23 
🟩⬜⬜🟩🟨:   20 🟨🟩⬜⬜⬜:   20 🟨⬜🟩⬜🟩:   19 🟨🟨🟩⬜⬜:   19 ⬜⬜🟨🟨🟩:   18 🟩⬜🟩⬜🟨:   18 
🟩⬜⬜🟩🟩:   18 🟩🟩⬜⬜⬜:   18 🟩⬜🟨🟨⬜:   17 ⬜⬜🟨🟨🟨:   17 ⬜⬜🟩🟨🟩:   15 ⬜🟩⬜⬜⬜:   15 
⬜🟩🟨⬜⬜:   15 ⬜⬜🟩🟩🟩:   14 🟩⬜⬜🟨🟩:   14 🟩⬜🟩🟩⬜:   13 🟩⬜🟩🟨⬜:   13 ⬜🟨🟩⬜🟩:   13 
🟩🟨⬜⬜🟨:   13 🟩🟨🟩⬜⬜:   12 ⬜⬜🟨🟩🟩:   12 🟨🟨⬜🟨⬜:   11 ⬜🟩🟩⬜⬜:   11 ⬜🟩⬜⬜🟩:   11 
⬜🟨🟨⬜🟩:   10 🟩🟨🟨⬜⬜:   10 🟨🟩🟩⬜⬜:   10 ⬜🟨⬜🟩🟩:    9 🟨⬜🟨⬜🟩:    9 🟨⬜⬜🟨🟩:    9 
🟩⬜🟨⬜🟩:    8 🟩🟨⬜⬜🟩:    7 🟨⬜🟨🟨⬜:    7 ⬜🟨🟨🟨⬜:    7 🟨🟨🟨⬜⬜:    7 🟨⬜🟩🟩🟨:    6 
🟩⬜🟨🟩⬜:    6 ⬜🟩🟩⬜🟩:    5 🟩🟩⬜⬜🟨:    5 ⬜🟩⬜🟩🟩:    5 🟩🟩⬜⬜🟩:    5 🟨🟩⬜⬜🟨:    5 
🟩🟩🟩⬜⬜:    5 ⬜🟨⬜🟨🟩:    4 🟨⬜⬜🟩🟩:    4 ⬜🟩🟨⬜🟨:    4 🟩🟩⬜🟩⬜:    4 🟨🟨⬜⬜🟩:    4 
🟩⬜🟩🟨🟩:    4 ⬜🟩🟨⬜🟩:    3 ⬜🟩⬜🟨⬜:    3 🟩🟨⬜🟨⬜:    3 🟩⬜🟩🟩🟩:    3 ⬜🟨🟩🟩🟩:    2 
🟩🟨🟩⬜🟩:    2 🟨⬜🟨🟩🟩:    2 ⬜⬜🟩🟩🟨:    2 🟩🟩🟨⬜⬜:    2 🟩🟩⬜🟩🟨:    2 🟨🟩🟨⬜⬜:    2 
⬜🟩⬜🟩⬜:    2 ⬜⬜🟩🟨🟨:    2 🟩🟨⬜🟩⬜:    2 ⬜🟨⬜🟩🟨:    1 ⬜🟩🟨🟨🟨:    1 ⬜🟨🟨⬜🟨:    1 
🟩⬜🟨🟩🟩:    1 ⬜🟨⬜🟨🟨:    1 🟩🟨🟨⬜🟩:    1 ⬜🟨🟨🟩⬜:    1 🟨🟩⬜🟨⬜:    1 🟩🟩🟩🟩🟩:    1 
🟨🟩⬜⬜🟩:    1 🟩🟨⬜🟩🟨:    1 ⬜⬜🟨🟩🟨:    1 🟩🟩🟩⬜🟩:    1 🟨🟩⬜🟩⬜:    1 🟨⬜🟩🟩🟩:    1 
🟨🟩🟩⬜🟨:    1 ⬜🟩🟨🟩⬜:    1 🟨🟨⬜🟩⬜:    1 ⬜🟨🟩🟩⬜:    1 🟩⬜🟩🟩🟨:    1 ⬜🟩🟨🟨⬜:    1

This means that, if we pick smile, we can get 101 different resulting patterns. The largest group we’re left with contains 316 words. So by guessing this single word, we can eliminate most of the words.

If we do this for all the words, my algorithm gives back that raise is the absolute best first word. This word returns 107 different patterns and the worst case is that we have 168 words left after the initial guess.

Another good candidate would be slate, this splits the words into more groups: 129. But the worst-case is a little bit worse, leaving just 221 words.

After `raise`, repeat.

After our first guess, we just continue with the following guesses. We calculate which words are still left and we go over all the words to find the one that gives us most information about the target word. Also: Don’t limit yourself to using/guessing just the possible words. For example look at the following case:

tares: 🟨⬜⬜⬜⬜
pilot: ⬜🟩⬜⬜🟨
dunsh: ⬜⬜⬜⬜🟩

There are now still eight possible words:

hitch, fifth, witch, aitch, bitch, fitch, gitch, mitch

Instead of guessing one of these words, often getting just a single piece of information… it’s much better to guess the word awful. This guess splits the possible words into 5 groups:

🟩⬜⬜⬜⬜: aitch
⬜🟨⬜⬜⬜: witch
⬜⬜🟩⬜⬜: fifth
⬜⬜🟨⬜⬜: fitch
⬜⬜⬜⬜⬜: hitch, bitch, gitch, mitch

If we get ⬜🟨⬜⬜⬜ in return we can conclude that the only option left to guess is witch and we’re done 🟩🟩🟩🟩🟩.

This algorithm allows me to solve all words within the given 6 tries you get with Wordle. For a single move I think this algorithm is optimal. However, we might be able to do better if we consider multiple moves into the future. It should be possible to extend this, build a tree and check the depth.

Perhaps raise is great for a single move, eliminating a lot of options, but what if the largest (worst-case) group is very hard to break up after that?

I haven’t tried to figure this out… but perhaps you will? What other algorithms can we come up with?

Bot tournament…?

Perhaps we can host a tournament where we have bots playing Wordle, taking turns solving and giving the opponent a next word to solve. We can even make the bots smart enough to recognise and counter certain words and/or tactics.

Sounds fun :)

Divide by three using shift and add

Fri, 31 Dec 2021 09:01:25 +0100

Today I stumbled across this excellent short video bij Mathologer:

This simple proof shows that 1/3 is the same as: 1/4 + 1/4^2 + 1/4^3 + 1/4^N…

As a programmer I wanted to try and program this.

Dividing by multiples of two (like 4^3) is extremely easy in binary, we just shift a number to the right. So if you have some integer X and we want to divide by 4, we do X >> 2, if we want to divide by 4^2 we shift X >> 4 etc.

This means we can just add these fractions together and approximate the result, just by dividing (shifting) up to >> 30 for an integer (which is 32-bit).

        int bignr = 612644632;

        // Approximate Using: 1/3 = 1/4 + 1/4^2 + 1/4^3 + etc
        int divless = (bignr >> 2) + (bignr >> 4) + (bignr >> 6) + (bignr >> 8) +
                (bignr >> 10) + (bignr >> 12) + (bignr >> 14) + (bignr >> 16) +
                (bignr >> 18) + (bignr >> 20) + (bignr >> 22) + (bignr >> 24) +
                (bignr >> 26) + (bignr >> 28) + (bignr >> 30);

        System.out.println(bignr + " / 3     = " + (bignr / 3));
        System.out.println(bignr + " + magic = " + divless);

        // 612644632 / 3     = 204214877
        // 612644632 + magic = 204214872 <- close, a bit too low

The accuracy isn’t great… but it’s pretty close. We’re always a bit below the actual sum because for each part we add, we lose a bit of information which we’re shifted away. To counter this we could first add things and shift the final result, making the accuracy a bit better:

        int bignr = 612644632;

        // Approximate Using: 1/3 = 1/4 + 1/4^2 + 1/4^3 + etc
        int divless = (bignr + (bignr >> 2) + (bignr >> 4) + (bignr >> 6) + (bignr >> 8) +
                (bignr >> 10) + (bignr >> 12) + (bignr >> 14) + (bignr >> 16) +
                (bignr >> 18) + (bignr >> 20) + (bignr >> 22) + (bignr >> 24) +
                (bignr >> 26) + (bignr >> 28) + (bignr >> 30)) >> 2;

        System.out.println(bignr + " / 3     = " + (bignr / 3));
        System.out.println(bignr + " + magic = " + divless);

        // 612644632 / 3     = 204214877
        // 612644632 + magic = 204214876 <- off by one haha

Probably a useless trick, but fun nonetheless.

Hex grid in single integer

Tue, 28 Dec 2021 11:19:31 +0100

AZsPCs: AP Math

Al Zimmermann hosts awesome Programming Contests every once in a while. They usually run for a long time (multiple months) and it allows mathematicians and programmers to compete in optimization problems. Usually the search space is very large and optimal solutions aren’t found for large N-values.

This time the contest was called “AP Math” and the goal was:

Given a hexagonal grid of size N, select the maximum amount of cells so no three cells form an arithmetic progression.

During this contest I didn’t find the right way to attack the problem, but I did discover a clever way to store all the hex coordinates in a singe integer value and do the math on that; which I wanted to share here.

Arithmetic progression?

The main goal in the contest is to select cells in a hex grid with this one rule “no three cells form an arithmetic progression”.

What does this mean? If you look at the example above for a N=3 hex grid you can see that we have to select cells, with the rule that no two points can have another cell in between, on the mid point.

Look at this website for more examples and a complete description: AP Math.

So to find solutions, we need a way to store the coordinates of this hex grid and a way to determine given two cells, which other cells are illegal.

Storing coordinates as single 32-bit integer

There are multiple ways to store the coordinates of an hexagonal grid. The best write-up I’ve found is located at redblobgames.com. I’m using an axial grid, so each coordinate has an x and y value.

But instead of working with an x and y value, I’m storing each coordinate as a single 32-bit integer.

To generate all of them I used the following (Java) code:

int N = 11;
int S = N + N - 1;

Set<Integer> indexes = new HashSet<>();
// Generate all hex points: (as single int)
for(int x = 0; x < S; x++) {
    for(int y = 0; y < S; y++) {
        // Cut off the two triangular edges to make a hex grid:
        if(x+y >= (N-1) && x+y < (3*N - 2)) {
            indexes.add(x << 16 | y);  //<-- store all the positions x, y both into a single integer value.
        }
    }
}

As you can see, I’m storing x and y in a single integer value, shifted 16 bits.

Now you might ask: how is storing each coordinate as a single 32-bit value useful? What does it add?

Well, suppose we have two coordinates from the complete list of all hex grid coordinates mentioned above. With these two indexes in our hex grid we can do the following:

// A bitmask to check for even numbers
int BOTH_EVEN_MASK = 1 << 16 | 1;

if (((index1 ^ index2) & BOTH_EVEN_MASK) == 0) {
    // Points index1 and index2 has the following midpoint:
    int invalidMid = (index1 + index2) >> 1;
}

int invalid1 = (index2 << 1) - index1;
int invalid2 = (index1 << 1) - index2;

Using very limited control code and little math we can calculate all three points that are needed for this contest. Given just the two coordinates we get the two or three points that are invalidated. This is much faster than any other coordinate system I tried. A single subtract of two indexes gives the x and y of the new point: less work for the CPU.

Turns out, I didn’t even need to use the actual x and y values anywhere, I could stay in this coordinate-system while searching. If you want to though, just mask and shift:

int x = index >>16 & 0xFFFF;
int y = index & 0xFFFF;

This was my little ‘hack’ for this contest. I might be able to use this xy as a single integer on more occasions, in other contests. It’s a nice tool to have in the toolbox.

Log4Shell / Leak4J

Tue, 14 Dec 2021 09:12:31 +0100

Bert Jan Schrijver and I did a livestream showcasing Log4Shell and talking about the vulnerability, check it out here:

What is Log4Shell?

Last couple of days (and nights) I’ve been studying the new (extremely dangerous) vulnerability in log4j2 called Log4Shell.

All versions of log4j-core from 2.0-beta9 to 2.14.1 are affected by this, and it’s a big one.

This vulnerability allows the attacker to remotely execute code on your system, with the ability to get complete control of the underlying servers.

Log4J has, for a long time, been the most used logging framework in the Java landscape. It’s extremely widely used and this attack has the most broad trigger you can imagine: It needs to log something.

The payload can be delivered in a LOT of ways, as long as it gets in a log statement. Either through user-controlled fields, HTTP requests, URLs, ANYTHING.

The attack

After writing some code (a malicious embedded LDAP server) I was able to reproduce the RCE (“Remote Code Execution”) attack on even the most basic project.

Here is an example: a simple REST endpoint in a Spring Boot starter project with a single line of logging

As you can see it downloads and executes a classfile I’m serving from the malicious LDAP server (running seperately) printing a message.

I’ll not be sharing the malicious code, it’s just too simple to set up and abuse. There are better and easier ways to check if your software is vulnerable. For example using this tool by Trend Micro.

Possible risks: 🚨

Risks of this vulnerability are:

Loss of ALL data
Ransomware
Backdoors
Botnet
Loss of AWS/Kubernetes keys/secrets
And the list goes on, and on, and on…

The fix

Option 1: If you haven’t already: upgrade log4j-core to version >= 2.16.0

Use version 2.16.0 instead of 2.15.0, this fixes the problem a bit more rigorously.

    <dependency>
        <groupId>org.apache.logging.log4j</groupId>
        <artifactId>log4j-core</artifactId>
        <version>2.16.0</version>
    </dependency>

Do this for all transitive dependencies as well (!).

Option 2: Another option is to launch the JRE with.

    -Dlog4j2.formatMsgNoLookups=true

But be AWARE this flag was put into log4j2 from 2.10.0 onwards. If you have an older version, this does not work.

Option 3: Remove JndiLookup.class

It is also possible to remove org.apache.logging.log4j.core.lookup.JndiLookup from your log4j JAR files. I don’t recommend doing this, if you want to go this route it’s probably easier to just upgrade to >= 2.16.0.

Not an option:

A newer Java version
Property: com.sun.jndi.ldap.object.trustURLCodebase

I’ve seen suggestions online that ‘newer’ Java versions are not affected, but this is just not true. Even with the latest Java versions and with trustURLCodebase set to false, they would still be able to steal data (for example environment variables) and deserialize objects already known to the classloader. This makes other deserialization RCEs trivial to launch.

It might be slightly harder/safer on a newer Java version, but it’s definitely NOT a fix.

To show this I’ve taken the latest version of Java 8 (1.8.311) and I’m using the log4j2 deserialization to craft something that opens the Calculator on my MacBook using other classes known to the vulnerable target:

Again: The payload is still being deserialized, on the latest Java version.

You’ve been compromised

Great, you’ve upgraded and fixed the issue. However: Don’t stop there, that’s just step one.

This leak was known and exploited for a long time, probably weeks before made publicly. And if even I can exploit it in a couple of minutes with my own malicious code: everybody can.

So here is what you need to do:

Step 1: Upgrade and fix the leak

Step 2: Assume everything was stolen: Rotate all your keys

Step 3: Go in and analyse the log files

Step 4: If you have set up infra-structure as code: rebuild your environment

Step 4: Redeploy all your applications

We HAVE to take this one seriously. I don’t want to hear or read a couple of months from now that some company forgot to patch their software. Not another

Sigh...

Thu, 14 May 2020 12:42:11 +0200

This morning I woke up and saw the following Tweet:

I'm a bit pissed right now! 😡 Why is the world so mean to me, to all of us developers, really? We've been so friendly! How often have we written "Hello, world"? And has anybody ever gotten as much as friendly nod in return?
— Nicolai Parlog (@nipafx) May 14, 2020

And yes, I agree.

The world has been a pretty shitty friend to us in 2020. Why should all the new programmers/developers have to welcome this messed up world with open arms?

I say: Screw You, World.

Your first program

My programming language of choice at the moment is Java. So that’s what I’ll be using for this “Screw You, World”.

To get started with Java there are a couple of things you’ll need to learn and install to get it running.

Let’s go!

Java JDK

There are a lot of places where you can download a so called “JDK”, which stands for Java Development Kit. This is needed for you to create and build your first Java program.

For now I won’t be bothering you with all the different releases and versions, we can just go to AdoptOpenJDK.

And download and install the latest version: OpenJDK 14 (latest)

REPL or Class

Now we have two different options, we can start the Java REPL (read eval print loop) to write Java code interactively, or we can store our code in a so called “class”-files and run it.

First we’ll look at the REPL, this is probably the easiest way to write your first “Screw You, World”.

Java REPL: jshell

The interactive code-writing tool in Java is called “jshell” and is launched from the Terminal/command line.

To do this, navigate to the JDK (usually there is a $JAVA_HOME variable for this) and start:

	
> $JAVA_HOME/bin/jshell

Next you’ll be presented with an interactive shell to write your code in:

> $JAVA_HOME/bin/jshell   
|  Welcome to JShell -- Version 14.0.1
|  For an introduction type: /help intro

jshell>

In this little editor you can write any valid Java code you want. Let’s grab the System class, get the out output stream (which defaults to the terminal), and write some text there for the world to see:

jshell> System.out.println("Screw You, World!")
Screw You, World!

jshell>

And that’s it. We’re done.

Using the REPL we’ve created a line of code which contains a message to 2020.

Java class files

The second way to write Java code is to use a text editor (or an IDE, an integrated development environment, like IntelliJ) and save the files, compile the files and run the files.

The code above has little to no real structure. In larger programs we’ll need to create some order in all the code you want to write and run. To do this Java has ‘classes’.

So open any text editor and write your first class file:

class ScrewYouWorld {
}

Save this file as ScrewYouWorld.java. This is the human-readable version of your code.

The next step is to take this code and process it, compile it. This turns the readable java file into a class file, which is made out of bytecode. It is still the same code, but optimized for machine readability.

To do this we call:

% javac ScrewYouWorld.java

Now we’ll get a file named ScrewYouWorld.class.

This class file is just en empty shell, it has no real code in it yet and there is not even a valid ‘starting point’. When programming in Java there are usually a lot of different classes that make up the application. The code however has to start somewhere, we’ll have to define a starting point.

In Java there is a very specific method we need to implement, this makes it a starting point. A method is a block of code we can call and it does something. The one used to start your code is always called main.

It looks like this:

class ScrewYouWorld {
    public static void main(String[] args) {

    }
}

Let’s break it down. First we have public, this means any class can access it, which is what we need to start our program.

Next we have static. There are TWO kinds of methods in Java, static and non-static/member methods. For now let’s just say that static methods are working on a class level, and non-static methods are working on the object level. This is a whole topic for another day. To start we have to make this static, from there we can create new instances.

Next we have void, this means that the method we are calling doesn’t return any results. Some methods calculate something and return a value, like a number or more complex answers.

main is a specific name, the name of our method.

The information between the parentheses are the method ‘arguments’. These are passed into the method. Just like arguments you supply to a program. In this case it is an array (list) of Strings.

This all becomes rather complicated quickly, but we don’t need to know the specifics right now.

Next we can add the line we wrote before to inside the method brackets. Classes add a little bit of structure/scaffolding to our applications.

If we put everything together we’ll end up with:

class ScrewYouWorld {
    public static void main(String[] args) {
        System.out.println("Screw You, World!");
    }
}

Save this file and compile to bytecode again:

> javac ScrewYouWorld.java

Now we can run this code and see the result. We take the compiled class and the java command understand that the main method should be called:

> java ScrewYouWorld      
Screw You, World!

Congratulations, we’ve just used two different ways to write and run your first Java program!

TODO: Add more “Screw You, World.” examples in different languages.