One of the things he mentioned was: If you don’t use caching, you are an idiot.
Where do websites cache?
There are multiple tiers where caching of websites is done, and is useful.
The best cache you can have is the cache inside the browser. If a website knows it has the latest version, it can just read it from disk. There is absolutely no reason to go online.
The second type of cache would be the proxy cache. As you would have guessed this is a proxy and it does caching. It sits between the user/browser and the internet gateway. This cache sees all the requests and stores pages that can be cached. If another user requests a webpage that hasn’t changed it can provide the page instantly.
Reversed proxy cache
You could also have a cache between the internet and the content providing server. If the server processes the request it might need to access databases and maybe other slow resources to build up the webpage. The resulting page can than be cached on the providing side in a “reverse” proxy cache. All subsequent requests can just be provided from the cache, as long as the page is still fresh.
Making pages cacheable
If you maintain a website, or you create web applications, you should be aware of caching. After Stefan’s rant, I’m completely convinced about that. If you don’t do anything all the requests will always go into the server and over the internet. There are HTML ways to control caching (META-Tags etc) but this just doesn’t work, and shouldn’t be used (!). So what could we do?
When sending a page back to the user you are able to set some HTTP headers. And “expires” is one of them.
This indicates that the current page is valid until the timestamp. Then it ‘Expires’. Easy!
The only problem is generating the timestamp, it can be a bit tricky. Also you’ll have to be sure you’ve set the time correct on your system. Also, the next time you update the page, you have to also update the timestamp!
With HTTP 1.1 there is a new class of headers called “Cache-Control”. These headers are more powerful than the Expires header.
To enable caching using Cache Control headers you can set:
The “max-age” is time in ms that the current page is valid. And by adding “must-revalidate” we tell the cache it should obey our max-age. If you don’t want an object to be cached you can use:
Refreshing cached data
The two methods described above will tell the cache if the content is cacheable. But what happens when the max-age or Expires timestamp expires? There are smarter ways to update the cache instead of getting the latest content from the server.
Websites should always set the response header called “Last-Modified”. This is a timestamp of the moment a webpage last changed.
When a cache has expired (max-age or Expires) and has to get a new version from the server it can set the request header “If-Modified-Since” and include the timestamp.
If the content on the server hasn’t been changed it’ll reply “304 Not Modified”. The cache can now keep the cached version.
With HTTP 1.1 there is also an improved method of doing the “Last-Modified”. Instead of using a timestamp (which is error prone), they’ve introduced the “ETag”. This is a tag that is completely customisable. Most of the time it will just be a hash of the content. The server sets the ETag as response header:
When a cache can no longer use the cached version (due to max-age or Expires) is will ask the server:
The term “If-None-Match” isn’t very clear, but is means “if-etag-changed-since” and works the same way as “If-Modified-Since”. When the ETag is the same the server will reply “304 Not Modified”, it won’t send the content back.
When you are working on a web application you could just add an ETag which is the MD5 of the returning content. If the content is the same, you don’t have to send the content over the line. The only drawback to this method is that you still need to generate the entire reply to calculate the MD5 hash to see if the content has changed…! But sometimes you’ll know in advance if the content has been changed.
I’m using WordPress and I’ve found the excellent plugin “WP Total Cache”.
It will involve a bit of tweaking, because only you can decide which stuff should be cached. But I think it worked out great, press F5 right now and you’ll probably be reading this from the browser cache.
The last two days I’ve been competing in a competition called Ludum Dare.
This is a short, 48 hour, contest. In this time you have to build an entire game based on a theme given at the start of the 48 hours. It is a good exercise is planning, scaling, hacking, imagining and just having fun! I really enjoyed it, and recommend you join LD24 four months from now.
For this game I decided to stick with Java. To make it playable for as many people I decided to make an Applet. It can easily become a standalone app, or maybe an Android app…!
I loved the old point-and-click games, from Dirty Larry to The Day of the Tentacle, from Monkey Island to Gobli(iiii)ns. So that was settled.
The big pro:
Not a lot of physics or game code.
The big con:
I’d have to brush off my paint skills because point-and-click adventures are filled with graphics and animation!
One big factor in games is music, and for this contest I took some midi control code I made some years ago. This was turned into a procedurally generated music generator. Every time you play you’ll hear something new.
With visitors coming to see our little baby girl on Sunday I decided to end early.
Here is my result, have fun playing the game: Itty-bitty botty!
The Real Katie
Today I stumbled upon the following blogpost:
The Real Katie - Lighten Up
Katie talks about the sexist jokes and remarks she regularly gets in the IT/programming world, and she is sick and tired of hearing “Come on, lighten up”.
The post is moving and shows how easy it is to offend people, not by a single remark, but by hundreds of similar remarks heard before.
Not an IT problem
There is one point I don’t agree with though. I don’t think it is fair to call this an IT/programmers problem.
Let me explain:
Obviously there are a lot of jerks, assholes and plain rude people around. Most of them are men, some are women. They pick on easy targets, the minorities. Sometimes the minority is a heavy male co-worker, sometimes it is the rare female programmer.
I fully agree, we should call the bullies out more. We all should do something about this problem. Her blogpost has re-opened my eyes again to that problem. But this isn’t a IT problem… it is a minority and rude people problem, a global social problem. It happens in all professions. Katie is just unlucky to be the minority in the field of work she loves.
More women in the IT
Also, I do agree that we could use more women in the IT world. At a young age we should teach boys and girls that there is no such thing as boy-jobs and girl-jobs, and both should learn the joy of programming! If we do that the problem of women being a minority in the IT world will disappear.
BUT that won’t solve the global social problem of assholes picking on minorities.
That is something we are all responsible for.
There will always be minorities and there will always be rude people (male or female). This is something we can’t change. You can however call them out and disapprove the behavior.
Our project is doing Scrum, and one of the main aspects of Scrum is having everything clearly visible. A great example is the scrumboard, a huge whiteboard filled with Post-It notes.
Post-It notes are perfect for this; small enough to be easy to handle; sticky enough so you can post them almost everywhere. I truly believe that without the Post-It note, Scrum wouldn’t be possible and probably wouldn’t even exist!
This all makes the real hero of the Agile movement: Arthur Fry. After somebody at 3M messed up a batch of glue, Arthur decided to add that glue to a piece of paper, creating the first Post-It notes.
This invention is much more important than the toilet, or wheel, or penicillin…! Celebrate and make the world aware of this unlikely hero, join us and celebrate Arthur Fry Day, this March 16th!
Moments ago this tweet caught my eye:
Devoxx 2011: "What Shazam doesn't want you to know!" by @royvanrijn is now freely available @ http://parleys.com/d/2869
That means everybody can now watch my talk without any subscription! If you want to learn how algorithms like Shazam work, be sure to watch this talk. It might be easier to understand than my blog post a year ago.
Without further ado:
Today I’ve been playing around with the Levenshtein distance. The Levenshtein distance is a number which measures the ‘distance’ between two strings. For example, the distance between “test” and “rest” is one.
A Levenshtein distance of one is the key element in a challenge I’ve been reading about. I first encountered it on williamedwardscoder’s blog.
The problem description:
Two words are friends if they have a Levenshtein distance of 1. That is, you can add, remove, or substitute exactly one letter in word X to create word Y. A word’s social network consists of all of its friends, plus all of their friends, and all of their friends’ friends, and so on. Write a program to tell us how big the social network for the word “causes” is, using this word list. Have fun!
Java solution (8.1 sec)
After some Googling and tweaking I decided to make an implementation based on the Trie structure. How this helps is excellently described by Steve Hanov. I’ve also had a peek in another Java based Trie implementation by Ximus.
I’ve been able to get the code below run in 8.1 seconds, which is pretty good. But I’ve read that there are Java implementations running in just 4 seconds…!? Maybe based on Levenshtein Automata?
The Orchard Planting contest from infinite search space is over. So it is time for a quick write-up.
The rules are simple, on a grid of integers, place N points on the grid to get as much 4 points on a line and never more then 4 points on a line.
My big break-through was when I figured out a way to improve the calculation speed of a solution, and make it possible to extend existing solutions (going back and forwards). To do this I used a unique vector (greatest common divisor vector) which is the same for all point on the same line:
Now we can evaluate the points:
If a point has three vectors that are the same, we have a line with four points! This can be checked easily if you sort the vectors and go through them once.
Also adding and removing points becomes very easy. A lot of the GCD calculations can be cached. To remove a point, just remove the vectors it made. And to add a point, calculate all the new vectors. So in the end it basically all boils down to a lot of GCD calculations and sorting.
Was this the fastest way to calculate solutions in this contest? I don’t know, but I was really pleased when I figured it out. With a better algorithm for picking possible numbers (instead of hill-climbing) and some more processor power I bet I could have ended a bit higher up the hill.
Also: Keep an eye out for the next contest, it is going to be an interesting one! January the 13th.
The guys at Devoxx/Parleys have already processed all the talks and post-processed them. So my talk it now available at Parleys.com.
There is one drawback though, the talk is currently for subscribers only. If you don’t have one you can only watch the first two (very nervous) minutes.
If you’ve attended Devoxx you will get a free subscription in the email.
(p.s. The intro movie for all the Devoxx ‘11 talks is made by me as well!)
Most internet videos are a waste of time, but this one isn’t! This is an interview of Stephen Colbert (out of his normal character) with Neil DeGrasse Tyson (Hayden Planetarium director, TV science host).
Today I’ve (again) seen this quote on Twitter:
“A project manager is someone who thinks that 9 pregnant women can create a baby in 1 month”
This obviously isn’t the case. The story demonstrates that adding more people to a team won’t (necessarily) make the team more effective. Because some processes can’t be cut into smaller pieces and taken up by more people. It will even cause a bit of overhead, more opinions and thus, more time.
What about one woman?
Everybody knows 9 women can’t create a baby in 1 month… so it isn’t fair to make that comparison. How about one pregnant woman and the project manager?
Four weeks into the ‘project’ the project manager has his monthly meeting with his pregnant wife:
Project manager: “How is everything going?”
Wife: “Oh, just fine, no problems at all”
Project manager: “Is there anything I can help you with?”
Wife: “No, currently not, everything is going like it is supposed to”
Project manager: “When do you think this baby-project is going to be completed?”
Wife: “Well, it takes 9 months, so 8 months from now!”
Project manager: “Hey, is everything still going fine?”
Wife: “Yes, still feeling fine.”
Project manager: “Is there anything I can help you with?”
Wife: “No, nothing that I can think of right now.”
Project manager: “Hey, still working on that baby?”
Wife: “Yes, I’m starting to have a bit of morning sickness now…”
Project manager: “Oh, can I help you with that?”
Wife: “Not really, it is just something I have to live with”
Project manager: “How does this affect the release date?”
Wife: “It doesn’t, I think… but I really wish we could release a bit sooner!”
Project manager: “Hey, still having that morning sickness issue?”
Wife: “A little bit, but it is almost gone now”
Project manager: “Still on track for the release date?”
Wife: “Yes, 5 months from now!”
Project manager: “How is it going? Morning sickness issue resolved?”
Wife: “Yes, but I need icecream and pickles.”
Project manager: “I’m on it. Any updates about the release date?”
Project manager: “Anything I can do for you right now?”
Wife: “Yes, I think you can start building that baby room”
Project manager: “When do you need it, when is the baby ready?”
Wife: “Three months from now.”
Project manager: “The baby room is almost ready, how is that baby going?”
Wife: “Just fine, just very tired. Two more months.”
Project manager: “I’ve got some more pickles for you!”
Wife: “You know I hate pickles :-( I’m tired…”
Project manager: “Sorry, how is the release coming along?”
Wife: “One more month.”
Project manager: “Is the baby there yet?”
Wife: “No, not yet, I’m tired but fine”
Project manager: “But…? What about the release?”
Wife: “Go away.”
Month #9 (plus one week)
Project manager: “What is the delay? Where is my baby?”
Wife: “It isn’t here yet.”
Project manager: “How come? How did this happen!? I kept asking about the release date and you kept saying it would be done a week ago!?”
Wife: “Get out.”
Month #9 (plus two weeks)
Wife: “I’m ready! I’ve got this lovely baby for you.”
Project manager: “Great, where did it go wrong with that planning?”
Project manager: “That babyroom has been ready for weeks now, and no baby!”
Project manager: “How can we keep this from happening next time? Scrum? Lean? Agile? RUP?”
Wife: “Screw you.”
This story also has an important message in it, I’ll just leave that as an exercise for the reader.
A couple of days ago I first encountered “bytebeat”. This is a new hype revolving algorithmic music and sounds. The basic idea is this:
This simple loop has one variable, t (time). And every iteration we use t to calculate some output. Next, we take the output and pipe it into 8-bit 8-kHz PCM channel (for example /dev/audio!).
A couple of examples in a YouTube video:
In Java you could do the following:
These simple one-liners are able to produce amazing sounds and music!
Here is one (including the best song I’ve created): http://t.co/9oognysS
The computer/synthetic sounds are also good for dubstep.
For more information on this subject:
As you might know I’m a big fan of math contests and basically just math in general. But only as long as it doesn’t involve long/hard equations! Math is all around us, we just don’t see it because we’ve been taught that math equals equations.
There is one person that keeps amazing me with her math related blog: Vi
When browsing this blog I came across a very good talk about programming and simplicity. What is simplicity? Things that are easy right? Wrong.
I won’t go into the subject, just watch this talk!!
I’m back home from Devoxx, I’m still alive (kind of) and it was great!
First of all, I had never been to Devoxx before, so it was all a new experience for me. And second, I was speaking at this conference, which is my first international talk! In this blogpost I’ll try to describe how it is, talking at Devoxx.
Before the talk, other sessions
There were a lot of good talks on Devoxx this year, mostly about HTML5, Android and new languages (like Clojure, Scala, Fantom, Ceylon). But to prepare for the talk I decided to stick with the sessions which had the well known international speakers (Joshua Bloch, Mark Reinhold, Brian Goetz, Chet Haase and of course the JavaPosse guys). Instead of learning new technical stuff I focused more on learning presentation skills instead.
The first day I also brought my camera with me, and created the following video (which is currently featured at the Devoxx.com frontpage):
My own session was planned on Friday, the last day of the conference, in the last slot of the day. Initially I didn’t like this spot (especially since it was the day after the Noxx party!), but when I saw the line of people waiting for popular sessions like Joshua Bloch’s and/or the JavaPosse I was very glad I wasn’t scheduled at the same time as them.
When I arrived at the venue Friday morning I noticed the room I was scheduled in was empty and being torn down… that didn’t look promising! But I quickly found out that ‘room 6’ was now in cinema room 4. And until the last moment the screens on the floor didn’t show my session and the session (from David Geary) before me. So I didn’t have a lot of hope, only my own colleagues would probably be able to find it…
Then suddenly David Geary was done with his talk (HTML5 Game Development: The most fun you can have in a browser), time flew by… time to go to the front of the stage and prepare the laptop. When connecting the laptop I saw a huge line of people, queuing to leave the room… sigh. So I went to the sound-guy, getting a microphone strapped to my head, and warned the cameraman that I have the habit of walking around like bored zoo animal. He was playing some World of Warcraft in between the sessions, so I challenged him: you can practice your aim on me instead the coming hour.
Wow! And it kept getting more and more crowded, in the end people were sitting on the steps because all the seats were taken. Then the timer was suddenly at 59:50, my talk had begun! The microphone was open and I started talking to the audience. Up to this point I wasn’t very nervous, but now I had to talk to all these people, in a foreign language (English). After the first joke landed, and the people reacted like I hoped, my confidence returned. During the talk I had to explain something with a bright orange stick, and before I knew it I was waving the stick around like a nerd jedi. Suddenly I remembered the poor cameraman, but one quick glance at the screen behind me and I could see he was doing an awesome job tracking my every move.
When the countdown reached 18 minutes, I was pretty much done with the talk, but I was anticipating a lot of questions and I had an anti-patent rant ready. For the next 13 minutes I answered a lot of good questions. Then I decided it was time, I took the liberty (being the last talk) to thanks the audience and tell them Devoxx was now over. Something like: “This is the end of our Devoxx for this year! Have a safe trip home, and hopefully I’ll see you all again next year!”
Afterwards a lot of people congratulated me, and I’ve seen some great tweets about the talk. It felt great and recommend it to everybody: Just do it!
The slides can be found here: http://www.slideshare.net/royvanrijn/what-shazam-doesnt-want-you-to-know
And the talk will be freely available (with video, slides and screen capture!) on Parleys.com some time in the future (could take a while, months).
Last week I wrote a blogpost on rewriting code and improving quality. And now we have some great news that will support my claims. To verify our project and convince management we are doing it the right way, we’ve send a copy of our code to TÜViT.
Most people from Europe will know TÜV. They test and certify cars, elevators and even nuclear power plants. They also verify IT security, quality, infrastructure, products, processes and their requirements. TÜViT is accredited by organisations and official bodies for the areas of IT security and IT quality.
Their verdict couldn’t have been better, last week we’ve received a banner with 5 out of 5 stars! Just 5% of the projects that are submitted to get tested get 5 stars. More information can be found in the official press release and in the Automatiserings Gids (Dutch).
Agile architecture and refactoring
For some reason getting these 5 stars makes me very proud. Normally I don’t care much about certification, they usually focus on the wrong issues. We know we are doing the right stuff, it feels good, and we’re making good progress. And this is just a perfect verification from an external source, this will also assure management we are on the right track.