Scala - Event Sourcing and Spray

Scala + NoSQL

Over the past 24 months I have been diving a bit deeper into Scala by way of using a new architecture. My dabbling started 4 years ago. I haven't touched Java for about 2.

I have been sick of, over the past 10 years, building the typical DB/App stack. If you track back about 5-6 years in my posts you'll notice I shifted to researching and investigating NoSQL enterprises. (db4o etc). The "ick" centred around the simple yet fundamental issue of the ORM Impedance Mismatch. Every man and dog has blogged and written about it and it is very prevalent.

One thing led to another and I found myself loving Scala and it's fresh way of code development. But never really landed on a great NoSQL solution. Being a Java hack I immersed myself in its ways (Scala that is) and attended ScalaDays 2012 to sure up my skills; There I met and chatted with a lot of people and found the 3 days brilliant.

At Scaladays I was researching my next "stack" and honed in on spray.io. After the talk Matthias Doenitz gave at ScalaDays, I caught up with him in the hallways asking a general question of

"I am an old GWT hack and want less complexity; what interfaces do you see being plugged into Spray,.."

to which he and a friend gave some tips to various JS libraries for me to check out (of which I have settled on Twitter Bootstrap). Matthias and / or the other (I don't recall who) made reference, if I wanted to walk on a new area, to check out the work in the Event Sourcing arena.

I read and could see it's benefits. So I figured I would read some more. I ended up watching the threads on the DDD/CQRS. That got me hooked - I have to say though that the CQRS is a simple yet massive theory and I wanted some good practical; it was to come.

About the same time, Martin Krasser posted some info on JAXB Scala Marshalling - I wanted some of that in my Spray application. Scala has excellent JSON marshalling using lift-json; alternatively you can use a built in library spray-json. Both of these worked well, however I wanted a "single" API definition class that I could expose as JSON or XML without me needing to code it twice. My sample application at the time was still bound to a Scala/Spring JPA and Hypersonic DB stack. 

Seeing the "sample" project that Martin Krasser had built, and his excellent blog posts on it, took me right into the eventsourced package where the JAXB marshalling was used. Martin Krasser together with colleagues released the early draft of this Event Sourcing package and after a few iterations it was fully embedded into the Akka way .. and since Spray was too, I have joined the two together ever so simply - and now have the base framework for my perfect "stack".

The Scala ES Spray Stack

So what does it look like ? At the front end, though I have some ways to go there, I have

• Twitter - Bootstrap, which talks to a
• JSON REST API; into
• spray.io routing; which delegates the "commands"; to
• eventsourced

The commands and the events that are journaled are Scala Case Classes, Annotated with JAXB annotations to support the Un/Marshalling in spray.

Akka Camel will come next, though because I am an old Apache Camel Hack - random contributions and use throughout time, I know what it does and well, so will slot it in later.

A Weak Schema on Historic Events

Event sourcing is great - in essence - keep everything you ever did. Everything. It is very BigData-esq, and actually beneficial for audit tracking. If I keep every Command/Event that the system responds and reacts with. I have a full "audit" trail of how it operates. CQRS gives me a benefit of no Database (can stick one on the "read" view if I want). However;

My last "piece" to the framework puzzle is the concept of supporting a weak schema. After you read all about Event Sourcing, you will quickly realise that "Cross Version" software support needs to be managed well. The typical "DB" stack doesn't have this challenge as much, simply because, unless coded for, all history is "thrown" away - and thus the problem is smaller - and DB upgrade scripts decide at "run" what is kept .. or what goes.. and the "change is usually irrevocable".

With Event Sourcing, the "history" is with you, and that is it's benefit. So to retain that benefit through future upgrades of your application, you need to support the older events, in what ever form they have. A few ways to achieve this are:

1. Retain, in code, the "V1, V2 and V3" objects that the event messages relate to;

Needless to say, this is quite complex. The amount of code you may have to manage over a long life span of the software could become messy. * discounted *

2. Upgrade the older Events, when you "upgrade" the Application.

This may work - but it feels wrong. Upgrading events to something they never were breaks the model. Adding a field to an event .. what should be the default of the value that was never supplied 4 years ago ? Needless to say, you will recognise this is often the DB way. That is okay. It does work.. there is another way. * discounted *

3. Translate the older events on the fly as they are read in..

My idea was to build a shim in the "serialisation layer" that translates older events V1, V2 up to V3 equivalent .. but again it kind of smells.. so that is not it. * discounted * .. kind of..  

4. A Weak Schema.. 

Greg Young, on the eventsourced mailing list pointed me (in two words .. "Weak Schema") to  look at this model .. quickly I ended looking at Google's protobuf. I had read it before but had not the need, until now.

Protobuf is all about "version" management across messages between systems. This is exactly what Google built it for. Between their index servers that may be running different versions of software. And as it turns out, it might be a brilliant fit. I have some "tests" to do, which for now I am going to park as the theory seems okay, and when the need warrants I will utilise it's power. 

I googled in earnest about working with a "weak schema" but there was not a lot to read, but it didn't take too long to work it out.

Let me give you an example: 

if we have a "Command" object that we will serialise to disk. It could look like this: 

"CorrectTheBirthDate(uuid,newBirthDate)"

In my fictitious application, imagine we serialise this to JSON. It may look like this.

Now imagine 3 years later we figure it is a good idea to record the "reason" why the Birth date had to change; to support this we add a new field for "reason". Simple enough, our Command object changes to CorrectTheBirthDate(uuid,newBirthDate,reason) and the newer JSON is serialised as you would expect.

Ok. So what happens when the system "replays" all the Commands. Well with protobuf, it just sees that the field is not supplied, so it doesn't deserialise "nothing" into the object. Instead (using ScalaBuff) the case class is marked with "Option[] fields, so that the value in that instance becomes "None".

If it were the other case where a field is dropped; then the Case Class will just never have the value "loaded in".

I am sure there are horrid edge cases lurking, but it feels right to let the serialisation layer deal with the problem .. how it knows .. and so long as the developers know the rules (when things are dropped, or what is added when) then coding can continue. The trick or benefit is that the "protobuf" default should be enough.

Serialising in this way means that regardless of the changes in my API's, Commands and Events. I will always have the History. Therefore I will always have the ability to scour the depths for stats and reports. Exactly my reasons for Event Sourcing it. (amongst so many others).

So, stay tuned. I will post a real sample application once I tidy up the mess and make a real UI to play with.

GWT SDK Not Installed - Eclipse

From time to time, I have seen the simple but annoying message.

GWT SDK Not Installed

the error message is annoying because I "have" it installed, which means, not so much "not installed", but more "not detected" by the Google Eclipse Plugin.

As I am busy working on many things I forgot what the resolution was and again, spent 1/2 an hour resolving the issue.

You may find this error when using Maven and GWT. Essentially, the GWT SDK Library shows up as a Classpath Container. "GWT SDK [2.4.0]" or similar.

The cause is because Eclipse locates the "gwt-*" libraries in your "Referenced Libraries" set, before it sees the actual Eclipse "GWT SDK [...]" reference.

The error message occurs when theses other GWT libraries are in your classpath, in this case the "referenced" libraries section; especially before the GWT one.

There are two fixes for this

1. Move the GWT SDK to the top

Move the GWT SDK "Classpath" Entry Container above your "Libraries" where GWT is found. The Google Eclipse Plugin finds the SDK based GWT Library first and all is well. To move it, manually edit the .classpath file and move the classpathentry line to the top.

2. Exclude the other GWT Libraries from your classpath.

If you are using maven, you'll know that the Maven Eclipse Plugin can't specify the order of classpath references in the .classpath file.

The trick then is to "exclude" the gwt-dev, gwt-user jars from the generated .classpath file, which you can do in the pom.xml. This way, ONLY the GWT libaries in the Eclipse GWT Container are the ones seen by the Google Eclipse Plugin and it all works well. Note also I have the GWT ClasspathContainer for GWT in there, so when I DO regenerate the .classpath and .project files, I don't have to regenerate things.

The example pom.xml is as follows:


I hope that helps you.

Crashplan Active Bandwidth Control #2

In a previous post on crashplan - Crashplan Active Bandwidth Control - I setup a sophisticated "control" mechanism. I have now simplified my approach down to understanding my family patterns and I think it works as a guaranteed performance "setup".

Essentially, between the hours of 23:45 and 06:30, my backup is using ALL the available bandwidth. But I go a little more than that.

Crashplan only has an "on, off" approach regarding times. I wanted a "Fast", "Slow" approach around times. I am using UNIX cron and a script I wrote in the previous version which logs into Crashplan and adjusts the bandwidth.

My /etc/crontab looks like this .. finely tuned over the past 2 months.


So this seems to work real well, I have been able to increase my Backups to now pushing out 90G per week. Pretty good going.

Sakis3g Control Script

I have found that my USB Modems, two of them, did not work out of the box with Ubuntu 11.10.
Not to worry, there is a great script that contains all the pieces for Ubuntu.

sakis3g

You can read all about it here - www.sakis3g.org and what it does. In essence, you use it in replace of your Network Manager or ppp/wvdial. Now I would prefer to use the built in method that the distribution has, however it plain does not work! Instead sakis3g and I have work to do.

I wanted a better way to control it, I created a command line control script for it for my modem. (Didn't like the GUI)

#!/bin/sh

BASE_CMD="/usr/local/bin/sakis3g"
OPTIONS="--term"
MODEMETC="OTHER='USBMODEM' USBMODEM='19d2:1003' USBINTERFACE='1' APN='3internet'"
COMMAND="$1"
shift

case $COMMAND in
status)
    $BASE_CMD $OPTIONS status
    ;;
connect)
    sudo $BASE_CMD $OPTIONS $MODEMETC connect 
    ;;
disconnect)
    sudo $BASE_CMD $OPTIONS disconnect 
    ;;
*)
    echo This is a wrapper to Sakis3g
    echo $0 connect
    echo $0 disconnect
    echo $0 status
    exit 1
esac

Crashplan Active Bandwidth Control

I love what random things I get up to over the holidays. Crashplan is a Cloud backup service that I use for all my backups. I have devised a way so that Crashplan starts backing up at FULL speed, when everyone is not at home, or are not doing anything on the internet. And then drops back down to a trickle backup when things become busy.

Essentially I have one computer, my server, and it has the Crashplan service running on it. The server is an Ubuntu based Linux distrubution. Nothing fancy there. What is interesting is how I have managed to control the Crashplan bandwidth it utilises. In the previous house, I had a good 18-20Mbps internet connection, so my backups were really nice. Now, I only have a 3Mbps connection so the "usage" is very important. If crashplan hogs the connection you reall notice it.

Crashplan has two ways of controlling it's backup usage. First is on a backup set basis, you set the time that it runs. Helpful, for the past 2 months, my backups have been scheduled for 12am to 7am. Works well, but at that rate, I will be completely backed up in about 12 months time... hrmm.

The other way Crashplan operates is by bandwidth limiting. This is how I roll it now.

Attempt 1 - See when they Connect

I needed to determine the best method to "ascertain" if someone was using the internet. I pondered looking at when the phones (we have 4 adults, 3 iPhones and 1 Android) are "on" or can be seen as that is a good indicator that someone is home. But of course it goes a bit more than that.

I did however, to test the theory, write a quick Perl script which was the start of monitoring when an iPhone connects to the WiFi. The Android apparently uses Zeroconf also, but I didn't test that.
Basically, each time the phone connects for the initial session, it sends a multicast out. If you are listening, you will see it. Simply, this perl script listens on the right port.
I quickly discounted this method when I realised that some devices just wont tell me they connected. And because I want to keep my network largely zero configuation, I want to just use DHCP and not have to bother with static entries, or Static leases for my clients. Which is when it hit me:

Attempt 2 - Scan the Known DHCP Range

The premise behind my final solution is simple. I have two types of devices in use in our household.

• Infrastructure - servers, printer, wirless and network switches / routers
• Clients - Laptops, phones (soon tablets)

All my infrastructure devices use static IP addresses, such as 1 to 39. All the clients, use a DHCP IP address. We have no desktops in the house, only laptops and when they are closed, they are not in use.

If they are not in the house, they are not in use. Same goes with the phones, they are either sleeping or not in the house.

The Solution

So, if all the laptops are closed and all the phones are out, I know that none of the "clients" will ping, and therefore I can ramp up the Crashplan bandwidth.

Solution to Attempt 2 - Monitor for Active Clients

I had a few things to solve. But for each I knew that Perl would be all that I would need.

Part 1 - Automatically Raise and Lower Crashplan Bandwidth

I needed to determine how Crashplan "stored" it's bandwidth rate. Turns out it is in a config file, but the web site can also change the config. So, the client (my Crashplan service running on my server) and the Crashplan Cloud keep in contact all the time· If I change the bandwidth on the client, it updates on the website, and vice versa.

I fired off a few support queries to Crashplan and they basically said that the config file I found (/usr/local/crashplan/conf/my.service.xml) is not editable by hand. And that the only supported way is via the Client (desktop Java App) or the Website.

Call WWW::Mechanize - I used this module to login to the Crashplan Website and change the bandwidth (kbps) based on my argument. Simple and it looks like this.

Not rabbit proof, but it works for my needs

Part 2 - Monitor for Activity

This was the hardest part, but as always, was solvable. The router (Sky Broadband Provider) I currently use "knows" what clients are using it. VERY handy. So like the Crashplan website solution above, I again use WWW::Mechanize to determine what IP addresses are currently using the router.

I do a filter out of all the ip addresses that do NOT fall inside my DHCP range.

I won't go into the details of using XPath on the HTML from the Router, but, you can see that it is quite succinct. Essentially the IP Addresses appear in a 2nd column in a table. I use an XPath expression to select the right values.

This approach is brittle in as much as, when / if I change my router then I need to code or come up with another way to determine which hosts, within the DHCP range, are currently "in use". I did first to this by just pinging all 60 possible IP addresses; works, brute force, but of course worked :-)

So .. now putting it all together. The script, a perl script, runs from Cron every minute. and performs the following tasks.

• Get the Current Rate from the Crashplan Config file
• Get all the current "hosts" that the Router knows
• Ping each host to MAKE sure they are active
• If we have an active host (one or more) - we need to go slow
  • If the current rate that Crashplan is running is more than our slow rate, tell Crashplan to slow down
  • If the current rate is slow, do nothing
• If we have no active hosts - we can go fast
  • If the current rate that Crashplan is running is less than our fast rate, tell Crashplan to go fast
  • If the current rate is fast, do nothing

Simple! And the full Script looks like below. Enjoy!

Of course, I did all this and know and realise that QoS could be used, but my Sky Broadband Router is just not that smart today and I haven't got the devices spare or the time to reconfigure to installl a dd-wrt based router or some such other. So perl scripts it is for now.