tag:blogger.com,1999:blog-34738718054043284052024-03-19T11:23:12.477+00:00RB TechRamon Buckland is a Java Solutions Architect and Finance IT Consultant based in Bristol, UK.
His experience is vast across all areas of IT. Focus is online development and architecting with the Java platform(s).Ramon Bucklandhttp://www.blogger.com/profile/16744583593281442424noreply@blogger.comBlogger73125tag:blogger.com,1999:blog-3473871805404328405.post-81484932198547915392023-07-18T09:50:00.002+01:002023-07-18T09:50:30.426+01:00Ubuntu 22.04 on older devices with no, or bad UEFI<p> Recently I had to install Ubuntu 22.04 server onto 3 laptops, which did not have any support for UEFI, or the UEFI support was broken. </p><p><br /></p><p>It was indicated when, after successful ubuntu install, the first boot, fails with no disk found. </p><p><br /></p><p>* BootDevice not found </p><p></p><p>etc. </p><p></p><p>the fix is simple enough. </p><p>1. disable the UEFI support in the BIOS</p><p>2. Beging the ubuntu (booting off the USB stick) install</p><p> but when you get to the first menu, choosing the keyboard layout, switch to a new console (ALT f2) and prepare the disk manually. </p><p>This will vary for some devices (which is the boot disk, partitiion layout etc) but the basics are</p><p>sudo fdisk /dev/sda</p><p># change the partition type to DOS (m for menu, look at the bottom) <br /></p><p># delete any old partitions <br /></p><p># create a new primary partition (2G in size for /boot)</p><p></p><p># create a second parition (remainder) for /</p><p># set partition one to be bootable ('a')</p><p># write the changes.</p><p> </p><p>3. now go back to console 1 (alt F1) and when you get to disk choice, </p><p>choose custom and select, and allocation the new parittions you created as </p><p>primary 1 - /boot</p><p>primary 2 - /</p><p>and then proceed. The installer will do the right thing and it will boot.</p><p> <br /></p><p><br /></p>Ramon Bucklandhttp://www.blogger.com/profile/16744583593281442424noreply@blogger.com0tag:blogger.com,1999:blog-3473871805404328405.post-37744436150319794752014-04-16T00:20:00.001+01:002014-04-16T00:20:22.388+01:00SBT not ready for the Corporate World<div dir="ltr" style="text-align: left;" trbidi="on">
I have been using SBT now for about two years and I really like it's features. I very much grok it.<br />
<br />
It is definitely growing up, in terms of new features and stability. However, of course there is an however, "I can't give this to my friends".<br />
<br />
It is just not ready for the corporate world. The likes of Gradle, and yes even Maven make it so much easier to explain the build process. Actually, I can explain the build process with gradle and maven. But SBT is SUCH a massive barrier. We really can do better folks.!<br />
<br />
I think I can base my gripes down to two areas: Syntax, and then a higher level, the complexity of creating a build setup.<br />
<br />
<b>Syntax</b><br />
In short, SBT has confusing and ugly syntax. (shocking really) Watching Martin Odersky's key note at the Scala 2013 days, he discussed the notion that we should be very careful with Symbolic vs alphabetic Method names.<br />
<br />
<a href="http://parleys.com/play/51c1994ae4b0d38b54f4621b/chapter35/about">http://parleys.com/play/51c1994ae4b0d38b54f4621b/chapter35/about</a> (see from 59:10)<br />
<br />
Here Martin discusses Alphabetic vs Symbolic method names. Can we have some normal reading config please ? Looked at a gradle build file lately ? I can explain that to a junior developer of 3 months very easy - but sbt. not a chance (I have tried). It just represents - "forget it - what do you know about node.js ?"<br />
<br />
<b>Build and Customisation Complexity</b><br />
<b><br /></b>
What Ruby on Rails and Maven and (some other stuff I am sure) have in common is the Convention over Configuration. Gradle does this well, where to extend a plugins execution, just define "more stuff for it to do".<br />
<br />
The way settings are applied in a .scala definition is almost appalling. It really needs a rethink as when I show people the various methods, you get blank stares or squizzed up faces.<br />
<br />
With SBT, I find I have to write some very complex logic if I want to move files, or process config files etc etc. I feel this comes from SBT lacking a "customer API". It has an Engineers API at the moment but ease of extending it's activities really requires some serious depth knowledge of Scala AND SBT.<br />
<br />
<b>My issue</b><br />
<br />
SBT is shaped as the defacto build too for Scala; yes Maven and Gradle can do it. But SBT does it better. Because it is defacto, and because we want new users to learn the languages, and because we want the new users to have good projects (for longevity of maintenance) we want them to use a build tool. But our option .. sticking them in front of SBT ?? oh dear.<br />
<br />
Scala is a fantastic language, simply my favourite and I know a LOT of languages and environments. Ant was a brilliant help, and Maven improved that massively (yes yes, ant v maven). So when I train new users (ok / so so developers) in a project, I don't want to confuse them with "take up", I just want to get them on the road to exploring. Training Java, it might be Spring, or Drools or JAX-RS or something else. With Maven (and now with Gradle as that is my preferred Java build tool) I can simply setup a build script, easily (and I mean easy), explain the script to someone, and get them on their way.<br />
<br />
The new-ish developer does not need to know much about Java to understand Maven and even less knowledge to understand Gradle as it reads better.<br />
<br />
But if I wanted to get someone up and running with Scala, giving them a default Build tool, I can show them an SBT build setup. Oops. that doesn't work.<br />
<br />
Parallel this poor experience with how Scala compares to Java, I can show them a case class and how simple that is, so at the high level, Scala as a language is easy, but SBT is just killing take up for the less experienced.<br />
<br />
I have read that sbt is like vi, once you grok it, you get it.. and yes I am definitely there. (vi is my favourite editor) - but please can we work on a new build definition language - and I really mean something that it's GOAL is user friendliness, to overlay on top of the vi-ness ?<br />
<br />
If we get that right, because SBT is a defacto build tool, we might get even more developers!<br />
<br />
Is there something in the works that will help SBT be a tool for the masses ? Because at the moment you need a degree and a year on Scala to understand how to do more complex things (and sometimes something basic).<br />
<br />
Every so often I do consider switching a project to Gradle but, of course, SBT just does work better with Scala and I like it, but I can't share a build setup with a new junior developer. I have tried and it just confuses to the point on non-productiveness<br />
<br />
So I have an itch, I may scratch it.. .. but I haven't yet<br />
<br />
There, I can have a coffee now.<br />
<br />
<rant-off/></div>
Ramon Bucklandhttp://www.blogger.com/profile/16744583593281442424noreply@blogger.com0tag:blogger.com,1999:blog-3473871805404328405.post-20027080094835023542014-03-15T01:47:00.002+00:002014-03-15T01:47:42.107+00:00akka-persistence and DDD<div dir="ltr" style="text-align: left;" trbidi="on">
I have been working for some years now on a few projects. In many respects just code to exercise the brain.<br />
<br />
About 3 years back I picked up event sourcing from a Scala Days conference and I am sold. now more than ever.<br />
<br />
So I have released the core components of my scala - akka-persistence - DDD (CQRS-esq) implementation over on github for all to peruse.<br />
<br />
In the coming weeks I will add a "Petstore" ala, sample application to the implementation (and it will be a case of eat your own dog food).<br />
<br />
<a href="https://github.com/rbuckland/io-straight-fw">https://github.com/rbuckland/io-straight-fw</a><br />
<br />
This ties together the very odd Uuid generator I blogged about about a year a go and Event Sourcing of objects.<br />
<br />
I am using AngularJS as my front library which works really well because with spray.io and the Jackson Marshalling, I get the Scala Case classess coming back to the browser as JSON with next to no effort at all.<br />
<br />
And because it is all in memory, the speed is just blindingly quick! Very very impressed.<br />
<br />
Thanks have to go to<br />
- Martin Krasser for eventsourced - and akka-persistence (it's successor)<br />
- Mathias Doentiz and the spray.io team<br />
- All the Typesafe Akka guys. Seriously brilliant stuff<br />
- The whole AngularJS team - job really well done.<br />
- Jackson JSON Marshalling. Truely the best JSON marshaller there is.<br />
(is that it ? ) .. for now yep.<br />
<br />
<br /></div>
Ramon Bucklandhttp://www.blogger.com/profile/16744583593281442424noreply@blogger.com0tag:blogger.com,1999:blog-3473871805404328405.post-24531039868406591732013-11-04T20:45:00.003+00:002013-11-04T20:49:32.035+00:00Ramon's Rough Guide to Akka - Top Tips 1<div dir="ltr" style="text-align: left;" trbidi="on">
<b>Ramon's Rough Guide to Akka - Top Tips 1</b><br />
<b><br /></b>
<i>And someone please tell me if the tips are wrong :-)</i><br />
<br />
Over the past 2 years in my spare down time, I have been using Akka for the core of my application. There is so much to grok, but it is also very simple.<br />
<br />
This will be a really short post, of things which I wish I had read somewhere, before diving into Akka.<br />
<br />
<u><b>1. </b>Don't pass ActorRef's around :-)</u><br />
<br />
The ActorRef (as I read tonight) is a handle to a specific instance of an Actor. So when that Actor dies an another takes it's place (or in my case during event sourcing replay) - The ActorRef will become stale.<br />
<br />
Instead, get the ActorRef (or an ActorSelection) from<br />
<br />
<span style="font-family: Courier New, Courier, monospace;"> system.actorSelection("name")</span><br />
<span style="font-family: Courier New, Courier, monospace;"> context.actorSelection("/orPath/name")</span><br />
<br />
(and I uncovered this due to the deprecation of actorFor).<br />
This will give you the Actor you need.<br />
<br />
I guess I passed the ActorRef in to places because I am not using any DI (like I used to always with Spring). So instead .. now system or the context can be always used.<br />
<br />
<br />
<u><b>2. </b>Read this new Post of Actor paths and Actor addressing!</u><br />
<br />
This explains (1) better than I can.<br />
<br />
<a href="http://doc.akka.io/docs/akka/snapshot/general/addressing.html">http://doc.akka.io/docs/akka/snapshot/general/addressing.html</a><br />
<br />
That's it for today.</div>
Ramon Bucklandhttp://www.blogger.com/profile/16744583593281442424noreply@blogger.com0tag:blogger.com,1999:blog-3473871805404328405.post-6105439382796279132013-10-11T16:30:00.001+01:002013-10-11T16:38:43.492+01:00Embedding a Primary Key in a UUID / GUID for CQRS / ES<p style="font-family: Helvetica, Arial, sans-serif; font-size: 14px;">For some time now I have been working on an Event Sourced system. Today I am going to describe my UUID/GUID primary key approach that I devised. Ignoring much about the event sourced technology and the merits thereof; primary key's are common and critical and their usage in an Event Sourced system is very critical.</p>
<p style="font-family: Helvetica, Arial, sans-serif; font-size: 14px;">The usual route for a primary key in a system is an increasing number (Person ID = 102).</p>
<p style="font-family: Helvetica, Arial, sans-serif; font-size: 14px;">With Event Sourcing and CQRS, the UUID or GUID is often identified as a good implementation choice for your primary key</p>
<p style="font-family: Helvetica, Arial, sans-serif; font-size: 14px;">If you are interested in the topic, then these two resources are a good read. (nothing to do with Event Sourcing)</p>
<ul>
<li>The Cost of GUIDs as Primary Keys - <a href="http://www.informit.com/articles/printerfriendly.aspx?p=25862">http://www.informit.com/articles/printerfriendly.aspx?p=25862</a></li>
<li>GUIDs as fast primary keys under multiple databases - <a href="http://www.codeproject.com/Articles/388157/GUIDs-as-fast-primary-keys-under-multiple-database">http://www.codeproject.com/Articles/388157/GUIDs-as-fast-primary-keys-under-multiple-database</a></li>
</ul>
<p style="font-family: Helvetica, Arial, sans-serif; font-size: 14px;">But I have a few issues with the UUID. For one, it's not very client friendly, nor is it easily memorable. I want a system wide unique reference for all my objects ; a UUID is great for that; and I want it to also function as a friendly and memorable primary key.. how ? Well here below is my solution.</p>
<p style="font-family: Helvetica, Arial, sans-serif; font-size: 14px;"><strong><span style="font-family: tahoma, arial, helvetica, sans-serif;">A little about Event Sourcing</span></strong></p>
<p style="font-family: Helvetica, Arial, sans-serif; font-size: 14px;">With event sourcing, the primary route to storage is not through the "Domain Object".</p>
<p style="font-family: Helvetica, Arial, sans-serif; font-size: 14px;">Most developers are familiar with an Object-Relational Mapping system (Hibernate, NHibernate etc). (Object persisted to a table). This usual approach is that a Domain object is persisted (from a class or class heirarchy structure) to a database table. Event Sourcing instead stores the "event" that creates or modifies the Domain Object, and not the Object itself. A rebuild or some other query is a replay of the Events stream that was persisted.</p>
<p style="font-family: Helvetica, Arial, sans-serif; font-size: 14px;">I explain all that to say that an Event Sourced (CQRS) system can be designed without "tradtional primary keys".</p>
<p style="font-family: Helvetica, Arial, sans-serif; font-size: 14px;">Each event has a key, and the objects created can have keys, but what is common is to use UUIDs across all domain objects.</p>
<p style="font-family: Helvetica, Arial, sans-serif; font-size: 14px;"><strong>My Design Goal</strong></p>
<p style="font-family: Helvetica, Arial, sans-serif; font-size: 14px;">I wanted some traditional primary keys; and I wanted the UUID. In short, the system should be simple to use; being both client friendly and a memorable primary key for use.</p>
<p style="font-family: Helvetica, Arial, sans-serif; font-size: 14px;">When you have a UUID as the primary key and it is used in API's and get's passed around the office as "Look up client 3422" you need this API to be simple.</p>
<p style="font-family: Helvetica, Arial, sans-serif; font-size: 14px;">An example: /person/fbe645f0-3031-41e3-aa6e-0800200c9a66 is just not a nice URI</p>
<p style="font-family: Helvetica, Arial, sans-serif; font-size: 14px;">However,</p>
<p style="font-family: Helvetica, Arial, sans-serif; font-size: 14px; padding-left: 30px;"><span style="font-family: 'courier new', courier;">/person/2078</span></p>
<p style="font-family: Helvetica, Arial, sans-serif; font-size: 14px;">or</p>
<p style="font-family: Helvetica, Arial, sans-serif; font-size: 14px; padding-left: 30px;"><span style="font-family: 'courier new', courier;">/person/5f9</span></p>
<p style="font-family: Helvetica, Arial, sans-serif; font-size: 14px;">They are simple.</p>
<p style="font-family: Helvetica, Arial, sans-serif; font-size: 14px;">What I have embarked on is to segment my UUID (my primary keys) so that I utilise the entire UUID space - a part for randomness, a part for the Group (or table ID if you like) and a part for the object; in this example case, the person.</p>
<p style="font-family: Helvetica, Arial, sans-serif; font-size: 14px;"><strong>The make up of my Custom UUID</strong></p>
<p style="font-family: Helvetica, Arial, sans-serif; font-size: 14px;">The UUID is a 128 bit number, represented as 32 hex characters (with some dashes for legibility)</p>
<p style="font-family: Helvetica, Arial, sans-serif; font-size: 14px;">The UUID specification reserves some bits for version and variant.</p>
<p style="font-family: Helvetica, Arial, sans-serif; font-size: 14px;">With much (actually 30 minutes) thought, I have decided to use Version 6. (it doesn't exist in the IETF RFC 4122, they just went 1, 2, 3, 4 and 5)</p>
<p style="font-family: Helvetica, Arial, sans-serif; font-size: 14px;">and just make the Variant 'a'.</p>
<p style="font-family: Helvetica, Arial, sans-serif; font-size: 14px;">but ignoring all that, what is special about my UUID is that I embed a primary key and a "group" ID into the UUID.</p>
<p style="font-family: Helvetica, Arial, sans-serif; font-size: 14px;">So, my UUID's look like</p>
<p style="font-family: Helvetica, Arial, sans-serif; font-size: 14px; padding-left: 30px;"><span style="font-family: 'courier new', courier;">pppppppp-pppp-6ggg-arrr-rrrrrrrrrrrr</span></p>
<p style="font-family: Helvetica, Arial, sans-serif; font-size: 14px; padding-left: 30px;"><span style="font-family: 'courier new', courier;">where p is the Primary key (sequential incremementing)</span></p>
<p style="font-family: Helvetica, Arial, sans-serif; font-size: 14px; padding-left: 30px;"><span style="font-family: 'courier new', courier;">where 6 is the Version (always 6)</span></p>
<p style="font-family: Helvetica, Arial, sans-serif; font-size: 14px; padding-left: 30px;"><span style="font-family: 'courier new', courier;">where g is the Group ID, (akin to an identifier of the table)</span></p>
<p style="font-family: Helvetica, Arial, sans-serif; font-size: 14px; padding-left: 30px;"><span style="font-family: 'courier new', courier;">where a is the variant (always a)</span></p>
<p style="font-family: Helvetica, Arial, sans-serif; font-size: 14px; padding-left: 30px;"><span style="font-family: 'courier new', courier;">where r is the random part</span></p>
<p style="font-family: Helvetica, Arial, sans-serif; font-size: 14px;">This is my working notes: </p>
<p style="font-family: Helvetica, Arial, sans-serif; font-size: 14px;"><!--?xml version="1.0" encoding="UTF-8" standalone="no"?--></p>
<div style="font-family: Arial;"><span style="font-family: 'Andale Mono';">00000000-0000-6000-a000-000000000000</span></div>
<div style="font-family: Arial;"><span style="font-family: 'Andale Mono';"> -----------------</span></div>
<div style="font-family: Arial;"><span style="font-family: 'Andale Mono';"> 60 bits Random bits (time or other)</span></div>
<div style="font-family: Arial;"><span style="font-family: 'Andale Mono';"> -</span></div>
<div style="font-family: Arial;"><span style="font-family: 'Andale Mono';"> x == a, b, 8 or 9</span></div>
<div style="font-family: Arial;"><span style="font-family: 'Andale Mono';"> ---</span></div>
<div style="font-family: Arial;"><span style="font-family: 'Andale Mono';"> group ID - of 12 bits value is (0 to 4095)</span></div>
<div style="font-family: Arial;"><span style="font-family: 'Andale Mono';"> </span><span style="font-family: 'Andale Mono';">- </span></div>
<div style="font-family: Arial;"><span style="font-family: 'Andale Mono';"> 6 == always 6 - (Version 4, Random UUID)</span></div>
<div style="font-family: Arial;"><span style="font-family: 'Andale Mono';">-------- ----</span></div>
<div style="font-family: Arial;"><span style="font-family: 'Andale Mono';">Primary key ID of 48 bits (max </span><span style="color: #444444; font-family: arial, sans-serif; font-size: small; line-height: 16px; text-align: left;">281,474,976,710,656)</span></div>
<p style="font-family: Helvetica, Arial, sans-serif; font-size: 14px;">So I have enough bits for a primary ID (halfway between Int *32 bits and Long *64 bits)</p>
<p style="font-family: Helvetica, Arial, sans-serif; font-size: 14px;"><strong>The Version</strong></p>
<p style="font-family: Helvetica, Arial, sans-serif; font-size: 14px;">I decided to use 6 as the Version, 1 - 3 are other uses, 4 is Random or Psuedo random, which is almost mine, but not quite and Version 5 is a SHA-1 Hash; the RFC 4122 stipulates the following</p>
<blockquote>
<p>Process of identifier assignment:</p>
<p>Generating a UUID does not require that a registration authority<br /> be contacted. One algorithm requires a unique value over space<br /> for each generator. This value is typically an IEEE 802 MAC<br /> address, usually already available on network-connected hosts.<br /> The address can be assigned from an address block obtained from<br /> the IEEE registration authority. If no such address is available,<br /> or privacy concerns make its use undesirable, Section 4.5<br /> specifies two alternatives. Another approach is to use version 3<br /> or version 4 UUIDs as defined below.</p>
</blockquote>
<p style="font-family: Helvetica, Arial, sans-serif; font-size: 14px;">The sentence I want to draw attention to is - "Generating a UUID does not require that a registration authority be contacted"</p>
<p style="font-family: Helvetica, Arial, sans-serif; font-size: 14px;">But then again, because I am using Version 6, it's probably not a UUID.</p>
<p style="font-family: Helvetica, Arial, sans-serif; font-size: 14px;">My reasoning to a Version 6 are:</p>
<ul style="font-family: Helvetica, Arial, sans-serif; font-size: 14px;">
<li>My UUID is not a Version 4, because it is only partially random ( a part of it )</li>
<li>It is not any other Version (1-3, or 5)</li>
<li>Version 6 wasn't used</li>
<li>If someone has an issue with my use, then I'll call it instead a LUUID, a Local UUID</li>
</ul>
<p style="font-family: Helvetica, Arial, sans-serif; font-size: 14px;">Moving on.</p>
<p style="font-family: Helvetica, Arial, sans-serif; font-size: 14px;"><strong>The Group ID</strong></p>
<p style="font-family: Helvetica, Arial, sans-serif; font-size: 14px;">I wanted to have a unique key that represents all objects across the space. In this way I can now have a REST URI that looks like</p>
<p style="font-family: Helvetica, Arial, sans-serif; font-size: 14px; padding-left: 30px;"><span style="font-family: 'courier new', courier;">/any/<uuid></span></p>
<p style="font-family: Helvetica, Arial, sans-serif; font-size: 14px;">and the system can appropriately redirect to the correct resource, by looking at the group ID. (pattern matching the -6xxx- )</p>
<p style="font-family: Helvetica, Arial, sans-serif; font-size: 14px;">For example, if we have the UUID</p>
<p style="font-family: Helvetica, Arial, sans-serif; font-size: 14px; padding-left: 30px;"><span style="font-family: 'courier new', courier;">000000231-22e-6aa1-a789-28ef27ab7c62</span></p>
<p style="font-family: Helvetica, Arial, sans-serif; font-size: 14px;">This is</p>
<ul style="font-family: Helvetica, Arial, sans-serif; font-size: 14px;">
<li>aa1 for the Group ID for 'person'</li>
<li>23122e for the primary key</li>
</ul>
<p style="font-family: Helvetica, Arial, sans-serif; font-size: 14px; padding-left: 30px;"><span style="font-family: 'courier new', courier;">/any/000000231-22e-6aa1-a789-28ef27ab7c62</span></p>
<p style="font-family: Helvetica, Arial, sans-serif; font-size: 14px;">and with a match we can redirect the request to</p>
<p style="font-family: Helvetica, Arial, sans-serif; font-size: 14px; padding-left: 30px;"><span style="font-family: 'courier new', courier;">/person/000000231-22e-6aa1-a789-28ef27ab7c62</span></p>
<p style="font-family: Helvetica, Arial, sans-serif; font-size: 14px;"><strong>Clashes in the Primary Key with a Distributed System</strong></p>
<p style="font-family: Helvetica, Arial, sans-serif; font-size: 14px;">CQRS guru Greg Young says of the UUID (and Event Sourced systems)</p>
<blockquote>
<p style="margin: 0px; font-size: 11px; font-family: Calibri;">Having the client originate Ids normally in the form of UUIDs is extremely valuable in distributed systems.</p>
</blockquote>
<p style="font-family: Helvetica, Arial, sans-serif; font-size: 14px;">And this is true, but I don't want to give the client that honour, I want to allocate them a UUID (for many reasons) - but that comes with a drawback of needing to cater for clashes. (there is always a tradeoff with IT)</p>
<p style="font-family: Helvetica, Arial, sans-serif; font-size: 14px;">For my UUIDs, the primary key is sequential, at point of allocation. (231-22e, 231-22f, 231-230 etc). Having the end of the "UUID" random (e.g.: <span style="font-family: 'courier new', courier; font-size: 14px;">-a789-28ef27ab7c62)</span>means I can cluster (multiple systems) and allow for "duplicates" even in the primary key space, (where two servers generate the same ID and Group ID).</p>
<p style="font-family: Helvetica, Arial, sans-serif; font-size: 14px;">Leaving the UUID as it is (with a duplicate in the PK ID space) is ok but not ideal. So we will need to cater for that. (and of course for the lottery day when two systems generate the same UUID even down to the random part!)</p>
<p style="font-family: Helvetica, Arial, sans-serif; font-size: 14px;">Let me explain that : Assume that I have a web application with two servers that generate "people",</p>
<p style="font-family: Helvetica, Arial, sans-serif; font-size: 14px;"> www.myco.com </p>
<ul>
<li>server1-myco; and</li>
<li>server2-myco</li>
</ul>
<div> </div>
<div>If server1-repo and server2-repo BOTH generate a UUID but the random part differs, e.g.</div>
<div> </div>
<div><span style="font-family: 'courier new', courier; font-size: 14px;">server1 - 000000<strong>231-a32</strong>-6aa1-a789-28ef27ab7c62 (Jim)</span></div>
<div><span style="font-family: 'courier new', courier; font-size: 14px;"><span style="font-family: 'courier new', courier; font-size: 14px;">server2 - 000000<strong>231-a32</strong>-6aa1-a25e-6ac5c6ef127c (Mary)</span></span></div>
<p style="font-family: Helvetica, Arial, sans-serif; font-size: 14px;">then, technically I have two unique ID's, but the initial shorter ID's clash. This is not exactly ideal - because I want my friendly PK ID's to be unique also. If I want to minimise that "duplicate", then here are some options.</p>
<ol>
<li>Single ID generation node (creates a single point of failure)</li>
<li>Regular re-assignment in case of a clash</li>
<li>Generate and check with peers</li>
<li>Block reserving</li>
</ol>
<p style="font-family: Helvetica, Arial, sans-serif; font-size: 14px;">Options 2, 3 and 4 will be my preferred. With a distributed system, I have that PK ID issue anyways, so it's not different because I am playing with UUID's.</p>
<p style="font-family: Helvetica, Arial, sans-serif; font-size: 14px;">But what I really like is how I can use the primary key all by itself, outside of the /UUID</p>
<p style="font-family: Helvetica, Arial, sans-serif; font-size: 14px;">for example. </p>
<blockquote>
<p style="font-family: Helvetica, Arial, sans-serif; font-size: 14px; padding-left: 30px;"><span style="font-family: 'book antiqua', palatino;"><br /></span></p>
<p style="font-family: Helvetica, Arial, sans-serif; font-size: 14px; padding-left: 30px;"><span style="font-family: 'book antiqua', palatino;">Dear Mr Client,</span></p>
<p style="font-family: Helvetica, Arial, sans-serif; font-size: 14px; padding-left: 30px;"><span style="font-family: 'book antiqua', palatino;">Welcome as a supplier to Company X. For future reference, your client ID is 23122e.</span></p>
<p style="font-family: Helvetica, Arial, sans-serif; font-size: 14px; padding-left: 30px;"><span style="font-family: 'book antiqua', palatino;">...</span></p>
<p style="font-family: Helvetica, Arial, sans-serif; font-size: 14px; padding-left: 30px;"><span style="font-family: 'book antiqua', palatino;"><br /></span></p>
</blockquote>
<p style="font-family: Helvetica, Arial, sans-serif; font-size: 14px;">or</p>
<p style="font-family: Helvetica, Arial, sans-serif; font-size: 14px; padding-left: 30px;"><span style="font-family: 'courier new', courier;">/person/23122e</span></p>
<p style="font-family: Helvetica, Arial, sans-serif; font-size: 14px;">I don't have to use the full UUID everywhere, but rather just where I need it (in the event sourced messages), and as a unique global ID on the system.</p>
<p style="font-family: Helvetica, Arial, sans-serif; font-size: 14px;">Debugging a system will be easier too, looking at logs, someone with a little knowledge will recognise 'people' UUID's vs 'building', or 'schedule' UUID's (because they will distinguish the group ID after '<span style="font-family: 'courier new', courier;">-6xxx-</span>' as the unique group identifier); in effect people will learn those 3 characters and identify what the group Id is.</p>
<p style="font-family: Helvetica, Arial, sans-serif; font-size: 14px;"><strong>UUID generation</strong></p>
<p style="font-family: Helvetica, Arial, sans-serif; font-size: 14px;">So now I hear you wonder, how do I generate these mythical UUID's ?</p>
<p style="font-family: Helvetica, Arial, sans-serif; font-size: 14px;">Simply really.</p>
<p style="font-family: Helvetica, Arial, sans-serif; font-size: 14px;">This is the call for the generating the UUID (reusing the java.util.UUID class, just giving two lower and upper longs (64 bits each)</p>
<p style="margin: 0px; font-size: 11px; font-family: Monaco; color: #4f76cb;"><span style="color: #000000;"> </span>/**</p>
<p style="margin: 0px; font-size: 11px; font-family: Monaco; color: #4f76cb;"> * Create a new UUID given some ID as the groupID and an already sequenced ID</p>
<p style="margin: 0px; font-size: 11px; font-family: Monaco; color: #4f76cb;"> */</p>
<p style="margin: 0px; font-size: 11px; font-family: Monaco;"> <span style="color: #931a68;">def</span> createUuid(groupId: Int, id: Long): Uuid = {</p>
<p style="margin: 0px; font-size: 11px; font-family: Monaco; min-height: 15px;"> </p>
<p style="margin: 0px; font-size: 11px; font-family: Monaco;"> <span style="color: #931a68;">val</span> <span style="color: #727aff;">randomBytes</span> = <span style="color: #931a68;">new</span> Array[Byte](<span style="color: #d0a3ff;">8</span>)</p>
<p style="margin: 0px; font-size: 11px; font-family: Monaco;"> secureRandom.nextBytes(<span style="color: #727aff;">randomBytes</span>)</p>
<p style="margin: 0px; font-size: 11px; font-family: Monaco;"> <span style="color: #931a68;">val</span> <span style="color: #727aff;">randomLong</span> = java.nio.ByteBuffer.wrap(<span style="color: #727aff;">randomBytes</span>).getLong()</p>
<p style="margin: 0px; font-size: 11px; font-family: Monaco; min-height: 15px;"> </p>
<p style="margin: 0px; font-size: 11px; font-family: Monaco; color: #4e9072;"><span style="color: #000000;"> </span>// groupID has to be 12bits as the 4L is going in over the top.</p>
<p style="margin: 0px; font-size: 11px; font-family: Monaco;"> <span style="color: #931a68;">return</span> Uuid(<span style="color: #931a68;">new</span> java.util.UUID(groupId | (<span style="color: #d0a3ff;">6L</span> << <span style="color: #d0a3ff;">12</span>) | (id << <span style="color: #d0a3ff;">16</span>), (<span style="color: #d0a3ff;">10L</span> << <span style="color: #d0a3ff;">60</span>) | (<span style="color: #727aff;">randomLong</span> >>> <span style="color: #d0a3ff;">4</span>)).toString())</p>
<p style="margin: 0px; font-size: 11px; font-family: Monaco; min-height: 15px;"> </p>
<p style="margin: 0px; font-size: 11px; font-family: Monaco;"> }</p>
<p style="margin: 0px; font-size: 11px; font-family: Monaco;"> </p>
<p style="margin: 0px; font-size: 11px; font-family: Monaco;"><span style="font-family: Helvetica, Arial, sans-serif; font-size: 14px;">The </span><span style="text-decoration: underline;">Primary Key</span><span style="font-family: Helvetica, Arial, sans-serif; font-size: 14px;"> is an incremental sequence on the code that creates a new person, or group, or employee (etc)</span></p>
<p><span style="font-family: Helvetica, Arial, sans-serif; font-size: large;">In Scala, it is simply a matter of adding in a Trait for the "class" you want ID's sequenced for/ UUID's generated for. e.g: I add </span><span style="font-family: Monaco; font-size: 11px; color: #931a68;">with</span><span style="font-family: Monaco; font-size: 11px;"> UuidGenerator[Person] </span><span><span style="font-family: Helvetica, Arial, sans-serif; font-size: large;">and this gives a newUuid() method</span><span style="font-family: Monaco;"><span style="font-size: 11px;"> </span></span></span></p>
<p style="margin: 0px; font-size: 11px; font-family: Monaco;"><span style="color: #931a68;">class</span> PersonProcessor(<span style="color: #931a68;">val</span> <span style="color: #0326cc;">repository</span>: Repository[Uuid, Person]) <span style="color: #931a68;">extends</span> AbstractProcessor[Person]</p>
<p style="margin: 0px; font-size: 11px; font-family: Monaco;"><span style="color: #931a68;">with</span> UuidGenerator[Person] </p>
<p style="margin: 0px; font-size: 11px; font-family: Monaco;">{ <span style="color: #0326cc;">this</span>: Emitter =></p>
<p style="margin: 0px; font-size: 11px; font-family: Monaco; min-height: 15px;"> </p>
<p style="margin: 0px; font-size: 11px; font-family: Monaco;"> <span style="color: #931a68;">def</span> klass = classOf[Person]</p>
<p style="margin: 0px; font-size: 11px; font-family: Monaco;"> </p>
<p style="margin: 0px; font-size: 11px; font-family: Monaco;">...</p>
<p style="margin: 0px; font-size: 11px; font-family: Monaco;"> createPerson(<strong>newUuid</strong>, <span style="color: #727aff;">cmd</span> )</p>
<p style="margin: 0px; font-size: 11px; font-family: Monaco;"> </p>
<p style="margin: 0px; font-size: 11px; font-family: Monaco;"><span style="font-family: Helvetica, Arial, sans-serif; font-size: 14px;">The implementation of that newUuid() method looks as follows:</span></p>
<p style="margin: 0px; font-size: 11px; font-family: Monaco;"><span style="font-family: Helvetica, Arial, sans-serif; font-size: 14px;"><br /></span></p>
<p style="margin: 0px; font-size: 11px; font-family: Monaco;"><span style="color: #931a68;">trait</span> UuidGenerator[D] {</p>
<p style="margin: 0px; font-size: 11px; font-family: Monaco; min-height: 15px;"> </p>
<p style="margin: 0px; font-size: 11px; font-family: Monaco;"> <span style="color: #931a68;">implicit</span> <span style="color: #931a68;">def</span> klass: Class[_]</p>
<p style="margin: 0px; font-size: 11px; font-family: Monaco; min-height: 15px;"> </p>
<p style="margin: 0px; font-size: 11px; font-family: Monaco;"> <span style="color: #931a68;">private</span> <span style="color: #931a68;">val</span> <span style="color: #0326cc;">ids</span> = Map.empty[String, Long]</p>
<p style="margin: 0px; font-size: 11px; font-family: Monaco; min-height: 15px;"> </p>
<p style="margin: 0px; font-size: 11px; font-family: Monaco;"> <span style="color: #931a68;">private</span>[<span style="color: #931a68;">this</span>]<span style="color: #931a68;">def</span> className = klass.getClass().getCanonicalName()</p>
<p style="margin: 0px; font-size: 11px; font-family: Monaco; min-height: 15px;"> </p>
<p style="margin: 0px; font-size: 11px; font-family: Monaco; color: #4f76cb;"><span style="color: #000000;"> </span>/**</p>
<p style="margin: 0px; font-size: 11px; font-family: Monaco; color: #4f76cb;"> * return the next ID</p>
<p style="margin: 0px; font-size: 11px; font-family: Monaco; color: #4f76cb;"> */</p>
<p style="margin: 0px; font-size: 11px; font-family: Monaco;"> <span style="color: #931a68;">def</span> newUuid:Uuid = {</p>
<p style="margin: 0px; font-size: 11px; font-family: Monaco;"> <span style="color: #931a68;">val</span> <span style="color: #727aff;">idKey</span> = className</p>
<p style="margin: 0px; font-size: 11px; font-family: Monaco;"> <span style="color: #931a68;">val</span> <span style="color: #727aff;">currentId</span> = <span style="color: #0326cc;">ids</span>.getOrElseUpdate(<span style="color: #727aff;">idKey</span>, <span style="color: #d0a3ff;">0L</span>)</p>
<p style="margin: 0px; font-size: 11px; font-family: Monaco;"> <span style="color: #0326cc;">ids</span> += (<span style="text-decoration: underline; color: #727aff;">idKey</span> -> (<span style="color: #727aff;">currentId</span> + <span style="color: #d0a3ff;">1</span>))</p>
<p style="margin: 0px; font-size: 11px; font-family: Monaco;"> <span style="color: #931a68;">return</span> Uuid.createUuid(klass, <span style="color: #727aff;">currentId</span> + <span style="color: #d0a3ff;">1</span>)</p>
<p style="margin: 0px; font-size: 11px; font-family: Monaco;"> }</p>
<p style="margin: 0px; font-size: 11px; font-family: Monaco;">...</p>
<p style="margin: 0px; font-size: 11px; font-family: Monaco;"><span style="font-family: Helvetica, Arial, sans-serif; font-size: 14px;"><br /></span></p>
<p style="margin: 0px; font-size: 11px; font-family: Monaco;"><span style="font-family: Helvetica, Arial, sans-serif; font-size: 14px;">And Uuid.createUuid looks like : </span></p>
<p style="margin: 0px; font-size: 11px; font-family: Monaco;"><span style="font-family: Helvetica, Arial, sans-serif; font-size: 14px;"><br /></span></p>
<p style="margin: 0px; font-size: 11px; font-family: Monaco;"> </p>
<p style="margin: 0px; font-size: 11px; font-family: Monaco;"> <span style="color: #931a68;">def</span> createUuid(klass: Class[_], id: Long): Uuid = createUuid(groupId(klass), id)</p>
<p style="margin: 0px; font-size: 11px; font-family: Monaco;"> </p>
<p style="margin: 0px; font-size: 11px; font-family: Monaco;"><span style="font-family: Helvetica, Arial, sans-serif; font-size: 14px;">The </span><span style="text-decoration: underline;">Group ID</span><span style="font-family: Helvetica, Arial, sans-serif; font-size: 14px;"> is simply CRC-12, or 12 bit CRC over the classname. This is because I had 12 bits to spare where I placed the groupId </span><span style="font-family: 'courier new', courier; font-size: 14px;">-6<strong>aa1</strong>- </span><span style="font-family: Helvetica, Arial, sans-serif; font-size: 14px;"> So given that all 'Person' domain objects extend from com.soqqo.system.domain.Person my groupId's are consistent there.</span></p>
<p style="margin: 0px; font-size: 11px; font-family: Monaco;"> </p>
<p>On bootstrapping my system, I make sure that all "in use" groupId's are not clashing on CRC12-ing them - could happen - and if it does I'll deal with that then. (just System.halt bootstrap .. and change code to suit)</p>
<p style="margin: 0px; font-size: 11px; font-family: Monaco;"> </p>
<p>This is the crc12 implementation in Scala. I haven't tested that brutally, but it DOES generate unique < 4096 ID's for random byte's passed in, so it is working the way I need it to.</p>
<p style="margin: 0px; font-size: 11px; font-family: Monaco;"> <span style="color: #931a68;">def</span> crc12(toHash: String) = {</p>
<p style="margin: 0px; font-size: 11px; font-family: Monaco; min-height: 15px;"> </p>
<p style="margin: 0px; font-size: 11px; font-family: Monaco;"> <span style="color: #4f76cb;">/**</span></p>
<p style="margin: 0px; font-size: 11px; font-family: Monaco; color: #4f76cb;"> * ************************************************************************</p>
<p style="margin: 0px; font-size: 11px; font-family: Monaco; color: #4f76cb;"> * Using direct calculation</p>
<p style="margin: 0px; font-size: 11px; font-family: Monaco; color: #4f76cb;"> * ************************************************************************</p>
<p style="margin: 0px; font-size: 11px; font-family: Monaco; color: #4f76cb;"> */</p>
<p style="margin: 0px; font-size: 11px; font-family: Monaco; min-height: 15px;"> </p>
<p style="margin: 0px; font-size: 11px; font-family: Monaco; color: #4e9072;"><span style="color: #000000;"> </span><span style="color: #931a68;">var</span><span style="color: #ff7671;">crc</span><span style="color: #000000;">:Int = </span><span style="color: #d0a3ff;">0xFFF</span><span style="color: #000000;">; </span>// initial contents of LFBSR</p>
<p style="margin: 0px; font-size: 11px; font-family: Monaco; color: #4e9072;"><span style="color: #000000;"> </span><span style="color: #931a68;">var</span><span style="color: #ff7671;">poly</span><span style="color: #000000;">: Int = </span><span style="color: #d0a3ff;">0xF01</span><span style="color: #000000;">; </span>// reverse polynomial</p>
<p style="margin: 0px; font-size: 11px; font-family: Monaco;"> <span style="color: #931a68;">var</span> <span style="color: #ff7671;">bytes</span> = toHash.getBytes()</p>
<p style="margin: 0px; font-size: 11px; font-family: Monaco; min-height: 15px;"> </p>
<p style="margin: 0px; font-size: 11px; font-family: Monaco;"> <span style="color: #931a68;">for</span> (<span style="color: #727aff;">b</span>: Byte <- <span style="text-decoration: underline; color: #ff7671;">bytes</span>) {</p>
<p style="margin: 0px; font-size: 11px; font-family: Monaco;"> <span style="color: #931a68;">var</span> <span style="color: #ff7671;">temp</span> = (<span style="color: #ff7671;">crc</span> ^ <span style="color: #727aff;">b</span>) & <span style="color: #d0a3ff;">0xff</span>;</p>
<p style="margin: 0px; font-size: 11px; font-family: Monaco; min-height: 15px;"> </p>
<p style="margin: 0px; font-size: 11px; font-family: Monaco; color: #4e9072;"><span style="color: #000000;"> </span>// read 8 bits one at a time</p>
<p style="margin: 0px; font-size: 11px; font-family: Monaco;"> <span style="color: #931a68;">for</span> (<span style="color: #727aff;">i</span> <- <span style="text-decoration: underline; color: #d0a3ff;">0</span> to <span style="color: #d0a3ff;">7</span>) {</p>
<p style="margin: 0px; font-size: 11px; font-family: Monaco;"> <span style="color: #931a68;">if</span> ((<span style="color: #ff7671;">temp</span> & <span style="color: #d0a3ff;">1</span>) == <span style="color: #d0a3ff;">1</span>) <span style="color: #ff7671;">temp</span> = (<span style="color: #ff7671;">temp</span> >>> <span style="color: #d0a3ff;">1</span>) ^ <span style="color: #ff7671;">poly</span>;</p>
<p style="margin: 0px; font-size: 11px; font-family: Monaco;"> <span style="color: #931a68;">else</span> <span style="color: #ff7671;">temp</span> = (<span style="color: #ff7671;">temp</span> >>> <span style="color: #d0a3ff;">1</span>);</p>
<p style="margin: 0px; font-size: 11px; font-family: Monaco;"> }</p>
<p style="margin: 0px; font-size: 11px; font-family: Monaco;"> <span style="color: #ff7671;">crc</span> = (<span style="color: #ff7671;">crc</span> >>> <span style="color: #d0a3ff;">8</span>) ^ <span style="color: #ff7671;">temp</span>;</p>
<p style="margin: 0px; font-size: 11px; font-family: Monaco;"> }</p>
<p style="margin: 0px; font-size: 11px; font-family: Monaco; min-height: 15px;"> </p>
<p style="margin: 0px; font-size: 11px; font-family: Monaco; color: #4e9072;"><span style="color: #000000;"> </span>// flip bits</p>
<p style="margin: 0px; font-size: 11px; font-family: Monaco;"> <span style="color: #ff7671;">crc</span> = <span style="color: #ff7671;">crc</span> ^ <span style="color: #d0a3ff;">0xfff</span>;</p>
<p style="margin: 0px; font-size: 11px; font-family: Monaco; min-height: 15px;"> </p>
<p style="margin: 0px; font-size: 11px; font-family: Monaco;"> <span style="color: #ff7671;">crc</span>;</p>
<p style="margin: 0px; font-size: 11px; font-family: Monaco; min-height: 15px;"> </p>
<p style="margin: 0px; font-size: 11px; font-family: Monaco;"> }</p>
<p style="margin: 0px; font-size: 11px; font-family: Monaco;"> </p>
<p><strong>Summary</strong></p>
<p>I hope this helps someone on any of the weird topics I have covered here. I will share the Uuid and UuidGenerator classes happily for anyone that wants them. They are anything special, but rather a lot of thinking about how I wanted my Uuid's to be utilised in the system.</p>
<p>My system entails:</p>
<p>1. spray.io - Web router on top of Spray Can<br />2. Event Sourced (now also known as akka-persistence)<br />3. AngularJS</p>
<p>Enjoy!</p>Ramon Bucklandhttp://www.blogger.com/profile/16744583593281442424noreply@blogger.com0tag:blogger.com,1999:blog-3473871805404328405.post-21381187560581211732013-07-02T19:50:00.000+01:002013-07-02T19:50:03.948+01:00Heroku and Gradle - and jetty-runner Configuration<p>
I recently worked on a project where we used Heroku as the deploment engine.
</p>
<p>
For speed of the project I chose Maven (simply because I know it well and it is VERY good).
</p>
<p>
Knowing also that Gradle is now becoming the "next build kid" on the block and, for me, recognising that Gradle is easier to configure, I set about the task today of switching the build to Gradle.
</p>
<p>
This was very simple, and the last "part of the puzzle" after replicating all the functionality was to setup the Heroku parts.
By default, we were using the Maven (jetty-runner) boot strapping. Arguably it is easier and lighter to run Jetty Embedded (as per <a href="https://github.com/heroku/devcenter-gradle">https://github.com/heroku/devcenter-gradle</a>) but I wanted to see what is required to use "jetty-runner"). This was more an excercise in build comparison, that it was "get it onto Heroku".
</p>
<p>
Heroku will detect a Maven pom.xml, and by default will run
</p>
<p>
<script class="brush:javascript" type="syntaxhighlighter"><![CDATA[
mvn package
]]></script>
</p>
<p>
This creates a war. Heroku then runs your app using "whatever" is in the Procfile. The Procfile they suggest looks like this:
</p>
<p>
<script class="brush:javascript" type="syntaxhighlighter"><![CDATA[
web: java $JAVA_OPTS -jar target/dependency/jetty-runner.jar --port $PORT target/*.war
]]></script>
</p>
<p>
The jetty-runner.jar gets in the "target/dependency/" folder due to this Maven Magic.
</p>
<p>
<script class="brush:xml" type="syntaxhighlighter">
<![CDATA[
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-dependency-plugin</artifactId>
<version>2.6</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>copy</goal>
</goals>
<configuration>
<artifactItems>
<artifactItem>
<groupId>org.mortbay.jetty</groupId>
<artifactId>jetty-runner</artifactId>
<version>${jetty.version}</version>
<destFileName>jetty-runner.jar</destFileName>
</artifactItem>
</artifactItems>
</configuration>
</execution>
</executions>
</plugin>
]]></script>
</p>
<p>
When Heroku detects a Gradle project, it runs
</p>
<p>
<script class="brush:javascript" type="syntaxhighlighter">
<![CDATA[
gradle stage
]]></script>
</p>
<p>
instead of "gradle package".
</p>
<p>
So to replicate the same with Gradle I had to write a copy task to get jetty-runner in there, and generate the war and attach that all to a "stage" task.
</p>
<p>
It is very easy when you look at it, the trick is in the knowing of the API that slows it down. But I resolved it in about 30 minutes.
</p>
<p>
In short, the changes for Gradle are as follows.
</p>
<p>
<b>1. Change your Procfile</b> (we will get the jetty-runner.jar in "build/libs") (the war goes there by default)
</p>
<p>
<script class="brush:javascript" type="syntaxhighlighter"><![CDATA[
web: java $JAVA_OPTS -jar build/libs/jetty-runner.jar --port $PORT build/libs/*.war
]]></script>
</p>
<p>
<b>2. Add a "new" configurations for the depenency of jetty-runner
</b></p>
<p>
<script class="brush:javascript" type="syntaxhighlighter"><![CDATA[
configurations {
runtimeOnly
}
]]></script>
</p>
<p>
<b>3. Add a new dependency for Jetty Runner</b> (note I also use newrelic so it goes in there too!)
</p>
<p>
<script class="brush:javascript" type="syntaxhighlighter"><![CDATA[
dependencies {
// to run our App on Heroku
runtimeOnly "org.mortbay.jetty:jetty-runner:8.1.1.v20120215"
// monitoring on Heroku
runtimeOnly "com.newrelic.agent.java:newrelic-agent:2.18.0"
...
}
]]></script>
</p>
<p>
<b>4. Create a new task which copies the jars from the dependencies "runtimeOnly"</b>
We also replicate the Maven method of renaming the jar to have no Version(s).
</p>
<p>
<script class="brush:javascript" type="syntaxhighlighter"><![CDATA[
task copyToLib << {
copy {
from configurations.runtimeOnly.copy().setTransitive(false)
into "$buildDir/libs"
rename { name ->
def artifacts = configurations.runtimeOnly.resolvedConfiguration.resolvedArtifacts
def artifact = artifacts.find { it.file.name == name }
"${artifact.name}.${artifact.extension}"
}
}
}
]]></script>
</p>
<p>
<b>5. Add a "stage" task, because that is what Heroku will run.
</b></p>
<p>
<script class="brush:javascript" type="syntaxhighlighter"><![CDATA[
task stage(dependsOn: ["clean", "war", "copyToLib"])
]]></script>
</p>
<p>
And that is it. When stage is run, it will create a war, and copy jetty-runner.jar into the build/libs folder.
</p>
<p>
Happy days.
</p>
Ramon Bucklandhttp://www.blogger.com/profile/16744583593281442424noreply@blogger.com0tag:blogger.com,1999:blog-3473871805404328405.post-26462475997172495372013-05-29T18:34:00.000+01:002013-05-29T18:34:09.094+01:00The hierarchy of the type X is inconsistent - A Classpath Issue<div dir="ltr" style="text-align: left;" trbidi="on">
<div>
Often the simple things will slow you down when developing.</div>
<div>
Today I had one such moment. It occurred a few days back where my Eclipse project was reporting the following error:</div>
<div>
<br /></div>
<div>
<div class="p1">
<span style="font-family: Courier New, Courier, monospace;">The hierarchy of the type LoggingFilter is inconsistent</span></div>
</div>
<div>
<br /></div>
<div dir="ltr" style="text-align: left;" trbidi="on">
The LoggingFilter was simply a class extending the Spring AbstractRequestLoggingFilter<br />
<br />
<br />
<script class="brush:java" type="syntaxhighlighter">
<![CDATA[
public class LoggingFilter extends AbstractRequestLoggingFilter {
]]>
</script>
<br />
<div class="p1">
Nothing really special was going on. I was kind of hoping it would be obvious but it wasn't.<br />
<br />
In short, my classpath was wrong where I had an import that did not (was older perhaps) match to a library that, somewhere in the stack, AbstractRequestLoggingFilter was depending on.<br />
<br />
I frequently used mvn dependency:tree to assess what this list was .. and nothing stood out. I checked the javax.servlet-api (yes I was using 3.0.1, and so was Spring). I checked my exclusion of commons logging (I use SLF4J), but that was okay.<br />
<br />
Eventually I looked inside the .classpath and to my mild horror I saw Spring 3.0.6 was included.
<br />
<script class="brush:xml" type="syntaxhighlighter">
<![CDATA[
<classpathentry kind="var" path="M2_REPO/org/springframework/spring-context/3.0.6.RELEASE/spring-context-3.0.6.RELEASE.jar" sourcepath="M2_REPO/org/springframework/spring-context/3.0.6.RELEASE/spring-context-3.0.6.RELEASE-sources.jar"/>
<classpathentry kind="var" path="M2_REPO/org/springframework/spring-core/3.2.3.RELEASE/spring-core-3.2.3.RELEASE.jar" sourcepath="M2_REPO/org/springframework/spring-core/3.2.3.RELEASE/spring-core-3.2.3.RELEASE-sources.jar">]]>
</script>
Looking back at the dependency:tree for Maven, it didn't show in the list.<br />
<br />
Which meant one of two things:<br />
1. mvn dependency:tree was wrong<br />
2. mvn eclipse:eclipse was wrong<br />
<br />
I looked first at dependency:tree, and then figured that perhaps Maven needed an update (I was using 3.0.4) and that took me to check release notes, of which I found : https://cwiki.apache.org/MAVEN/maven-3x-compatibility-notes.html#Maven3.xCompatibilityNotes-DependencyResolution<br />
<br />
and a magic note that , dependency:tree does not work according to maven's resolution.<br />
So no worries, I ran debug and I saw that .. yes Maven was "seeing" 3.0.6 but it was also excluding it.<br />
So that meant that eclipse:eclipse was wrong.. a quick pom change and (2.9 eclipse plugin) and it was all good.<br />
<br />
mvn eclipse:eclipse excluded 3.0.6 as expected and included 3.2.3 as needed.<br />
<br />
Jetty however is still bootstrapping with 3.0.6 .. so I may have to specifically find which of the depenedencies is trying to include it, exclude it and then forcibly include spring-context.. rather than rely on the transitive.<br />
<br />
Moral to the post .. check everything .. and assume nothing! </div>
<br /></div>
</div>
Ramon Bucklandhttp://www.blogger.com/profile/16744583593281442424noreply@blogger.com0tag:blogger.com,1999:blog-3473871805404328405.post-11462972649462917602013-03-08T23:23:00.001+00:002013-03-08T23:23:47.338+00:00Scala - Event Sourcing and Spray<div dir="ltr" style="text-align: left;" trbidi="on">
<h4 style="text-align: left;">
<span style="font-family: Arial, Helvetica, sans-serif;">Scala + NoSQL</span></h4>
<span style="font-family: Arial, Helvetica, sans-serif;">Over the past 24 months I have been diving a bit deeper into Scala by way of using a new architecture. My dabbling started 4 years ago. I haven't touched Java for about 2.</span><br />
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">I have been sick of, over the past 10 years, building the typical DB/App stack. If you track back about 5-6 years in my posts you'll notice I shifted to researching and investigating NoSQL enterprises. (db4o etc). The "ick" centred around the simple yet fundamental issue of the <a href="http://en.wikipedia.org/wiki/Object-relational_impedance_mismatch">ORM Impedance Mismatch</a>. Every man and dog has blogged and written about it and it is very prevalent.</span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">One thing led to another and I found myself loving Scala and it's fresh way of code development. But never really landed on a great NoSQL solution. Being a Java hack I immersed myself in its ways (Scala that is) and attended ScalaDays 2012 to sure up my skills; There I met and chatted with a lot of people and found the 3 days brilliant.</span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">At Scaladays I was researching my next "stack" and honed in on <a href="http://spray.io/">spray.io</a>. After the talk Matthias Doenitz gave at <a href="http://skillsmatter.com/podcast/scala/spray-rest-on-akka">ScalaDays</a>, I caught up with him in the hallways asking a general question of</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><br /></span></div>
<div>
<span style="background-color: #cccccc; color: blue; font-family: Arial, Helvetica, sans-serif;">"I am an old GWT hack and want less complexity; what interfaces do you see being plugged into Spray,.."</span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">to which he and a friend gave some tips to various JS libraries for me to check out (of which I have settled on <a href="http://twitter.github.com/bootstrap/">Twitter Bootstrap</a>). Matthias and / or the other (I don't recall who) made reference, if I wanted to walk on a new area, to check out the work in the Event Sourcing arena.</span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">I read and could see it's benefits. So I figured I would read some more. I ended up watching the threads on the <a href="http://groups.google.com/group/dddcqrs">DDD/CQRS</a>. That got me hooked - I have to say though that the CQRS is a simple yet massive theory and I wanted some good practical; it was to come.</span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">About the same time, <a href="http://krasserm.blogspot.co.uk/">Martin Krasser</a> posted some info on JAXB Scala Marshalling - I wanted some of that in my Spray application. Scala has excellent JSON marshalling using lift-json; alternatively you can use a built in library spray-json. Both of these worked well, however I wanted a "single" API definition class that I could expose as JSON or XML without me needing to code it twice. My sample application at the time was still bound to a Scala/Spring JPA and Hypersonic DB stack. </span></div>
<div>
<br /></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">Seeing the "sample" project that Martin Krasser had built, and his excellent blog posts on it, took me right into the </span><a href="https://github.com/eligosource/eventsourced" style="font-family: Arial, Helvetica, sans-serif;">eventsourced</a><span style="font-family: Arial, Helvetica, sans-serif;"> package where the JAXB marshalling was used. </span><span style="font-family: Arial, Helvetica, sans-serif;">Martin Krasser together with colleagues released the early draft of this Event Sourcing package and</span><span style="font-family: Arial, Helvetica, sans-serif;"> after a few iterations it was fully embedded into the Akka way .. and since Spray was too, I have joined the two together ever so simply - and now have the base framework for my perfect "stack".</span></div>
<h4 style="text-align: left;">
<span style="font-family: Arial, Helvetica, sans-serif;">The Scala ES Spray Stack</span></h4>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">So what does it look like ? At the front end, though I have some ways to go there, I have</span></div>
<div>
<ul style="text-align: left;">
<li><span style="color: #351c75; font-family: Courier New, Courier, monospace;">Twitter - Bootstrap</span><span style="font-family: Arial, Helvetica, sans-serif;">, which talks to a</span></li>
<li><span style="color: #351c75; font-family: Courier New, Courier, monospace;">JSON REST API</span><span style="font-family: Arial, Helvetica, sans-serif;">; into</span></li>
<li><span style="font-family: Courier New, Courier, monospace;">spray.io</span><span style="font-family: Arial, Helvetica, sans-serif;"> routing; which delegates the "commands"; to</span></li>
<li><span style="font-family: Courier New, Courier, monospace;">eventsourced</span></li>
</ul>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">The commands and the events that are journaled are Scala Case Classes, Annotated with JAXB annotations to support the Un/Marshalling in spray.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">Akka Camel will come next, though because I am an old Apache Camel Hack - random contributions and use throughout time, I know what it does and well, so will slot it in later.</span></div>
</div>
<h4 style="text-align: left;">
<span style="font-family: Arial, Helvetica, sans-serif;">A Weak Schema on Historic Events</span></h4>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">Event sourcing is great - in essence - keep everything you ever did. Everything. It is very BigData-esq, and actually beneficial for audit tracking. If I keep every Command/Event that the system responds and reacts with. I have a full "audit" trail of how it operates. CQRS gives me a benefit of no Database (can stick one on the "read" view if I want). However;</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">My last "piece" to the framework puzzle is the concept of supporting a weak schema. <i>After you read all about Event Sourcing, you will quickly realise that "Cross Version" software support </i>needs to be managed well. The typical "DB" stack doesn't have this challenge as much, simply because, unless coded for, all history is "thrown" away - and thus the problem is smaller - and DB upgrade scripts decide at "run" what is kept .. or what goes.. and the "change is usually irrevocable".</span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">With Event Sourcing, the "history" is with you, and that is it's benefit. So to retain that benefit through future upgrades of your application, you need to support the older events, in what ever form they have. A few ways to achieve this are:</span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="color: blue;">1. Retain, in code, the "V1, V2 and V3" objects that the event messages relate to;</span></span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="color: blue;"><br /></span></span></div>
<div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">Needless to say, this is quite complex. The amount of code you may have to manage over a long life span of the software could become messy. <span style="color: red;">* discounted *</span></span></div>
</div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div>
<span style="color: blue; font-family: Arial, Helvetica, sans-serif;">2. Upgrade the older Events, when you "upgrade" the Application.</span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">This may work - but it feels wrong. Upgrading events to something they never were breaks the model. Adding a field to an event .. what should be the default of the value that was never supplied 4 years ago ? Needless to say, you will recognise this is often the DB way. That is okay. It does work.. there is another way. </span><span style="color: red; font-family: Arial, Helvetica, sans-serif;">* discounted *</span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div>
<span style="color: blue; font-family: Arial, Helvetica, sans-serif;">3. Translate the older events on the fly as they are read in..</span></div>
<div>
<span style="color: blue; font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">My idea was to build a shim in the "se</span><span style="font-family: Arial, Helvetica, sans-serif;">rialisation layer" that translates older events V1, V2 up to V3 equivalent .. but again it kind of smells.. so that is not it. </span><span style="color: red; font-family: Arial, Helvetica, sans-serif;">* discounted * </span><span style="font-family: Arial, Helvetica, sans-serif;">.. kind of.. </span><span style="color: red; font-family: Arial, Helvetica, sans-serif;"> </span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="color: blue;">4. A Weak Schema.. </span></span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="color: blue;"><br /></span></span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">Greg Young, on the eventsourced mailing list pointed me (in two words .. "Weak Schema") to look at this model .. quickly I ended looking at Google's <a href="https://developers.google.com/protocol-buffers/docs/overview">protobuf</a>. I had read it before but had not the need, until now.</span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">Protobuf is all about "version" management across messages between systems. This is exactly what Google built it for. Between their index servers that may be running different versions of software. And as it turns out, it might be a brilliant fit. I have some "tests" to do, which for now I am going to park as the theory seems okay, and when the need warrants I will utilise it's power. </span></div>
<div>
<br /></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">I googled in earnest about working with a "weak schema" but there was not a lot to read, but it didn't take too long to work it out.</span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">Let me give you an example: </span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">if we have a "Command" object that we will serialise to disk. It could look like this: </span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">"CorrectTheBirthDate(uuid,newBirthDate)"</span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">In my fictitious application, imagine we serialise this to JSON. It may look like this.</span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div>
<script class="brush:javascript" type="syntaxhighlighter">
<![CDATA[
command: "CorrectTheBirthDate",
values : {
uuid: "58128010-8843-11e2-9e96-0800200c9a66",
newBirthdate: "1987-10-04"
}
]]>
</script>
</div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">Now imagine 3 years later we figure it is a good idea to record the "reason" why the Birth date had to change; to support this we add a new field for "reason". Simple enough, our Command object changes to CorrectTheBirthDate(uuid,newBirthDate,reason) and the newer JSON is serialised as you would expect.
</span></div>
<div>
<script class="brush:javascript" type="syntaxhighlighter">
<![CDATA[
command: "CorrectTheBirthDate",
values : {
uuid: "58128010-8843-11e2-9e96-0800200c9a66",
newBirthdate: "1987-10-04",
reason: "mistake on the birth certificate. client informed us of the correction"
}
]]>
</script>
</div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">Ok. So what happens when the system "replays" all the Commands. Well with protobuf, it just sees that the field is not supplied, so it doesn't deserialise "nothing" into the
object. Instead (using <a href="https://github.com/SandroGrzicic/ScalaBuff">ScalaBuff</a>) the case class is marked with "Option[] fields, so that the value in that instance becomes "None".</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">If it were the other case where a field is dropped; then the Case Class will just never have the value "loaded in".</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">I am sure there are horrid edge cases lurking, but it feels right to let the serialisation layer deal with the problem .. how it knows .. and so long as the developers know the rules (when things are dropped, or what is added when) then coding can continue. The trick or benefit is that the "protobuf" default should be enough.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">Serialising in this way means that regardless of the changes in my API's, Commands and Events. I will always have the History. Therefore I will always have the ability to scour the depths for stats and reports. Exactly my reasons for Event Sourcing it. (amongst so many others).</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">So, stay tuned. I will post a real sample application once I tidy up the mess and make a real UI to play with.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
</div>
Ramon Bucklandhttp://www.blogger.com/profile/16744583593281442424noreply@blogger.com0tag:blogger.com,1999:blog-3473871805404328405.post-50360369653903822312012-04-09T11:19:00.000+01:002013-03-08T23:33:15.360+00:00GWT SDK Not Installed - Eclipse<div dir="ltr" style="text-align: left;" trbidi="on">
<div dir="ltr" style="text-align: left;" trbidi="on">
From time to time, I have seen the simple but annoying message.
<br />
<blockquote>
GWT SDK Not Installed</blockquote>
the error message is annoying because I "have" it installed, which means, not so much "not installed", but more "not detected" by the Google Eclipse Plugin.
<br />
<br />
As I am busy working on many things I forgot what the resolution was and again, spent 1/2 an hour resolving the issue.<br />
<br />
You may find this error when using Maven and GWT. Essentially, the GWT SDK Library shows up as a Classpath Container. "GWT SDK [2.4.0]" or similar.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgC_Y9Ts5nCOXmJnCvWzT6SOG_Psl9zyYjodDXhpI8UOmxYCUvyg8myidy3LSRrO2qlQ5XpqbT5w-gClvEqhGd-C4ZgasGdE9mGYAIfg6hEVnGlN-cYY3PdMcgMqyc8ONX4y2I30VRO_g/s1600/Screen+Shot+2012-04-09+at+11.05.18.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="107" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgC_Y9Ts5nCOXmJnCvWzT6SOG_Psl9zyYjodDXhpI8UOmxYCUvyg8myidy3LSRrO2qlQ5XpqbT5w-gClvEqhGd-C4ZgasGdE9mGYAIfg6hEVnGlN-cYY3PdMcgMqyc8ONX4y2I30VRO_g/s320/Screen+Shot+2012-04-09+at+11.05.18.png" width="320" /></a></div>
<br />
The cause is because Eclipse locates the "gwt-*" libraries in your "Referenced Libraries" set, before it sees the actual Eclipse "GWT SDK [...]" reference.<br />
<br />
The error message occurs when theses other GWT libraries are in your classpath, in this case the "referenced" libraries section; especially before the GWT one.<br />
<br />
There are two fixes for this<br />
<br />
<b>1. Move the GWT SDK to the top</b><br />
<br />
Move the GWT SDK "Classpath" Entry Container above your "Libraries" where GWT is found. The Google Eclipse Plugin finds the SDK based GWT Library first and all is well. To move it, manually edit the .classpath file and move the classpathentry line to the top.<br />
<br />
<script class="brush:xml" type="syntaxhighlighter">
<![CDATA[
<classpathentry kind="con" path="com.google.gwt.eclipse.core.GWT_CONTAINER"/>
]]>
</script><b>
2. Exclude the other GWT Libraries from your classpath.</b><br />
<br />
If you are using maven, you'll know that the Maven Eclipse Plugin can't specify the order of classpath references in the .classpath file.<br />
<br />
The trick then is to "exclude" the gwt-dev, gwt-user jars from the generated .classpath file, which you can do in the pom.xml. This way, ONLY the GWT libaries in the Eclipse GWT Container are the ones seen by the Google Eclipse Plugin and it all works well. Note also I have the GWT ClasspathContainer for GWT in there, so when I DO regenerate the .classpath and .project files, I don't have to regenerate things.<br />
<br />
The example pom.xml is as follows:<br />
<br />
<br />
<script class="brush:xml" type="syntaxhighlighter">
<![CDATA[
<plugin>
<groupid>org.apache.maven.plugins</groupId>
<artifactid>maven-eclipse-plugin</artifactId>
<version>2.8</version>
<!- - see http://maven.apache.org/plugins/maven-eclipse-plugin/eclipse-mojo.html
for more information - ->
<configuration>
<downloadsources>true</downloadsources>
<downloadjavadocs>true</downloadjavadocs>
<projectnatures>
<projectnature>org.scala-ide.sdt.core.scalanature</projectnature>
<projectnature>org.eclipse.jdt.core.javanature</projectnature>
<projectnature>com.google.gwt.eclipse.core.gwtNature</projectnature>
</projectnatures>
<buildcommands>
<!- - <buildcommand>org.eclipse.jdt.core.javabuilder</buildcommand> - ->
<buildcommand>org.scala-ide.sdt.core.scalabuilder</buildcommand>
<buildcommand>com.google.gdt.eclipse.core.webAppProjectValidator</buildcommand>
<buildcommand>com.google.gwt.eclipse.core.gwtProjectValidator</buildcommand>
</div>
<!- - **** See my GWT Eclipse Referenced Container HERE **** - - >
<classpathcontainers>
<classpathcontainer>com.google.gwt.eclipse.core.GWT_CONTAINER</classpathcontainer>
<classpathcontainer>org.scala-ide.sdt.launching.SCALA_CONTAINER</classpathcontainer>
<classpathcontainer>org.eclipse.jdt.launching.JRE_CONTAINER</classpathcontainer>
</classpathcontainers>
<!- - **** HERE ARE the Exclusions **** - - >
<excludes>
<exclude>com.google.gwt:gwt-user</exclude>
<exclude>com.google.gwt:gwt-dev</exclude>
<exclude>org.scala-lang:scala-library</exclude>
<exclude>org.scala-lang:scala-compiler</exclude>
</excludes>
<sourceincludes>
<sourceinclude>**/*.scala</sourceinclude>
<sourceinclude>**/*.java</sourceinclude>
</sourceincludes>
]]>
</script>
I hope that helps you.
</div>
</div>Ramon Bucklandhttp://www.blogger.com/profile/16744583593281442424noreply@blogger.com0tag:blogger.com,1999:blog-3473871805404328405.post-45504210819642386592012-04-09T00:31:00.000+01:002012-04-09T00:33:52.981+01:00Crashplan Active Bandwidth Control #2In a previous post on crashplan - <a href="http://www.blogger.com/rbtech.blogspot.co.uk/2011/12/crashplan-active-bandwidth-control.html">Crashplan Active Bandwidth Control</a> - I setup a sophisticated "control" mechanism.
I have now simplified my approach down to understanding my family patterns and I think it works as a guaranteed performance "setup".<br />
<br />
Essentially, between the hours of 23:45 and 06:30, my backup is using ALL the available bandwidth. But I go a little more than that.<br />
<br />
Crashplan only has an "on, off" approach regarding times. I wanted a "Fast", "Slow" approach around times. I am using UNIX cron and a script I wrote in the previous version which logs into Crashplan and adjusts the bandwidth.<br />
<br />
My /etc/crontab looks like this .. finely tuned over the past 2 months.<br />
<br />
<br />
<script class="brush:perl" type="syntaxhighlighter">
<![CDATA[
CPLAN_LOW_RATE=30
CPLAN_MID_RATE=50
CPLAN_HI_RATE=4096
# m h dom mon dow user command
# every night, go fast
45 23 * * * root /opt/crashplan-rate-controller-1.0/adjust-wan-rate.pl $CPLAN_HI_RATE
# 2nd shift catch all in case I changed it post 23:45 and then went to bed forgetting to "up it again"
# 15 3 * * * root /opt/crashplan-rate-controller-1.0/adjust-wan-rate.pl $CPLAN_HI_RATE
# every morn, go slow
45 6 * * * root /opt/crashplan-rate-controller-1.0/adjust-wan-rate.pl $CPLAN_LOW_RATE
# every school pickup time go a bit faster
45 15 * * * root /opt/crashplan-rate-controller-1.0/adjust-wan-rate.pl $CPLAN_MID_RATE
30 16 * * * root /opt/crashplan-rate-controller-1.0/adjust-wan-rate.pl $CPLAN_LOW_RATE
# around dinner time go a bit faster
30 17 * * * root /opt/crashplan-rate-controller-1.0/adjust-wan-rate.pl $CPLAN_MID_RATE
30 18 * * * root /opt/crashplan-rate-controller-1.0/adjust-wan-rate.pl $CPLAN_LOW_RATE
# every sunday go fast - we are at church these times.
00 9 * * 7 root /opt/crashplan-rate-controller-1.0/adjust-wan-rate.pl $CPLAN_HI_RATE
00 13 * * 7 root /opt/crashplan-rate-controller-1.0/adjust-wan-rate.pl $CPLAN_LOW_RATE
]]>
</script>
So this seems to work real well, I have been able to increase my Backups to now pushing out 90G per week. Pretty good going.
<img src="http://a8.sphotos.ak.fbcdn.net/hphotos-ak-ash4/389657_10150643818571193_669286192_9699219_386317827_n.jpg"/>Ramon Bucklandhttp://www.blogger.com/profile/16744583593281442424noreply@blogger.com0tag:blogger.com,1999:blog-3473871805404328405.post-52467613739427875012012-01-10T13:02:00.000+00:002012-01-10T13:02:56.714+00:00Sakis3g Control ScriptI have found that my USB Modems, two of them, did not work out of the box with Ubuntu 11.10.<br />
Not to worry, there is a great script that contains all the pieces for Ubuntu.<br />
<br />
<br />
<pre>
sakis3g
</pre>
You can read all about it here - <a href="http://www.sakis3g.org">www.sakis3g.org</a> and what it does.
In essence, you use it in replace of your Network Manager or ppp/wvdial. Now I would prefer to use the built in method that the distribution has, however it plain does not work! Instead sakis3g and I have work to do.
<br/>
<br/>
I wanted a better way to control it, I created a command line control script for it for my modem. (Didn't like the GUI)
<pre>
#!/bin/sh
BASE_CMD="/usr/local/bin/sakis3g"
OPTIONS="--term"
MODEMETC="OTHER='USBMODEM' USBMODEM='19d2:1003' USBINTERFACE='1' APN='3internet'"
COMMAND="$1"
shift
case $COMMAND in
status)
$BASE_CMD $OPTIONS status
;;
connect)
sudo $BASE_CMD $OPTIONS $MODEMETC connect
;;
disconnect)
sudo $BASE_CMD $OPTIONS disconnect
;;
*)
echo This is a wrapper to Sakis3g
echo $0 connect
echo $0 disconnect
echo $0 status
exit 1
esac
</pre>Ramon Bucklandhttp://www.blogger.com/profile/16744583593281442424noreply@blogger.com0tag:blogger.com,1999:blog-3473871805404328405.post-21467347808797597172011-12-29T23:50:00.001+00:002011-12-29T23:50:20.003+00:00Crashplan Active Bandwidth Control<br />
I love what random things I get up to over the holidays. Crashplan is a Cloud backup service that I use for all my backups. I have devised a way so that Crashplan starts backing up at FULL speed, when everyone is not at home, or are not doing anything on the internet. And then drops back down to a trickle backup when things become busy.<br />
<br />
Essentially I have one computer, my server, and it has the Crashplan service running on it. The server is an Ubuntu based Linux distrubution. Nothing fancy there. What is interesting is how I have managed to control the Crashplan bandwidth it utilises. In the previous house, I had a good 18-20Mbps internet connection, so my backups were really nice. Now, I only have a 3Mbps connection so the "usage" is very important. If crashplan hogs the connection you reall notice it.<br />
<br />
Crashplan has two ways of controlling it's backup usage. First is on a backup set basis, you set the time that it runs. Helpful, for the past 2 months, my backups have been scheduled for 12am to 7am. Works well, but at that rate, I will be completely backed up in about 12 months time... hrmm.<br />
<br />
The other way Crashplan operates is by bandwidth limiting. This is how I roll it now.<br />
<br />
<h2>Attempt 1 - See when they Connect</h2>
I needed to determine the best method to "ascertain" if someone was using the internet. I pondered looking at when the phones (we have 4 adults, 3 iPhones and 1 Android) are "on" or can be seen as that is a good indicator that someone is home. But of course it goes a bit more than that.<br />
<br />
I did however, to test the theory, write a quick Perl script which was the start of monitoring when an iPhone connects to the WiFi. The Android apparently uses Zeroconf also, but I didn't test that.<br />
Basically, each time the phone connects for the initial session, it sends a multicast out. If you are listening, you will see it. Simply, this perl script listens on the right port.
<br />
<script class="brush:perl" type="syntaxhighlighter">
<![CDATA[
#!/usr/bin/perl
use strict;
use IO::Socket::Multicast;
use Net::ARP;
use constant GROUP => '224.0.0.251';
use constant PORT => '5353';
my $device = q{eth1};
my $sock = IO::Socket::Multicast->new(Proto=>'udp',LocalPort=>PORT) || die "could not creat listen: $!\n";
$sock->mcast_add(GROUP) || die "Couldn't set group: $!\n";
while (1) {
my $data;
next unless $sock->recv($data,1024);
my $remoteIp = $sock->peerhost();
system ('ping -c 1 ' . $remoteIp);
print "\nReceived: [[" . $sock->peerhost() . "]] " . Net::ARP::arp_lookup("",$sock->peerhost());
}
]]>
</script>
I quickly discounted this method when I realised that some devices just wont tell me they connected. And because I want to keep my network largely zero configuation, I want to just use DHCP and not have to bother with static entries, or Static leases for my clients. Which is when it hit me:
<h2>Attempt 2 - Scan the Known DHCP Range</h2>
<br/>
The premise behind my final solution is simple. I have two types of devices in use in our household.
<ul>
<li>Infrastructure - servers, printer, wirless and network switches / routers</li>
<li>Clients - Laptops, phones (soon tablets)</li>
</ul>
<p>
All my infrastructure devices use static IP addresses, such as 1 to 39. All the clients, use a DHCP IP address.
We have no desktops in the house, only laptops and when they are closed, they are not in use.
</p>
<p>
If they are not in the house, they are not in use. Same goes with the phones, they are either sleeping or not in the house.
</p>
<p>
<h3>The Solution</h3>
So, if all the laptops are closed and all the phones are out, I know that none of the "clients" will ping, and therefore I can ramp up the Crashplan bandwidth.
</p>
<h2>Solution to Attempt 2 - Monitor for Active Clients</h2>
<p>
I had a few things to solve. But for each I knew that Perl would be all that I would need.
<h3>Part 1 - Automatically Raise and Lower Crashplan Bandwidth</h3>
<p>I needed to determine how Crashplan "stored" it's bandwidth rate. Turns out it is in a config file, but the web site can also change the config. So, the client (my Crashplan service running on my server) and the Crashplan Cloud keep in contact all the time· If I change the bandwidth on the client, it updates on the website, and vice versa.
</p>
<p>
I fired off a few support queries to Crashplan and they basically said that the config file I found (/usr/local/crashplan/conf/my.service.xml) is not editable by hand. And that the only supported way is via the Client (desktop Java App) or the Website.
</p>
<p>
Call WWW::Mechanize - I used this module to login to the Crashplan Website and change the bandwidth (kbps) based on my argument. Simple and it looks like this.
</p>
<script class="brush:perl" type="syntaxhighlighter">
<![CDATA[
#!/usr/bin/perl
use strict;
use WWW::Mechanize;
my $kbps = $ARGV[0];
my $computer_id = 99999999; # determine what your computer ID is (look at the URLs on the web
my $change_url = 'https://www.crashplan.com/account/mycomputers/manage_config.vtl?reset=true&computer=';
my $login_url = 'https://www.crashplan.com/account/login.vtl';
my $logout_url = 'https://www.crashplan.com/logout.vtl';
my $username = 'your_magic_username';
my $password = 'your_magic_password';
# login to Crashplan
my $mech = WWW::Mechanize->new();
$mech->get($login_url);
$mech->submit_form(
form_name => 'loginForm',
fields => { 'loginForm.username' => $username ,
'loginForm.password' => $password },
);
die unless ($mech->success);
$mech->get($change_url . $computer_id);
$mech->submit_form(
form_name => 'manageConfigForm',
fields => { 'manageConfigForm.bandwidthIdleKbps' => $kbps }
);
die unless ($mech->success);
$mech->get($logout_url);
]]>
</script>
<p>
Not rabbit proof, but it works for my needs
</p>
<h3>Part 2 - Monitor for Activity</h3>
<p>
This was the hardest part, but as always, was solvable. The router (Sky Broadband Provider) I currently use "knows" what clients are using it. VERY handy. So like the Crashplan website solution above, I again use WWW::Mechanize to determine what IP addresses are currently using the router.
</p>
<p>
I do a filter out of all the ip addresses that do NOT fall inside my DHCP range.
</p>
<script class="brush:perl" type="syntaxhighlighter">
<![CDATA[
my $subnet = '192.168.1';
my $dhcp_lower = '40';
my $dhcp_upper = '100';
my $sky_attached_devices_url = 'http://192.168.1.1/sky_attached_devices.html';
my $sky_logout_url = 'http://192.168.1.1/sky_logout.html';
my $sky_username = 'magic_admin_username';
my $sky_password = 'magic_router_password';
sub get_sky_attached_devices {
my $mech = WWW::Mechanize->new();
$mech->credentials( $sky_username => $sky_password );
$mech->get($sky_attached_devices_url);
my $root = HTML::TreeBuilder->new_from_content($mech->content);
my @attached_hosts = $root->findvalues( '//table[tr/th/text() = "MAC Address"]/tr/td[position() = 2]/text()' );
my @dhcp_hosts;
$mech->get($sky_logout_url);
foreach my $host (@attached_hosts) {
$host =~ /$subnet\.(\d+)/;
push(@dhcp_hosts,$host) if ( $1 >= $dhcp_lower && $1 <= $dhcp_upper );
}
return @dhcp_hosts;
}
]]>
</script>
<p>
I won't go into the details of using XPath on the HTML from the Router, but, you can see that it is quite succinct. Essentially the IP Addresses appear in a 2nd column in a table. I use an XPath expression to select the right values.
</p>
<p>
This approach is brittle in as much as, when / if I change my router then I need to code or come up with another way to determine which hosts, within the DHCP range, are currently "in use". I did first to this by just pinging all 60 possible IP addresses; works, brute force, but of course worked :-)
</p>
<p>
So .. now putting it all together. The script, a perl script, runs from Cron every minute. and performs the following tasks.
<ul>
<li>Get the Current Rate from the Crashplan Config file</li>
<li>Get all the current "hosts" that the Router knows</li>
<li>Ping each host to MAKE sure they are active</li>
<li>If we have an active host (one or more) - we need to go slow
<ul>
<li>If the current rate that Crashplan is running is more than our slow rate, tell Crashplan to slow down</li>
<li>If the current rate is slow, do nothing</li>
</ul>
</li>
<li>If we have no active hosts - we can go fast
<ul>
<li>If the current rate that Crashplan is running is less than our fast rate, tell Crashplan to go fast</li>
<li>If the current rate is fast, do nothing</li>
</ul>
</li>
</ul>
<p>
Simple! And the full Script looks like below. Enjoy!
</p>
<p>
<script class="brush:perl" type="syntaxhighlighter">
<![CDATA[
#!/usr/bin/perl
use strict;
use Net::Ping;
use WWW::Mechanize;
use HTML::TreeBuilder::XPath;
use Sys::Syslog qw(:standard :macros);
use XML::XPath;
my $base_path = '/opt/crashplan-rate-controller';
my $adjust_rate_command = $base_path . "/" . 'adjust-wan-rate.pl';
my $slow_rate = 50;
my $fast_rate = 4096;
my $subnet = '192.168.1';
my $dhcp_lower = '40';
my $dhcp_upper = '100';
my $sky_attached_devices_url = 'http://192.168.1.1/sky_attached_devices.html';
my $sky_logout_url = 'http://192.168.1.1/sky_logout.html';
my $sky_username = 'magic_admin_username';
my $sky_password = 'a_magic_password';
my $crashplan_config_file = '/usr/local/crashplan/conf/my.service.xml';
#
# load the current rate that Crashplan is running as / on
my $current_rate = &get_current_rate;
#
# load the ip addresses we want to see
my @hosts_to_check = &get_sky_attached_devices;
#
# check each one
my $ping = Net::Ping->new("icmp",3);
my $have_active_hosts = 0;
my $active_hosts = '';
for my $host (@hosts_to_check) {
if ($ping->ping($host)) {
$active_hosts .= $host . " ";
$have_active_hosts = 1;
}
}
chop($active_hosts);
if ($have_active_hosts) {
# print "Current = $current_rate , slow = $slow_rate\n";
if ($current_rate > $slow_rate) {
system($adjust_rate_command,$slow_rate);
syslog(LOG_INFO, "[crashplan] Lowered - " . $slow_rate . "kbps - hosts active (". $active_hosts . ")");
} else {
# syslog(LOG_INFO, "[crashplan] Keeping - " . $current_rate . "kbps - hosts active (". $active_hosts . ")");
}
} else {
if ($current_rate < $fast_rate) {
system($adjust_rate_command,$fast_rate);
syslog(LOG_INFO, "[crashplan] Raised to - " . $fast_rate . "kbps - No active Hosts");
}
}
exit;
sub get_sky_attached_devices {
my $mech = WWW::Mechanize->new();
$mech->credentials( $sky_username => $sky_password );
$mech->get($sky_attached_devices_url);
my $root = HTML::TreeBuilder->new_from_content($mech->content);
my @attached_hosts = $root->findvalues( '//table[tr/th/text() = "MAC Address"]/tr/td[position() = 2]/text()' );
my @dhcp_hosts;
$mech->get($sky_logout_url);
foreach my $host (@attached_hosts) {
$host =~ /$subnet\.(\d+)/;
push(@dhcp_hosts,$host) if ( $1 >= $dhcp_lower && $1 <= $dhcp_upper );
}
return @dhcp_hosts;
}
#
#
sub get_current_rate {
my $xp = XML::XPath->new(filename => $crashplan_config_file);
my $rate = $xp->getNodeText('/config/serviceBackupConfig/highBandwidthRate');
# the value is stored in KBytes per second on the config file.
# the web iface at crashplan.com accepts a kbps however.
# the 8 is to bring the KBytes in the file to Kbits
return $rate->string_value * 8;
}
]]>
</script>
</p>
<p>
Of course, I did all this and know and realise that QoS could be used, but my Sky Broadband Router is just not that smart today and I haven't got the devices spare or the time to reconfigure to installl a dd-wrt based router or some such other. So perl scripts it is for now.
</p>Ramon Bucklandhttp://www.blogger.com/profile/16744583593281442424noreply@blogger.com0tag:blogger.com,1999:blog-3473871805404328405.post-55827067219347604612011-10-27T12:33:00.002+01:002011-10-27T12:33:53.673+01:00Neo4J - 2nd Look - Setting a Primary Key on Nodes<h2>Primary Key</h2>
<p>
In my last post I considered the lack of Primary Key like Id's as something I need to solve. My use case is
<p>
<blockquote>
The application I will be building out will have, after all is said and done a really simple Web Interface with REST type URLs. So .. for example, I will be able to do.
<br/>
http://myservice.co.uk/superwebapp/mySpecialThing/detailedView/55
<br/>
The 55 there will result in a query to Neo4J to locate the "MySpecialThing" object with ID of type 55 and display it.
</blockquote>
<p>
I also considered using a UUID across objects which is also good, but not really what I was after. I want a class of objects to all share an identifier. It has a lot of use. To solve the problem I arrived at the following solution.
</p>
<h2>Solution Outline</h2>
<ul>
<li>All Domain Objects extend fro super type of AbstractLongDomain which has getId()/setId() (Long)</li>
<li>An Aspect wrapped around the getId() looks for a null value and if no Id is found. It creates one</li>
<li>In the aspect creation of an ID involves talking to a singleton IdManager for a "nextId()"</li>
<li>nextId() on the manager looks to it's cache HashMap<String,IdObject> to see if it has an IdObject that knows what the next Id is</li>
<li>IdObject self persists to the repository after each call (** this could be slow.. see how we go)</li>
</ul>
<h2>To the Code</h2>
All my Objects extend the AbstractLongDomain
<script type="syntaxhighlighter" class="brush:java"><![CDATA[
public abstract class AbstractNamedDomainObject extends AbstractLongDomain {
]]></script>
The ID Object holds a "per" class Long Id, so each time an ID is needed, one of these objects gives one out.
<script type="syntaxhighlighter" class="brush:java"><![CDATA[
package org.soqqo.luap.model;
import org.springframework.data.neo4j.annotation.Indexed;
import org.springframework.data.neo4j.annotation.NodeEntity;
/**
* Simple Pojo, one per "class" set we want to track individual Id's on idKey is
* typically the Canonical Class Name.
*
* @author rbuckland
*
*/
@NodeEntity
public class IdObject {
@Indexed
private Long nextId;
@Indexed
private String idKey;
public IdObject(String idKey) {
this.idKey = idKey;
}
public Long getNextId() {
try {
if (nextId == null) {
nextId = 1L;
}
return nextId++;
} finally {
this.persist();
}
}
}
]]></script>
Next we have the IdManager that is managed as a Spring Singleton Bean. Its job is to return an id based on the "class" that needs an Id via getNextId(Class klass).
<script type="syntaxhighlighter" class="brush:java"><![CDATA[
package org.soqqo.luap.model;
import java.util.HashMap;
import java.util.Iterator;
import java.util.Map;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.soqqo.luap.server.repository.neo4j.IdObjectRepository;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Component;
/**
* IdManager is a Singleton managed bean which looks after locating the
* idObjects in the repository.
*
* It will load them up if it does not have as reference for one. And will
* retain a reference in the map to the ones it has already loaded.
*
* This ensures we are not going crazy to finding them in the repository every
* time.
*
* @author rbuckland
*
*/
@Component
public class IdManager {
Logger log = LoggerFactory.getLogger(IdManager.class);
@Autowired
private IdObjectRepository idObjectRepository;
// our internal cache
private Map<String, IdObject> idObjects;
/**
* returns a new Id
*
* @param klass
* @return
*/
public Long getNextId(Class<?> klass) {
if (idObjects == null) {
idObjects = new HashMap<String, IdObject>();
log.debug("creating new cache map");
}
// have we a cached version ?
if (!idObjects.containsKey(klass.getCanonicalName())) {
// find it in the repo
Iterator<IdObject> idObjectIter = idObjectRepository.findAllByPropertyValue("idKey", klass.getCanonicalName()).iterator();
if (idObjectIter.hasNext()) {
// there was one in the repo, so we will store it in our HashMap
idObjects.put(klass.getCanonicalName(), idObjectIter.next());
log.debug("Loaded a new idObject[" + klass.getCanonicalName() + "] from the repo to cache");
} else {
// none, so add one to the repo and store it in our hashmap for
// later use
log.debug("Created a new idObject[" + klass.getCanonicalName() + "] to the repo & to cache");
idObjects.put(klass.getCanonicalName(), (IdObject) new IdObject(klass.getCanonicalName()).persist());
}
}
// return a new id
return idObjects.get(klass.getCanonicalName()).getNextId();
}
/**
* In Unit tests, within each method the repository is flushed as it's
* transactional per method.
*
* So we use this method to reset for each Unit test. Otherwise our cached
* objects become stale resulting in org.neo4j.graphdb.NotFoundException:
* Node[3] not found. (or which ever node it is)
*/
public void flush() {
idObjects = new HashMap<String, IdObject>();
}
}
]]></script>
The idRepository you see there is a simple Spring Data Neo4J Repository which has aspect-magic dust sprinkled on it to make the actual implementation.
<script type="syntaxhighlighter" class="brush:java"><![CDATA[
package org.soqqo.luap.server.repository.neo4j;
import org.soqqo.luap.model.IdObject;
import org.springframework.data.neo4j.repository.GraphRepository;
import org.springframework.data.neo4j.repository.NamedIndexRepository;
public interface IdObjectRepository extends GraphRepository<IdObject>, NamedIndexRepository<IdObject> {
}
]]></script>
<p>
Because I am not sure if neo4j is the "best" place to store the Id's (though it is the logical) I created a simple idGenerator interface which is simply what the aspect will call and talk to. One implementation (the only) is the Neo4JBackedIdGenerator which uses the id objects and idmanager above.
</p>
<p>
So first the interface for the IdGenerator
</p>
<script type="syntaxhighlighter" class="brush:java"><![CDATA[
package org.soqqo.luap.model;
public interface IdGenerator {
public Long nextId(Class<? extends AbstractLongDomain> class1);
}
]]></script>
And then the actual Neo4JBackedIdGenerator which is created and managed as a spring bean.
<script type="syntaxhighlighter" class="brush:java"><![CDATA[
package org.soqqo.luap.model;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Component;
@Component
public class Neo4JBackedIdGenerator implements IdGenerator {
Logger log = LoggerFactory.getLogger(Neo4JBackedIdGenerator.class);
@Autowired
IdManager idManager;
@Override
// @Transactional(propagation = Propagation.REQUIRED)
public synchronized Long nextId(Class<? extends AbstractLongDomain> klass) {
Long id = idManager.getNextId(klass);
log.debug("Allocated a new Id("+ id +") for [" + klass.getCanonicalName() + "]");
return id;
}
}
]]></script>
<p>
I will have to play with the transactional semantics on this one. I recall a horrid situation which a similar design but via stored procs many moons ago where we had the sproc that generate Id's wrapped in transactions. They needed to be in their own transaction to ensure that mass object thread creation would not get stuck on a lock. (just an area I know can be sticky and bite.. so I put the @Transactional in there and commented out to remind me.
</p>
<p>
So last, we have the AspectJ which wraps our getId(). All the domain objects extend org.soqqo.luap.model.AbstractLongDomain which means we will get the Id creation and generation for free each time getId is called. (technically a catch here is that setId doesn't get checked if called manually on setting an Id. It could I guess look into the repo to see if the Id is already used.
</p>
<script type="syntaxhighlighter" class="brush:java"><![CDATA[
package org.soqqo.luap.model;
import org.aspectj.lang.ProceedingJoinPoint;
import org.aspectj.lang.annotation.Around;
import org.aspectj.lang.annotation.Aspect;
import org.aspectj.lang.annotation.Pointcut;
import org.springframework.beans.factory.annotation.Autowired;
/**
* This class is also managed as a bean having the idGenerator of choice
* injected in.
*
* It needs the <bean id="longIDGeneratorAspect"
* class="org.soqqo.luap.model.LongIDGeneratorAspect" factory-method="aspectOf"
* /< Configuration in a Spring aspect.
*
* @author rbuckland
*
*/
@Aspect
public class LongIDGeneratorAspect {
@Autowired
IdGenerator idGenerator;
@Pointcut("execution(Long org.soqqo.luap.model.AbstractLongDomain.getId())")
public void doIDGeneration() {
}
@Around("doIDGeneration()")
public Object generateId(ProceedingJoinPoint pjp) throws Throwable {
Object val = pjp.proceed();
if (val == null) {
// calling the generator to get a new value
AbstractLongDomain ald = (AbstractLongDomain) pjp.getThis();
ald.setId(idGenerator.nextId(ald.getClass()));
val = ald.getId();
}
return val;
}
}
]]></script>
And finally the Unit Test code to show that it all works
<script type="syntaxhighlighter" class="brush:java"><![CDATA[
package org.soqqo.luap.model.tests;
import org.junit.Assert;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.soqqo.luap.model.IdManager;
import org.soqqo.luap.model.core.Member;
import org.soqqo.luap.model.party.Person;
import org.soqqo.luap.server.repository.neo4j.MemberRepository;
import org.soqqo.luap.server.repository.neo4j.PersonRepository;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.test.annotation.DirtiesContext;
import org.springframework.test.context.ContextConfiguration;
import org.springframework.test.context.junit4.SpringJUnit4ClassRunner;
import org.springframework.transaction.annotation.Transactional;
@RunWith(SpringJUnit4ClassRunner.class)
@ContextConfiguration({ "/model-test-context.xml" })
@Transactional
public class Neo4jDomainTest {
@Autowired
MemberRepository memberRepository;
@Autowired
PersonRepository personRepository;
@Autowired
IdManager idManager;
@Test
//@DirtiesContext (flush the idManager instead, it is quicker!)
public void persistedMemberIsRetrievable() {
idManager.flush();
Member frank = new Member("Frank", "Low").persist();
Person jim = new Person("Jim", "Fow").persist();
Member retrievedMember = memberRepository.findOne(1L);
Assert.assertEquals("retrieved member matches persisted one", frank, retrievedMember);
Assert.assertEquals("retrieved member first name must match", "Frank", retrievedMember.getFirstName());
Person retrievedPerson = personRepository.findByPropertyValue("firstName", "Jim");
Assert.assertEquals("retrieved member matches persisted one", jim, retrievedPerson);
Assert.assertEquals("retrieved member first name matches", "Fow", retrievedPerson.getLastName());
Assert.assertEquals("id's should be 1 each", new Long(1L), frank.getId());
Assert.assertEquals("id's should be 1 each", new Long(1L), jim.getId());
}
@Test
//@DirtiesContext
public void persistedMembersGetNewIds() {
idManager.flush();
Member frank = new Member("Frank", "Low").persist();
Member jim = new Member("Jim", "Fow").persist();
Assert.assertNotSame("id's should not be equal", jim.getId(), frank.getId());
}
}
]]></script>
<p>
Note the use of @DirtiesContext, because the Neo4J is transactional, after each test the contents are dumped, which means that the idManager which has the HashMap cache becomes stale and it is singleton and has a lifecycle of the test class, not just the method. So the fix is either..manually flush() the cache or use @DirtiesContext which tells spring to re-build the context file. Both work but manually flushing my HashMap (new() ) is 2 seconds faster (0.037s for the test) than spring is at rebuilding.
</p>
<p>
The 2nd last piece to show is my test context file - model-test-context.xml
</p>
<script type="syntaxhighlighter" class="brush:xml"><![CDATA[
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:context="http://www.springframework.org/schema/context"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:neo4j="http://www.springframework.org/schema/data/neo4j"
xmlns:tx="http://www.springframework.org/schema/tx"
xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-3.0.xsd
http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context-3.0.xsd
http://www.springframework.org/schema/data/neo4j http://www.springframework.org/schema/data/neo4j/spring-neo4j-2.0.xsd http://www.springframework.org/schema/tx http://www.springframework.org/schema/tx/spring-tx-2.5.xsd">
<context:annotation-config/>
<context:spring-configured/>
<bean id="longIDGeneratorAspect" class="org.soqqo.luap.model.LongIDGeneratorAspect" factory-method="aspectOf" />
<context:component-scan base-package="org.soqqo.luap"/>
<neo4j:config graphDatabaseService="graphDatabaseService"/>
<neo4j:repositories base-package="org.soqqo.luap.server.repository"/>
<bean id="graphDatabaseService" class="org.neo4j.test.ImpermanentGraphDatabase"/>
<tx:annotation-driven mode="aspectj"/>
</beans>
]]></script>
The very last piece is what my Maven POM looks like because a lot of people like to see that...
Hopefully these are the correct relevant bits.
<script type="syntaxhighlighter" class="brush:xml"><![CDATA[
<dependencies>
<!-- @DateTimeFormat -->
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-context</artifactId>
</dependency>
<!-- @Configurable -->
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-beans</artifactId>
</dependency>
<!-- @Transactional -->
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-tx</artifactId>
</dependency>
<!-- Joda Time Date and time formats -->
<dependency>
<groupId>joda-time</groupId>
<artifactId>joda-time</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.data</groupId>
<artifactId>spring-data-neo4j-aspects</artifactId>
</dependency>
<!-- Neo4j Libraries -->
<dependency>
<groupId>org.neo4j</groupId>
<artifactId>neo4j-kernel</artifactId>
<type>test-jar</type>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<plugins>
<!-- the eclipse plugin interacts with the aspectj-maven-plugin
BUT ONLY if the ajdtVersion config value is set (remove it and it won't) -->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-eclipse-plugin</artifactId>
<version>2.8</version>
<configuration>
<downloadSources>true</downloadSources>
<!-- The ajdtVersion configuration parameter is optional. The valid values are none, 1.4, and 1.5. none indicates that AJDT should not be enabled even though Aspectj is enabled in maven. 1.4 generates the org.eclipse.ajdt.ui.prefs file in the .settings directory. 1.5 (or later) includes the configuration into the .classpath file and is the default value. -->
<ajdtVersion>1.5</ajdtVersion>
<additionalProjectnatures>
<projectnature>org.eclipse.ajdt.ui.ajnature</projectnature>
<projectnature>org.eclipse.jdt.core.javanature</projectnature>
<projectnature>org.springframework.ide.eclipse.core.springnature</projectnature>
</additionalProjectnatures>
</configuration>
</plugin>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>aspectj-maven-plugin</artifactId>
<version>1.2</version>
<dependencies>
<dependency>
<groupId>org.aspectj</groupId>
<artifactId>aspectjrt</artifactId>
<version>${aspectj.version}</version>
</dependency>
<dependency>
<groupId>org.aspectj</groupId>
<artifactId>aspectjtools</artifactId>
<version>${aspectj.version}</version>
</dependency>
</dependencies>
<executions>
<execution>
<goals>
<goal>compile</goal>
<goal>test-compile</goal>
</goals>
</execution>
</executions>
<configuration>
<outxml>true</outxml>
<aspectLibraries>
<aspectLibrary>
<groupId>org.springframework</groupId>
<artifactId>spring-aspects</artifactId>
</aspectLibrary>
<aspectLibrary>
<groupId>org.springframework.data</groupId>
<artifactId>spring-data-neo4j-aspects</artifactId>
</aspectLibrary>
</aspectLibraries>
<source>1.6</source>
<target>1.6</target>
</configuration>
</plugin>
</plugins>
</build>
This is the versions of libraries I am using today
<properties>
<spring.version>3.0.6.RELEASE</spring.version>
<aspectj.version>1.6.12.M1</aspectj.version>
<slf4j.version>1.6.1</slf4j.version>
<gwt.version>2.3.0</gwt.version>
<jetty.version>7.4.1.v20110513</jetty.version>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<neo4j.version>1.5.M02</neo4j.version>
<spring-data-graph.version>2.0.0.M1</spring-data-graph.version>
</properties>
And these are the repositories I have configured.
<repositories>
<repository>
<id>jetty-sonatype</id>
<name>Sonatype Repository</name>
<url>https://oss.sonatype.org/content/groups/jetty</url>
</repository>
<repository>
<id>spring-maven-release</id>
<name>Spring Maven Release Repository</name>
<url>http://maven.springframework.org/release</url>
</repository>
<repository>
<id>spring-maven-milestone</id>
<name>Spring Maven Milestone Repository</name>
<url>http://maven.springframework.org/milestone</url>
</repository>
<repository>
<id>JBoss Repo</id>
<url>https://repository.jboss.org/nexus/content/repositories/releases</url>
<name>JBoss Repo</name>
</repository>
</repositories>
]]></script>
I am happy to provide all this as a ZIP or push it up to github if people want to see more of it.Ramon Bucklandhttp://www.blogger.com/profile/16744583593281442424noreply@blogger.com0tag:blogger.com,1999:blog-3473871805404328405.post-21075246724603216562011-10-26T12:33:00.000+01:002011-10-26T12:33:28.709+01:00Neo4J - My First Look with Spring Data Graph<h2>
Spring Data Neo4J</h2>
<br />
I have been working on some proof of concept code and decided on a clean route to using NoSQL.
Of course there are many choices and because the application I am working on is highly connected around relationships. (not boyfriend girlfriend types) I figured I would look at Neo4j. Given my favourite library of the year is <a href="http://www.springsource.org/spring-data">spring-data</a> I would take a look at the recently release spring-data-neo4j library (formerly called Spring Data Graph).
<br />
<br />
Spring Data provides some funky interface abstraction over your data store, be that an RDBMS, or other type of storage like NoSQL forms.<br />
<br />
<h2>
Specifics to Neo4j and Spring Data Neo4J</h2>
A Quick Overview:
<br />
<br />
<ul>
<li>Neo4J allows you to store POJOs without the need of a Schema.</li>
<li>POJOs are tied together using Relationships</li>
<li>Neo4J Understands a Node and a Relationship (that is it)</li>
<li>Spring Data Graph makes the "Node storage" and "relationship" tie-ing really simple with Annotations</li>
</ul>
Let's look at that last point in detail.
spring-data-neo4j uses a few special annotations, not unlike JPA's annotations.
<br />
<h2>
Declaring a Node</h2>
A Node is declared with the following annotation
<br />
<pre class="brush:java">@NodeEntity
public class MySpecialPojo {
@Indexed
Long id;
@Indexed(indexType=IndexType.FULLTEXT, indexName = "search")
String textField;
//...
}
</pre>
Effectively these annotations make some magic happen. One of the big magic happen things is to do with some special methods() you will find on the objects.
If you include the right "stuff" in your Maven POM for spring-data-neo4j, you will get some good stuff happening.
Effectively some "DAO/repository" style methods get woven into your domain objects.
<br />
<pre class="brush:java"> // for free we get .persist() Which wraps up a call to neo4j and put my Pojo as a Node down to Neo4j.
MySpecialPojo special = new MySpecialPojo(1,"Some Data").persist();
MySpecialPojo foundSpecial = this.pojoRepository.findByPropertyValue("textField", "Some Data");
</pre>
You will also have .remove() and other fun stuff. The best document I have found (as it is very new (2.0.0.M1) is the following PDF. <a href="http://static.springsource.org/spring-data/data-graph/snapshot-site/reference/pdf/spring-data-neo4j-reference.pdf">Spring Data Neo4J - Good Relationships</a><br />
<br />
<h2>Missing Identity or Mimicking(sp?) a Primary Key</h2>
The application I will be building out will have, after all is said and done a really simple Web Interface with REST type URLs.
So .. for example, I will be able to do
<pre>
http://myservice.co.uk/superwebapp/mySpecialThing/detailedView/55
</pre>
The 55 there will result in a query to Neo4J to locate the "MySpecialThing" object with ID of type 55 and display it. The problem I have is a two fold
<ul>
<li>Neo4J just stores objects and does not have "primary keys" other than a "nodeId". </li>
<li>The NodeId is collection wide. So Pojo1 shares the incremental nodeIds with Pojo2.
</ul>
<br/>
spring-data-neo4j adds (via an aspect ITD) a getNodeId() method to my POJOs but I don't want to depend on these for my "primary key" (future proofing my app if I move from Neo4J to something else).<br/>
<br/>
So I want a Class wide "Id" so that when an object is persisted it has an ID for it.<br/> I may be thinking about this wrong, and should just mold and accept a collection (database/store) wide ID system.
<br/>
I am heading down this path
<pre class="brush:java">
public class Foo {
@NodeId(collection=Foo.class)
private Long id;
}
// or
@NodeId(collection=Foo.class)
public class Foo extends AbstractLongIdDomainObject {
// .... get/setId() is found on super class.
}
</pre>
This would then store and manage an ID, like we used to in RDBMS days when they did not have Primary Key AUTO INCREMENT or IDENTITY type stuff.
The way you would implement Primary ID's is to have a table that stores the ID for "each collection" and a lookup stored-procedure or code that would "get" the next ID for you (within a transaction for example).<br/>
<br/>
<br/>
I have not perfected this, but I figured I would show what I was thinking. Plus I will do some more reading, (maybe Neo4J has some config to allow per class type nodeIds. * I always go the hard way first *<br/>
<br/>
<br/>
<pre class="brush:java">
@NodeEntity
public class IdManager {
/**
* This map holds an idObject per "className" for each object we want to "have a Primary KEY id for"
*/
@Indexed
private Map<String,IdObject> idCollection;
}
@NodeEntity
public class IdObject {
public Long getNextId() {
return nextId;
}
public void setNextId(Long nextId) {
this.nextId = nextId;
}
}
</pre>
<br/>
So with this magic code I would then annotate as above with my custom (@NodeId) and some magic happens using aspects and stuff to weave in the "next" ID when an object is "created" and about to be stored through spring-data-neo4j.
<h2>Can't Default Values</h2>
Probably just by how the aspects interact with the get/set methods for your fields, I found that you cannot default a fields value like you can with JPA.
<pre class="brush:java">
@NodeEntity
public class Foo {
private Long someNumber = 1L;
// .. getter setters
}
</pre>
If you create this object, and set the someNumber to 55 for example. And then fooInstances.persist() and then retrieve it from the repository, it will not have the value of 55, but the value of 1. !! Annoying. So that is okay .. but I think the apsects that "populate" the fields are going in too early or something. I have a test case that shows this happening but it was in a complex Aspect( because of my above primary key playing) so it may be a special case. I'll see.
<h2>Summary</h2>
Really nice and I like the simplicity that neo4j and spring-data give. GO away SQL.Ramon Bucklandhttp://www.blogger.com/profile/16744583593281442424noreply@blogger.com0tag:blogger.com,1999:blog-3473871805404328405.post-44150687530978006282011-09-16T13:51:00.004+01:002011-10-26T10:30:43.417+01:00Released a New Open Source LibraryFor a while it has slightly or mildly annoyed me how clunky creating random data can be.<br />
You see all manner of approaches from Importing CSV files and XML to loading up JSON objects and the like.
<br />
What I have wanted was a library that allows random or specific generation into your Java POJO domain model which you then simply persist using your preferred storage layer (Hibernate, JPA, Neo4j) etc. <br/>
DBUnit is fantastic and I have often used <a href="http://www.generatedata.com/">http://www.generatedata.com</a> to generate XML files to upload through DBUnit. <br />
<br />
Well know I sctarched that itch over the last day to create the aptly named<br />
<br />
random-data-generator - <a href="http://code.google.com/p/random-data-generator">http://code.google.com/p/random-data-generator</a><br />
<br />
In short, see the project page as it explains most of how it works. It is a library for use in your code for "seeding" objects with random data.<br />
<br />
Enjoy.Ramon Bucklandhttp://www.blogger.com/profile/16744583593281442424noreply@blogger.com0tag:blogger.com,1999:blog-3473871805404328405.post-90489153551824165502011-09-15T11:29:00.001+01:002011-09-15T11:31:40.703+01:00Love *NixI Still love *nix after all these years. It only took me 1 minute to convert lines like<br /><br /><pre><br />1880,"Mary",0.072381,"girl"<br /></pre><br />to what I wanted.<br /><pre><br />$ grep \"girl\" baby-names.csv | head | cut -d, -f2 | sed 's/"//g;s/$/,F/g'<br />Mary,F<br />Anna,F<br />Emma,F<br />Elizabeth,F<br />Minnie,F<br />Margaret,F<br />Ida,F<br />Alice,F<br />Bertha,F<br />Sarah,F<br /></pre>Ramon Bucklandhttp://www.blogger.com/profile/16744583593281442424noreply@blogger.com0tag:blogger.com,1999:blog-3473871805404328405.post-62408166063833502452010-04-23T21:22:00.001+01:002010-04-23T21:28:55.839+01:00GWT2 and MavenGWT 2 and Maven - Close I have been working with the new GWT 2 code. Especially the new IDE Plugin for eclipse. <br><br> One things I remember doing with older 1.5 --> 1.7 code was run a separate Jetty container and pass -noserver to the GWT Dev mode, so that I could control <br><br> (a) where the compiled JS files were and went to<br><br> (b) running a servlet separate from the hosted app was nice. <br><br> With the new in-IDE mode and the new browser plugin, I found it is almost there. Essentially, my issue I am resolving at the moment is a maven thing.
<ol>
<li> I don't want the GWT Hosted mode to put compiled *.js files (etc) in my src/main/webapp. Primarily because this folder is in Source Control.</li>
<li> My Maven compiler plugin writes out the WAR when i do mvn package. </li>
<li> I want (1.) solved by having it splay out the Compiled files to target/some-ignorable-directory and use the magic Jetty ALIAS do-hickey to "combine" src/main/webapp and target/gwt-hosted-war.tmp </li>
</ol>
achievable, yes/. The problem comes that .. In the lanucher .. I try: -war "${workspace_loc:sample-gwt-webapp/target}/gwt-hosted-war.tmp" <br />
<blockquote>-war "${workspace_loc:sample-gwt-webapp/target}/gwt-hosted-war.tmp" </blockquote>But, that directory does not actually have the WEB-INF files in it. <br><br>
How could I solve it:
<ol>
<li>symlinks src/main/webapp/gwt/* to target/tmpfilder/gwt with Linux and Mac.. not portable to Windows .. except for linkd </li>
<li>before starting the Jetty Container, copy the src/main/webapp to target/tmp-war/ .. not nice for code changes of running jetty in src/main/webapp </li>
<li>Make changes to the dev classes .. (you can provide a new ServletContainer .. but only that .. and it's not in there where the checking is done) </li>
<li>use the -noserver and use Jetty (I want to run just on JVM for debugging) .. </li>
</ol>
bummer.Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-3473871805404328405.post-53635486288269971882010-03-26T22:47:00.001+00:002010-03-26T22:48:13.022+00:00OSGiA good read this. <br />
<br />
<a href="http://njbartlett.name/blog/2010/03/17/osgi-and-how-it-got-that-way/">http://njbartlett.name/blog/2010/03/17/osgi-and-how-it-got-that-way/</a><br />
<br />
<br />
I am looking at using OSGi to provide the plugin capability to this new Scala, JPA, GWT, Spring App.Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-3473871805404328405.post-91913601685219680232010-03-20T23:15:00.005+00:002010-03-21T13:03:43.936+00:00Scala, JPA, GWT, Spring, Eclipse, Maven - Part 1About to embark on something ambitiuos (I suspect), I figured I would detail through some blog posts how I go.<br /><br />I am building an application (secret squirrel until it releases) that I want Scala to be it's core language.<br /><br />The GUI I want in Java (it just works so well), Eclipse is just the default IDE for me, Maven my absolute must build tool and then all the goodness of Spring and Hibernate as usual.<br /><br />My first challenge (beyond learning Scala) is to get Eclipse, Maven and Scala to co-operate. The project will have Java source and Scala Source mixed in.<br /><br />GWT, as per my other standard projects will be in it's own module anyway.<br /><br />So.. here goes. This is just the "I am trying this" post. I'll let you all know how I go.<br /><br /><a href="http://rbtech.blogspot.com/2010/03/scala-jpa-gwt-spring-eclipse-maven-p2.html">Read on for Part 2</a>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-3473871805404328405.post-59461379498785134302009-12-30T11:25:00.002+00:002009-12-30T11:31:59.367+00:00Changing GMail AccountsI recently decided to ditch my old gmail account in favour of my new gmail account, which is my personal domain name I have had for years (Google Apps for Domain).<br /><br />In doing so I needed to port all my mail from the old gmail account to the new gmail account.<br /><br />I setup the appropriate forwards etc, been running that way for about a month now.<br /><br />The best way I have found to "Move" all the mail from old to new is.<br /><br />Setup BOTH the GMail accounts in Mail (OSX on the Mac) as an IMAP account. The IMAP bit of course is important. Once they are setup, and you can see the mail in both of them.<br /><br />Simply: Drag and Drop. Go to Sent Mail, in the old. select all and drag it into the Sent Mail on the new. All the dates and everything is preserved. Similarly, the same for Drafts (yes I had over 50 drafts) .. and All Mail, that caps in at 10,000 emails.<br /><br />Just drag and drop to the same folder in the new account; Leave it running over night of course as it takes a LONG time. And without needed to state the obvious, the faster your connection, the faster this is. It moves the mail from the OLD to the new, VIA your computer.Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-3473871805404328405.post-87196392416572925162009-09-12T23:57:00.002+01:002011-10-27T09:42:04.238+01:00Eclipse Galileo java.lang.ClassNotFoundException with JUnit Tests<strong>This post describes my problem, and the solution</strong><br />
Recently I upgraded to Eclipse Galileo 3.5 on my Ubuntu dev box. All was well and I noticed no degradation or changes (new, good or bad) .. until ..<br />
Until I went to run my first testcase in my new <strong>ete</strong> library. I received the beautiful<br />
<pre class="brush:java">java.lang.ClassNotFoundException</pre>
.. on the JUnit Test case! :-) yuck.This was the stacktrace in it's entirety<br />
<pre class="brush:java">Class not found org.soqqo.ete.source.data.EhCachedDataRowListTest
java.lang.ClassNotFoundException: org.soqqo.ete.source.data.EhCachedDataRowListTest
at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:276)
at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.loadClass(RemoteTestRunner.java:693)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.loadClasses(RemoteTestRunner.java:429)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:452)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
</pre>
<span style="color: black;"><span style="font-size: 100%;"><span style="font-family: 'times new roman';">It really threw me because on the command line, Maven2 (mvn test) test cases still worked fine. So I knew I havd an Eclipse problem. The upgrade was my key, ie: it was within 20 minutes of starting Galileo that this error occured.<br /><br />So I ran my test cases in Eclipse 3.4 and it worked. 3.5 I say.. but then, that was too simple. There must be something up.<br /><br />Much googling around for a good few hours each day and I found nothing. It then struck me that it was NOT eclipse, but rather a plugin I had in Eclipse. As it turned out it was the opposite of that: In that I did NOT have a plugin.<br /><br />In the project, in the Maven 2 POM I declare some extra eclipse settings, to use the aspectJ Eclipse nature and also something for Spring as part of the Spring IDE. When you run</span></span></span>
<blockquote>
<span style="color: black;"><span style="font-size: 100%;"><span style="font-family: 'courier new';">mvn eclipse:eclipse</span></span></span></blockquote>
<span style="color: black;"><span style="font-size: 100%;"><span style="font-family: 'courier new';"></span>the .classpath and .project files generated have all you relevate bits in there for eclipse.<br />I had this in the POM.</span></span><br />
<pre class="brush:xml">
<plugin>
<artifactId>maven-eclipse-plugin</artifactId>
<configuration>
<additionalProjectnatures>
<projectnature> org.eclipse.ajdt.ui.ajnature </projectnature>
<projectnature> org.eclipse.jdt.core.javanature </projectnature>
<projectnature> org.springframework.ide.eclipse.core.springnature </projectnature>
</additionalProjectnatures>
<buildcommands>
<buildcommand> org.eclipse.ajdt.core.ajbuilder </buildcommand>
<buildcommand> org.springframework.ide.eclipse.core.springbuilder </buildcommand>
</buildcommands>
<classpathContainers>
<classpathContainer>org.eclipse.jdt.launching.JRE_CONTAINER</classpathContainer>
<classpathContainer>org.eclipse.ajdt.core.ASPECTJRT_CONTAINER</classpathContainer>
</classpathContainers>
</configuration>
</plugin>
</pre>
<span style="color: black;"><span style="font-size: 100%;"><span style="font-family: 'times new roman';">
My guess was that I did not have the AspectJ plugin and the Spring Builder plugins, so the build paths were all screwey, which meant that Eclipse classpaths for JUnit were failing.<br /><br />I was right, a quick uncomment of the whole extra customisation on the eclipse maven plugin in the pom.xml, and rerun<span style="font-style: italic;"> mvn eclipse:eclipse </span>and I was up and running again.<br /><br />cheers. Hope this finds someone else a short path back to productivity.</span></span></span>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-3473871805404328405.post-53485805811476731302009-07-07T10:51:00.001+01:002009-07-07T10:52:48.678+01:00Coffee .. so disappointingOk, my first post since moving to the UK. I just walked down the road to my closest, best possible coffee house thus far, and it was Starbucks. How disappointing. not to be dismayed, I will find a solution one day.<br /><br />All is well, lots to read and understand at my new role. (but without coffee, my goodness, how will I survive ?)Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-3473871805404328405.post-65534984262744725082009-02-18T13:16:00.002+00:002009-02-18T13:22:10.701+00:00Importing old mail into GMailI have a client you needed a good 4 years of emails imported into GMail.<br /><br />At first I thought, okay there must be a nice way of doing this. Then I found some python scripts which did some magic, and I thought .. ahh it works. And then I found without testing or checking, that perhaps they were not what i was after.<br /><br />I needed the following to occur<br />(a) the emails appeared as if it was created when it was<br />(b) it showed it was sent by or received from as per the original <br />(c) there was no alteration to subjects or anything etc.<br /><br />So I managed to try a "rule" in Mac mail to forward all mail under certain criteria to the gmail address, and then apply the rule to the mails in question. But this was a bit dicky. It added the Fwd: subject prefix and amm mail appears as "me".<br /><br />I then thought about it ad it was simple... <br /><br />Just attach the Gmail mailbox in question as an IMAP folder in mac mail.<br /><br />Import (or no need if already there) the old email from wherever to mac mail. Then .. hat kiddies ? .. drag and drop the mail from the various folders to the IMAP folders.<br /><br />Simple. A quick check and the mail appears as if sent in 2007, from whom, to whom. (this was the sent mail) .<br /><br />Nice. No scripts or crap.. just time waiting to MacMail to do it's thing. I suspect my client will need to expand their data limit with Google.. but hey, what a service!Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-3473871805404328405.post-66017666708478881622009-02-06T00:19:00.004+00:002009-02-06T01:15:52.810+00:00Maven2 dependency classifier failsGot a beef with Maven2 classifier today. The classifier is an extra tag/property on the artifact (jar) identifier. Where it is used often is with -sources.jar and -javadoc.jar for a particular jar.<br /><br />An example of it's use is provided on the maven doco site<br /><br /><blockquote><br /><span style="font-weight: bold;">classifier</span>:<br />The classifier allows to distinguish artifacts that were built from the same POM but differ in their content. ...<br /><br />As a motivation for this element, consider for example a project that offers an artifact targeting JRE 1.5 but at the same time also an artifact that still supports JRE 1.4. The first artifact could be equipped with the classifier jdk15 and the second one with jdk14 such that clients can choose which one to use.<br /></blockquote><br />I have two jars which are JDK specific, one depends on the other (version). This means I have 4 jars.<br /><br /> bcprov-jdk16-141.jar<br /> ^ depends - bcpg-jdk16-141.jar<br /><br /> bcprov-jdk15-141.jar<br /> ^ depends - bcpg-jdk15-141.jar<br /><br />This is of course, bouncy castle, and am using it with camel-crypto component for Camel. Bouncy castle comes in a version for each JDK, one for JDK 1.5, one for JDK 1.6. Great! The classifier is a case for this.<br /><br />The code I will be writing primarily depends on bcpg (PGP implementation). So if I just depend on it, it will (should) pull in the parent bcprov.<br /><br />A few things I noted when creating the pom changes was that<br />(first) The name of the bouncycastle jar is names as<br /><br /><span style="font-family:courier new;"> bcprov-jdk16-141.jar</span><br /><br />Maven puts the classifier "after" the version.<br /><blockquote>arbitrary string that - if present - is appended to the artifact name just after the version number.</blockquote>So .. under maven, the bouncy castle jar should be<br /><br /><span style="font-family:courier new;"> bcprov-141-jdk16.jar</span><br /><span style="font-family:courier new;"> bcprov-${version}-${classifier}.jar</span><br /><br />At first, no worries, the bcprov jar up on ibiblio is old anyway and is missing the other jar I need which is bcpg, the PGP implementation. (the bit I want to add for Camel).<br /><br /><span style="font-weight: bold;">Putting this altogether:</span><br />in the root pom, we need to get a "property" version string so we can use it in the classifier.<br /><br /><span style="color: rgb(0, 0, 153);font-size:85%;" ><span style="font-family:courier new;"> <!-- set a property for the jdk version we are using based on the match of the jdk in use --></span><br /><span style="font-family:courier new;"></span><span style="font-family:courier new;"> <profile></profile></span><br /><span style="font-family:courier new;"> <id>jdk16</id></span><br /><span style="font-family:courier new;"> <activation><jdk>1.6</jdk></activation></span><br /><span style="font-family:courier new;"> <properties><jdk-version>jdk16</jdk-version></properties></span><br /><span style="font-family:courier new;"> </span><br /><span style="font-family:courier new;"> <profile></span><br /><span style="font-family:courier new;"> <id>jdk15</id></span><br /><span style="font-family:courier new;"> <activation><jdk>1.5</jdk></activation></span><br /><span style="font-family:courier new;"> <properties><jdk-version>jdk15</jdk-version></properties></span><br /><span style="font-family:courier new;"> </span></span><br /><br />In doing this, we will have ${jdk-version} as a property. This leads to a dependency like<br /><br /><span style="color: rgb(51, 51, 255);font-family:courier new;font-size:85%;" > <br /><br /> <dependency><br /> <groupid>bouncycastle</groupid><br /> <artifactid>bcpg</artifactid><br /> <version>${bouncycastle-version}</version><br /> <classifier>${jdk-version}</classifier><br /> </dependency></span><br /><br />The last bit to resolve is to install the renamed jars into my local repository and have bcpg depend on bcprov. But this is where maven fell over. One classifer jar and another, share the same "non" classifier pom.<br /><br /><blockquote style="font-family:courier new;"><span style="font-size:85%;">mvn install:install-file -DgroupId=bouncycastle -DartifactId=bcpg -Dversion=141 -Dclassifier=jdk16 -Dpackaging=jar -Dfile=/home/rbuckland/datastore/java/bcpg-jdk16-141.jar</span></blockquote><br /><br />This will then put in the renamed jar bcpg-jdk16-141.jar as bcpg-141-jdk16.jar.<br /><br />Where it falls over. Maven2 has one pom for all classif(ied)er versions. So the listing for the repository is thus<br /><br /><span style=";font-family:courier new;font-size:85%;" ><span style="color: rgb(0, 153, 0);"></span><blockquote><span style="color: rgb(0, 153, 0);">ls /home/rbuckland/.m2/repository/org/bouncycastle/bcpg/141/</span><br /><span style="color: rgb(0, 153, 0);">bcpg-141-jdk15.jar</span><br /><span style="color: rgb(0, 153, 0);">bcpg-141-jdk16.jar</span><br /><span style="color: rgb(0, 153, 0);">bcpg-141.pom</span></blockquote><span style="color: rgb(0, 153, 0);"></span></span><br />Not a pom for each. So in the pom (above) i need to declare the dependency on bcprov, but of course , in this repository pom for bcpg,<br /><br />(a) specificing no classifier, the dependency does not get picked up<br />(b) tried ${classifier} .. hopeful<br />(c) tried ${pom.classifier} .. hopeful<br /><br />I gave up. So instead the workaround is that the "version" of the var encompasses the JDK version instead. ie:<br /><br /><span style="color: rgb(51, 51, 255);font-size:85%;" ><span style="font-family:courier new;"> <dependency></span><br /><span style="font-family:courier new;"> <groupid>bouncycastle<br /></groupid></span><br /><span style="font-family:courier new;"> <artifactid>bcpg</artifactid></span><br /><span style="font-family:courier new;"> <version>${jdk-version}-${bouncycastle-version}</version></span><br /><span style="font-family:courier new;"> </span><br /></span><br />And then we can do what we need (of note, with this, the original naming of the jar by bouncycastle team stays).<br /><br />Ah well.Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-3473871805404328405.post-54557037568756695892009-01-23T11:35:00.004+00:002009-01-25T02:26:08.237+00:00Apache Camel - Something FreshI have been working with Apache Camel for a good 8-9 months now, and have been committing as much as I can more recently. It is amazing as it feels like a breath of fresh air from the complexities of jbi in the servicemix arena. <br /><br />Of course, many who work on and with servicemix also work on and work with camel and that is why I find camel refreshing. Simply that it is all the goodness of servicemix without the layer of jbi. Not to knock jbi of course, but it was very complex working with myriads of SU and SA Zip files that sometimes did nothing more than bloat your .m2 repository. Ahh those were the days. (edit: I will add that I am looking forward to the new ServiceMix 4.0 kernel as it , from what I can tell, is making large task of simplifying the architecture).<br /><br />So what am I using Camel for ? Recently a client has had some iteresting gaps in a project that neither the global provided ETL PowerCenter informatica or locally developed scripts were sufficing or could suffice the gap. <br /><br />Primarily we have been camel riding inside AMQ so we get good cluster support (potential) but it's the out of box nature that I enjoy (used to it from the servicemix days).<br /><br />So .. good work Camel devs, keep it up, and hopefully I can keep it up.Unknownnoreply@blogger.com0