RB Tech

Sakis3g Control Script

2012-01-10T13:02:00.000Z

I have found that my USB Modems, two of them, did not work out of the box with Ubuntu 11.10.
Not to worry, there is a great script that contains all the pieces for Ubuntu.


sakis3g

You can read all about it here - www.sakis3g.org and what it does.In essence, you use it in replace of your Network Manager or ppp/wvdial. Now I would prefer to use the built in method that the distribution has, however it plain does not work! Instead sakis3g and I have work to do.

I wanted a better way to control it, I created a command line control script for it for my modem. (Didn't like the GUI)


#!/bin/sh

BASE_CMD="/usr/local/bin/sakis3g"
OPTIONS="--term"
MODEMETC="OTHER='USBMODEM' USBMODEM='19d2:1003' USBINTERFACE='1' APN='3internet'"
COMMAND="$1"
shift

case $COMMAND in
status)
    $BASE_CMD $OPTIONS status
    ;;
connect)
    sudo $BASE_CMD $OPTIONS $MODEMETC connect 
    ;;
disconnect)
    sudo $BASE_CMD $OPTIONS disconnect 
    ;;
*)
    echo This is a wrapper to Sakis3g
    echo $0 connect
    echo $0 disconnect
    echo $0 status
    exit 1
esac

Crashplan Active Bandwidth Control

2011-12-29T23:50:00.001Z

I love what random things I get up to over the holidays. Crashplan is a Cloud backup service that I use for all my backups. I have devised a way so that Crashplan starts backing up at FULL speed, when everyone is not at home, or are not doing anything on the internet. And then drops back down to a trickle backup when things become busy.

Essentially I have one computer, my server, and it has the Crashplan service running on it. The server is an Ubuntu based Linux distrubution. Nothing fancy there. What is interesting is how I have managed to control the Crashplan bandwidth it utilises. In the previous house, I had a good 18-20Mbps internet connection, so my backups were really nice. Now, I only have a 3Mbps connection so the "usage" is very important. If crashplan hogs the connection you reall notice it.

Crashplan has two ways of controlling it's backup usage. First is on a backup set basis, you set the time that it runs. Helpful, for the past 2 months, my backups have been scheduled for 12am to 7am. Works well, but at that rate, I will be completely backed up in about 12 months time... hrmm.

The other way Crashplan operates is by bandwidth limiting. This is how I roll it now.

Attempt 1 - See when they Connect

I needed to determine the best method to "ascertain" if someone was using the internet. I pondered looking at when the phones (we have 4 adults, 3 iPhones and 1 Android) are "on" or can be seen as that is a good indicator that someone is home. But of course it goes a bit more than that.

I did however, to test the theory, write a quick Perl script which was the start of monitoring when an iPhone connects to the WiFi. The Android apparently uses Zeroconf also, but I didn't test that.
Basically, each time the phone connects for the initial session, it sends a multicast out. If you are listening, you will see it. Simply, this perl script listens on the right port.
I quickly discounted this method when I realised that some devices just wont tell me they connected. And because I want to keep my network largely zero configuation, I want to just use DHCP and not have to bother with static entries, or Static leases for my clients. Which is when it hit me:

Attempt 2 - Scan the Known DHCP Range

The premise behind my final solution is simple. I have two types of devices in use in our household.

Infrastructure - servers, printer, wirless and network switches / routers
Clients - Laptops, phones (soon tablets)

All my infrastructure devices use static IP addresses, such as 1 to 39. All the clients, use a DHCP IP address.We have no desktops in the house, only laptops and when they are closed, they are not in use.

If they are not in the house, they are not in use. Same goes with the phones, they are either sleeping or not in the house.

The Solution

So, if all the laptops are closed and all the phones are out, I know that none of the "clients" will ping, and therefore I can ramp up the Crashplan bandwidth.

Solution to Attempt 2 - Monitor for Active Clients

I had a few things to solve. But for each I knew that Perl would be all that I would need.

Part 1 - Automatically Raise and Lower Crashplan Bandwidth

I needed to determine how Crashplan "stored" it's bandwidth rate. Turns out it is in a config file, but the web site can also change the config. So, the client (my Crashplan service running on my server) and the Crashplan Cloud keep in contact all the time· If I change the bandwidth on the client, it updates on the website, and vice versa.

I fired off a few support queries to Crashplan and they basically said that the config file I found (/usr/local/crashplan/conf/my.service.xml) is not editable by hand. And that the only supported way is via the Client (desktop Java App) or the Website.

Call WWW::Mechanize - I used this module to login to the Crashplan Website and change the bandwidth (kbps) based on my argument. Simple and it looks like this.

Not rabbit proof, but it works for my needs

Part 2 - Monitor for Activity

This was the hardest part, but as always, was solvable. The router (Sky Broadband Provider) I currently use "knows" what clients are using it. VERY handy. So like the Crashplan website solution above, I again use WWW::Mechanize to determine what IP addresses are currently using the router.

I do a filter out of all the ip addresses that do NOT fall inside my DHCP range.

I won't go into the details of using XPath on the HTML from the Router, but, you can see that it is quite succinct. Essentially the IP Addresses appear in a 2nd column in a table. I use an XPath expression to select the right values.

This approach is brittle in as much as, when / if I change my router then I need to code or come up with another way to determine which hosts, within the DHCP range, are currently "in use". I did first to this by just pinging all 60 possible IP addresses; works, brute force, but of course worked :-)

So .. now putting it all together. The script, a perl script, runs from Cron every minute. and performs the following tasks.

Get the Current Rate from the Crashplan Config file
Get all the current "hosts" that the Router knows
Ping each host to MAKE sure they are active
If we have an active host (one or more) - we need to go slow
- If the current rate that Crashplan is running is more than our slow rate, tell Crashplan to slow down
- If the current rate is slow, do nothing
If we have no active hosts - we can go fast
- If the current rate that Crashplan is running is less than our fast rate, tell Crashplan to go fast
- If the current rate is fast, do nothing

Simple! And the full Script looks like below. Enjoy!

Of course, I did all this and know and realise that QoS could be used, but my Sky Broadband Router is just not that smart today and I haven't got the devices spare or the time to reconfigure to installl a dd-wrt based router or some such other. So perl scripts it is for now.

Neo4J - 2nd Look - Setting a Primary Key on Nodes

2011-10-27T12:33:00.002+01:00

Primary Key

In my last post I considered the lack of Primary Key like Id's as something I need to solve. My use case is

The application I will be building out will have, after all is said and done a really simple Web Interface with REST type URLs. So .. for example, I will be able to do.
http://myservice.co.uk/superwebapp/mySpecialThing/detailedView/55
The 55 there will result in a query to Neo4J to locate the "MySpecialThing" object with ID of type 55 and display it.

I also considered using a UUID across objects which is also good, but not really what I was after. I want a class of objects to all share an identifier. It has a lot of use. To solve the problem I arrived at the following solution.

Solution Outline

All Domain Objects extend fro super type of AbstractLongDomain which has getId()/setId() (Long)
An Aspect wrapped around the getId() looks for a null value and if no Id is found. It creates one
In the aspect creation of an ID involves talking to a singleton IdManager for a "nextId()"
nextId() on the manager looks to it's cache HashMap to see if it has an IdObject that knows what the next Id is
IdObject self persists to the repository after each call (** this could be slow.. see how we go)

To the Code

All my Objects extend the AbstractLongDomainThe ID Object holds a "per" class Long Id, so each time an ID is needed, one of these objects gives one out.Next we have the IdManager that is managed as a Spring Singleton Bean. Its job is to return an id based on the "class" that needs an Id via getNextId(Class klass).The idRepository you see there is a simple Spring Data Neo4J Repository which has aspect-magic dust sprinkled on it to make the actual implementation.

Because I am not sure if neo4j is the "best" place to store the Id's (though it is the logical) I created a simple idGenerator interface which is simply what the aspect will call and talk to. One implementation (the only) is the Neo4JBackedIdGenerator which uses the id objects and idmanager above.

So first the interface for the IdGenerator

And then the actual Neo4JBackedIdGenerator which is created and managed as a spring bean.

I will have to play with the transactional semantics on this one. I recall a horrid situation which a similar design but via stored procs many moons ago where we had the sproc that generate Id's wrapped in transactions. They needed to be in their own transaction to ensure that mass object thread creation would not get stuck on a lock. (just an area I know can be sticky and bite.. so I put the @Transactional in there and commented out to remind me.

So last, we have the AspectJ which wraps our getId(). All the domain objects extend org.soqqo.luap.model.AbstractLongDomain which means we will get the Id creation and generation for free each time getId is called. (technically a catch here is that setId doesn't get checked if called manually on setting an Id. It could I guess look into the repo to see if the Id is already used.

And finally the Unit Test code to show that it all works

Note the use of @DirtiesContext, because the Neo4J is transactional, after each test the contents are dumped, which means that the idManager which has the HashMap cache becomes stale and it is singleton and has a lifecycle of the test class, not just the method. So the fix is either..manually flush() the cache or use @DirtiesContext which tells spring to re-build the context file. Both work but manually flushing my HashMap (new() ) is 2 seconds faster (0.037s for the test) than spring is at rebuilding.

The 2nd last piece to show is my test context file - model-test-context.xml

The very last piece is what my Maven POM looks like because a lot of people like to see that... Hopefully these are the correct relevant bits.I am happy to provide all this as a ZIP or push it up to github if people want to see more of it.

Neo4J - My First Look with Spring Data Graph

2011-10-26T12:33:00.000+01:00

Spring Data Neo4J

I have been working on some proof of concept code and decided on a clean route to using NoSQL. Of course there are many choices and because the application I am working on is highly connected around relationships. (not boyfriend girlfriend types) I figured I would look at Neo4j. Given my favourite library of the year is spring-data I would take a look at the recently release spring-data-neo4j library (formerly called Spring Data Graph).

Spring Data provides some funky interface abstraction over your data store, be that an RDBMS, or other type of storage like NoSQL forms.

Specifics to Neo4j and Spring Data Neo4J

A Quick Overview:

Neo4J allows you to store POJOs without the need of a Schema.
POJOs are tied together using Relationships
Neo4J Understands a Node and a Relationship (that is it)
Spring Data Graph makes the "Node storage" and "relationship" tie-ing really simple with Annotations

Let's look at that last point in detail.spring-data-neo4j uses a few special annotations, not unlike JPA's annotations.

Declaring a Node

A Node is declared with the following annotation

@NodeEntity
public class MySpecialPojo {

    @Indexed
    Long id;

    @Indexed(indexType=IndexType.FULLTEXT, indexName = "search")
    String textField;
    //...
}

Effectively these annotations make some magic happen. One of the big magic happen things is to do with some special methods() you will find on the objects.If you include the right "stuff" in your Maven POM for spring-data-neo4j, you will get some good stuff happening.Effectively some "DAO/repository" style methods get woven into your domain objects.

        // for free we get .persist() Which wraps up a call to neo4j and put my Pojo as a Node down to Neo4j.
        MySpecialPojo special = new MySpecialPojo(1,"Some Data").persist();

        MySpecialPojo foundSpecial = this.pojoRepository.findByPropertyValue("textField", "Some Data");

You will also have .remove() and other fun stuff. The best document I have found (as it is very new (2.0.0.M1) is the following PDF. Spring Data Neo4J - Good Relationships

Missing Identity or Mimicking(sp?) a Primary Key

The application I will be building out will have, after all is said and done a really simple Web Interface with REST type URLs.So .. for example, I will be able to do


http://myservice.co.uk/superwebapp/mySpecialThing/detailedView/55

The 55 there will result in a query to Neo4J to locate the "MySpecialThing" object with ID of type 55 and display it. The problem I have is a two fold

Neo4J just stores objects and does not have "primary keys" other than a "nodeId".
The NodeId is collection wide. So Pojo1 shares the incremental nodeIds with Pojo2.

spring-data-neo4j adds (via an aspect ITD) a getNodeId() method to my POJOs but I don't want to depend on these for my "primary key" (future proofing my app if I move from Neo4J to something else).

So I want a Class wide "Id" so that when an object is persisted it has an ID for it.
I may be thinking about this wrong, and should just mold and accept a collection (database/store) wide ID system.
I am heading down this path


public class Foo { 
   @NodeId(collection=Foo.class)
   private Long id;
}

// or 
@NodeId(collection=Foo.class)
public class Foo extends AbstractLongIdDomainObject { 
    // .... get/setId() is found on super class.
}

This would then store and manage an ID, like we used to in RDBMS days when they did not have Primary Key AUTO INCREMENT or IDENTITY type stuff. The way you would implement Primary ID's is to have a table that stores the ID for "each collection" and a lookup stored-procedure or code that would "get" the next ID for you (within a transaction for example).

I have not perfected this, but I figured I would show what I was thinking. Plus I will do some more reading, (maybe Neo4J has some config to allow per class type nodeIds. * I always go the hard way first *



@NodeEntity
public class IdManager { 

    /**
     * This map holds an idObject per "className" for each object we want to "have a Primary KEY id for"
     */
    @Indexed
    private Map idCollection;

}
@NodeEntity
public class IdObject {

    public Long getNextId() {
        return nextId;
    }

    public void setNextId(Long nextId) {
        this.nextId = nextId;
    }

}

So with this magic code I would then annotate as above with my custom (@NodeId) and some magic happens using aspects and stuff to weave in the "next" ID when an object is "created" and about to be stored through spring-data-neo4j.

Can't Default Values

Probably just by how the aspects interact with the get/set methods for your fields, I found that you cannot default a fields value like you can with JPA.


@NodeEntity
public class Foo { 

   private Long someNumber = 1L;
   // .. getter setters
}

If you create this object, and set the someNumber to 55 for example. And then fooInstances.persist() and then retrieve it from the repository, it will not have the value of 55, but the value of 1. !! Annoying. So that is okay .. but I think the apsects that "populate" the fields are going in too early or something. I have a test case that shows this happening but it was in a complex Aspect( because of my above primary key playing) so it may be a special case. I'll see.

Summary

Really nice and I like the simplicity that neo4j and spring-data give. GO away SQL.

Released a New Open Source Library

2011-09-16T13:51:00.004+01:00

For a while it has slightly or mildly annoyed me how clunky creating random data can be.
You see all manner of approaches from Importing CSV files and XML to loading up JSON objects and the like.
What I have wanted was a library that allows random or specific generation into your Java POJO domain model which you then simply persist using your preferred storage layer (Hibernate, JPA, Neo4j) etc.
DBUnit is fantastic and I have often used http://www.generatedata.com to generate XML files to upload through DBUnit.

Well know I sctarched that itch over the last day to create the aptly named

random-data-generator - http://code.google.com/p/random-data-generator

In short, see the project page as it explains most of how it works. It is a library for use in your code for "seeding" objects with random data.

Enjoy.

Love *Nix

2011-09-15T11:29:00.001+01:00

I Still love *nix after all these years. It only took me 1 minute to convert lines like


1880,"Mary",0.072381,"girl"

to what I wanted.


$ grep \"girl\" baby-names.csv  | head | cut -d, -f2 | sed 's/"//g;s/$/,F/g'
Mary,F
Anna,F
Emma,F
Elizabeth,F
Minnie,F
Margaret,F
Ida,F
Alice,F
Bertha,F
Sarah,F

GWT2 and Maven

2010-04-23T21:22:00.001+01:00

GWT 2 and Maven - Close I have been working with the new GWT 2 code. Especially the new IDE Plugin for eclipse.

One things I remember doing with older 1.5 --> 1.7 code was run a separate Jetty container and pass -noserver to the GWT Dev mode, so that I could control

(a) where the compiled JS files were and went to

(b) running a servlet separate from the hosted app was nice.

With the new in-IDE mode and the new browser plugin, I found it is almost there. Essentially, my issue I am resolving at the moment is a maven thing.

I don't want the GWT Hosted mode to put compiled *.js files (etc) in my src/main/webapp. Primarily because this folder is in Source Control.
My Maven compiler plugin writes out the WAR when i do mvn package.
I want (1.) solved by having it splay out the Compiled files to target/some-ignorable-directory and use the magic Jetty ALIAS do-hickey to "combine" src/main/webapp and target/gwt-hosted-war.tmp

achievable, yes/. The problem comes that .. In the lanucher .. I try: -war "${workspace_loc:sample-gwt-webapp/target}/gwt-hosted-war.tmp"

-war "${workspace_loc:sample-gwt-webapp/target}/gwt-hosted-war.tmp"

But, that directory does not actually have the WEB-INF files in it.

How could I solve it:

symlinks src/main/webapp/gwt/* to target/tmpfilder/gwt with Linux and Mac.. not portable to Windows .. except for linkd
before starting the Jetty Container, copy the src/main/webapp to target/tmp-war/ .. not nice for code changes of running jetty in src/main/webapp
Make changes to the dev classes .. (you can provide a new ServletContainer .. but only that .. and it's not in there where the checking is done)
use the -noserver and use Jetty (I want to run just on JVM for debugging) ..

bummer.

OSGi

2010-03-26T22:47:00.001Z

A good read this.

http://njbartlett.name/blog/2010/03/17/osgi-and-how-it-got-that-way/

I am looking at using OSGi to provide the plugin capability to this new Scala, JPA, GWT, Spring App.

Scala, JPA, GWT, Spring, Eclipse, Maven - Part 1

2010-03-20T23:15:00.005Z

About to embark on something ambitiuos (I suspect), I figured I would detail through some blog posts how I go.

I am building an application (secret squirrel until it releases) that I want Scala to be it's core language.

The GUI I want in Java (it just works so well), Eclipse is just the default IDE for me, Maven my absolute must build tool and then all the goodness of Spring and Hibernate as usual.

My first challenge (beyond learning Scala) is to get Eclipse, Maven and Scala to co-operate. The project will have Java source and Scala Source mixed in.

GWT, as per my other standard projects will be in it's own module anyway.

So.. here goes. This is just the "I am trying this" post. I'll let you all know how I go.

Read on for Part 2

Changing GMail Accounts

2009-12-30T11:25:00.002Z

I recently decided to ditch my old gmail account in favour of my new gmail account, which is my personal domain name I have had for years (Google Apps for Domain).

In doing so I needed to port all my mail from the old gmail account to the new gmail account.

I setup the appropriate forwards etc, been running that way for about a month now.

The best way I have found to "Move" all the mail from old to new is.

Setup BOTH the GMail accounts in Mail (OSX on the Mac) as an IMAP account. The IMAP bit of course is important. Once they are setup, and you can see the mail in both of them.

Simply: Drag and Drop. Go to Sent Mail, in the old. select all and drag it into the Sent Mail on the new. All the dates and everything is preserved. Similarly, the same for Drafts (yes I had over 50 drafts) .. and All Mail, that caps in at 10,000 emails.

Just drag and drop to the same folder in the new account; Leave it running over night of course as it takes a LONG time. And without needed to state the obvious, the faster your connection, the faster this is. It moves the mail from the OLD to the new, VIA your computer.

Eclipse Galileo java.lang.ClassNotFoundException with JUnit Tests

2009-09-12T23:57:00.002+01:00

This post describes my problem, and the solution
Recently I upgraded to Eclipse Galileo 3.5 on my Ubuntu dev box. All was well and I noticed no degradation or changes (new, good or bad) .. until ..
Until I went to run my first testcase in my new ete library. I received the beautiful

java.lang.ClassNotFoundException

.. on the JUnit Test case! :-) yuck.This was the stacktrace in it's entirety

Class not found org.soqqo.ete.source.data.EhCachedDataRowListTest
java.lang.ClassNotFoundException: org.soqqo.ete.source.data.EhCachedDataRowListTest
at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:276)
at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.loadClass(RemoteTestRunner.java:693)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.loadClasses(RemoteTestRunner.java:429)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:452)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)

It really threw me because on the command line, Maven2 (mvn test) test cases still worked fine. So I knew I havd an Eclipse problem. The upgrade was my key, ie: it was within 20 minutes of starting Galileo that this error occured.

So I ran my test cases in Eclipse 3.4 and it worked. 3.5 I say.. but then, that was too simple. There must be something up.

Much googling around for a good few hours each day and I found nothing. It then struck me that it was NOT eclipse, but rather a plugin I had in Eclipse. As it turned out it was the opposite of that: In that I did NOT have a plugin.

In the project, in the Maven 2 POM I declare some extra eclipse settings, to use the aspectJ Eclipse nature and also something for Spring as part of the Spring IDE. When you run

mvn eclipse:eclipse

the .classpath and .project files generated have all you relevate bits in there for eclipse.
I had this in the POM.



  maven-eclipse-plugin
  
    
       org.eclipse.ajdt.ui.ajnature 
       org.eclipse.jdt.core.javanature 
       org.springframework.ide.eclipse.core.springnature 
    
    
       org.eclipse.ajdt.core.ajbuilder 
       org.springframework.ide.eclipse.core.springbuilder 
    
    
      org.eclipse.jdt.launching.JRE_CONTAINER
      org.eclipse.ajdt.core.ASPECTJRT_CONTAINER

My guess was that I did not have the AspectJ plugin and the Spring Builder plugins, so the build paths were all screwey, which meant that Eclipse classpaths for JUnit were failing.

I was right, a quick uncomment of the whole extra customisation on the eclipse maven plugin in the pom.xml, and rerun mvn eclipse:eclipse and I was up and running again.

cheers. Hope this finds someone else a short path back to productivity.

Coffee .. so disappointing

2009-07-07T10:51:00.001+01:00

Ok, my first post since moving to the UK. I just walked down the road to my closest, best possible coffee house thus far, and it was Starbucks. How disappointing. not to be dismayed, I will find a solution one day.

All is well, lots to read and understand at my new role. (but without coffee, my goodness, how will I survive ?)

Importing old mail into GMail

2009-02-18T13:16:00.002Z

I have a client you needed a good 4 years of emails imported into GMail.

At first I thought, okay there must be a nice way of doing this. Then I found some python scripts which did some magic, and I thought .. ahh it works. And then I found without testing or checking, that perhaps they were not what i was after.

I needed the following to occur
(a) the emails appeared as if it was created when it was
(b) it showed it was sent by or received from as per the original
(c) there was no alteration to subjects or anything etc.

So I managed to try a "rule" in Mac mail to forward all mail under certain criteria to the gmail address, and then apply the rule to the mails in question. But this was a bit dicky. It added the Fwd: subject prefix and amm mail appears as "me".

I then thought about it ad it was simple...

Just attach the Gmail mailbox in question as an IMAP folder in mac mail.

Import (or no need if already there) the old email from wherever to mac mail. Then .. hat kiddies ? .. drag and drop the mail from the various folders to the IMAP folders.

Simple. A quick check and the mail appears as if sent in 2007, from whom, to whom. (this was the sent mail) .

Nice. No scripts or crap.. just time waiting to MacMail to do it's thing. I suspect my client will need to expand their data limit with Google.. but hey, what a service!

Maven2 dependency classifier fails

2009-02-06T00:19:00.004Z

Got a beef with Maven2 classifier today. The classifier is an extra tag/property on the artifact (jar) identifier. Where it is used often is with -sources.jar and -javadoc.jar for a particular jar.

An example of it's use is provided on the maven doco site

classifier:
The classifier allows to distinguish artifacts that were built from the same POM but differ in their content. ...

As a motivation for this element, consider for example a project that offers an artifact targeting JRE 1.5 but at the same time also an artifact that still supports JRE 1.4. The first artifact could be equipped with the classifier jdk15 and the second one with jdk14 such that clients can choose which one to use.

I have two jars which are JDK specific, one depends on the other (version). This means I have 4 jars.

bcprov-jdk16-141.jar
^ depends - bcpg-jdk16-141.jar

bcprov-jdk15-141.jar
^ depends - bcpg-jdk15-141.jar

This is of course, bouncy castle, and am using it with camel-crypto component for Camel. Bouncy castle comes in a version for each JDK, one for JDK 1.5, one for JDK 1.6. Great! The classifier is a case for this.

The code I will be writing primarily depends on bcpg (PGP implementation). So if I just depend on it, it will (should) pull in the parent bcprov.

A few things I noted when creating the pom changes was that
(first) The name of the bouncycastle jar is names as

bcprov-jdk16-141.jar

Maven puts the classifier "after" the version.

arbitrary string that - if present - is appended to the artifact name just after the version number.

So .. under maven, the bouncy castle jar should be

bcprov-141-jdk16.jar
bcprov-${version}-${classifier}.jar

At first, no worries, the bcprov jar up on ibiblio is old anyway and is missing the other jar I need which is bcpg, the PGP implementation. (the bit I want to add for Camel).

Putting this altogether:
in the root pom, we need to get a "property" version string so we can use it in the classifier.


<profile>
<id>jdk16</id>
<activation><jdk>1.6</jdk></activation>
<properties><jdk-version>jdk16</jdk-version></properties>

<profile>
<id>jdk15</id>
<activation><jdk>1.5</activation>
<properties><jdk-version>jdk15</jdk-version></properties>

In doing this, we will have ${jdk-version} as a property. This leads to a dependency like

<dependency>
<groupid>bouncycastle</groupid>
<artifactid>bcpg</artifactid>
<version>${bouncycastle-version}</version>
<classifier>${jdk-version}</classifier>
</dependency>

The last bit to resolve is to install the renamed jars into my local repository and have bcpg depend on bcprov. But this is where maven fell over. One classifer jar and another, share the same "non" classifier pom.

mvn install:install-file -DgroupId=bouncycastle -DartifactId=bcpg -Dversion=141 -Dclassifier=jdk16 -Dpackaging=jar -Dfile=/home/rbuckland/datastore/java/bcpg-jdk16-141.jar

This will then put in the renamed jar bcpg-jdk16-141.jar as bcpg-141-jdk16.jar.

Where it falls over. Maven2 has one pom for all classif(ied)er versions. So the listing for the repository is thus

ls /home/rbuckland/.m2/repository/org/bouncycastle/bcpg/141/
bcpg-141-jdk15.jar
bcpg-141-jdk16.jar
bcpg-141.pom

Not a pom for each. So in the pom (above) i need to declare the dependency on bcprov, but of course , in this repository pom for bcpg,

(a) specificing no classifier, the dependency does not get picked up
(b) tried ${classifier} .. hopeful
(c) tried ${pom.classifier} .. hopeful

I gave up. So instead the workaround is that the "version" of the var encompasses the JDK version instead. ie:

<dependency>
<groupid>bouncycastle
</groupid>
<artifactid>bcpg</artifactid>
<version>${jdk-version}-${bouncycastle-version}</version>

And then we can do what we need (of note, with this, the original naming of the jar by bouncycastle team stays).

Ah well.

Apache Camel - Something Fresh

2009-01-23T11:35:00.004Z

I have been working with Apache Camel for a good 8-9 months now, and have been committing as much as I can more recently. It is amazing as it feels like a breath of fresh air from the complexities of jbi in the servicemix arena.

Of course, many who work on and with servicemix also work on and work with camel and that is why I find camel refreshing. Simply that it is all the goodness of servicemix without the layer of jbi. Not to knock jbi of course, but it was very complex working with myriads of SU and SA Zip files that sometimes did nothing more than bloat your .m2 repository. Ahh those were the days. (edit: I will add that I am looking forward to the new ServiceMix 4.0 kernel as it , from what I can tell, is making large task of simplifying the architecture).

So what am I using Camel for ? Recently a client has had some iteresting gaps in a project that neither the global provided ETL PowerCenter informatica or locally developed scripts were sufficing or could suffice the gap.

Primarily we have been camel riding inside AMQ so we get good cluster support (potential) but it's the out of box nature that I enjoy (used to it from the servicemix days).

So .. good work Camel devs, keep it up, and hopefully I can keep it up.

Stick with the masses ( Eclipse 3.4.1 XPCOM error -2147467262 )

2009-01-12T00:46:00.003Z

There is a lot to be said about sticking with the masses, going with the crowd so to speak. Especially when you are riding the crowd (crowd) surfing for a different purpose .. huh ?

Let me explain ..

I have been an active user of Linux for the best part of 14 years. In that time, I have sat on the bleeding edge of changes and sometimes suffered. But, also I had time to kill, the fix and fix and patch time was kind of fun.

These days I have a lot less time that I allocate to that discretionary IT time budget, as I have a 2 girls and my beautiful wife with whom I enjoy spending time.

I still mass large amounts of work into my time, but when something goes wrong, ancillary to my task, (or, better put - my tools fail), then i get cranky.

Just on Friday I was prepping to make some changes to camel-file/ftp, adding support for commons-vfs and my eclipse failed to load.

It worked on the Thu, but Fri, nothing. My error said look at workspace/.metadata/.log

In there I find an obscure message...


!ENTRY org.eclipse.ui 4 0 2009-01-12 14:23:58.273
!MESSAGE Unhandled event loop exception
!STACK 0
org.eclipse.swt.SWTException: Failed to execute runnable (org.eclipse.swt.SWTError: XPCOM error -2147467262)

As it turned out, this was caused by an incompatibility with my eclipse and (possible Firefox or something in a mozilla stack).

What was my fix ? Well it turns out, the workspace info page at the first startup of eclipse uses a browser (no worries). But it had fundametal issues launching my system default browser. To get past the bug .. you need to unpack a jar

(/opt/eclipse/plugins/org.eclipse.ui.intro_3.2.201.v20080702_34x.jar)

and edit the plugin.xml, commenting out all extension elements, repack it and restart eclipse with a -clean.

See here for details (this link was the 5th attempt at me getting around the problem).

So why stick with the masses ? Well, I can only guess, but I suspect my running of Ubuntu amd64 is the cause of this eclipse issue biting me. Or, put it mildly, the error has not been tested much on this platform and I got stung, I could be wrong, but I can only guess that, surely others have seen the problem, I am not the only one running eclipse on Linux.

Where it failed for me is that Ubuntu automatically updated (something) and then my eclipse broke. I haven't got the time to work out what updated in the last week due to the auto update manager .. ? question: is there such a place to see the install log.. dpkg must know).

Anyways, my problem is gone and I am back to my usual eclipse-ing self. (minus the intro page, but i have never used that anyway).

bash commands, xml cleaning and Java Searching

2008-07-17T04:33:00.010+01:00

A lot of you are Linux users, some Mac and *NIX and the others, I want you to be *NIX. (perhaps use cygwin for a bit).

This post is primarily about two things .. helping you in the command line and helping you develop.

I am, as most people know a Java developer (by trade) and work a lot with XML. I found two things annoyed me for some time a year or two back.

1. Finding the jar of some class I had not seen (and it may be in my m2 repository)
2. Cleaning my XML files (especially the pom.xml files) so they were neat.

I solved both of these with two seperate bash functions. A bash function acts like command on the command line, so if you have



function foo () { echo "foo and Hello World together" }

in your .bashrc, when you "source ~/.bashrc or open another shell, the command "foo" will work.

Cobling this together with some good old unix commandline foo, you can do amazing stuff (I remember the first complex function I wrote was a wrapper for df (disk filesystem) and using awk to show the percentage of usage (of about 2MB) that I had used on the UNE CS Dec Alpha Server, mihi and later neumann).

Sadly I don't have that command anymore, but I don't need it either as df -h gives you the wonderful details. I digress.

Back to functions; it is all well and good to have them, but if you don't remember them, or how to run them, they become dusty (and your self help shell foo rusty).

To solve this problem, I created my own simple help system, "myhelp" nothing robust, nothing fancy but it all works within the one .bashrc file. The principle is simple. Any function I write for me, adheres to two principles.

It calls a "help" function first off, which checks if the first argument is --help and promptly displays the help and exits the function.
After the first { on the same line as the "function ..." brace, i add a comment # myFunction which signifies it is special and help can be listed for it. (and that is is my function)

Now to the functions themselves.
1. First the XML Cleaner.

#
# Use XMLLint to reformat the XML
#
function xmlclean () { # myFunction
myShowHelp $1 "xmlclean" " - Reformat the XML Document" || return

doc=$1;
tmpDoc=/tmp/xcleaner.$$.xml
cp ${doc} ${tmpDoc} && xmllint --format ${tmpDoc} --output ${doc} && rm ${tmpDoc}
}

The indentation is a little skewed (I'll try and resolve that later). This bash script uses xmllint

sudo apt-get install libxml2-utils

Fairly simple, in fact I THINK that the xmllint can do an inline replacement, but hey mine works.

so .. formatting an XML file is simply ..

rbuckland@ld630:~$ xmlclean foo.xml

2. Now for locating classes in Jars ..

#
# find a java class in the Repo
#
function jfind () { # myFunction
myShowHelp $1 "jfind" " - Find a classname(substring also) in Maven2 jars. The classname is passed to egrep on the jar contents" || return
find ~/.m2/repository/ -name '*.jar' | xargs -l1 -ixx sh -c "jar tvf xx | egrep $1 && echo xx"
}

An example of it in action:

rbuckland@ld630:~$ jfind DxmlExecutor
1842 Fri Jul 04 09:09:06 EST 2008 com/mivira/dxml/core/DxmlExecutor.class
/home/rbuckland/.m2/repository/com/mivira/dxml/dxml/0.9.1a/dxml-0.9.1a.jar

I won't go into what it is doing (it's pretty simple) just run each command seperately and you will get it all.

3. Now for the help system,

To show your custom commands you have written, type the following.

rbuckland@ld630:~$ myhelp
jfind - Find a classname(substring also) in Maven2 jars. The classname is passed to egrep on the jar contents
xmlclean - Reformat the XML Document
rbuckland@ld630:~$

To see the help for one command ..

rbuckland@ld630:~$ xmlclean --help
xmlclean - Reformat the XML Document

The myhelp works simply by looking for all "function name () { #myFunction" lines in your .bashrc and runs functionName --help for each one.

here is the two help functions you need in the .bashrc.

# declare -F shows the functions, and the shopt 'extdebug' is meant to show the line numbers
# of the functions (and src file) but it didn't so I am going with a more fool proof
# way of identifying my functions .. grep the .bashrc for all function lines that have the
# comment on the same line (hack but works)
#
function myhelp () {
for i in `grep myFunction ~/.bashrc | cut -f2 -d' '`
do
$i --help
done
}

function myShowHelp() {
if [ $1 == '--help' ]; then
echo $2 $3
return 1
fi
return 0
}

That's it. Now when I add a new function. All I have to do is create it according to my template which is

function mynewcommand () { # myFunction
myShowHelp $1 "mynewcommand" "description" || return

# do all the work here
}

and that's it, re-source my .bashrc (source ~/.bashrc) and test it out. and myhelp just automagically picks it up for me.

Done!

Addendum: As is is often the case, looking at my old code I make imporevements and posting this blog entry did just that.

I figured that maven related tasks should be prefixed with m2. and also I added another helper for maven.

Here is my full set of relevant helpers.
Cheers


#-----------------------------------------------------------------------
# declare -F shows the functions, and the shopt 'extdebug' is meant to show the line numbers
# of the functions (and src file) but it didn't so I am going with a more fool proof
# way of identifying my functions .. grep the .bashrc for all function lines that have the 
# comment on the same line (hack but works)
#
function myhelp () {
  for i in `grep myFunction ~/.bashrc | cut -f2 -d' '`
  do
     $i --help
  done
}

function myShowHelp() {
   if [ $1 == '--help' ]; then
     echo $2 $3
     return 1
   fi
   return 0
}

#
# print out all the matching jars in the repo
#
function m2showversions () { # myFunction
    myShowHelp $1 "m2showversions" " - Simply lists all the jar filenames that match. Infer from that the versions you have on disk"

    (cd ~/.m2/repository  && find * -name "*$1*")
}

#
# find a java class in the Repo
#
function m2find () { # myFunction
    myShowHelp $1 "m2find" "  - Find a classname(substring also) in Maven2 jars. The classname is passed to egrep on the jar contents" || return
    find ~/.m2/repository/ -name '*.jar' | xargs -l1 -ixx sh -c "jar tvf xx | egrep $1 && echo xx"
}

#
# Use XMLLint to reformat the XML
#
function xmlclean () { # myFunction
    myShowHelp $1 "xmlclean" "     - Reformat the XML Document" || return

    doc=$1;
    tmpDoc=/tmp/xcleaner.$$.xml
    cp ${doc} ${tmpDoc} && xmllint --format ${tmpDoc} --output ${doc} && rm ${tmpDoc}
}
#-----------------------------------------------------------------------

hope that helps

Hoax Emails - Internet Banking Fraud

2008-03-25T11:18:00.004Z

In my time of collecting and monitoring SPAM I have noticed that I have not received many Internet Banking Fraud / Hoax emails. It might just be me but it seems a bit odd that the only one that I have received are for Internet Banking sites which I use. So, for example, I use ANZ Internet Banking and just today received my 3rd Internet Banking Hoax Email, this one was for ANZ.

So how do the fraudsters (server registered in Chile (.cl) ) know that I bank ANZ, and why don;t I get NAB, St George or Bank Brunei Email Hoaxes ? The answer is that they KNOW I bank ANZ.

What would be some ways they obtain this list ? I can certainly think of some. And I will add also, ways for the Banks to prevent it, and also for us to be mindful (as users).

Buying the list
At first you would hope it doesn't occur, but when you think about it, it is quite plausible. A high enough price put up entices someone with access to "sell" the list.

Imagine that of 100,000 emails sent out, only 100 people (.001%) provide their details. And if from this, the fraudsters were able to transfer an average of $500, that's a quick easy $50,000.

Of course, some accounts have $50,000 in them to start with, so the bounty is much more of course.

So let's go with $50,000. We can pay $5,000 to the list provider, all they have to provide is Email Addresses of valid customers, nothing more.

$5k is not a lot, but if it were sold for $5k to 3 "fraud" groups, then the insider pockets $15k.

A security breach is 90% of the time an insider job and is the one thing that banks must watch out for.

So who could be an insider ? Well simply put anyone with access to the email list.

This could be:

Anyone in IT with sufficient privileges (to the test, development, or production database).
Anyone in Marketing who can extract / export a list of email addresses for bulk email
Anyone in Customer Servicing with this type of access
The Vendor of the Banking System who is given the Database in original form for issue resolution (and of course then, any of the vendors employees)

So the insider is an issue. We know it, but it really is as scarey as that. So many IT systems are left open or "available" for reading but all users. I have certainly worked with my fair share that do.

Moving On, so, if it's not an insider job, how else could it be done ?

The ISP Proxy Server
All requests that I make from my ISP (ie: at home) are logged, not only my ISP at home, but at work also.

Might seem odd, but this data is worth money.

At the ISP, they just have to marry my email address and a list of known "banking" logon sites that I access, then they have a decent list. Of course it is not ALL users, but a big ISP (Telstra Bigpond in Australia) would be a nice size and, sadly, the bigger the ISP, the more un-suspecting the user will be and more likely to click on that link.

At a stretch
Virus / URL Logger / Spy
So this one is kind of logical, but probably not done. If there is software on my PC which is "logging" my surfing, they could use that. But if it's on my PC, then well, might as well log the logon to the internet banking site also.

So How do We Prevent it ?
I'll talk about us and the Bank.
For us

(a) Change banks to one which provides
(i) SMS Bank Pay Anyone Verification (With the CommonWealth Bank - not ANZ) for any new Pay Anyone Account not in my list, I need to verify it with an SMS code which is sent to my Mobile phone. That way, I must have my phone with my to pay anyone.

(ii) Secure ID Number Generators for login
HSBC in Australia and HSBC Commercial in the UK use these. A new number is generated every 60 seconds, and I need to supply my ID, password and this number to login.

Fraudsters won't target banks which use either of these (a,i or a,ii) , they either can't get in, or can't do anything once they are in.

(b) NEVER, EVER click on a link in an email for Internet Bank, for changing password, or for reactivating an account, or any activity. Go through your banks website directly, or ring them.

(c) Change banks to someone that DOES provide good access.

For the banks

Where do you start ?

First - Secure internal Access

Perform a full audit of all access methods to the "customer data" at length. This will show holes on how data can come and go, and don't think that just stopping USB access will prevent a user from getting large lists of data out. Where there is a will, there is a way. a quick UUENCODE and a Cut'n'Paste to a pastebin.ca will dump data out in next to no time. If that doesn't seem plausible, what about some steganography ?

Second - Audit Access

Goes without saying, but you would be suprised. Audit all access to the data, so that if data does get out, you know when and by whom (hopefully).

Third - Provide Decent Security Features

It is almost ignorant for banks to NOT have a decent security feature such as SMS verification of Number Generator Fobs for use. They can tell us all they like what NOT to do, but really there are technologies out there which now prevent this type of attack from working, so just get to it an implement it.

There is MUCH more to be said on this topic, but that will get you thinking for now.

Linux - Ubuntu RAID, Modules, my NAS and Fun

2007-12-23T11:13:00.001Z

Ahh to be minimalist.

It all started over 3 months ago. I have taken to decluttering our house, a personal goal. (I should have set a KPI or two). One task which leads me down this path is digitizing my music and movies.

Many years ago, Sara and I decided to never rent another DVD, but to buy them. We don't buy the latest, unless we "really want it" but will buy the DVD's on special etc, or ex-rentals. Needless to say, from the past 3 years of DVD buying we have a very impressive collection, tipping the 200 mark. Digitising this lot of DVDs has caused a ripple into the Linux world. As you can gather, it takes up lots of space.

To start with, I got rid of some old servers, but kept their hard drives, as my goal is (has been) to setup a home NAS (Network Attached Storage) as the central location for all CDs and DVDs.

So, sitting in front of me is the glorious resurrected "icemaster" (i have to find my chroming again). I have the usual mechano (tip for extending drive bays) and the following drives. 40G x 3, 80G x 3 1 x 160G and 1 x 200G.

My layout is essentially to have a 40G RAID1 / and the rest in LVM as RAID5 devices (split up by like sized partitions).

To have so many drives, of course I needed to extend the IDE channels. (not upgrading to SATA yet.. maybe another few years). I purchased off eBay (new) 2 ITE 8212 IDE RAID cards. The Linux module these use is the it821x or, in the newer Ubuntu 7.10 series, the libata based pata_it821x module. This is where the fun begins.

The card supports RAID0, RAID1, RAID1+0 and JBOD, and just a standard IDE bus. I wanted the latter as I am doing all the RAID in software using the mdadm tools. (very nice).

I have been bouncing around with a few hurdles, and I will detail them each in turn, and how I have solved the, I am sure someone will come up with a similar problem.

So to give a rundown on the hardware we are using.

It's a P4 1.8Ghz, 256M RAM (yes is low spec, but it doesn't need to crunch galaxies, just serve files).
I have a Wireless RT61 based Network Card. (used to always user the rt2x00 serialmonkey module.
There is the standard USB and AGP display card, and an extra 10/100 NIC (8139too if I recall).

Problem #1 - noticing the sd[*] instead of hd[*]
So when I first dived into installing Ubuntu 7.10, I noticed that my IDE harddrives were now appearing as /dev/sd[a-z] drives, instead of the older /dev/hd[a-z] devices. This is now because of the new libata subsystem, supporting PATA and SATA (and SCSI ??) drives. This had a little flow on effect, but largely okay. (just a symantec changed, and I like it actually)

Problem #2 - Recognising the IT8212 PCI RAID Controller.
My first figured I would install onto one of the 40G hard drives, non RAID1 and standard swap of 2G and then setup my RAID and go from there. As I started before the IT8212 cards arrived from Hong Kong (yes it was cheaper then walking to the hardware shop), I actually setup a 150G RAID on booting 40G partition and transferred all my movies and music (and Gigs of data) onto the RAID array. This was on the 2 IDE buses, ie, no CDROM (DVD) after the initial install.

.. then the cards arrived.

The IT8212 by default, in Ubuntu uses the pata_it821x module. In older documentation people have asks some odd questions of this device, but it all seemed to centre around it being used as a RAID controller, ie doing RAID in hardware (good) , but I am using RAID5, which it didn't support.

On first boot, only the primary drives showed up. ie, if the it8212 had a 80G,40G and 200G,160G. Then linux would detect the 80G and 200G only. (odd I thought, must be config).

So into the BIOS of the card, and changed the config to IDE, from RAID (stripe).
Come back out, reboot Linux. no change.

Problem #3 - detected order of the /dev/sd[a-z] changed
Now, I'll pause for a moment from probem #2 and describe #3. when the drives were plugged in, my two boot drives (2 x 40G) on the first IDE bus, were moved from sda and sdb to AFTER the detected drives on the PCI card. For example.

before the IT8212 is plugged in the drives were
/dev/sda (40G)
/dev/sdb (40G)
/dev/sdc (DVD Drive)

After plugging in the card (with the problem #2 still occuring) you see the following

/dev/sda (80G)
/dev/sdb (200G)
/dev/sdc (40G)
/dev/sdd (40G)
/dev/sde (DVD Drive)

Now, the fix to this problem is possibly two areas. One is more robust, the other I had NO success, but new it would work if I persisted.

Solution to Problem #3
When Ubuntu installs, it uses UUID's in the fstab and grub to refer to a harddrive instead of using the known (possibly changing) device location. Let me explain as I see it. A UUID is a Universally Unique Idenitifer and can be created and used as a ID for something. We use them in software / computers all the time. In this case, the Harddrive has a UUID. I don't exactly know who creates them for the drive, but they do exist. Actually, it's the partitions of the disk that have the UUID that we are interested in, the UUID of the harddrive (I assume there is one) is not that important.

So, to make it so it doesn't matter "what" your root drives come under, you use the UUID everywhere to refer to it. that way, when it changes (ie moves, unplugs on the bus etc) it will still be a known device. To locate the uuid for a partition, just use vol_id

once you have the UUID, pop it in the fstab and grub.
/etc/fstab

UUID=897ef-ab67c6-7785-abcb2 / (etc)

and /boot/grub/menu.lst

Now once I had tested that I could (re)boot back in regardless of the "sd-ness" of the drive, I could go back to solving problem #2

A little better (I feel) than the UUID=... is to use LABELs .. this seems to work nicer. But it could clash if you later add a new drive from another system which has a matching label.

So make it easier for me, I just prefixed the label with the short version name of the server. so

icem-root
icem-swap

Getting initramfs to recognise my LABELs (and the UUIDs) was a tad hard.

Solution to Problem #2
Problem #2 was actually simple. and I learnt something too.

The PCI RAID controller had to be told, via the module to load in passthrough mode so it was recognised as just an IDE bus and not a RAID controller.

With the change to libata, the it821x module changed names (and code) to pata_it821x. I read some places about the noraid=1 option for the module. So I tried a few things. Firstly.
if the module (pata_it821x or it821x) is compiled into the kernel, and not as a module, then getting this paratemer into the module is via the kernel parameters (ie, in the /boot/grub/menu.lst)
so .. kernel image ... ro it821x.noraid=1 ...
note the '.' dot in between the noraid and the module name.

I intially got stumped between the pata_ and the (sans pata_) other module.

the pata_ is the newer module, so for Ubuntu 7.10, the it821x doesn't exist (isn't compiled). but I also worked out that the "kernel" line is NOT for modules loaded on the fly.

so .. after I realised that my 7.10 had no it821x, i tried kernel.... pata_it821x.noraid=1
This would have worked had the moduled been compiled into the kernel. not to be.
So .. it was actually meant to be done the way I knew from way back when days.

/etc/modprobe.d/options

option pata_it821x noraid=1

That makes the it821x show the drives on the bus as "just" drives and NOT for linux to detect them as RAID drives. YAY! they all showed up.

Nokia N95 8G, iSync, 3 (Three), music, internet and my Mac

2007-12-17T00:47:00.000Z

So, recently I have switched over my phone service to 3 (three) for the sole purpose of being a roaming IT worker. Since beginning of October I have found myself without internet access in a lot of places, so I switched to being mobile using three's Internet plans.

To do this, of course it necessitated the phone upgrade (actually, my previous phone was a good 3 year run, the shoicking iMate K-JAM (don't buy)). so .. back to Nokia I went.
The Nokia N95 8GB is the upgraded version to the N95 with the major difference being (for me) all the N95 stuff (software etc) didn't work.

So .. what I wanted was to have this N95 as my modem through bluetooth. To be able to sync calendar, pull photos off it (5MP camera, nice) and to have some stuff to listen to such as my usual Java Posse).

TO do this, seemed simple enough, but there were a few hurdles.

First. Internet access, the most important.
Turns out I had to locate som different modem scripts.
The following was my source of succcess.

<<==-- Internet Access --==>>
Download the Modem Scripts from Ross Barkman (thanks!)
You want the ones labeled Scripts for Nokia 3.5G (HSDPA) phones (26kB): Nokia 3.5G Scripts.

Once you have put them in place ("/Library/Modem Scripts")
you will have three new scripts to use. Nokia HSDPA CID[1-3]

Pair you phone through th Bluetooth preference panel.
Once paired, go into the Network Preference Page.

The following settings are what I have used.

[PPP]
Service Provider:
Account Name: 3netaccess
Password:
Telephone Number:
Alternate Number:

In PPP Options, the following are "ticked"

[v] Disconnect when user logs out
[v] Disconnect when switching user accounts
[v] Redial if busy
[v] Send PPP echo packets
[v] Use TCP header compression

[TCP/IP]
Configure IPv4: [Using PPP]

[Proxies]
** no proxies configured (i don't use any)

[Bluetooth Modem]
Nokia HSDPA CID1

I have unticked the following

[ ] Enable error connection and compression in modem
[ ] Wait for dial tone before dialling

Now that that is all done, see if you can connect :-) the "3netaccess" account name, I found that from the following link

<<==-- iSync--==>>
So, Internet Access was probably the easiest (1/2 hour research).
iSync proved to be a good week battle of trying many iSync plugins until I happened upon the only one that worked which was this.

iSync Plugin for Nokia N95 8GB

Thank you Jussi Edlund!

With this iSync plugin installed iSync works.

<<==-- Music and Pictures--==>>
So, Nokia, they have this "almost" right .. the camera appears in iPhoto, but the phone doesn't appear as an iTunes "player" (probably can't do that).
But anyway .. from Europe, download the following ..

Nokia Multimedia Transfer

Hope that all helps someone!
Have a great day.

Firefox Files (Crap) on the Desktop on OSX

2007-09-18T01:02:00.000+01:00

When you use Firefox and OSX, you will come to realise a VERY annoying feature / quirk of it's file management.

If I download a file, I can choose where it goes (change preferences to set this feature) and that is all good, however, if I click on, say a PDF, to view it (and say "Open with Preview") then the file is downloaded, wait .. on my desktop, and left there. After sometime, you will notice a heck load of files that you have viewed, Excel Spreadsheets, PDFs, word documents, mp3 etc. and they just make a mess of what is supposed to be a clean machine (clean lines, clean desktop :-).

So there are two solutions to this problem.

1. Changing the directory

Firefox saves these files based on the "setting" in Safari about where files get downloaded to.
So the quick fix here is to open up Safari (never used it, except for downloading firefox :-). Go to preferences, go to the General tab and change the download directory to somewhere other than the Desktop.

Now, I actually changed it to /tmp/ (I actually have a sym link in my home directory for tmp (always have done this) .. so firrefox now dumps all these "saved" files, to /tmp/ and /tmp gets purged every shutdown/reboot.. yay! No more mess.

You could of course, make a directory in ~/ called Downloads, and then you can peruse and remove at your leisure. Might be handy to do that when you needed to "review" a file you viewed whilst "online".

2. Tell Firefox to delete these files when it shuts down
In firefox about:config, you need to add a new config parameter that tells firefox to remove these files when it shuts down.
The key is app.helperApps.deleteTempFileOnExit and needs to be a boolean.

I haven't tried this way, but don't like it much because it means, if firefox crashes, or the mac crashes, firefox won't delete the files for that session. And also, my firefox is often open for weeks at a time, never closed, so the files will still grow in number.

Of course, you could combine the 2.

GWT Security Concerns - Object Injection

2007-08-02T14:14:00.000+01:00

GWT opens a new way of building webapps with it's different AJAX development method(ology) and it also opens a new area of "hackable" software.

I will term this new area "Object Injection".

GWT makes the AJAX development easier as the code is written in a type safe language (Java) and "compiled" out to JavaScript, dealing with all the browser inconsistencies and nuances behind the scenes. You still have the ability to write JavaScript via a Java Native Interface (JNI) and "look and feel" is done through CSS.

The process of writing a GWT web application is to write your GUI web interface or just parts of it in Java , and using the GWT compiler (a java app) compile your Java to JavaScript.

What the GWT does, is trawls your code and converts what it finds to JavaScript. So if you use ObjectA as a POJO to make changes to a Drop Down Select Box (for example) then this ObjectA will be exposed (someway) as JavaScript.

One of the niceties that the GWT Java-to-JavaScript model brings up is that your direct domain model objects can be worked on at the front interface without the need of DTOs, a code/time saving benefit.

So .. code might look something like (simplified by not showing the Asyn Callback()

MyDomainObejct obj = finder.findDOByID(myId);
myTextBox.setText(obj.getName());

Now this code above would be compiled to JavaScript and, even with the GWT J2JS compiler set to "DETAILED" and not OBF(USCATED)" you get some pretty hefty JavaScript.

This is some real JavaScript from the GWT in Detailed Mode, showing a "BusinessEntity" object. (a POJO if you like)

function pkg_BusinessEntity(){
}

_ = pkg_BusinessEntity.prototype = new java_lang_Object();
_.java_lang_Object_typeName = package_pkg_ + 'BusinessEntity';
_.java_lang_Object_typeId = 4;
_.pkg_BusinessEntity_abn = null;
_.pkg_BusinessEntity_commencementDate = null;
_.pkg_BusinessEntity_isCurrent = false;
_.pkg_BusinessEntity_isRegisteredForGST = false;
_.pkg_BusinessEntity_name = null;
_.pkg_BusinessEntity_postcode = null;
_.pkg_BusinessEntity_registeredState = null;

So, where is the security concern ?

Well in some cases, such as most of my world, the domain model is fairly significant, with trading / transaction objects and user accounts with passwords, and money and settings etc etc.

So if, in a GWT class, someone were simply to expose the DAO (Data Access Object) service such as

service.storeObject(myUserObject);

Then, essentially we have, assuming there is no or not sufficient checks, a direct way to store objects into the domain (database) from the browser.

I will call this "Object Injection" in the same vain as "SQL Injection". An Object injection hole is could really be termed as a

Poorly Coded RPC interface which does not check values and who and why of what it is "doing".

Of course, finding an Object Injection hole in a GWT application is going to prove fairly complex to do (but surely not as hard as searching for a buffer over/under run). The types of tasks that a would-be inquisitive person might have to do is as follows.

De-Obfuscate the javascript to find the "meaningful" objects that are being worked with
Dump out all the "Service" methods and try and tie these to where they are being used in the GUI (Web Application).
Identify the types of objects which would provide benefit of "changing", such as Bank Accounts, Limits, restrcitions, user account saving (like storing settings etc) and locate how the interface is working with the RPC method calls
Write some Javascript to create a generated GWT object .. and send it to the RPC API.

The code that you write in Java to be compiled to JavaScript, is now available for all to see.
Anything you do, can be repeated, out of step with simple JavaScript calls.

If parts of the domain model are exposed, this is a good place to start hacking.

The problem is no different with current non-GWT AJAX development. It is just that GWT does a lot of smoke and mirror compiling for you and if you unwittingly create holes in your Java RPC methods (without sufficient checks), then those holes are exposed for all to see (and play with).

So what techniques can we use to prevent us making such a holes. The following would be my quick and dirty set of rules.

1. Don't expose the domain model to the GWT compiler.

Why ? Well for a few reasons

(a) It will expose your internal workings of the application to the browser, and this is one place where the would-be nefarious person can start looking to analyse your application to understand its structure, thus weaknesses. You might think, "Gee, that's a lot of work when this JavaScript is obfuscated", but think for a second, is there anywhere in you application that one can profit (monetarily or otherwise) from your application, giving them an incentive to start the work ?

(b) To expose a Domain Model Object, means you probably want to "change" and store it, which means the DAO service (or similar) will have to take an object and save it to the database. Simply, find that DAO Service and the right object that it requires, and I can then start poking in (Object Injection) other object values to see if I change some stuff at the server.

2. Have Users / Accounts and Password - and Audit them
If you use usernames/passwords and sufficient checks in the right places you will have the ability to log who is doing what. Another way to think of this is, minimise the amount of GWT you use to people who are not authenticated against your application.

A way that I always achieve this these days is to use Acgei Security with Spring and webfilters over the whole application. This way I know which users are hitting which RPC methods and could audit appropriately.

3. Expose ONLY that which needs to be - Use DTO's, simple APIs, push "work" to the server, Separate Projects.

Use DTO's to push and pull your data back and forth. This way you can check the DTO values before storing and also know what is being exposed at the front end.

Use Simple API's - Make you RPC methods simple (ie, don't push too much back and forth). This is probably just a KISS principal and will help in making sure you don't inadvertantly expose that which shouldn't be.

Move as much logic as you can to the server. This ties in with the previous point. A simple API logically means that the complex stuff is on the server. This way, code which does not need to be seen, is not seen. Plus the burden on the browser *our unknown environment* is kept at a minimum (which has an added bonus because we don't want the help desk of our application getting complain calls saying the application slows down their browser thus tarnishing the "company".)

Separate Projects - One method I am using in my GWT apps is to have a separate project for GWT compiled code, for server (RPC and JSPs etc) and business / domain model. I use Maven 2 and a parent project compiles and assembles all 3 pieces together. So three projects, but ONLY one is the GWT compiled code, so I know that, anything potentially that goes in this client project, is going to be in JavaScript. Anything OUTSIDE that project, is hidden from the browser.

4. Secure all RPC interfaces
Use Aspects or Filters (I use Acegi Security) to secure your GWT RPC methods. You will be able to lock down the method calls to the user and also ensure that the "objects" are allowed to be transported that are.

I hope this little entry helps people see the potential issues and raise awareness in their designs about this potential security "feature".

It is nothing new, of course, but the way in which GWT development is done, can have an impact on the "bad applications" we write if we are not careful.

GWT Fun

2007-07-25T18:52:00.000+01:00

Over the past 4 weeks I have been building an impressive list of what to do an what not to do when it comes to GWT. In my last post I mentioned an issue with JSP 2.0 and GWT.

Tip #1
Tonight, I came across a very similar issue with IE6. (might be IE7 problem also).

the Script tag which you include the GWT module with looks like

<script .../>

But , if that is how you do it, IE won't like it. In fact it will render a big blank page with no error and "Done." down the bottom.

You won't even see an ounce of the "page" even if GWT is not the primary content.

The quick fix is to change the single script tag to have an open and close (and put some space between them).

That took all of about 20 minutes to discover.

So far, my verdict on GWT is 80% for and 20% against. I am certain the 20% against, ie things that I can find I don't like, are because I have not yet worked out the "good" or proper way of doing things.

Tip #2

Some people have mentioned about needing the rpc interfaces in the client package and that they don;t like it. Well I didn't like it either. I have an rpc package.

I found a sneaky (smelly) way of just having arbitrary "code" for GWT to find.

Create a "BaseGWT.gwt.xml" and point the entry-point class at an empy implementation of EntryPoint. But in this gwt.xml, set all the source folders that you want.

Then in the "other" modules, inherit it. Now, the compiler (maven2) complains and errors, but because of a bug in the maven2 GWT plugin, it keeps going (and doesn't fail) and the other modules happily inherit this dummy GWT module and all the trappings (other source folders).

When I find a better way of doing this, I'll holler.

GWT and JSP 2.0 (Oh the Pain!)

2007-06-20T02:54:00.001+01:00

So it's been four weeks now and I'm moving along at a steady pace with GWT.
I found by shear accident that a new feature added into GWT 1.4 is that you don't need the gwt.js anymore or the <meta..> tag in the HTML and you can just include the module.nocache.js file directly. (good change guys).

I have been trying to host my modules (GWT modules) into an already existing JSP page that is written in JSP 2.0 of the XML variety. I find these good because it means you are forced to create valid HTML markup. Anyways, combining the two, GWT and XML JSPs meant I was completely unable to get the GWT modules to attach and appear on the page.

I was getting JavaScript Errors, j.write is not a function. Now, the GWT compiler is compiling by default using the OBFuscation setting. Which means that the JavaScript, is well, obfuscated, a little hard to read. Not knowing yet what the other settings are to change it (I think there is detailed and something else I saw) I dove (breifly) into the JS to see what j.write was all about.

It took a bit to track down the problem.

Essentially weird things happen to your markup, it gets all compressed to one line (obfuscated HTML) and open, no content, close tag elements get changed to /> elements to make it more valid XML. For those that have worked with JSP XML documents (ie, the <jsp:root> ...</jsp:root> variety) will know what I am talking about.

The problem is that a jsp marked up this way just fails straight away with GWT in the works. I recalled a similar problem I had two years ago that was making javascript fail because the <script> tags written in the jsp as open <script> and closing </script> with no content in the middle (ie, a <script src="somefile.js"></script>) will get collapsed in the browsers view of the HTML as <script .. />. ie, just one tag.

To get around this JS error, back then, was to add a CDATA and put a comment delimeter in the CDATA. This way the JSP XML parser would preserve the open and close tags.

This looked like the following ..

<script src="somfile.js"><![CDATA[]]></script>

Which will bre rendered correctly as

<script src="somfile.js"></script>

And the problem, can't remember what it was, but javascript loading issues, would work.

Alas, this was not my problem with GWT as I changed the <script ...></script> and the <iframe.../> to have this CDATA hack and it didn't resolve my j.write is not a function error.

So .. the quick simple test, cut and past the output of the JSP that has the GWT <script></script> in it, and put that as an HTML file and try that. So, refresh my broken page, view source, cut and paste the ugly format (hard to read for others) HTML and put it into a new .html file and load that, it worked.

So I have two files, tester.jsp which has the <jsp:root> top and bottom and then a html file which is the "output" viewed from the browser of this jsp file. So as far as the browser is concerned, hitting the /tester.jsp and /tester.html renders exactly the same looking HTML.

Which lead me to only one conclusion, it was not "format" of the HTML presented that the browser was barfing on, (and then GWT not loading) but rather the Headers that the server was sending that caused the problem.

Loading up firebug (just clicking my little firebug icon in firefox.. thank you thank you firebug authors), I inspected the headers of the tester.jsp and tester.html, low (lo) and behold,

tester.jsp : Content-Type: text/xml;charset=UTF-8
tester.html: Content-type: text/html

This then leads to the quick fix, change the content type for my JSP.

<jsp:root version="2.0" jsp="http://java.sun.com/JSP/Page">
<jsp:directive.page contenttype="text/html; charset=UTF-8"/>

And then all is good.

So, what is happening here ? It would appear that the browser (Firefox 2.0 on Mac OSX in this instance) is not creating a DOM that the GWT javascript can interact with when the content type is XML as opposed to when the content type is HTML.

This is evidenced by the line in question in the GWT module.nocache.js which is

j=document;

And a few lines in,

j.write("<script>...</script>")

From this called attempt to write out some dynamic javascript to the page, the specific error message obtained is as follows

j.write is not a function

Which just shows that the DOM is not how it is supposed to look. I am sure someone from the firefox community could explain that one a little more. (to how it structures the DOM based on content-type).

So this one was not because of GWT pain, but JSP pain :-)

All good so far. I have a few more improvements to the m2 GWT plugin which I will post in the next few days.

GWT and Maven 2, OH the pain!

2007-06-09T17:30:00.000+01:00

GWT - Maven2 and Eclipse .. Ouch!

Using GWT for all of 3 weeks now, I am finding it is nice. It's logical, well thought out and simple.

But, I think there is a little way to go when it comes to using some de jure standards. I am a big Maven 2 fan because in an Enterprise (captial E for 'ooh' fancy) environment, standards go a LONG way and Maven 2 dictates a nice and well understood way of doing things.

So, what's me beef with GWT ? Simply this, gosh it has been hard to get it working with Maven 2.

Now, part of this is because I am still trying to get my head around how it works. I have been using the applicationCreator script to setup a quick hello world sample, then move that into the src/main/java and src/main/resources folders.
But what does that do to my src/main/webapp ? where do the images get pulled from ? why don't index.jsp load in the GWTShell (when it's tomcat underneath) ? So many questions.

So, there is a maven 2 plugin. I am currently using the 1.5.2 version of it.

Google Code Homepage : http://code.google.com/p/gwt-maven/
Maven 2 Plugin Doc : http://gwt-maven.googlecode.com/...
A list of some problems : http://code.google.com/p/gwt-maven/wiki/FAQ
Maven Repo : http://gwt-maven.googlecode.com/...
SVN Repository : http://gwt-maven.googlecode.com/.../

and as it turns out, there are a few bugs and features which need some work. (I wish I saw that FAQ page first before diving in)

The first issue I cam across was the "plural" vs "singular" naming convention. In Maven 2, a configuration item for a plugin that requires more than one value is pluralised .. ie

<options>
<option>...</option>
<option>...</option>

</options>

What's the problem ?

Well, the com.totsp.gwt plugin needs a configuration line like

<compiletarget>org.sobbo.ui.Home</compiletarget>

However, this config value in the plugin source needs an [] array or targets. (logical yes) but m2 plugins, dictate that the property name is then plural (compileTargets) otherwise you get this kind of error message.

[INFO] Failed to configure plugin parameters for: com.totsp.gwt:maven-googlewebtoolkit2-plugin:1.5.3-SNAPSHOT

(found static expression: 'org.sobbo.ui.client.Home' which may act as a default value).

Cause: Cannot assign configuration entry 'compileTarget' to 'class [Ljava.lang.String;' from 'org.sobbo.ui.client.Home', which is of type class java.lang.String

so .. the AbstractGWTMojo has to change to have a plural of configurationTargets (there must be a way else how would they be using it right now ?

It turns out there is an issue logged for this .. http://code.google.com/p/gwt-maven/issues/detail?id=37

Anyways, what's the next pain ? well to checkout the src from the subversion repository, you get this nice little gem.

$ svn co http://gwt-maven.googlecode.com/svn/trunk/maven-googlewebtoolkit2-plugin/

svn: REPORT request failed on '/svn/!svn/vcc/default'
svn: REPORT of '/svn/!svn/vcc/default': 400 Bad Request (http://gwt-maven.googlecode.com)

BANG, and there is my pain. This looks like a straight up Google SVN problem. Funny enough, I have an SVN downloaded script that works to "download" src from SVN repositories using wget.

See the end of this post for the shell script.

So .. my third issue ? Well there was some unusual "Java Execution Mojo Bootstrapping" stuff going on, and it's broken. Essentially when trying to launch java to compile (testing, shell etc) it (the GWT maven plugin) couldn't find it (java) and died. I found the problem (after downloading the src and corrected it) and logged this issue with a patch @ GWT M2 Plugin Issue 43

So .. is it ready ? This Maven 2 plugin really needs to be working, maven 2 is big and GWT / M2 / Spring / Eclipse is pretty important (to me at least).

I'll keep battling with it because this way I can probably make it a little better. I hope that it gets better.

So, that SVN downloader script, it's not completely "safe" because the revision could change by a third party part way through your download, but hey, it works.
#!/bin/sh
# Author: Ramon Buckland ramon#at#thebuckland.com
# Quick hack script to pull the latest SVN Revision version from a repo repo when you have no SVN tools
# or SVN through ISA proxy servers are just not working ..
#

function usage {
echo "$0

echo "svn-url: SVN URL is a URL to the trunk or a tag you interested in"
echo " eg: http://svn.apache.org/repos/asf/incubator/servicemix/trunk/"
echo "product-name: A short name of the product so that we can create a directory for you"
echo " eg: servicemix"
echo "(also, set the http_proxy=http://hostrunning-ntlmaps:port)"
exit
}

if [ "X$1" == 'X' ]; then
usage
fi

if [ "X$2" == 'X' ]; then
usage
fi

if [ "X$http_proxy" == 'X' ]; then
echo "WARNING: http_proxy is not set. Do you need it ?"
fi

SVN_REPO=$1
PRODUCT_NAME=$2

mkdir ${PRODUCT_NAME}-svn-pid-$$
cd ${PRODUCT_NAME}-svn-pid-$$

# nv=non-verbose
# nH=noHost directory created
# -np=no ascend to paremt dirs
# --cut-dirs=3
# -erobots=off .. don't look at robots.txt to see what you are and aren't allowed to do
# -m mirror

wget -nv -nH -np --cut-dirs=3 -erobots=off -m ${SVN_REPO}

NEWDIR=`grep Revision index.html | grep h2 | cut -d':' -f1 | cut -d'>' -f2 | tr ' ' _`
cd ..
mv ${PRODUCT_NAME}-svn-pid-$$ ${PRODUCT_NAME}-svn-${NEWDIR}
find . -type f -name index.html | xargs rm

Hope that helps someone. I use it a bit here and there.

Switching Domains

2007-06-05T23:48:00.001+01:00

In my new role I have been pouring my brain into a new domain (industry) for the past 2 weeks.

For the years I have been writing code, moving from one language to another has never a big issue. ie, from Pascal to Modula-2, C to C++, Perl in the middle, Java from the dawn of ages, dabble in C#, poor over XML and become a quasi-guru in XSLT's. So when a new language comes about, it's a matter of learning a new language construct or few (expansion of the language domain) and then give it a crack. Being bi-lingual is not too difficult. It is probably because not a lot of things have been written under the sun recently that bend the brain.

For the most part, the "spoken language description" or nomenclature of the languages is the same ie: objects, methods, procedures, closures, fucntions, instance variables, parameters etc. So only when someone bakes up a new "thing" and gives it a name, do you have a learning exercise. This is to exclude the obvious of learning a new feature or third party system which is ever-on-going.

For those who have been coding for many years in many languages you will know what I mean. It's fun looking at new languages and it's not long before you are proficient. For ths most part, coding in another language is only slowed by the environment in which it has to operate, so OS, execution platform (JVM, native) the IDEs, no IDE, dynamic, compiled etc.

So .. how about switching the domain in which you are coding for? This, I have not done often. I have currently side stepped from funds management and trading over to insurance. I knew there would be challenges and yes there are. Insurance is another whole industry unto itself, there certainly are links and in the FM and Trading game you come across these links a bit.

It's not so much the concepts of insurance, they are easy, or easy enough, but it's the nomenclature that is used and when it applies, and who owns what. My view point is that I MUST be proficient in understanding the business before I can really code. Sometimes you won't have that luxury, perhaps due to time pressure. But if you work for an ISV or interact with a department, not being able to speak their language gives you no credibility and also makes you untrust worthy. The business domain experts will instantly pick up on this.

I am studying insurance like it's a new degree for me.

.. it's real Domain Driven Design back again (for me). It's fun. It's fun because it's forcing me to think outside the square and, as an outsider (at the moment) see how things work and then twist my brain around it to understand how it hangs together.

A policy, quotations, indications, endorsements, declines.. the list goes on. and to write a system I first have to understand the domain.

It reminds me of 7 years ago when I was first cutting systems for fund management. What a battle in the brain that was. I am finding this new domain a bit easier now than then. Perhaps because I know the right types of questions.

I think working with business people who are amenable to your plight (lack of knowledge) helps, of course :-)

So.. I encourage anyone who faces new domains, often or not often, study it, learn it. You will be noted as a good developer simply because it shows you actually do care. And, BTW, this goes for not only coding, but Systems support and DBA's and everyone.

We are like Vets, we have to know a lot in order to do our job.

Eclipse Mylar ... Wow!

2007-05-26T03:31:00.000+01:00

So, Subclipse is installed. I noticed (well was forced to notice) that Subclipse required Mylar before it would install. Okay .. dependency yicky, but, quickly sorted out by installing Mylar.

But, as Always, I MUST know what something is (which is a painful character trait as it does get me distracted often) and Mylar quickly became on top of my list. (Talk about being distracted, at 9:50am today (Saturday) I was starting the day for prototyping an idea I had last night at 2am using GWT, and now I am blogging, not yet into my prototype).

So, where was I.. Mylar, Mylar after it installed in Eclipse actually prompted me to look at the Getting Started Webinar, which was good because I was just about to minimise eclipse and go looking myself.

Mylar loaded a browser in Eclipse, without hassle, and on the Mac (love it when things work).. so I went along for the journey. It's a 40 minute presentation by Mik Kersten. All I can say is wow. It truely is something to be seen, even for small shops you should be using this stuff.

What is it ? Well you really need to see the webinar, but in summary it helps you do your work by showing you the things you are working on (automatically) and not involving you in the stuff you don't need to see right now. Eclipse has excellent integration hooks into the whole environment for controlling stuff (views and issues and notes etc), and Mylar has just gone one step further and controls all of this for you based on issue or tasks that you are working on (which you choose). So what you see, is only what you NEED to see, and it has the quick flick buttons to show all if you know you need to see something, and then, once you looked at that item, its now in your "filtered" view (if that's an ugly term for it).

It is far more than just filtering too, because integration into the Issue Tracking software is built in for Bugzilla, Jira and Trac which means team collaboration and project deadlines are at your fingre tips to make you more productive.

Really, watch the webinar (link is at the bottom) and you will see what I mean.

Here is (Ramon's dodgey list) the QUICKEST way to get the most out of Mylar.

Install Trac (Bug tracking software) somewhere in your org (or use Jira if you can it's GREAT!)
Install Eclipse 3.2.2
Setup Subversion
Install Mylar then Subclipse (the Subversion plugin for Eclipse)
.. and watch this webinar

In my previous employment, issue tracking was managed by Tracker (Merant) and now the newer version TeamTrack. The benefits for Java Developers using Eclipse with integrated source repository and issue tracking software is just brilliant, and it's complete dark ages if you aren't. (now that I have said that I will have to practice what I preach hey).

Seriously, watch that Webinar and you will see why I have just become so excited. If anyone has worked on a project that is big, and you have to collaborate, it, Mylar, MUST be a MUST. If you aren't on a big project, then use it anyway because

You are probably WANTING to make it big and you might as well start with a good foundation; and
A good practice of coding ONLY leads to good quality code which relates to more time with your family and THAT is important.

Netbeans users, I see that there is something called ALM (Application Lifecycle Management). Sorry NB peoples, I MUST not get distracted, I might look at it another day. Someone can surely comment on the differences between Mylar and ALM. Not being a big Netbeans user it's lower on my priority list, sorry.

Back to Mylar, what's the name deal ..

Mylar is a term oft heard around the spacey dudes that watch the sun.

and Mik explains in his webinar

Mylar for the Sun - Aluminised film used to avoid blindness when staring at an eclipse.
Mylar for Eclipse - Task focused UI to avoid information blindness when staring at Eclipse.

So in closing

To the Mylar Team

Brilliant tool guys and what an idea. A product feature like this deserves an award and it shows that you have thought long and hard about what you wanted it to be.

And .. oh, what a fantastic name for your project .. I absolutely loved the definition.

Good Job!

I wish there was something like mylar for my web browsing which would have removed the crud I waft through daily so that I would have seen Mylar earlier (like 7 months ago) :-) Excellent job Mylar Team.. just brilliant.

References
The Mylar Webinar - https://admin.adobe.acrobat.com/_a300965365/p46246963
A Podcast from EclipseCon 2007 - http://www.eclipse.org/resources/resource.php?id=366
Netbeans ALM - http://www.intland.com/products/cb-download.html
Trac Issue tracking - http://trac.edgewall.org/
Jira Issue Tracking - http://www.atlassian.com/software/jira/
Subversion - http://subversion.tigris.org/
Subclipse - http://subclipse.tigris.org/
The Mylar Homepage - http://www.eclipse.org/mylar/index.php

Can I just say again .. Mik and the team, this deserves the biggest pat on the back of your careers. Fantastic!!! A Killer Plugin for an Already Brilliant IDE. You have made me love Eclipse even more... good work.

My old avatar, and my iPod

2007-05-25T18:12:00.000+01:00

So, following on from the previous post, I have "reposted" all the OLD blog entries 03 - 04 (not lots, but you know, something for the kiddies).

Anyway, I also found my old avatar. Not so much an avatar as it is me back 3 years ago. :-)

It took a while to repost. iPod is still trying to find it's bad sectors... the long story is that I spent $160 on a 60G iPod Harddriver from the US.. and it doesn't work .. ARGH.. THAT really frustrates me. I might just go out and buy an iPod and be done with tinkering I think.

The WayBackMachine (internet archive)

2007-05-25T17:13:00.000+01:00

Having recently moved my blog over to blogspot, I have wanting to "retrieve" all my old blog posts. Some are just historical and one is just funny.

In irony of a good IT professional, I was unable to find my blog backups, so instead, a waking (6:55am) thought prompted me to look on the Internet Archive (aka The Wayback Machine).

Of course, the posts were there. (this story has a good ending). So I am now proceeding to copy my old posts over to "here". For those bored, you can take a look at the RSS feeds and see which "old" posts have "modified" dates of 2007. (told you it's for the bored).

One post in particular that I didn't want to copy verbatim in is the following..

Wed, 9th Apr 2003

web load testing tools   [web development] -  ramon - ramon@thebuckland.com @ 09:09:11 

Here is a list of Web Testing tools that I have found
Comments on the ones I have used:

Siege
Open Source

OpenSTA
Open Source

MS Web Appplication Stress Tool (MS WAST)
Free for Download, Usage Persuant to EULA

Load (by PushToTest) - No longer Maintained

TestMaker (by PushToTest)

So I have copied it in because, technically (or chronologically) it is my first public blog posting. Bit boring, but just hilighting it's contents, testing has well moved on from here now.

Two items when it comes to GUI Unit and load testing comes to mind (actually 3).
First is that with (Google Web Toolkit) GWT, the unit testing framework is built in. (A plus of course).
Second is the mozilla initiative to provide a Java testing framework for web pages. I have not yet had the pleasure (possibly) of using it, but suffice to say it looks great.

Of course, the third is really just a note, that since this 2003 posting, I have hammered grinder, weighed jmeter even more and, .. the browsers are finally starting to behave like you want them to .. THANK you FireFox!!)

(as I type this I hear the horrid sound of
mke2fs -c -v /dev/sda

clicking away on my iPod because the HDD has completely died. .. Im scanning for the bad blocks on Linux so that I can recreate the partition map skipping the majority (with buffers) of the bad sectors. Let' see what I come up with .. (there is a MUCH bigger story to this, one which I hope has a happy ending).

MacOSX and .. and ..

2007-05-23T12:28:00.000+01:00

So it's been 2 days now that I have been using the Mac for my primary machine.
I have used Mac's for a while in the past, but primarily as test machines, or in my earlier days as a network admin, supporting users on them.

Now having officially switched, I can say I like it, certainly as much as I thought I would.

So, how odd is this, I have switched from, now Windows but from Linux. Now .. it's not Linux that I don't like. In fact I love it, but it's that as a development platform, the mac powerbook is just right.

I installed eclipse on monday and used it for about 5 hours today. I had some odd problems trying to install subclipse, and also the spring ide plugin.

I had downloaded 3.2.2 and Eclipse was saying that a particular version of some package was not installed. The odd thing was that it was the stable version of the plugin(s).

Because people all around me have been talking netbeans, I figured I would give it a good try.
I have never liked the concept that ant is at it's heart, especially since I have been a maven advocate for the past 2 years. Anyways, I'll see how it goes.

So far, I have followed the setup for a maven project in netbeans, and have seen how they hook goal execution into the ant subsystem. Not a bad way of doing it.

I created a fairly comprehensive domain model today (30+ classes) and I want to now see this as a model, so I couldn't install the Eclipse UML modelling (see above) .. which is why I am trying netbeans. Perhaps I will have better luck.

In theory the install for the UML Modelling went well, but poor old netbeans didn't register that the plugin was installed.

All is not fair in war. Some things are not as they seem (as they are described) .. but never get disheartened, there will be a way around or through this.

Desire vs Need - Part 2

2007-05-17T13:31:00.000+01:00

In building a proof of concept, a sound architecture is abstract from it's infrastricture. In this view, the project 014 (my codename) will be put on to an ODBMS and an RDBMS.

Now, being the good timesheeting developer that I am. I will break up my development time into framework, and then also time the effort for the data access layer, and then the time for implementing the datastore.

For this datastore, I will choose one for the prototype (RDBMS or ODBMS) and then as a research (non-official) will choose the other.

I will literally ask my magic 8-ball which one to start with.

I do expect that the ODBMS will come out trumps, but lets see.

So .. the need vs desire is simply that the I know RDBMS systems extremely well which is the "need" for time to market (prototype) a time thing) and the "desire" (see previous posts)
to use an ODBMS instead.

I actually expect that developing with the RDBMS will take longer than with the ODBMS, as quite simply, it looks simpler.

So what should I choose ?
Well .. you have to compare like fruits, so
db4objects embedded has to be compared with Hibernate+Derby
db4objects client/server has to be compared with Hibernate+MySQL.

I am NOT comparing speed, I will just compare effort. I will do that afterwards.

One of the many reasons some people jump to Ruby on Rails and other "X" fancy platforms is because of all this "crap" we have to put up with. (Ruby and ActiveRecords apparently does away with the Hibernate Crud..) it just still feels wrong. So, if we like Java as much as we say we do, we MUST do something about it. Sun will, but it will probably take them about 2 years longer than 6 months it would take "us" to make change happen. After all, we are the users who "sell" this stuff.

Desire vs Need .. ODBMS vs RDBMS

2007-05-14T14:36:00.001+01:00

I have been on the think track of not using SQL for sometime now and have completely come to th conclusion that it is entirely possible. Possible in the sense that a decent well founded ODBMS will suit my need.

I mentioned a few things that needed to be investigated in order for the switch to be solid.

1. Administration - So presumably, the chosen ODBMS (Server, Platform) will provide a level of administration, such as object browsing, log, db maintenence etc What evere the underlying "architecture" is, there has to be tools to see it, so it is not a black hole. One of the nicesties of an RDBMS is it is not a black hole in as much as SQL from any compliant tool gets you to your data, a plus.

2. Monitoring - How can we monitor this beast. I put this here because in my line of work, there are clear distinctions between those who build and those whome maintain and the maintainers want a system, that, well, can be maintained. So it's important.

SO, where is this desire and need. The desire, I want to switch, the need. The project I am embarking on does not have the funds to splash around and this puts it squarley into the hands of OSS for the chosen ODBMS. If you look around, it leaves only a few choices. After you weed the crappy and incompatible ones out, the clear winner is db4objects. But ahh, Here is my catch, db4o is GPL based, and you HAVE to code with it, ie, embed it. Which means that I can't use it, unless you buy their commercial license.

What is lacking is a non-viral license ODBMS to the quality of db4objects (Apache 2, Mozilla, LGPL) such that it suits the needs of this "commercial" yet cost constrained project.

So, there is my desire (ODBMS) and need (RDBMS). I must (need) have a running architecture with which to start.

I must make sure that this "platform" chosen (Hybernate, MySQL (PostgreSQL)) is "removeable" for the day (if) when a compatible ODBMS comes available, or the project at hand makes some money to warrant a change.

Hrmm, certainly has you thinking. I will look further into the db4o world. Here is a thought: if db4o were running as a server, in the SAME way that MySQL runs as a server, then there is no impact to your licenses. db4o is well within it's platform, wrapped in server code, .. hrmm .. cheeky. Let me take look at that.

So, the server code could be GPL'd next to db4o.

KISS - RDBMS and ODBMS

2007-05-12T10:24:00.000+01:00

Well, you can guess the title right there (Keep it Simple Stupid). Another title could have been (for controversey). Why the heck are we still dealing with RDBMS platforms and architectures ?
So living in the Object world of Java I have come in contact with many databases. Sybase, MSSQL, PostgreSQL, MySQL and some Oracle (in that order of exposure).

In each case I have had to deal with the RDBMS and the Relational Mapping layer (RM).

Experience has whispered to me that .. "there has to be a better way", but certainly until about a year ago, shy on stating that there has to be a better way. I have known about ODBMS's for as long as I have known RDBMS's but have never worked with one.

Certainly, legacy (and I mean that in the "ODBMS to RDBMS sense") platforms have to stay RDBMS and that is where OR mapping stays, but .. I want a simpler system.

I blogged some time back that my goal for the next project is to make it sans-SQL. Now why on earth would I want to do that ? Well for one, I want to reduce complexity. It's that simple really (Keep It Simple Stupid).

This is such a "hot" topic (many people have an opinion) that I am going out on a little (hopefully stong) limb and stating that RDBMS platforms are just simply getting long in the tooth and now is certainly a best time to move away from the RDBMS.

Why (why are you being so nuts, why move away from proven technology, why ?)
I have just finished listening to a very good talk by Ted Neward on TheServerSide
The Webcast Talk - Object/Relational Mapping and the Vietnam of Computer Science - Expert Webcast

This talk centres around the idea that when working in the Object space, we should just not use RDBMS's. For background reading, the comment of OR Mapping being the "Vietnam of Computer Science" is found here at his blog.

One of the good quotes, which aligns with my previous post is this.
Ted was talking about the "lure" of OR Mapping

"[OR Mapping has a]... selling point that says, I don't have to think about SQL, I don't have to think about tables, I don't have to think about manually unwrapping this stuff myself.. phew .. I don't have to worry about these issues anymore, and then to get bit by them .. at the 11th hour"

So true, with Hibernate, OJB, and JPA (etc etc), SQL is still there when you have to do complex stuff. It is still there when Crystal Reports or JReports or platform X comes into it.

I remember many a time sitting at night trying to tune that sproc that was mapped to my business service pouring over SQL, just to get my data right and quick. That HAS to Cost something.

There has to be an extraordinary TCO attached to using OO and an RDBMS, one which could be lost if we remove the RDBMS altogether.

Of course, there are problems, how to we make sure the ODBMS is quick, and efficient, and what do we do when we want reports ? (These answers I am going to find out)

Ted mentioned, rightly, that it all depends what is on top of the paltform, is it Objects, or relational Data. What is the priority of the platform ?

Back to the TCO. So If I have no DBA (only Operations looking after the "service" and "data" tier) and I have no need for SQL developers (if there are sprocs involved) and I don't have to worry about OR layers .. It has to mean that I have less to worry about, surely, which means that "time" to market, TCO etc is lower. It just has to .. It has to mean a simpler system.

This KISS principle, A very interesting WebCast that.

So where to from here ? I have decided that my projects will not use *SQL. So that means I have to choose an ODBMS platform.

Two issues that were rasied in the Webcast which are VERY real are the following.

- How do you administer the chosen ODBMS ?
- How do you monitor the chosen ODBMS ?

Very true, kind of a worry (possibly). We'll see.

So glad things have moved on...

2007-05-10T15:43:00.000+01:00

Having always been in IT for most of my (post age 13) life. I take notes, and lots of them.

Here is a snippet of some notes I took about 5 - 6 years ago. It's like my Own personal "change control" sheet. Before I even really knew what the heck a change control "form" was.

Oh, the icemaster is my "server" almost always. I have a gorgeous old 60's chrome that belongs on the back of a car, which is script says icemaster. Funny thing is I found it in the inside of the freezer box of a 60's fridge. It is
fantatic and has moved with me to all my servers over the years...

Why do I say things have moved on. Well now Ubuntu (and other distros just make this stuff so much easier). I don't need to worry about installing stuff to any great detail like you used to have to.

I remember sitting down for 6 hours with Adam Spann (Druss) and working on Slackware with was running kernel 1.2.28 in 1994. Gosh those were the days of 2 hour kernel compiles !!!

I don't think I have even compile a kernel for Linux in the last 18 months. whoa.. is that even happening, wait till I have kids, I might forget XML.

soo .. here you are .. snippet of 6 years ago

icemaster - P120 32M RAM .. lots of disk space to play with as well

2001 07 12
After much fluffing around and installing Debian 2.2r3
I have a working systems
To do .. install SCSI modules for the SCSI cards
Configure the snd card it's conflicting on IRQ10
so I need to get it off there at some stage.

2001 07 12
Install Xvnc (vncserver) .. gives us X windows vnc Access .. Neato!
tar it out
xmkmf
make World
...
cd Xvnc
make World

make install and make install.man
Edit the /etc/vnc.conf file and uncomment the fontPath settings ..
All is well .. now let's get some X interface happening

2001 07 12
Installed samba 2.2.1a with default options into /usr/local/samba and /etc/samba/conf
/var/log/samba/%m.log

Have 'make'd apache 1.3.20 .. it isn't installed yet.

2001 07 16
Installed apache 1.3.20
Attempted to connect .. no go ..
after a few mins ..
Did an netstat -an
Ahh there the bugger is .. port 8080 ,Must have been because I was compiling as a user
Ahh well .. fixed now.

2001 07 16
Setup swat
/etc/services ---> swat 901/tcp
/etc/inetd.conf ---> swat stream tcp nowait.400 root /usr/local/samba/bin/swat swat

Now .. killall -HUP inetd
DOH! .. bad username/password ..
Looked aorund using google.. possible that it needs PAM configs to auth root
Seems to need:
icemaster:/home/rbuckland/software/samba-2.2.1a/source/pam_smbpass# less /etc/pam.d/samba
auth required /lib/security/pam_pwdb.so nullok shadow
account required /lib/security/pam_pwdb.so

Couldn't find pam_pwdb.so... Try net for 2 mins
Nope .. use dselect .. Ahh there it is. ** NOPE STILL NOT WORKING **

-- Recompiling samba
~/software/samba-2.2.1a/src/
./configure --with-pam
cd src
make
make install
... YESS!

2001 07 16
Setting up jakarta-tomcat (compiling it)
+ Installed j2sdk 1.3.1

2001 07 17
Setup a symlink from /etc/samba/smb.conf --> /usr/local/samba/lib/smb.conf
so that swat can use the same file (quick fix)

... and it went on, but I have snipped it

A close friend .. replaces Rod Johnson

2007-05-10T15:08:00.000+01:00

Was just chatting with Vishal Puri, a very close friend and colleague of mine.
He said the following ..
--------------------------

Vishal: hehe all the best!

me going to india

on 29th may

me: Way ! When did this happen

Vishal: yeahh yesterday

confirmed only today

me: did u tell me and I wasn't listening ?

Vishal: speaking in place of Rod Johnson

--------------------------

I like the casual statement .. speaking "in place of" Rod. Now who gets a chance like that. :-) He left working with me to go and work for I21. Ish pish.

Good job Vishal, you'll do well.

http://www.sda-india.com/conferences/jax-india/speakers.php

Vendor lock in

2007-05-10T14:46:00.000+01:00

Big Appservers annoy me. Websphere, WebLogic .. even JBoss sometimes. But, you know what. They are good. Now .. there's a revoke of statements.

I have used JBoss from early V2 days and now use V4. If I could everything would just run on Tomcat but in banking circles this is just not a goer (wish it could be).

Brands, and branding sells.

A Colleague made a (un)true statement today. .. spoken from the client view ..

"It doesn't matter, we want Websphere because it scales".

Now the statement is not the (un)true part, but he said it tonque in cheek.

WAS scales, it does and that's why our clients will use it, want it.. think they want it. But what about when you have a standalone (runs on a JVM (thanks Spring)) app and then you run it in Websphere and it has a 300% degradation. ie, 1s response becomes 3s.

I suspect that the app is not "plugged" well into WAS 6.1. It's ServiceMix that we are talking here..

I mean where do you start, we can tune our code, but obviously the code runs well stand alone, so it's off to WAS to see what's going on.

I'm not acually the dev doing the performance testing. That is another one of our merry men. It's a hard task and it is only solved by starting at the top.

Back to Application Servers. I hate vendor lock in. It's a necessary evil though, I make no quarms about using a product when it's good. Weblogic 9 was great to dev with. Websphere 5.1 was clunky. 6 seems better. But any appserver, Java is meant to be portable and when you have a "from the start" idea that you use one, you might as well go MicroSoft and get far better "rubbing shoulders" experience than you will with Java.

A few things leae a bad taste in my mouth, one of them is the jboss-*.xml files that have to be in wars, ears etc .. That's just annoying. :-) (Maven solves that but so I don't have to worry about it, so did ant to think of it)

So , I am a technologist who chooses product or library X because it works.
I hate something that doesn't work.

Stuff I found works, absolutely brilliantly I might add is, Spring and Maven 2.
These two products alone have changed the way I view Java Development.

So How do I avoid vendor lockin there ?
Well, Maven doesn't go near my code, it sits on the edge, ie Ant and Maven co-exist for sure.

Spring, (some, not many) people have this conception that Spring is through your code. to me, if spring is in your code it's because you either found something there that really helps, or you have not thought well about how to keep it out.

Can't wait to see how Java Dev'ing on the Mac goes. Vishal says it's brilliant. My brother swears by it (but he's a designery type so he would ;-).

Bottom of the food chain

2007-05-10T14:23:00.000+01:00

Free Stuff websites. That's the bottom of the food chain.. and now I landed there.

Partly because my iPod was stolen many a year ago. But let me tell you more..

These websites work by offering a "free" price in exchange (there, it's no longer free) of you completing an "offer" from an advertiser and then referring X number of friends to the site, X being based on the gift you choose.

These sites do work, simply. The offer might cost me, anywhere between $1 - $20 or more, and if I sign up for a "subscription" to the advertiser, then more of course.

The pinch is that the "gift wanter" may just take up an offer. I'm sure there are kickbacks.
Now why am I blogging this in the tech side ?
Well .. I got lured, I have chosen an iPod as my free gift, I have to have 8 "friends" do the same thing, they go to the site and accept an offer. Then I get my iPod.

(Shameless link to "my" referral) http://gifts.freepay.com/?r=38178588

So here's the tech bit.

I chose the "advertiser" that had the most non-boring thing, which was a subscription $10/wk to mobile phone games. I made sure there was an opt out option .. (scoured the advertisers website first) .. and then went through the "sign up" phase.

I entered my mobile #, it sends an SMS asking me to confirm, I confirmed with OK.

The website, in the mean time is "waiting for" my OK SMS. The SMS obviously gets picked up because it moves to stage 2 which is "success" and sends me a "welcome SMS".

Now I go back to the gifts.freepay website and refresh my "you have not completed the offer" stage. ... No change.

Well .. I refreshed once or twice and then I went to help, help says to wait 1 - 15 days to see acknowledgement from the "advertiser" that accepted the offer. ... bummer...

Then, sure enough, an email arrives in my inbox which says my offer has been confirmed. There's the technology bit. I shouldn't be too surprised, but interestingly enough, it was server to server communication. ie, the mobile phone company notified the freepay website that my offer was accepted.

Anyways, click on my link, sign up. I get an iPod. ;-)

And this the title of this post.

OJB, Hibernate and the Cage

2007-05-07T23:48:00.000+01:00

I have been working for one company for the past 2.5 years and back about 3-4 years a go. The ORM platform chosen was OJB. Now Apache OJB (from the DB Project) had quite a potential lead in the ORM world.

It was logical in it's design, goals for JDO compatibility etc. (it made them all).

Over time, users waned and moved over, or took up hibernate. Hibernate having the market share takes the cake now for

Performance
Configuration Ease
Options
Assistance
Stability

and last and NOT least

6. Plugability

This is OJB's main killer.
In my 2.5 years working with it I can honestly say that it is the worst Java library I have ever used, ever had problems with, and more than ever passionately wanted to move off.

Now this is not a diss to the authors of OJB. By all means, they did a great job. But some things just died in there and before you know it, the project is like a car running on rims with no tyres trying to drive the nullabor. Just ill equipped for a big task.

My main pet peaves for OJB are the following

1. Class loading issues

Deep in the bowels of OJB it all starts with an OJB.properties file. This is the beginning of a painful end. OJB locates this from the classpath and then inside this properties file, locates youre repository.xml file (akin to *.hbm.xml).

Now, the repository.xml has all the settings for the mappings, but it also requires you to declare you datasource (can be a JNDI) but also transaction demarcationb (the OJB way) and a few other nasties.

Coding MUST be done such that infrastcuture is a side concern. ? What do I mean. Write your code, but write it like it;s on an island. Don't include code, logicthat binds you to a database, or a way of communicating across the wire.

This classloading issues gets ugly in many ways. It I wanted to merge repository files together, I have to do this by either, writing an adaptor class to join them all up or .. just hand (or ant,maven) hack them together.

But .. remember, a repository file is for one database.. how do you do two databases ? (seperate schemas) ..

Know, that we worked it out .. how, by ensuring that one OJB for one schema was in a sibling or seperate classloader.

2. Inconsistencies between versions (minor versions)

The Apache Portable Runtime project has a very nice succint way to describe versions. http://apr.apache.org/versioning.html
OJB certainy did not follow any of these guidlines.

The library is still stuck on 1.0.4, 1.0.2 is about 3 years old and 1.0.3 was just skipped because it was full of bugs (for us).

OJB 1.0.4 seems stable enough, this was after a massive code change that occured to our base product to make it work.. oh the pain.

Now the observant among you would say that you shouldn't have to change code if you change your ORM tool. Sadly we had to because of the legacy Java Code, but even still. There was horrid work to be done just trying to work with the incompatibilities.

Why upgrade then ? Well we found some things didn't work in 1.0.2 (something with key Sequences and Sybase (yes another groan, see my Sans SQL Post) and 1.0.4 supported it.

From the site itself (on the news section)

12/2005 - OJB 1.0.4 released

Contains bug fixes and new features. For more details see release-notes.

Argh ..

3. It's just plain dead

The site is tumble weed city. Everywhere you look on OJB it just smells of a house that is falling.
Where I work(ed soon) we have a large application that relies, depends, is hard coded to, OJB, it is an inherited platform and we now know what not and what to do with OJB.

Some initial analysis has gone into the replacing it with Hibernate, but a lot of the other code is going to be shelved so it is almost not worth it now.

All the new work has a clear separation of ORM and Domain so that way it can easily be shifted in or out as you see fit. (thanks Spring).

I think the project (OJB) is dead, or should be left alone.
The mailing lists are a dieing world.
http://mail-archives.apache.org/mod_mbox/db-ojb-user/
Summary

Why did I mention hibernate in the title ? Well, Im never using OJB again that is why.

OJB users now and future, your mileage may vary, because you might now how to make OJB work sanely. The thing is, it was not obvious to me or about 6 other developers I worked with who also have a bad taste in their mouth from this seemingly harmless library.

Have fun, and keep the code away from the hardware :-)

More Thoughts .. Moving from that proverbial database

2007-05-03T14:07:00.000+01:00

I posted back in November my desire to move away from the database, somehow.

Well .. my chance has come .. I will still use it, but let's not have any SQL shall we. I am moving work-house.

So, Starting this new company in a few weeks time, I have the opportunity to not rely on a Database. Let's read that anotherway. I won't (want) have to write another piece of SQL.

Now, I actually have a great fondness of SQL. I actually becomes quite a proficient Sybase Tuner (knowing the profiler better than the walk home) back in 2002 - 2004. So I actually like the stuff and SQL is certainly a way to shift sheer lare amounts of data.

But why another language, I will already have CSS, XHTML, JavaScript, Java and a hack of XML for Maven, config, oh and Im sure my favourite XSLT language will land somewhere in the next big app I write.. so .. where does that leave SQL.

Well I am going to go dry .. I will look for all options, (logical), where I don't have to have any SQL. It is just another language, that when I am hiring or training or handing over, that I don't want to be the only one in the team that knows it so well that I get scared looking at others working on it ..
(told you it was a rambling).

So what are the options ?
How can you develop an enterprise app, that HAS a database but has no SQL (ie auto-gen'd SQL).

Here are some avenues to pursue.
- ODBMS (most likely)
- Auto Gen of OR Mappings
- Direct cache storage of Objects (eww!)

What happens when my objects grow, what about the old data etc ? That will be fun. I don't want to have to write "update" SQL (remember, sansSQL

Shoot em down, clear em out, get them bugs !

2007-05-03T13:22:00.000+01:00

Listening to #115 of the Java Posse in traffic this morning (taking the car to the mechanics), the boys have mentioned it many times before, the findBugs plugin.

I finally remembered again to go back to the podcast listing http://javaposse.com and take a look.

FindBugs looks great, I quickly ran it across two projects I had open in Eclipse and it certainly picked up a 3 in one and about 28 in the other.

Going by class (I should go by LOC) the 28 bugs on 60 classes is a bad.

The other is 20 classes and 3 bugs .. good thing this second project is recent :-) and the other 3 years active.

Good job for that plugin I say.

JEE and Spring

2007-05-02T01:15:00.000+01:00

I was in discussion with a colleague the other day who had not used Spring before. He was describing to me how Spring is not needed now because of JEE5.
It suddenly occured to me that, anyone who hasn't used Spring doesn't really understand how it works.

Se we were on the discussion of View handlers.. so the bit that finally renders (X)HTML in the browser.

Here is my theory : Spring has become well known for a lightweight IOC Container and famous for it. But people seem to miss the point that this is one area of Spring. I can take the Spring Portlet code and pop it right next to the JEE stack and it's all good.

This is akin to saying let's run struts in a JEE stack.
Same as me saying, let's run SpringMVC in a JEE stack

Interesting.. Anyways.

The Ideal Java Development Environment - Post 1

2007-04-30T00:59:00.000+01:00

Starting a new job in a few weeks, my mind has slightly been drawn to what is the quickest way to setup a fledgling company with a stable build system and a structured project and development lifecycle for Java.

Now many of you will be very observant to note that I have tagged few big issues in that first paragraph. Yes, that is the problem.. or the focus of my attention for a short number of days (what, he's mad, setting up something in a short number of days).

So where do you start ?

My new company I am working for primarily exists to raise funds for charities, technically we are not non-for-profit, because it is not 100% of profit which goes to the charities, but let's just say it is more that 40%.

So .. costs are critical, we need to keep them fairly low, and software development certainly sits at the high end of draining resource (money).

Of course being a major supporter, user and contributor of Open Source, it will sit majorly within the realms of my new job.

At a brief glance, these are the items for which I either (a) know intricately and will bring to the table, or (b) know I need to look at them.

So lets start shall we, here is a quick run down of items I will either be using, or will investigate ..
Build and Development

Eclipse - (for many years now)
Maven 2 - Without a doubt
Subversion - goes without saying
Continuum - (CI - Continuous Integration) - Would love to use Bamboo (atlassian)

Some things I want to take a look at (suspect they may be of benefit)

Bamboo
Buildix (ThoughtWorks Out-of-the-Box of all the above)
Tracs (it's in the Buildix platform)
Artifactory for Maven - Definitely a Winner here

Some standard "stuff" I will always add to the Java projects

Checkstyle via Maven
Maven Site Build (customise it)
Spring (Security, JDBCTemplate)

Some stuff I will certainly look at

Maven Dependency Graphs http://philhoser.blogspot.com/2007/01/dependency-graph-for-maven-2.html

So that is a brief list of items that will form the new "bucko" workshop.
I'll keep you all updated on this as I grow this area.
There are a few things I am missing there, but I'll keep you all posted on that front.

Oh, one last thing I do want to add it, when the team becomes >=2 developers.
and that is hooked up to the CI .. build notification.

A lot can be said on this topic. Here is a place for it.

http://www.pragmaticautomation.com/cgi-bin/pragauto.cgi

I have seen a whacky lamp at a lighting shop in Balgowlah, hideous. $100 or so .. and it would be great to "turn" it on, when it all goes bad. It's a foul looking angel on the back of a dragon. Perhaps the ubiquitous (sp?) lava lamp will do.

I will post the steps I take in the next few postings to detail what I setup. But It wont be for a few more posting yet because I am still with my current (and fantastic employer) so need to be honourable there.

Aargh ! Letting Others 0wn U

2007-04-30T00:22:00.000+01:00

Having a few issues with the blogs lately.

Hopefully this can be the last place of my blog. The blogger.com / blogspot would not let me migrate the old account across to my recently heavily used gmail account.

So I have ditched the old in favour of the new.

Installing Ubuntu Server 6.10 on Dell PowerEdge 2400 Server

2007-03-18T11:53:00.000Z

After much pain in trying to start the install, I was finally able to get the install underway.

What was the problem ?

I have 4 disks (2x 9 and 2 x 18G) and the installer was not recognising the logical disk that I setup on the RAID array (2 x 8 as one, RAID1 or 0)

What was the "possible" solution ?

Apparently it is a common found problem with these MegaRAID adaptors.

Essentially I2O is a specification used in these RAID arrays.

Personally I don't care too much, I just need my server running.

Apparently, the i2o module (i2o_core) for linux loads up first and then the megaraid module can't find the disks when it loads.

A few people found ways around it by manually loading the modules in the right
order (RHE, Fedora).

My answer for Ubuntu 6.10 server:

I found a tip on the RHEL "release" notes for RHEL 5 beta 2.

http://www.linuxcompatible.org/Red_Hat_Enterprise_Linux_5_Beta_2_s76179.html

It states: Change the megaraid mode from I2O to Mass Storage.

I did that .. (Ctrl-M when it pops up) and then went back through the install. No problem.

I can see my 9G and then also my 2 x 18G's.

Nice.

Thanks Red Hat. (Sorry I left you at RE 4.2)

Java and Schema Generation

2007-01-08T02:05:00.000Z

You'd think that generating an XML schema from a set of Java Classes would be a simple thing. But gee it is hard.

I have spent the better part of four hours and three coffees pouring over the inner workings of the JAXB 2 SchemaGenerator classes. Odd stuff this, I know I am close because I have it working for simple types, but I am trying to work out how I get it to handle a complex map of objects. Hrmm..

[[ Edit ]]

I did work it out in the end. It was horrid code, in that there was so much of it which I suspect means that I didn't have it right. (it does do what I needed it for though so, close that door).

Thoughts .. Moving away from the DB

2006-11-18T05:02:00.000Z

This is often the topic of an OO developer. How do I distance myself from the database platform, but not lose, and only gain, as would be the goal.

So I want to not have to rely on a DBA.. I mean, whay have two models. One for the database and one for the code.

The problems I still see are

- Invested expertise in DB coding (SQL sprocs etc)

- Speed of Batch processing. (one of these days Im going to bench mark some hardcore processing and see how I can get the same performance from a platform)

Client requirements, speed, experience dictate that we are tied to the RDBMS, Oracle, DB2, Sybase, SQL Server, great platforms. I myself have extensive experience in knowing how to get the diamonds out of the Sybase Parser, but not practising that black art any more I often wonder what my next "personal" project architecture would be.

James Strachan mentioned to me the other day how cool it would be to have XPath (Xquery) on top of Jaxb2. That is a cool thought.

The XML Content repository JSR-170 is a good read if anyone ever thinks of an XML database :-) that is essentially what it is.. I will certainly look at it again as a thought.

SOA, SOAP, RMI, TLA, FLA and Horses

2006-09-06T00:03:00.000+01:00

Picking that horse. I'm not a betting man at all, never been trackside, but some describe technology like betting a horse.
Do your homework and choose, then hope it wins, maybe needing a good relationship with your bookie (I am not sure who the bookie is in this analogy).

My own take is technology far much more than betting, (and the punters would say the same about horses).
But in fairness, some people get burnt, myself included, when then chose the wrong horse.

So taking some pages from anyones book, do your research, and, if your're
at the other end when the horse is scrapped, don't give up and move on.

My point is, a technology, platform can be chosen which ensures you don't have to worry, too much, about what is going to
come up against if you follow a few simple guide lines.

Ramon's Tech Tip #1 - Horses will be scrapped

What do you do when the horse you backed is scrapped ? Same as you do in business, learn from it.
It may have been the best horse when you started, and sure, it got you to now.

Ramon's Tech Tip #2 - Choose the Horse, Jockey, Race History and the Silks

You trusted your instinct and your fellow peers, but something went wrong.

The way I apprach a choice area is to talk to people who would agree with me, easy. I also
get opinions from people who I know won't like it, two outcomes come from that approach, (1) I will learn something, like that
they are definitely wrong and I am right, or that they actually have a valid point. (2) They show I am wrong and in fact they
do like it anyway.

The other approach I do is to research the other technology camps to find out where they are heading. In this I mean .NET, LAMP, Java,
bright people with bright ideas working in another language, what do they see the future in X area to be ?

Silks ? Well, go with what you like, at the end of the day you will have to be the decider.

Ramon's Tech Tip #3 - Think of the future bets

All good and well when you have to decide for a short term item (1 - 2 years), but what about choosing a
direction when you yourself won't even be looking after it.

A well guided decision that will impact others beyond you must be well thought out and planned. Don't rush into
any decision lightly (Sub tip #3a). I have recently been involved in the decision making process that will see a platform
being used well into the next 3 - 4 years. I can be good and say I will be around in that time, most likely I will, but what if I am not?
I need to be considerate of the fact that the choices we make now, take us that far into the future, expecting that things will change.

Ramon's Tech Tip #4 - Follow the Jockey

If you don't get too friendly and passionate about Mega-Deployed Platform AlphaX, you will be happy to change to Simple-Deployed Platform AlphaX
when you need to. My tip here, make the choices such that each component is not interdependent on another, don't couple things so tightly
that it means cutting off when all you wanted to do was change cars

Ramon's Tech Tip #5 - Horses fall over

Expect it to change. In all our dealings with IT, the one thing that is guaranteed is that something will change.
Be it that it will become obsoleted or improved.

I have just been involved in a project where the first cut of our product was available for no more than 9 months
and then we saw a radical change in how it should be offered to clients.
One key component to the easing of this transition is that it was well engineered from the onset to cope with the change.
So the best way to describe this is, write it so it can be changed, but write it to last. Loosely coupled.

in the .NET land I have seen comments that interfaces should be removed, but in the land of dependency injection. Interfaces
are king. Sure interfaces may change and need to be redrawn but so will a tightly coupled system. I can tell you
that IoC Frameworks are becoming King, and, bottom line, they save money. and I don't mean small cost cutting here
and there. I mean in the order of weeks of work for teams of developers. Relating to $1k's to $10k's for large projects.
Prove me in this and I'll be quiet, but they really work and are not just some Astronaut engineering Oxygenless fad.

Ramon's Tech Tip #6 - Don't sweat when you lose

Let's face it, and that's it, you will have to face it one time or another.
Horses collapse, decisions made may die. It may not have been a bad horse, it may not have been a mistake.
It could have been due to external factors such as another horse on the track destroying yours, or a big comet
taking out the whole race.
In any case, accept it and move on. The person that hangs on to the, "oh but it was good" and doesn't learn from it
doesn't learn. Sometimes things were good and there are hard factors that caused them to fail, such as humans.
So .. learn from it, analyse what you did wrong and choose another horse.

Ramon's Tech Tip #7 - You're betting for someone else

I once had a colleague that, when I asked him what he wanted to be doing in 3 - 5 years time, said, "I don't want to be doing the 'next IT project' which just involves another set of requirements and tasks". He wanted to get out.
Now sure, that's valid, and he probably needed to move on. My take on IT and the next project similar to the last
is that I am pleasing a client. The client is always right, and IT forgets that. Sadly there has been this thing that the techy dude,
sys admin, software engineer, whoever, knows better than business person X because they know something. Knowledge is king, sharing it with someone else and letting them make a decision is power to the king.

It's always said, and often in retail. The client is always right. Well that is also so true in IT. IT department, make the client the person that pays your wage. Software guys, the client is the one who is funding the project.

IT from it's onset has always been around to "solve" a problem, make a buck, help someone do task X. The key thread in all these is that there is a client, and there is the problem they want solved.
They are the ones that will dictate what needs to be done, how much to spend and the cut off point at
which foobar-option-ABC is implemented using pure methodology Reverse Vegemite Toast.

Upgrading to Kernel 2.6

2004-05-08T18:08:00.000+01:00

Due to various technicality issues arising from incompatibilities with my kernel 2.5, I was forced to upgrade to 2.6 (on the laptop here)

So a quick check, and then by force having to reboot back to 2.4 to do it,
I upgraded, and a few things didn't work. I needed to add a line in /etc/netowkr/interfaces
eth2 inet dhcp auto

# or similar and /etc/modules
psmouse

The first, (eth2) my PCMCIA wireless network card moved to eth2 (when under 2.4 and 2.5)
it was eth1.

At the psmouse, it's a new module needed for the PS2 type interfaces, of which my Trackpad on the Vaio lappy
is one.

So .. a quick, 40 mins of play and Im up to 2.6.5 .. not bad for a days work.

Thanks Linux (and debian) for making this all possible.!
(love it)

Some JS Links to Fancy

2004-04-08T18:09:00.000+01:00

Always looking for some links to deal with JS, specifically, obfucating it.

See this link for more links on compressing, stripping, crunching and obfuscating JavaScript (ECMAScript)

Optimizing JavaScript

This link (from above link) has worked best at the moment (didn't break the 23k JS file we had)

Script Squisher

Sybase output parameter issue with ColdFusion

2004-03-11T16:02:00.000Z

Using Sybase 12.5 we have a stored procedure that
returns a string in an output parameter.

The Stored procedure is declared like

create proc sp_foobar
@pc_char1 char(10),
@pi_int1 int,
@pvc_vc1 varchar(512),
@pc_char2 char(6),
@pvc_vc2 varchar(40),
@pvc_vc3 varchar(255),
@pb_bit1 bit,
@pvc_vc4 varchar(20),
@pvc_return_code varchar(30) output
as

...

In ColdFusion, we were calling the stored procedure as follows.

But this throws an error,

[Macromedia][Sybase JDBC Driver][Sybase]Operand type clash: VARCHAR is incompatible with VOID TYPE

and it highlights the last cfprocparam (the output variable).as the issue

I worked out that if you pass it a value="" parameter, it stops it from failing.. ie

The sproc, in this example as part of it's internal logic/contract returns "SPROC_SUCCESS" as it's varchar, in @pvc_return_code. What Coldfusion puts in this
v_return_code is now equal to only the first letter. (ie 'S')

Hrmmm, .. so .. I worked a way around this one also... A Fix for this BUG .. make the 'value=""' a string as long as the string you expect to be returned. eg,

and then you would get #v_return_code# equal to "SPROC_SUCCESS"

Odd, any takers? Someone SURELY has seen this before.

(In)Sane Scanner for Linux

2003-11-10T15:59:00.000Z

I'm blogging this one because I have needed it before today (this time being the 2nd time I have needed to know this info).

I have a BENQ 4300U USB Scanner, was $70 8 months a go and is still great.

Making it work with Linux is not too hard as some might think.
install the sane drivers .. (apt-get install sane) debian-esq

apt-get install xsane (the front gui scanning tool)

and now the firmware,

The sane driver (snapscan.sf.net) needs to know
how to talk to the scanner.
This is all in the scanner driver/bios/something file (i don't care much
for these details. I just need it to work .. gosh I'm changing :-), in the past
I would investigate everything..

so .. find this file on your windows install .. reference the table at snapscan.sf.net
My file is the u176v046.bin

How do I know this ?
Easy .. the device has a number, when you plug the USB scanner in, the linux USB drivers will
detect it and throw a message in /var/log/messages about such event.

In the log message, is the device number, which will correspond to the snapscan device id they know
about.. egc:

# dmesg | grep -i usb
... skip all this to the last lines ..
hub.c: new USB device 00:07.2-1, assigned address 3
usb.c: USB device 3 (vend/prod 0x4a5/0x20b0) is not claimed by any active driver.

vendor id and product ID .. 0x4a5/0x20b0

These quite cheerily map to the table entry at the aforementioned driver website.

So Search for this file on your windows boxen.
Mine of course is u176v046.bin

So .. I created a directoy /usr/lib/sane/firmware (anyone know where else this is meant to go, seems good to me)

and then edit the /etc/sane.d/snapscan.conf file ,
at the top set the path to your firmware file.

restart xsane and hey presto ..

BUT, something has changed since I did this on the laptop.

As I understand it, previously, sane used to use an object file
scanner.o method to talk to the scanner.
now it is all via /proc/bus/usb, which debian rightly warned me,
only root has access to.

So, a quick check in /usr/src//Documentation/usb
grep usbfs reveals proc_usb.txt, in there reveals
that you set devmode=0666 for the mount and the device entry points will get
-rw-rw-rw-

so my now /etc/fstab entry for the /dev/usbfs looks like

none /proc/bus/usb usbfs defaults,devmode=0666 0 0

Hope that all helps someone (and me next time I have to do it again and forgot
how I did it last time).

cheerios.

1RY CPU coolers for AMD are hard to find

2003-10-11T17:58:00.000+01:00

I am currently in the process of building a lowish spec 1RU server for a project I'm working on.

The Server spec is an AMD XP 2000+ 512MB RAM and IDE Disks. (big whoop I hear - I told you it was low spec).

SO anyways, a 1RU server needs a 1RU CPU heat sink and fan.

the AMD Chip was the lowest in the Athlon range at the moment. I think it's around 1.6Ghz
but, being the lower end, puts it in the workstation class of course, so do you think I
could find a CPU heat sink and fan for it easy. Nope, issue was (here is Aussie World) that
most suppliers won;t stock it, not a "fast moving item".

Since it is a 'desktop' class CPU most people just answered, "Nope, don't have any, won't have any"

So anyway, thanks to the nice mag I buy monthly (just to keep my ear on the hardware ground)
Computer Market (not related to www.computermarket.com.au)
(where was I.. ). Yeah, this mag, I find an ad
for some guy selling lots of Fan stuff. This is after 2 weeks of calling about 20 suppliers,
going straight to thermaltake.com.au in Melbourne even and asking them. (it's one of their fans I was
after)

This guy, www.comparts.com.au had one of the Tsunami fans I wanted.
A Socket 462 AMD Athlon 1U (1RU) Heatsink and fan.So $40 later it should be in my mailbox on Monday.

I'd just like to add that so far service from there is great!

Also, the 1RU case I have purchased (if your interested also)
was a good buy..
1RU server case: Maccase 101 1RU ATX Case
From: Online Marketing Pty Ltd (Sydney - Often good prices)
Direct Product Listing here

Had a birthday the other day

2003-10-08T17:57:00.000+01:00

As you do most every year, I have had my 28th birthday.

Got myself a CD or two
* Grandaddy's 2003 album, title escapes me. Fine listening there
* The Beatles White Album (laying now)

some parfume (is it perfume for guys or aftershave ?) some nice
smelly Gucci Stuff that is.. and an egg cooker. Bit of a funny
one that..

Growing up my mum had a Sunbeam Egg Center. It cooked your eggs. :-)
It was shaped like a flying saucer (like a bad B-Grade Movie FLying Saucer).

The eggs sat it a tray (5 or 6 I think). You filled the lid (had a measure
in it) to your desired 'softness', poured that water into tha base,
turned it on and waited.

I gues it detects a rise in temp to know when it's boiled off all the water.

Anyway, a few months back, that got me thinking (I was never getting the
cooked egg thing right by guessing the timing).
I grabbed a saucepan, put in 1/2 cup of water, but on top of the saucepan
the 'asian style' steamer (made of bamboo). Put in my 4 eggs (standard issue
700 grammers).

Put on the lid, gas on high, and waited.

I waited probably 5 minutes, the trick I figured was, given I got the amount
of water correct, all I had to do was listen and detect the lack of
water in my pot.

First few goes weren;t too bad, overcooked them really. I like
a soft/mushy centre.

SO .. I worked out that 1/2 a cup of water was just right, and you
could just hear when there was no more boiling
water, thus eggs a ready.

BUT, I wanted a better way. My next thought (untested) is to put
in the saucepan (with the water), some light balls, like ceramic
balls the size of marbles. This way, they would get tossed by the water
making a sound until the water is no more, then you know the eggs
are ready. reduce the amount of water to soften your eggs etc..

So little beknownst to I, Sara had bought me a 'real egg cooker'

It's shaped like a chicken, fancy that. and it cheeps when the eggs are
ready.

Ahh the joys of the XBox

2003-10-08T17:56:00.000+01:00

Just today, I've noticed my xbox has stopped a responding on the network.
Not hardware related as the EvoX menu responds with pings (Phew).

Bit of a fiddle here and there, looks like my XBox ethernet card is failing
to be picked up by the kernel.

No matter, on to debian xbox 0.4.1 I say.

I installed this on my brothers XBox and it went ithout a hitch,
my issue is going to be if it will allow me to install over the top of the existing filesystem
since I have my full datastore on there.

I _think_ it's time to get the datastore (aka my 80G drive) _out_ of the XBox and back
into the server. Im a little nervous about not having that nice control
of the drive when it's in the XBox. Putting it on
the server again means I can still have nfs'd.

Ahh ramblings. .. and All I want to do is get on with the other big project
Im working on.

Bah!

A Nifty SQL for T-SQL (Sybase and MS-SQL)

2003-08-21T17:52:00.000+01:00

A nifty SQL for T-SQL (Sybase and MSSQL) [web development] - ramon - ramon@thebuckland.com @ 12:31:24

When playing with SQL I often find that I create some massive SQL statement .. like


select blah,blah, (case foo when ... end)
from ...
and i need to put that into a temp table.. what should the temp table look like was always the next step. So .. thinking about it
I decided to hack up my solution.

Insert the select statement into a #temp_table
select *
into #my_tmp_table
from ...
and then I wrote this whopper to extract out of the sys tables
what the Temp table looks like as a CREATE statement

It's a bit crude but it will give the basic overview of what you want

select sc.name +  ' ' + st.name +
       (case st.name
     when 'varchar' then
         '(' + convert(varchar,sc.length) + ') ' + (case status when 0 then 'not null' else 'null' end) + ','
     when 'numeric' then
         '(' + convert(varchar,sc.prec) + ',' + convert(varchar,sc.scale) + ') ' + (case status when 0 then 'not null' else 'null' end) + ','
     when 'decimal' then
         '(' + convert(varchar,sc.prec) + ',' + convert(varchar,sc.scale) + ') ' + (case status when 0 then 'not null' else 'null' end) + ','
     else
         ' ' + (case status when 0 then 'not null' else 'null' end) + ','
      end) ,-- sc.*
from tempdb..syscolumns sc
  inner join systypes st
  on st.usertype = sc.usertype
where id in (select id
           from tempdb..sysobjects
          where name like "#transaction%")
HTH.

Ramon

My XBOX is Up

2003-06-08T17:48:00.000+01:00

xbox is up [xbox] - ramon - ramon@thebuckland.com @ 15:15:25

we finally did it. My Mod took a very long to to complete (all of about three weeks I would guess).

What Happened I here you ask?
The CromWell BIOS that came with my Modchip did not sit too well with
heat issues.

Each time Tux the penguin started to go across the screen, he would stop
about halfway and freeze. This became worse and worse upon each succession of boot.
After about two or three goes. No go. The XBOX would show the red circle
as if a hardware failure had occurred.

Disable the Modchip however, and all was well.

I spoke with a few people on IRC (Wilber #ozxchip) whom pointed me to info
regarding the cromwell and this very issue.

My solution was to re flash the chip with a later version, or try
a different BIOS altogether.

So I decided to go the programmer route.. being a hands on that I am
I decided to make the CheapLPC Programmer at this site
This was most wonderfully pointed out to me by GreenGian (#ozxchip) whom happens to run
that site. He had some good tips.

So off I went on my merry way.

The Programmer proved to be a tough nut to crack for me.

Spending about $15 all up on parts I started with the power circuit.
My brother helped out there where I used a circuit with a voltage regulator (opted
not to use USB because, well I didn't).

The voltage circuit was perfect, 3.32v all the time (could go up or
down in increments of about .01v according to my multi-meter.

Day 4
So once that was complete, I then had the lovely task of putting the
whole thing together

Day 6
After a few days of rest The circuit was complete and the weekend was here.
I sat down a late Friday evening with my trusty multi-meter, my beauty Laptop
and the programmer.

After a bit of sorting stuff out, I had the milksop software pumping
out some data for me.

On Windows XP. milksop was showing 1100 fairly consistency (1100 being the output
of it's testing procedure).

Knowing I was to be expecting 1111 I decided to ditch XP and I put the
programmer into the back of my trusty Linux Server of 2 years.

Now here it was worse, 1100 1010 0011 1101 and stuff like that ..
but consistently unusual.

I slowly (after about 2 hours, and many conversations with my brother again) worked
out that the issue was that I had some software (a kernel module paraport)
listening in on my conversations on the alp0 port and was stuffing
my readings.

telinit 1 into safe (root user only) mode
and check ..

better but not good. 1101 and then 1010 on and off.

.. now Here I thought, aha! Software on the XP was stuffing
my readings as well. and sure enough, at 3am. It all came together..
the Programmer testing software showed up 1111 and n o 1010 every 20 seconds.

(I shutdown the print spooler).

So onto re-flashing my modchip.

no go either
The actually flashing never succeeded, The milksop software wonderfully
written by Andy Green could not detect my device and I had to concede
defeat on my programmer (5am).

I packed up and washed the dishes.

9am rolls around and I chucked all my gear in bags
and hopped on a train (little to know trains weren't running and I would
be on a 2.5 hr buss trip) I made my way to the Central Coast at my brothers place.

Install fest of XBOX v1.1 and v1.0
Jono and I discussed the dilema, he being a fully qualified electrical and
electronic engineer, in his wisdom said, "Forget the programmer, we will spend
hours on this thing as it's hard stuff".

Our other option: Mod his v1.0 XBox (Which we had to do anyway) and flash my
chip.

The v1.0 Mod
Jono started on his mod, got the solder out of the LPC header (I didn't have to
do that thankfully) and put in the pins.

Identifying the d0 port on the back he started to fix his wire, and after a
few bad takes, BANG! the d0 point just came right off the board and also broke the
track.

Bugger! But Jono, determined as he is, stuck at it, With my trusty B and W camera
lens from an old video camera helping as his eye, he fixed that track (even De-soldered
the CAP next to the d0 to get at the point so he could put in his wire).

After about 2 hours of fun (about 4am Sat just gone) he had the mod installed.
Scary stuff, we almost lost the XBox then.

Onto flashing my chip
Thankfully i brought 3 CD-RW's with me and we proceeded to burn the ozxflash.

After about an hour, we had my chip flashed with Xecutor-2 BIOS and .. it worked!

I was able to boot and also play my games still.

So next thing was to get my 80G drive in there. I booted trusty EvolutionX
and found the FTP server, Backed up the data and then went on
to do the HDD swap trick.

It goes horribly wrong 2
Now, I've been working IT for about 9 year and in my time, I have hot swapped
one or two drives. (that weren't meant to be hot swapped) and it never did
what I wanted and I was always worried about damaging them.

Well wouldn't you know it. I hot swap this drive following their instructions
very carefully and it .. well I now have a dead 10G harddrive. (I'm yet
to determine what exactly has gone wrong, but Jono suspects that I have damaged the buffer
chip of something like that on the PCB of the drive).

When you boot a PC with this 10G in, it detects OK, but then says (for eg:) Secondary Master Hard
Disk Fail. A clear indication that the computer was not able to contact the drive.

Unbeknownst to us. I continued on thinking, this HDD swap didn't work, there
must be another way to get my drive in the machine.

I plugged the 10G back into my XBox after the failed attempts and ..

XBox no longer liked my drive
The XBox would start it's little BIOS routine and make a beautiful BRRRRR
sound (brief, about a 1/2 a second or so) before it shows the X logo.

Noticing this straight away, I got worried. of course, the XBox only gives
error codes and all I could determine was... the DVD would no
longer eject, The XBox would boot to my MS Dashboard (I hadn't
installed EvolutionX Dashboard) and it would no longer boot off any DVD (play games etc).

where's my paddle?
When I disabled the XBox BIOS, i still got the brrr sound but then I would get an error
code of 6 and a notice to contact Tech support.

Not to be out done. I decided, put this drive in Jono's XBox and see how
it goes... same thing .. BRRRRR.

I took the drive out (11PM there abouts) and put it in the spare PC we had and that's
when I first saw the "Secondary Master Hard Disk Fail" having seen this more than a few thousand
times over the past 9 year I knew exactly what had happened, my heart sank. I had done in
my HDD (which incidently I was going to flog off on EBay after I had unlocked it)..

Not to worry .. I had the 80G to put in anyways! :-)

3am and I fall asleep
So we put the 80G in my machine and it booted CD-RW's again, put it in Jono's XBox and same thing.

I did some fishing around and Found reference to Slayers ISO Tools. Looked like it
could format my HDD, Brilliant!.

Download that and burnt it to a CD-RW (about 0th or 10th CD-RW we had reburmt that evening).

Loaded it up and it had a few options for what i was looking for. One, Format Drive and
install EvoX.. Select .. wait..

Looks good.

Put it on My XBox .. Error 13. No Default DashBoard found!!! Bugger!

I think, from memory it was at about this time that I fell asleep.
I was sitting on the couch and .. next thing I know, I woke up to Jono was asking me a question.
I was out to it. I had been up since 7am on the Friday morning and it was now 2am on Sunday morning. That makes about 43 hrs (I had about a 40min sleep on the bus).

That brief moment of sleep ... it woke me up.. I thought more, Jono and I talked
more and about an hour later decided to call it a night.

A Remote Control
I woke at about 10am that morning. (Jono was up much earlier trying to fix his 2nd TV's
remote control so we could have 2 TV's to be doing installs and setups with.).

Why a remote ? Well the previous night we tried to get his little TV to work but the TV (as like many, don't have the TV/AV switch on the TV, only the remote, and as a result we weren't able to
switch the TV to AV to see XBox. (and had no Video Player to act as an AV to RF converter) ah well.

This brief attempt at fixing the remote didn't work ( I suspect Jono spent a good hour
or two doing that :-).

Config Magic
Enter Config Magic. So I jumped on IRC again and spoke with some guy
called I_am_zim on #xbox-scene. He mentioned this software "Config Magic" but didn't
mention what it was. I jumped onto the old XBox place and pulled it down.

After a quick Format and burn of the CD_RW we saw this cool tool booting up
on Jono's XBox. We put my 80G drive into his and played abit .. inadvertently locking
it to his XBox ..
well little did we know .. this cost us about 4 hours of pain and grief,

A locked drive can only stay locked for so long
Locking my 80G Drive to his XBox . meant that now .. we were seeing
a different error code on my XBox .. 9 I think it was. well many boots
later I started to get this sick feeling in my throat that, not only had I
destroyed my 10G drive, but I had also stuffed the IDE Bus on My XBox as well.

We tried all sorts of stuff, but nothing would make my XBox boot.
Sometimes I would see an error 16 and at one stage, we saw this Error
page but it was Half there Cut diagonally, top left, to bottom right
and the bottom half was black (It was the Xecutor2 error page) and the error
number was 16.

OF course, the PC we had could not read my 80G drive (BIOS was not supportive :-)
and that just made stuff hard.

The drive worked in Jono's, not in mine.

Finally, we started at the beginning again.

Downloaded Cromwell 1.22dev from xbox-linux.sf.netand flashed my chip (In the 1.0 XBox). With this BIOS we could actually
see what was going on.

We put in a 1.2G Hard Drive into my XBox and booted Cromwell. It locked up .. Not too worry.. grabbed my trusty BIG fan and .. whoohoo! it went further, even booted the debian install CD.

SO .. this proved that my XBox was still okay, but there was something wary with the 80G HardDrive. (and my 10G hee hee).

Put the 80G in and debian install showed all this BUS Errors, failed to read from
disk error seeking etc etc .. now I should have clicked at this stage but not
being around XBox stuff for long it wasn't obvious.

Incidently, the 1.2G Drive freaked out Xeceutor2 BIOS (and native XBox BIOS, possibly because
it is too small and they were trying to read past the end of the drive looking for
the non-partition table FATX partitions.

So a few more plays with trying various things, (seeing if I could reformat my drive in another
machine (which required flashing it's BIOS to support 80G drives) and then

a Road to freedom
and then... (a quick toilet stop) and bang! it hit me .. we had locked my drive
to Jono's XBox.

So .. fired up this cool tool Config Magic and unlocked it, re-flashed my BIOS BACK
to Xecutor2.

We did this because even though the 1.22dev still booted okay it locked up
through an attempt at install on the 1.2G drive which was to me a heat problem again, not unlike the 1.18 stuff I saw. With the 1.18 I used the fan as well but never had ANY luck, so they must be close
to fixing that issue because the 1.22dev worked so much better.

After a re-flash, and confirmed that my drive worked in mine, we backed up
Jono's drive and then copied stuff onto my 80G Drive and then I finally
after 2 weeks and then a MEGA install fest of 16 hours .. had my XBox booting
EvolutionX! (not wanted that, as it is running Linux solely).

Whoa!.

Unplug that DVD for me could you?
Oh did I mention, because Jono's box was 1.0, he had a Thomson drive and none
of the CD-Rw's we had would boot on that drive (additional to the fact that Jono's
CD-RW drive was freezing when it was trying to burn his CD-RW's he had ..).

We used my Philips Drive (comes with most of the v1.1 XBoxes apparently) and decided
to between the two XBox's. I think I swapped that DVD out about 20 or 30 times.

so because we had three Verbatim CD-RW's we rotated then on a number system.
1 2 and 3 and a sheet that showed what was on them. (go pen and paper).

The Stuff we were rotating on three CD's over the whole time was

EvolutionX
Ozxflash
bios's for Ozxflash
Slayers tool (yup, I_am_zim said it was useless)
Config Magic < -- this saves the world. Debian Install CD Debian Boot CD All of the time, a Quick Erase using CD Creator was okay to make the CD. We used Disk At Once.2 speed.

When I finally booted debian install on my XBox with my 80G drive attached
It started okay, but wouldn't login to X properly.. no probs..
I flicked to Console and started the install of XBOXLinuxInstall.

It seemed to come up okay but . then halfway through copying the
files saw all these errors everyone and then about unable to read from
device hdb (the DVD) ..

I though, ahh, maybe she'll be okay.. but when I rebooted into the install
vi wouldn;t work.
"Bus Error"

was the err message.
No worries.

ssh the config files off and edit them on the server and transfer back ..
"Bus Error"

ifup eth0
"Bus Error"

hrmmm, maybe those errors had something to do with my bus errors.
jono's Linux Install worked okay. (he used CD-RW 1 I think, whereas mine
was from CD-RW 3, just the numbering system).

I decided to call it a day at Jono's and came home.

I wake up and did a few chores. (make chicken soup).

and reformatted (COMPLETE) one of Jono's CD-RW's he had .. lMEDIA was the brand.

Waited, the reburnt Ed'd Debian to the CD and ..

it installed .. no errors I have no BUS errors.

AND ..

here I am .. typing out this beautiful story of sadness and joy on my IBM
USB keyboard looking at a fuzzy Mozilla Browser on my 48cm colour TV with my
XBox pulling down deb relevant to debian unstable whilst listening to Sting
on my CD player (not the XBox yet, coming).

And I hope this was a nice read and you picked up some tips from the
pain Jono and I felt.

:-)

Problems booting the Cromwell BIOS on the XBox

2003-06-01T15:13:00.000+01:00

So I installed the modchip all okay. But no luck being able to boot the cromwell BIOS (GPL BIOS for the XBox).

I found a survey for failed cromwell booting info so go there and help out.

I will be tryong to flash up to a later version at some stage soon.

in the mean time, it's still an XBox (shame im not much of a gamer :-)

Drummers Click Alternative

2003-05-24T17:45:00.000+01:00

Drummers click alternative [myworld] - ScottiouS - @ 14:45:02

I've found a great solution to the annoying beep in your ear that keeps you in time.

A friend let me in on this one which I haven't tried yet but I'm sure that you will be interested Ramon.

Well you know the shock packs you buy for games which you strap to your back and it pumps a low level sound or something into your back. Well simply get one and plug it straight into your click track's headphones output and it'll pump the click into the pack. Neat hey!

You can buy them from JB Hi-Fi for about $20

And when your not using it on ur drums plug it into the tv out and feel the tremble of the movies you watch.

Music via JavaScript - (Reading the floppy drive from the browser ? :-)

2003-05-24T17:42:00.000+01:00

Umm,

Click on this link to take you to some weird javascript that uses your floppy
to make music, and yes I wrote it ;)

floppy music

(just couldn't resist could you.. :-)
and They said I'd grow up!

MOding the XBox with an OZXChip

2003-05-21T17:41:00.000+01:00

Reason: to run linux..
details so far.

I mod'd the xbox.

The new BIOS screen loaded up straight away.. no issues. (so far)
(yet to boot linux or put in new harddrive)

A good note I read was, before you mod the xbox, burn a copy of
a music CD and see if you can use the standard Xbox music copy
thingo to 'rip' it to the harddrive, if you can then the CD
brand you used should be okay, some DVD drives don't like CD-R's

I used a cheapey CDR and it ripped okay.. I have heaps of CDRW's
which are said to have far less problems.

I put the BIOS disable/enable switch at the back
(drilled hole in plastic and metal sheeting)
I completely dismantled it to do this though.

There was a few tricks, in all.

1. the picture in 4b of instructions does not give a clear view
of which pin to solder the wire onto.

This piccy helped me
modchip pikky

2. my wire was a little taught for my liking if I went around the board
so I ran it through one of the spare holes in the LPC plug where the
modchip goes.

the wire can be soldered to the top or bottom,(i think)
but the bottom has a lot better access and less obstacles.

3 to remove the metal tray, you have to remove the front panel
after you have removed the motherboard, usb ports, powersupply.

the front panel is clipped on the sides and three clips along the front
inside (easy to see with everything removed)
I just gently prised one side clip and the closest front clip to that
side to pop it off.

4. The fan, unclips at the bottom, either side, and slides up and
unclips again at the bottom to come out.

the back of my xbox looks like

(x) (o)
[vga/soundport] [net]

(x) is the switch.. it's high and clear of the board inside and
plenty of room there, I think it's nicer
at the back.

(o) is a dummy hole (only in metal, not in plastic)
next to the installed toggle, i put 2nd hole in the metal
sheet, but not in the plastic, (in case i want to add that 2nd toggle
for BIOS protection or something other later).

ramon

Paper jams on HP LaserJet and Canon LBP Printers (a free fix)

2003-05-19T17:39:00.000+01:00

Does you Laser printer take too many sheets ?

Does it Jam when it takes too many sheets ;)

This might be your answer, HP has it.

Found a fix for it (a bit left field).
Apparently there is a fault with some
HP lasejets and the fix is provided by HP
HP Support.

Read more from these URL's google

The fix is a tool they send out, for Free and you use this tool to fix the printer.

Basically, Our printer is a Canon LBP-660 and about a year a go it started taking
2 then 3 then more sheets of paper.

I scouted aroud for a DIY fix and found a reference somwehere for paper jams
and the HP laserjet. Knowing that some printer engines are the same (manufactured
by same company) I did some more googling and found that the canon printer I have
is similar to a HP one (just ngine wise, not firmware interface etc).

So, thought I'd give it a go, (it was free).

The parcel took about 2 months to arrive.. but, it worked.

The 'thing' that HP gives away is like a sticky label that is installed
into the printer using this tool they provide. (weird)

.. why post this now? Well 9 months has past and the printer is at it again
and luckily I have found the URL. so .. when I need to find it next time
I can just look on my website ;)

I can't remember which of the printer 'tool' version I ordered. Maybe it
was for the 6L that I used. The difference is in the shape of the tool
that they provide.

The tool is made of cardboard

First Sync in linux with my m505

2003-04-21T17:33:00.000+01:00

First thing would be to get the palm pilot syncing. I have a Palm m505
and it comes with a USB cradle only. First thought was that I would need
to go out an buy a serial sync cable to get the console.

But thanks to some awesome work, there is the visor module (created from
the original HandSpring Visor as it was the first Palm to have USB
as it's syncing technology.

So .. to get syncing...
I used this site to do my first linux sync.
http://home.earthlink.net/~cleryd/

Best part of all this was simple steps. Because I run debian (any many
other distros support this also), all I had to do was



      modprobe visor

This loaded the module into the kernel.
and I saw this spew up on the console, always a good sign (no errors)



      usb.c: registered new driver serial

      usbserial.c: USB Serial support registered for Generic

      usbserial.c: USB Serial Driver core v1.4

      usbserial.c: USB Serial support registered for Handspring Visor

      usbserial.c: USB Serial support registered for Palm 4.0

      usbserial.c: USB Serial support registered for Sony Clié 3.5

      usbserial.c: USB Serial support registered for Sony Clié 4.0

      visor.c: USB HandSpring Visor, Palm m50x, Sony Clié driver v1.5

next bit was to get the pilot-link

(thanks debian ...)



      apt-get install pilot-link

I notice there is another 'type' of syncing software, coldsync.
Didn't / haven't tried it, will tell more when I know more I guess.

So now, the test, let's back this baby up.



      mkdir somedir

      cd somedir

      pilot-xfer -p /dev/ttyUSB1 -b .



      * Press the HotSync *

:-( didn't work..

oh wait, I need the core usb stuff loaded...
(* took a guess, ohci (either that or uhci, there is a way to
work out what type of bridge you have, it's all in the lspci
from memory) *)



      modprobe usb-ohci

That worked !!

So now , try again,



      pilot-xfer -p /dev/ttyUSB1 -b .

      * Press the HotSync button, whammo! *

I saw this on the console..



      usbserial.c: Palm 4.0 converter detected

      visor.c: Palm 4.0: Number of ports: 2

      visor.c: Palm 4.0: port 1, is for Generic use and is bound to ttyUSB0

      visor.c: Palm 4.0: port 2, is for HotSync use and is bound to ttyUSB1

      usbserial.c: Palm 4.0 converter now attached to ttyUSB0 (or usb/tts/0 for devfs)

      usbserial.c: Palm 4.0 converter now attached to ttyUSB1 (or usb/tts/1 for devfs)

Now, notice a few things here, I had no idea if the Palm was on
ttyUSB0 or ttyUSB1. But I just followed what was done before,
and lo and behold, it was the right one because if you see,
The USB interface must have two logical ports, which the visor kernel module
must map to the 'serial like' devices USB0 and 1
.. and the "HotSync use and is bound to ttyUSB1" says it all.

cool!

Now on to compiling my first linux kernel into this puppy and blow it up!

Satellite SM200D Installation Instructions for Smoothwall (and IPCop)

2003-03-20T15:38:00.000Z

Okay..

After a huge delay, I have finally managed to put the smoothwall/satellite mod together...

smoothwall_sm200d_1.0.tgz - my v1.0[415k]
list_of_new_files.txt - A list of the files I have changed/added.

The original drivers.
SM200D SkyMedia Telemann v2.20 drivers - For SM200Dxx Series.

Helps to see what I did if you diff each file against the original smoothwall file.
It really wasn't that hard. Just a lot of working out. So use what I ahev done and go forth!

eg: tar cvfz /tmp/originals.tgz -T list_of_new_files.txt
diff -r

What's in there ?

Compiled 2.20 Text Drivers (app and module) against the 2.2.25 Kernel (smoothwall kernel in 2.2.25) (using debian of course)

Built for smoothwall 1.0 patch level 4.

Control scripts to load the module and change firewall rules (at start up)

A web interface to put in the settings you need.

How do I use them
Backup your smoothie, (of it's a new one, don't worry you've got the CD ;)

patch it to level 4. (if they go beyond level 4, let me know and I'll check
this tar file for conflictions).

Setup smoothwall to be a normal dialin machine. (Red Modem is Dialup)

Using scp (scp, WinSCP etc etc), copy the tgz to /tmp

cd / && tar xvfz /tmp/

Go to the smoothwall admin and you will see a new link under [dialup] satellite. Put your settings in there.
I just realised that the IP address is irrelevant. leave it as 1.1.1.1 if you want. I used to use it where the dialup account was not meant for dialup (was given a different IP address via PPP)

Email me if you have any issues - ramon_at_thebuckland.com and I'll see if I can help.

How did I compile them
1. Loaded up a 2.2.25 kernel in debian to match that of smoothwall. (this isn't necessary I believe)
2. Obtained the kernel headers for 2.2.25.
3. Downloaded the drives from telemann's site.
4. Ran the INSTALL script once.
5. Edited the INSTALL script and commented out the 'gzip lines for each section'
6. edited drv/drv.mak and added -I/usr/src/kernel-headers-2.2.25/include
to the end of CFLAGS (after the -O)
7. Ran the INSTALL again.
8. This did create some warnings, but all googling showed that they were okay.
... but I can't test these drivers as I no longer have this card. This is just a service to you :-)
hope you appreciate it.
9. collected the bits together (in the Release directory)
10. Used the smoothwall 1.0 patched to level4.
11. Added my mods created back in April last year.

Some notes
I have noticed that there is a mysterious 2.30 version of the driver but I was never able
to find it. A friend pointed me to an smdstats program that someone wrote. Text based, similar to the Windows stuff, shows the signal strength. (Excercise for the reader to get it into smoothwall).

This 'hack' is not suggested to work against a smoothwall > 1.0 pl 4.

I don't have a card, so these drivers may not work. But they error saying they can't
find a card okay so that is always good.

If you want to donate me a card ;) i can test them and get them working proper.

Extra bits
I used to use my satellite service with a dialup ISP that did not know about it.
What this meant was that the dialup account I used did not have the satellite IP address.
It was dynamic.

Basically, on top of this patch above, what I did was added some lines to the /etc/ppp/ip-up
to create an alias interface so my packets would go out with the source address of my satellite

Add this after
system('/usr/bin/killall', 'dnrd');
system @commandlist



# change the ppp0 to ppp0:0 for satellite

&readhash("${swroot}/satellite/settings", \%satsettings)



if ($satsettings{'ENABLED'} eq 'on')

{



$localip = $satsettings{'IP_ADDRESS'};

$interface = $interface . ':0';



# ifconfig ppp0:0  netmask 255.255.255.255 pointopoint 

er-IP>



&log("Satelite: < ".$interface."> is now ".$satsettings{'IP_ADDRESS'});

system(@cmd,('/sbin/ifconfig',$interface, $localip,'netmask',

         '255.255.255.255','pointopoint',$remoteip));



system(@cmd,('/sbin/route','del','default'));

system(@cmd,('/sbin/route','add','default','gw',$remoteip,'dev',$interface));



}

Cheers!
Ramon