bash commands, xml cleaning and Java Searching

A lot of you are Linux users, some Mac and *NIX and the others, I want you to be *NIX. (perhaps use cygwin for a bit).

This post is primarily about two things .. helping you in the command line and helping you develop.

I am, as most people know a Java developer (by trade) and work a lot with XML. I found two things annoyed me for some time a year or two back.

1. Finding the jar of some class I had not seen (and it may be in my m2 repository)
2. Cleaning my XML files (especially the pom.xml files) so they were neat.

I solved both of these with two seperate bash functions. A bash function acts like command on the command line, so if you have




function foo () { echo "foo and Hello World together" }



in your .bashrc, when you "source ~/.bashrc or open another shell, the command "foo" will work.

Cobling this together with some good old unix commandline foo, you can do amazing stuff (I remember the first complex function I wrote was a wrapper for df (disk filesystem) and using awk to show the percentage of usage (of about 2MB) that I had used on the UNE CS Dec Alpha Server, mihi and later neumann).

Sadly I don't have that command anymore, but I don't need it either as df -h gives you the wonderful details. I digress.

Back to functions; it is all well and good to have them, but if you don't remember them, or how to run them, they become dusty (and your self help shell foo rusty).

To solve this problem, I created my own simple help system, "myhelp" nothing robust, nothing fancy but it all works within the one .bashrc file. The principle is simple. Any function I write for me, adheres to two principles.

1. It calls a "help" function first off, which checks if the first argument is --help and promptly displays the help and exits the function.
   
2. After the first { on the same line as the "function ..." brace, i add a comment # myFunction which signifies it is special and help can be listed for it. (and that is is my function)
   

Now to the functions themselves.
1. First the XML Cleaner.

#
# Use XMLLint to reformat the XML
#
function xmlclean () { # myFunction
myShowHelp $1 "xmlclean" " - Reformat the XML Document" || return

doc=$1;
tmpDoc=/tmp/xcleaner.$$.xml
cp ${doc} ${tmpDoc} && xmllint --format ${tmpDoc} --output ${doc} && rm ${tmpDoc}
}

The indentation is a little skewed (I'll try and resolve that later). This bash script uses xmllint

sudo apt-get install libxml2-utils

Fairly simple, in fact I THINK that the xmllint can do an inline replacement, but hey mine works.

so .. formatting an XML file is simply ..

rbuckland@ld630:~$ xmlclean foo.xml

2. Now for locating classes in Jars ..

#
# find a java class in the Repo
#
function jfind () { # myFunction
myShowHelp $1 "jfind" " - Find a classname(substring also) in Maven2 jars. The classname is passed to egrep on the jar contents" || return
find ~/.m2/repository/ -name '*.jar' | xargs -l1 -ixx sh -c "jar tvf xx | egrep $1 && echo xx"
}

An example of it in action:

rbuckland@ld630:~$ jfind DxmlExecutor
1842 Fri Jul 04 09:09:06 EST 2008 com/mivira/dxml/core/DxmlExecutor.class
/home/rbuckland/.m2/repository/com/mivira/dxml/dxml/0.9.1a/dxml-0.9.1a.jar

I won't go into what it is doing (it's pretty simple) just run each command seperately and you will get it all.

3. Now for the help system,

To show your custom commands you have written, type the following.

rbuckland@ld630:~$ myhelp
jfind - Find a classname(substring also) in Maven2 jars. The classname is passed to egrep on the jar contents
xmlclean - Reformat the XML Document
rbuckland@ld630:~$

To see the help for one command ..

rbuckland@ld630:~$ xmlclean --help
xmlclean - Reformat the XML Document

The myhelp works simply by looking for all "function name () { #myFunction" lines in your .bashrc and runs functionName --help for each one.

here is the two help functions you need in the .bashrc.

# declare -F shows the functions, and the shopt 'extdebug' is meant to show the line numbers
# of the functions (and src file) but it didn't so I am going with a more fool proof
# way of identifying my functions .. grep the .bashrc for all function lines that have the
# comment on the same line (hack but works)
#
function myhelp () {
for i in `grep myFunction ~/.bashrc | cut -f2 -d' '`
do
$i --help
done
}

function myShowHelp() {
if [ $1 == '--help' ]; then
echo $2 $3
return 1
fi
return 0
}

That's it. Now when I add a new function. All I have to do is create it according to my template which is

function mynewcommand () { # myFunction
myShowHelp $1 "mynewcommand" "description" || return

# do all the work here
}

and that's it, re-source my .bashrc (source ~/.bashrc) and test it out. and myhelp just automagically picks it up for me.

Done!

Addendum: As is is often the case, looking at my old code I make imporevements and posting this blog entry did just that.

I figured that maven related tasks should be prefixed with m2. and also I added another helper for maven.

Here is my full set of relevant helpers.
Cheers

#-----------------------------------------------------------------------
# declare -F shows the functions, and the shopt 'extdebug' is meant to show the line numbers
# of the functions (and src file) but it didn't so I am going with a more fool proof
# way of identifying my functions .. grep the .bashrc for all function lines that have the
# comment on the same line (hack but works)
#
function myhelp () {
for i in `grep myFunction ~/.bashrc | cut -f2 -d' '`
do
$i --help
done
}

function myShowHelp() {
if [ $1 == '--help' ]; then
echo $2 $3
return 1
fi
return 0
}

#
# print out all the matching jars in the repo
#
function m2showversions () { # myFunction
myShowHelp $1 "m2showversions" " - Simply lists all the jar filenames that match. Infer from that the versions you have on disk"

(cd ~/.m2/repository && find * -name "*$1*")
}

#
# find a java class in the Repo
#
function m2find () { # myFunction
myShowHelp $1 "m2find" " - Find a classname(substring also) in Maven2 jars. The classname is passed to egrep on the jar contents" || return
find ~/.m2/repository/ -name '*.jar' | xargs -l1 -ixx sh -c "jar tvf xx | egrep $1 && echo xx"
}

#
# Use XMLLint to reformat the XML
#
function xmlclean () { # myFunction
myShowHelp $1 "xmlclean" " - Reformat the XML Document" || return

doc=$1;
tmpDoc=/tmp/xcleaner.$$.xml
cp ${doc} ${tmpDoc} && xmllint --format ${tmpDoc} --output ${doc} && rm ${tmpDoc}
}
#-----------------------------------------------------------------------


hope that helps

Hoax Emails - Internet Banking Fraud

In my time of collecting and monitoring SPAM I have noticed that I have not received many Internet Banking Fraud / Hoax emails. It might just be me but it seems a bit odd that the only one that I have received are for Internet Banking sites which I use. So, for example, I use ANZ Internet Banking and just today received my 3rd Internet Banking Hoax Email, this one was for ANZ.

So how do the fraudsters (server registered in Chile (.cl) ) know that I bank ANZ, and why don;t I get NAB, St George or Bank Brunei Email Hoaxes ? The answer is that they KNOW I bank ANZ.

What would be some ways they obtain this list ? I can certainly think of some. And I will add also, ways for the Banks to prevent it, and also for us to be mindful (as users).

Buying the list
At first you would hope it doesn't occur, but when you think about it, it is quite plausible. A high enough price put up entices someone with access to "sell" the list.

Imagine that of 100,000 emails sent out, only 100 people (.001%) provide their details. And if from this, the fraudsters were able to transfer an average of $500, that's a quick easy $50,000.

Of course, some accounts have $50,000 in them to start with, so the bounty is much more of course.

So let's go with $50,000. We can pay $5,000 to the list provider, all they have to provide is Email Addresses of valid customers, nothing more.

$5k is not a lot, but if it were sold for $5k to 3 "fraud" groups, then the insider pockets $15k.

A security breach is 90% of the time an insider job and is the one thing that banks must watch out for.

So who could be an insider ? Well simply put anyone with access to the email list.

This could be:

• Anyone in IT with sufficient privileges (to the test, development, or production database).
• Anyone in Marketing who can extract / export a list of email addresses for bulk email
• Anyone in Customer Servicing with this type of access
• The Vendor of the Banking System who is given the Database in original form for issue resolution (and of course then, any of the vendors employees)

So the insider is an issue. We know it, but it really is as scarey as that. So many IT systems are left open or "available" for reading but all users. I have certainly worked with my fair share that do.

Moving On, so, if it's not an insider job, how else could it be done ?

The ISP Proxy Server
All requests that I make from my ISP (ie: at home) are logged, not only my ISP at home, but at work also.

Might seem odd, but this data is worth money.

At the ISP, they just have to marry my email address and a list of known "banking" logon sites that I access, then they have a decent list. Of course it is not ALL users, but a big ISP (Telstra Bigpond in Australia) would be a nice size and, sadly, the bigger the ISP, the more un-suspecting the user will be and more likely to click on that link.

At a stretch
Virus / URL Logger / Spy
So this one is kind of logical, but probably not done. If there is software on my PC which is "logging" my surfing, they could use that. But if it's on my PC, then well, might as well log the logon to the internet banking site also.

So How do We Prevent it ?
I'll talk about us and the Bank.
For us

(a) Change banks to one which provides
(i) SMS Bank Pay Anyone Verification (With the CommonWealth Bank - not ANZ) for any new Pay Anyone Account not in my list, I need to verify it with an SMS code which is sent to my Mobile phone. That way, I must have my phone with my to pay anyone.

(ii) Secure ID Number Generators for login
HSBC in Australia and HSBC Commercial in the UK use these. A new number is generated every 60 seconds, and I need to supply my ID, password and this number to login.

Fraudsters won't target banks which use either of these (a,i or a,ii) , they either can't get in, or can't do anything once they are in.

(b) NEVER, EVER click on a link in an email for Internet Bank, for changing password, or for reactivating an account, or any activity. Go through your banks website directly, or ring them.

(c) Change banks to someone that DOES provide good access.

For the banks

Where do you start ?

First - Secure internal Access

Perform a full audit of all access methods to the "customer data" at length. This will show holes on how data can come and go, and don't think that just stopping USB access will prevent a user from getting large lists of data out. Where there is a will, there is a way. a quick UUENCODE and a Cut'n'Paste to a pastebin.ca will dump data out in next to no time. If that doesn't seem plausible, what about some steganography ?

Second - Audit Access

Goes without saying, but you would be suprised. Audit all access to the data, so that if data does get out, you know when and by whom (hopefully).

Third - Provide Decent Security Features

It is almost ignorant for banks to NOT have a decent security feature such as SMS verification of Number Generator Fobs for use. They can tell us all they like what NOT to do, but really there are technologies out there which now prevent this type of attack from working, so just get to it an implement it.


There is MUCH more to be said on this topic, but that will get you thinking for now.