We have Twig!

I have just made my first commit to Twig on Google Code: http://code.google.com/p/twig-persist/

This is my first open source project and in all likelihood my last. I worked quite solidly for two weeks on it although at first I imagined it would be 3 or 4 days work. So probably one of my more accurate predictions actually!

My meagre unit tests pass and now I can spend some time actually using it on a real project before alerting the rest of the app engine crew that there has been a shift in the force. JDO is not my father... it is a piece of shit.

GWT - Unhelpful RPC serialization message

There are a number of causes for this error message:

com.google.gwt.user.client.rpc.SerializationException: Type ‘blah.blah.blah’ was not included in the set of types which can be serialized by this SerializationPolicy or its Class object could not be loaded. For security purposes, this type will not be serialized.

Check: I have a no args constructor (private is OK)

Check: I have implemented IsSerializable

Check: It is on my source path for my module - in

What else could be wrong?

It turned out that one of the fields within my class was not Serializable due to not implementing IsSerializable.

Recover deleted photos on a Mac for free!

Oops... someone deleted all the photos on the camera after our holiday instead of just one. It was Mr or Ms nobody. Fear not, because as long as you don't fill the camera with new photos the old ones will still be there - just hidden from the operating system.

Firstly you need a usb card reader so your computer can get direct access to the "drive". I bought a cheap one from Tescos here in Thailand for 140 baht (3 quid) that reads my large CF card and many others.

I downloaded a free demo of a program called PhotoRecovery for Mac by AppleXsoft that found my photos but wanted 50 bucks to give them to me. That gave me the confidence that they were indeed still accessible so I googled on the interweb a bit more and came up with this:

http://www.cgsecurity.org/wiki/PhotoRec

It is a command line tool you can download or you can install it straight from the brilliant MacPorts by typing

sudo port install testdisk



TestDisk is the package that contains the photorec command.

Now type "photorec" and the UI kicks in to guide you through the process of selecting the drive etc.

I chose the drive by guessing from its size - my card was 1GB so I chose the 991 MB option

Then I had to guess a filesystem type - I chose EFI for lack of a better idea.

Then I chose where to save the files and the program started doing its thing. Like so:

and low and behold the damn thing worked like a charm and spat out all my deleted photos into the folder I chose.

Sing hallelujah!

Dropbox as source code repository

I have just started to use Mercurial as my SCM on my latest project and so far it seems great. When you have a functioning unit of change you commit it to your development local (or remote) repository and when you have an entire feature ready you can push it to your "central" repository for testing. In theory repositories should not have half finished non-compiling work in them.

But often I move from my Desktop to my laptop in the middle of development while everything is still up in the air. I would also like a backup of my work in progress without having to commit to the repository. This is where Dropbox comes in.

Install Dropbox from www.getdropbox.com and it creates a folder that is always synchronised to the server as long as you are connected to the internet. It is free for up to 2GB of storage, private unless I choose to share files, and has simple revision control to revert back or download an older version from the web interface. Brilliant and simple.

I created a symbolic link from ~/Dropbox to my project folder and suddenly every time I save a file it is synchronised with the version on the server. I don't need to remember to commit small changes and always have the latest version when I change dev machines.

The best part is that I am no longer tempted to check in half backed code.

Google App Engine with Maven

Google App Engine has an Eclipse plugin to help with uploading and enhancing your JDO classes. It also sets up the dependancies and exposes the GAR API. Very handy. But it doesn't play nicely with maven because the two of them assume a different project layout. Maven expects your webapp source to live under src/main/webapp/ and it builds an exploded war under target/ . Meanwhile, the Google App Engine plugin for eclipse is hardcoded to look for both source and target under the root folder war/

The google team will probably allow this configuration soon but until then you can update your maven pox.xml like this:

<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-war-plugin</artifactId>
<version>2.0</version>
<configuration>
<directory>war</directory>
<warSourceDirectory>war</warSourceDirectory>
<webappDirectory>war</webappDirectory>
</configuration>
</plugin>

You may also need to create a appengine-web.xmlfile under war/WEB-INF/ to get the plugin to work if you are using appengine on an existing project.

Maven error

In case someone comes accross the same problem - I came hit this exception and fixed it by deleting my target directory and clean-ing the project from Eclipse to rebuild the classes.

[INFO] [war:war]
[INFO] Packaging webapp
[INFO] ------------------------------------------------------------------------
[ERROR] FATAL ERROR
[INFO] ------------------------------------------------------------------------
[INFO] modelEncoding : modelEncoding
---- Debugging information ----
message : modelEncoding : modelEncoding
cause-exception : com.thoughtworks.xstream.mapper.CannotResolveClassException
cause-message : modelEncoding : modelEncoding
class : org.apache.maven.plugin.war.util.WebappStructure
required-type : org.apache.maven.model.Dependency
path : /webapp-structure/dependenciesInfo/org.apache.maven.plugin.war.util.DependencyInfo/dependency/modelEncoding
line number : 1108
-------------------------------
[INFO] ------------------------------------------------------------------------
[INFO] Trace
com.thoughtworks.xstream.converters.ConversionException: modelEncoding : modelEncoding
---- Debugging information ----
message : modelEncoding : modelEncoding
cause-exception : com.thoughtworks.xstream.mapper.CannotResolveClassException
cause-message : modelEncoding : modelEncoding
class : org.apache.maven.plugin.war.util.WebappStructure
required-type : org.apache.maven.model.Dependency
path : /webapp-structure/dependenciesInfo/org.apache.maven.plugin.war.util.DependencyInfo/dependency/modelEncoding
line number : 1108
-------------------------------
at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:63)

Plotting graphs on the Mac

I am developing an iPhone pedometer application that uses the accelerometer to count paces. My tests generate a lot of data - 40 readings per seconds - that I wanted to visualise some how to allow me to develop an algorithm to count the paces. After a test run with my POC app I had a file with about 100,000 lines of float data.

I first tried Apples Numbers which is a part of iWork which chocked on the data and became impossibly slow to use. It also does not have many options regarding the presentation of the graph.

After a quick Google search I found gnuplot which can be installed using macports like this

: sudo port install gnuplot

However it would not start on my Leopard machine until I reinstalled X11 from here:

http://xquartz.macosforge.org/trac/wiki

after installing the latest X11 everything ran perfectly. It plotted my 100k points in a split second making Numbers look like the bloated eye candy that I have come to expect from iWork. The degree of flexibility is amazing and the ability to use standard unix file manipulation commands adds even more options. But I was a bit frustrated that I could not use transparency in the current version (4.2.4) and it was a pain in the ass to change the background colour; you need to set a command line property when you launch it.

Then I found Plot which seems to have a lot of the options that gnuplot has but all packaged up into a nice UI.

http://plot.micw.eu/

It was easy to change background colours and set transparency etc. I think this will be very useful in visualising my data points and creating some pretty graphs for documentation pages. It does not handle 3D graphs but that is not a problem for me.

The options in Plot are astounding - curve fitting, smoothing, many buttons that have complicated sounding names. It all makes me wonder why anyone would create such an elaborate program for free. Perhaps they make money from donations? Or perhaps they don't think about money quite as much as I do. Or maybe that is the reason that the last version was released over two years ago and has not reached 1.0 yet.

But good workMichael Wesemann of Berlin! The world is a slightly better place due to your free program.

Thread dump in Jetty

Normally if you have a problem with dead-locked threads or you want to know what is taking so long in a running application you can get the JVM to do a thread dump for you by sending the QUIT signal like this:

First get the process id using

ps | grep java

then send the quit signal

kill -QUIT <your-pid>

But this was not working for me on my server running Jetty 6.1 as installed by the .deb installation. The logs showed nothing.

After a bit of searching on the interweb I found the new (to me) method of generating stack traces using

sudo jstack <pid>

This was slightly more helpful and told me:

"Unable to open socket file: target process not responding or HotSpot VM not loaded"

At last I had something to go on and found that jstack must be run as the same user as the java process - which in the case of Jetty as installed by the debian install files is "jetty"

sudo -u jetty jstack <your-pid>

worked a treat and so did

sudo -u jetty kill -QUIT <your-pid>

Debugging in Eclipse very slow

I hit a problem developing with App Engine which took me a long time to figure out.  I have been flipping between projects so didn't really notice when GAE and GWT started so run so slowly.  I have also been experimenting with JRebel and messing with JVM memory parameters.  Well "suddenly" every started running really really slowly and App Engine would take around 10 minutes to start up and run my 6 async parallel queries.  Was it the new version of App Engine Java introducing a concurrency problem?  Was it that my local datastore local_db.bin was too big - holy shit its 10MB!

After a lot of kafuffle I finally removed all break points and viola!  Every thing was lightening fast again!  What a massive waste of time.

Actually I have just added a really nifty optimisation to Twig for when you are merging multiple parent queries (relation index entities) which is needed for Target Rooms geo search.  It now asks each child query which entities it wants and gets rid of duplicates before sending off a single request for all of them.  When any of the children needs more parent entities it asks this EntitySupplier for more which in turn asks all of the children again if they need any more.

It has speed up the geospatial search HUGELY!  I only recently realised that the complex part of the query (searching 6 blocks of geospatial data in parallel and sorting it by price) was completing in about 50ms!!!  The simple part (bulk get of the parent results) was taking between 200 and 1200ms to do 6 bulk gets.  What a surprise.  You can actually see in the Google App Engine Status page that some days each simple get (not even bulk get) can take like 500ms.

Now Twig combines all bulk gets for each child query into a single bulk get which has made a huge improvement.