Plotting graphs on the Mac

I am developing an iPhone pedometer application that uses the accelerometer to count paces. My tests generate a lot of data - 40 readings per seconds - that I wanted to visualise some how to allow me to develop an algorithm to count the paces. After a test run with my POC app I had a file with about 100,000 lines of float data.

I first tried Apples Numbers which is a part of iWork which chocked on the data and became impossibly slow to use. It also does not have many options regarding the presentation of the graph.

After a quick Google search I found gnuplot which can be installed using macports like this

: sudo port install gnuplot

However it would not start on my Leopard machine until I reinstalled X11 from here:

http://xquartz.macosforge.org/trac/wiki

after installing the latest X11 everything ran perfectly. It plotted my 100k points in a split second making Numbers look like the bloated eye candy that I have come to expect from iWork. The degree of flexibility is amazing and the ability to use standard unix file manipulation commands adds even more options. But I was a bit frustrated that I could not use transparency in the current version (4.2.4) and it was a pain in the ass to change the background colour; you need to set a command line property when you launch it.

Then I found Plot which seems to have a lot of the options that gnuplot has but all packaged up into a nice UI.

http://plot.micw.eu/

It was easy to change background colours and set transparency etc. I think this will be very useful in visualising my data points and creating some pretty graphs for documentation pages. It does not handle 3D graphs but that is not a problem for me.

The options in Plot are astounding - curve fitting, smoothing, many buttons that have complicated sounding names. It all makes me wonder why anyone would create such an elaborate program for free. Perhaps they make money from donations? Or perhaps they don't think about money quite as much as I do. Or maybe that is the reason that the last version was released over two years ago and has not reached 1.0 yet.

But good workMichael Wesemann of Berlin! The world is a slightly better place due to your free program.

Thread dump in Jetty

Normally if you have a problem with dead-locked threads or you want to know what is taking so long in a running application you can get the JVM to do a thread dump for you by sending the QUIT signal like this:

First get the process id using

ps | grep java

then send the quit signal

kill -QUIT <your-pid>

But this was not working for me on my server running Jetty 6.1 as installed by the .deb installation. The logs showed nothing.

After a bit of searching on the interweb I found the new (to me) method of generating stack traces using

sudo jstack <pid>

This was slightly more helpful and told me:

"Unable to open socket file: target process not responding or HotSpot VM not loaded"

At last I had something to go on and found that jstack must be run as the same user as the java process - which in the case of Jetty as installed by the debian install files is "jetty"

sudo -u jetty jstack <your-pid>

worked a treat and so did

sudo -u jetty kill -QUIT <your-pid>

Debugging in Eclipse very slow

I hit a problem developing with App Engine which took me a long time to figure out.  I have been flipping between projects so didn't really notice when GAE and GWT started so run so slowly.  I have also been experimenting with JRebel and messing with JVM memory parameters.  Well "suddenly" every started running really really slowly and App Engine would take around 10 minutes to start up and run my 6 async parallel queries.  Was it the new version of App Engine Java introducing a concurrency problem?  Was it that my local datastore local_db.bin was too big - holy shit its 10MB!

After a lot of kafuffle I finally removed all break points and viola!  Every thing was lightening fast again!  What a massive waste of time.

Actually I have just added a really nifty optimisation to Twig for when you are merging multiple parent queries (relation index entities) which is needed for Target Rooms geo search.  It now asks each child query which entities it wants and gets rid of duplicates before sending off a single request for all of them.  When any of the children needs more parent entities it asks this EntitySupplier for more which in turn asks all of the children again if they need any more.

It has speed up the geospatial search HUGELY!  I only recently realised that the complex part of the query (searching 6 blocks of geospatial data in parallel and sorting it by price) was completing in about 50ms!!!  The simple part (bulk get of the parent results) was taking between 200 and 1200ms to do 6 bulk gets.  What a surprise.  You can actually see in the Google App Engine Status page that some days each simple get (not even bulk get) can take like 500ms.

Now Twig combines all bulk gets for each child query into a single bulk get which has made a huge improvement.

Firefox Radio Button Alignment

I have been doing a bit of front end work on a web site and while I am still developing ideas I have been viewing my results in Safari because that it allows me to use cool CSS 3 effects like -webkit-box-shadow and -webkit-border-radius for rounded corners. This is great for design mock ups because it is so much quicker than creating bitmaps and then maybe deciding not to use them.

I can see that WebKit also has a lot more install for Safari users with gradients, masks and more. But my current version of Safari does not include these features yet.

But then I noticed that when I looked at my page on FireFox all the radio buttons seemed out of alignment and looked very messy. I wasted hours trying to figure out if my style sheet was the culprit. Then I noticed that even Google looked weird so it was not something I was doing to mess up the formatting:

Safari:

Internet Explorer:

Firefox:

Notice how the radio buttons seem to float above the text.

After a bit of experimenting with various alignment options I found that setting vertical-align: text-top made the alignment look OK in Safari and Firefox but terrible in IE.

For now I have given up on Firefox and just accept that its looks a bit shit.

Copying music off an iPod

My external hard drive broke taking with it my iTunes library. All my music is also on my 80GB iPod but iTunes annoyingly does not let you copy music from your iPod back to your computer. I guess it was one of the conditions Apple negotiated with the record companies when setting up the iTunes store.

When my iPod is connected to my Mac Mini it shows in the Finder as an external hard drive but the music folder is hidden. From the command line

john@mini:~$ ll /Volumes/John\ Patterson’s\ iPod/
total 64
drwxr-xr-x   4 john  staff   136B 30 May 19:55 Backup
drwxr-xr-x   2 john  staff    68B  1 Jan  2000 Calendars
drwxr-xr-x   2 john  staff    68B  1 Jan  2000 Contacts
-rw-r--r--@  1 john  staff   1.0K 16 Dec  2007 Desktop DB
-rw-r--r--@  1 john  staff     2B 16 Dec  2007 Desktop DF
drwxr-xr-x   3 john  staff   102B  1 Jan  2000 Notes
drwxr-xr-x   6 john  staff   204B 18 May  2008 Photos
drwxr-xr-x   2 john  staff    68B  1 Jan  2000 Recordings
drwxr-xr-x@ 11 john  staff   374B 16 Dec  2007 iPod_Control

Now we can see the iPod_Control directory that contains all the music
Simply copy the music onto your local disc:

john@mini:~$ cp -r /Volumes/John\ Patterson’s\ iPod/iPod_Control/Music/ /Volumes/External/Music/iPod

The folders are still hidden so you need to remove the hidden attribute like this

xattr -d com.apple.FinderInfo /Volumes/External/Music/iPod/F*

And when that is done drag the whole folder onto your iTunes icon and all the music files will be imported into your current library. If you have iTunes set up to keep your music folders organised it will rename the cryptically named music files and create correctly named folders for you. Neat.

Watching BBC online overseas

I need to watch the All Black playing Ireland on Sunday but here in Spain I have no idea where to watch the game with English commentary. BBC makes a lot of video content available on their website but only if you are in the UK.

I have servers running in the UK and knew it must be possible to somehow use them as a proxy so that the British Broadcasting Commissioners think that I too am in old Blighty. But how to hook it all up?

After a bit of Googling on the interweb I discovered that OpenSSH can be used as a SOCKS proxy which creates a secure connection to any server you can ssh into and then fires off your requests from there. OpenSSH is the flavour of SSH that runs on Mac OS X so all I had to do was type this at the terminal to start the proxy:

ssh -ND 9999 myserver.co.uk

The -N just tells ssh that I don't want to send any commands - just set up the proxy and listen on local port 9999.

Then you need to tell your browser to use the SOCKS proxy. Safari proxy settings are just the system wide network preferences. Just click on advanced and then Proxies and add a SOCKS proxy at localhost port 9999 and voila!. But I don't want all my browsing to use the proxy - most sites should connect directly.

So I installed the handy little FireFox plugin, FoxyProxy, which can selectively send some requests through the proxy and not others. It looks pretty flexible but I simply set it to use my proxy for all URL's and then used Safari as normal for direct browsing.

Now I can watch Top Gear on BBC too! And also, when I look at my website which has advertising including Google AdSense and Yahoo PPC ads I can see the same ads that users of my site in the UK see!

Nice.

Filtering resources in Eclipse

When you have a lot of files in an Eclipse 3.4 workspace things pretty much grind to a halt. I have a folder containing 200,000 images for a web project I'm working on and I need to include in the workspace. Starting Eclipse with this folder takes forever and Subversive hangs trying to to something with all the files.

I filed a bug report on this issue 3 years ago! https://bugs.eclipse.org/bugs/show_bug.cgi?id=84988. Eclipse development seems to move slowly.

I can exclude files from the Package Explore using a filter but the problem remains. Under the covers Eclipse is still monitoring these files at the resource level.

I found this plugin which claims to do what I want but I could not get it to work for me.

https://sourceforge.net/projects/eclipseresfltr/

Some people have posted about adding a "resource filter" to filter out files at a lower level

In the Resource Navigator (Window -> Show View -> Navigator) you can choose to filter resources that match .* or *.class but you cannot add your own filters. These default ones are added by plugins as extension points.

But you can add your own filters by messing with an existing Plugin and adding a filter definition to its plugin.xml. Just find a plugin that is not in a jar file - they are signed so you cannot alter their contents. I chose the jdt plugin org.eclipse.jdt.debug_3.4.0.v20080604 and added this to its plugin.xml

<extension point="org.eclipse.ui.ide.resourceFilters">
<filter selected="false" pattern="my-large-folder"></filter>
</extension>

However, this does NOT stop the resources from being scanned for updates for some reason. It just hides the folder from view but my machine was still pegged on close to 100% CPU.

Then I re-read one of the comments on my bug report above and finally solved the problem.

1. Move the large folder outside your workspace.
2. In the Package Explorer New -> File
3. Enter the name of the large folder as the filename
4. Click Advanced and check "Link to file in the filesystem"
5. Enter /dev/null (on Mac OS X) and click finish
   
6. Move the large folder back to your workspace.
   

Now the null file hides your folder. I also added a filter to the Package Exploer so the linked null file does not show. Now when I refresh the workspace or restart my machine no longer hangs. Nice one.