Wed, 20 Aug 2008

Error Message Hell
If there's one thing anyone that works with computers hates, it's an error message that is misleading or vague. "Syntax Error", "Bad Command Or File Name", "General Protection Fault", and so forth have haunted us for ages; kernel panics, strange reboots, devices that just don't seem to be recognised by the system, and programs mysteriously disappearing likewise. The trend has been to give people more information, and preferably a way to understand what they need to do to fix the problem.

I blog this because I've just been struggling with a problem in Django for the last day or so, and after much experimentation I've finally discovered what the error really means. Django, being written in Python, of course comes with huge backtraces, verbose error messages, and neat formatting of all the data in the hopes that it will give you more to work with when solving your problem. Unfortunately, this error message was both wrong - in that the error it was complaining about was not actually correct - and misleading - in that the real cause of the error was something else entirely.

Django has a urls.py file which defines a set of regular expressions for URLs, and the appropriate action to take when receiving each one. So you can set up r'/poll/(?P\d+)' as a URL, and it will call the associated view's method and pass the parameter poll_id to be whatever the URL contained. In the spirit of Don't Repeat Yourself, you can also name this URL, for example:

url(r'/poll/(?P\d+)', 'view_poll', name = 'poll_view_one')

And then in your templates you can say:

<a href="{{ url poll_view_one poll_id=poll.id }}">{{ poll.name }}</a>

Django will then find the URL with that name, feed the poll ID in at the appropriate place in the expression, and there you are - you don't have to go rewriting all your links when your site structure changes. This, to me, is a great idea.

The problem was that Django was reporting that "Reverse for 'portal.address_new_in_street' not found." when it was clearly listed in a clearly working urls.py file. Finally, I started playing around with the expression, experimenting with what would work and what wouldn't in the expression. In this case, the pattern was:

new/in/(?P\d+)/(?P[A-Za-z .'-]+)

When I changed this to:

new/in/(?P.+)/(?P.+)

It suddenly came good. And then I discovered that the the thing being fed into the 'suburb_id' was not a number, but a string. So what that error message really means is "The pattern you tried to use didn't match because of format differences between the parameters and the regular expression." Maybe it means that you can have several patterns with the same name that will try to match based on the first such pattern that does so. But until then, I'll remember this; and hopefully someone else trying to figure out this problem won't butt their head against a wall for a day like I did.

posted at: 16:13 | path: /tech/web | permanent link to this entry

Tue, 29 Jul 2008

Django 101
At work I've started working on a portal written in Python using the Django framework. And I have to say I'm pretty impressed. Django does large quantities of magic to make mothe model data accessible, the templating language is pretty spiffy (it's about on a par with ClearSilver, which I'm more familiar with - each has bits that the other doesn't do), and the views and url mapping handling is nice too. I can see this as being a very attractive platform to get into in the future - I'm already considering writing my Set Dance Music Database in it just to see what it can do.

So how do I feel as a Perl programmer writing Python? Pretty good too. There are obvious differences, and traps for new players, but the fact that I can dive into something and fairly quickly be fixing bugs and implementing new features is pretty nice too. Overall, I think that once you get beyond the relatively trivial details of the structure of the code and how variables work and so on, what really makes languages strong is their libraries and interfaces, and this to me is where Perl stands out with its overwhelmingly successfull CPAN and Python, while slightly less organised from what I've seen so far, still has a similar level of power.

About the only criticism I have is the way the command line option processing is implemented - Python has tried one way (getopt) which is clearly thinking just like a C programmer, and another (optparse) which is more object oriented but is hugely cumbersome to use in its attempt to be flexible. Neither of these hold a candle to Perl's GetOpt::Long module.

posted at: 13:53 | path: /tech/web | permanent link to this entry

Tue, 15 Jul 2008

The lost limericks list
After that post, I thought I'd just check which category I'd put my previous limericks in. To my horror, I discovered that I hadn't blogged them at all, but had (merely) posted them to the Linux Australia list. So I rescued them and posted them here for posterity.

That wonderful man Andrew Tridgell
Over SaMBa keeps permanent vigil.
SMB, it is said,
He decodes in his head,
And CIFS 2 will some day bear his sigil.

The great LGuest programmer Rusty,
Is virtually never seen dusty.
He eats 16K pages,
And has done so for ages,
Yet his moustache is clean and not crusty.

That marvellous girl Pia Waugh
Is certainly hard to ignore.
With her leet ninja moves,
Open Source just improves -
All Linux Australians show awe!

The Wireless Jonathan Oxer
After the three limericks I wrote about Tridge, Pia and Rusty, the conversation came up on #linux-aus about whether I could make a similar epgiram for Jon Oxer, former Linux Australia president, front-line hardware hacker and all-round good guy. It took me two months, but in an email to Jon I finally cracked it, packing much more into the rhyme than I originally thought would be possible:

The wireless Jonathan Oxer,
Waves his hand and his front door unloxer.
A remote-control loo,
And home theatre too -
If you as me, his whole house just roxor!

We tune to podcasting James Purser,
Long known as a rhymer and verser.
With his darling wife Karin
They are not known as barren:
Three children now stare at their cursor.

Send your suggestions of who should be next under the pen to paulway@mabula.net

posted at: 09:50 | path: /tech | permanent link to this entry

Sun, 15 Jun 2008

Common code in ClearSilver 001
I've been using ClearSilver as a template language for my CGI websites in earnest for about half a year now. I decided to rewrite my Set Dance Music Database in it and it's generally been a good thing. Initially, though, I had two problems: it was hard to know exactly what data had been put into the HDF object, and it was a pain to debug template rendering problems by having to upload them to the server (surprisingly, but I think justifiably, I don't run Apache and PostgreSQL on my laptop so as to have a 'production' environment at home).

I solved this problem rather neatly by getting my code to write out the HDF object to a file, rsync'ing that file back to my own machine, and then test the template locally.

I knew that ClearSilver's Perl library had a 'readFile' method to slurp an HDF file directly into the HDF object, and a quick check of the C library said that it had an equivalent 'writeFile' call. So happily I found that they'd also provided this call in Perl. My 'site library' module provided the $hdf object and a Render function which took a template name; it was relatively simple to write to a file derived from the template name. That way I had a one-to-one correspondence between template file and data file.

Then I can run ClearSilver's cstest program to test the template - it takes two parameters, the template file and the HDF file. You either get the page rendered, or a backtrace to where the syntax error in your template occurred. I can also browse through the HDF file - which is just a text file - to work out what data is being sent to the template, which solves the problem of "why isn't that data being shown" fairly quickly.

Another possibility I haven't explored is to run a test suite against the entire site using standard HDF files each time I do a change to make sure there aren't any regressions before uploading.

Hopefully I've piqued a few people's interest in ClearSilver, because I'm going to be talking more about it in upcoming posts.

posted at: 11:10 | path: /tech/web | permanent link to this entry

Wed, 02 Apr 2008

Stupid Error 32512
For a while my brother's been having a problem with his MythTV setup - the mythfilldatabase script won't run the associated tv_grab_au script when run automatically, but will work just fine when run manually. In the logs it says:

FAILED: xmltv returned error code 32512.

After a lot more searching, and a good deal of abuse on the #mythtv-users channel on freenode.org, I finally found some information about shell exit codes, and it turns out that 127 is "command not found". In other words, mythfilldatabase at that point is trying to call the tv_grab_au grabber and not finding it. On my brother's machine, this is because sh under root does not get the path /usr/local/bin, which is where the grabber is stored.

(It works on my machine because I run it from a script which picks a random time, and includes /usr/local/bin in the path.

So there are two solutions, as I see it:

1) Put tv_grab_au in /usr/bin/.

2) Run mythfilldatabase from cron using a script which includes /usr/local/bin in the path.

Given the bollocking I got in #mythtv-users for suggesting something so crude and hackish (in the words of Marcus Brown, mzb_d800) as cron, I guess I'll have to go with option 1. But here's hoping that this blog entry helps someone else out there - almost every post on the mythtv-users email list that mentions 32512 never mentions a solution...

posted at: 11:40 | path: /tech | permanent link to this entry

Tue, 18 Mar 2008

Standard Observations
Simon Rumble mentioned Joel Spolsky's post on web standards and it really is an excellent read. The fundamental point is that as a standard grows, testing any arbitrary device's compliance with it it grows harder. Given that, for rendering HTML, not only do we have a couple of 'official' standards: HTML 4, XHTML, etc., but we also have a number of 'defacto' standards - IE 5, IE 5.5, IE 6, IE 7, Firefox, Opera, etc. etc. etc ad nauseam. For a long time, Microsoft has banked on their desktop monopoly to lever their own defacto standards onto us, but I think they never intended it to be because of bugs in their own software. And now the chickens are coming home to roost, and they're stuck with either being bug-for-bug compatible with their own software (i.e. making it more expensive to produce) or breaking all those old web pages (i.e. making it much more unpopular).

I wonder if there was anyone in Microsoft Internet Explorer development team around the time they were producing 5.0 that was saying, "No, we can't ship this until it complies with the standard; that way we know we'll have less work to do in the future." If so, I feel doubly sorry for you: you've been proved right, but you're still stuck.

However, this is not a new problem to us software engineers. We've invented various test-based coding methodologies that ensure that the software probably obeys the standard, or at least can be proven to obey some standard (as opposed to being random). We've also seen the nifty XSLT macro that takes the OpenFormula specification and produces an OpenDocument Spreadsheet that tests the formula - I can't find any live links to it but I saved a copy and put it here. So it shouldn't actually be that hard to go through and implement, if not all, then a good portion of the HTML standard as rigorous tests and then use browser scripting to test its actual output. Tell me that someone isn't doing this already.

But the problem isn't really with making software obey the standard - although obviously Microsoft has had some problem with that in the past, and therefore I don't feel we can trust them in the future. The problem is that those pieces of broken software have formed a defacto standard that isn't mapped by a document. In fact, they form several inconsistent and conflicting standards. If you want another problem, it's that people writing web site code to detect browser type in the past have written something like:

if ($browser eq 'IE') {
    if ($version <= 5.0) {
        write_IE_5_0_HTML();
    } elsif ($version <= 5.5) {
        write_IE_5_5_HTML();
    } else {
        write_IE_HTML();
    }
    ...
}

    } elsif ($version <= 6.0) {
        write_IE_6_0_HTML();

Joel doesn't identify Microsoft's correct response in this situation. The reason for this is that we're all small coders reading Joel's blog and we just don't have the power of Microsoft. It should be relatively easy for them to write a program that goes out and checks web sites to see whether they render correctly in IE 8, and then they should work together with the web site owners whose web sites don't render correctly to fix this. Microsoft does a big publicity campaign about how it's cleaning up the web to make sure it's all standard compliant for its new standards-compliant browser, they call it a big win, everyone goes back to work without an extra headache. Instead, they're carrying on like it's not their fault that the problem exists in the first place.

Microsoft's talking big about how it's this nice friendly corporate citizen that plays nice these days - let's see it start fixing up some of its past mistakes.

posted at: 22:41 | path: /tech/web | permanent link to this entry

Fri, 14 Mar 2008

Beat Counter Project
For a variety of reasons, I'm looking for a library that can not only determine the BPM of a song but count how many beats and bars are in it, excluding the introduction and finish of the song where there may be no actual music. (In other words it's not just a case of dividing the track length in real-number minutes by the BPM). Furthermore, one application has the complication of working with music that isn't four quarter-beats in a bar (i.e. 4/4 notation) - it might be 2/4, 3/4, 6/8 or 12/8. I do know this ahead of time - mostly - but automatic detection would be nice. The other application will require millisecond-precision locations of each beat, and must be able to compensate for tempo changes in the song.

So I've started a MicroPledge project for it and pledged $100US of my own money. The project must run on Linux, Mac OS-X and Windows and must also use an open source license, preferably the GPLv3. But I guess this is a bit of a bleg (thank you Mary Gardiner for introducing me to that term :-) if anyone knows of such a thing or some project that I can add code to.

Now to watch it fade into obscurity as I cast around for some way to write the thing myself...

posted at: 15:23 | path: /tech | permanent link to this entry

Sun, 02 Mar 2008

Floor Wax, Dessert Topping, Make-Up, Mould Release...
As a woodworker, I use Carnauba Wax mixed with lemon oil on my wood turning pieces to give them a nice shine that's also dust-proof and preserves the wood, preventing it from drying out and cracking. And as a student of popular culture, I've seen the reference to the Saturday Night Live sketch about Shimmer, the revolutionary product that's both a floor wax and a dessert topping.

So it amused and amazed even me to find out that Carnauba Wax is all this and more. It's the product of the Carnauba Palm, has a melting point way higher than most waxes, and is harder than concrete in pure form. It is used both in woodworking and in car polishes for its high-gloss, protective coating, but in that capacity (and because it's edible) it's also used as an ingredient in some cake icings and on the coatings of Tic-Tacs and othe candy to make them glossy. Likewise it's used in products such as lipsticks and blushes for the glossy, resilient coating. With a solvent in a can, it's sprayed into moulds for epoxy resin products such as semiconductors to make sure the product breaks free from the mould easily; because it's not soluble in water or alcohol it can be used in liquid epoxy casting too.

And to think that most people think that Shimmer doesn't exist...

posted at: 10:38 | path: /tech | permanent link to this entry

Sat, 02 Feb 2008

LCA 2008 Google Party Mix
The day finally came, and though I was a ball of sweaty clothing from giving my Lightning Talk I was ready to do some mixing for the LCA 2008 Google Party. Afterward, thanks to some pre-prepared scripts, I put the mix up on my torrent server pretty soon afterward. If you want it, you can download the mix via BitTorrent or read the track listing. All the music is Creative Commons licensed and therefore my mix is also similarly licensed; I'll work out the exact license code when I've looked at the licenses on all the music, but for now I will release the mix under a Creative Commons 3.0 By-NC-SA license.

Thank you to Peter Lieverdink and the LCA 2008 team for allowing me to mix at LCA - I had a great time doing it. And my collection hat (thank you Stewart Smith) raised $24.30 to donate to the artists. I reckon that's pretty good for something completely voluntary where most people hadn't been really getting into the music much (that I could see). Now to work out how to donate it...

posted at: 20:48 | path: /tech/lca | permanent link to this entry

Thu, 31 Jan 2008

Network Interactionativity
For some reason, on certain access points at LCA - for instance the one in the St. Mary's common room - I need to set my MTU to 1000 (i.e. down from 1500) in order to get Thunderbird to do secure POP. Everything else works fine, but Thunderbird just sits there timing out. I discovered this by watching the Wireshark log and noticing packet fragments disappearing (i.e. some packets where the tcp fragment analysis couldn't find parts of the packet to reassemble). Hopefully this isn't also causing Steve Walsh to pick up his specially sharpened LAN cable and hunt me down...

posted at: 23:16 | path: /tech/lca | permanent link to this entry

On to other things
After spending four hours or so working on my hackfest entry, I was less than optimistic. My entry had yet to even be compiled on the test machines, and it still had huge areas of code that were completely unimplemented. When I went into the common room at St Mary's, Nick from OzLabs recognised me and helpfully mentioned that someone else not only had their code completely running but was in the process of optimising it. I promptly resigned.

I say "helpfully" sincerely there. It is a bit of a pity that my ideas won't see the light of day this hackfest, and that I won't be in the running to win whatever prizes they might offer. But since I don't have a snowball's chance in a furnace of winning anyway that's hardly a real disappointment. And I can go to bed with a clear head and prepare for my lightning talk and the Irish Set Dancing and mixing I plan to do at the Google party, which realistically are much higher priorities.

I do hope that we get to see the winning solutions, though...

posted at: 22:17 | path: /tech/lca | permanent link to this entry

Hackfesty?
I've decided to have a more serious look at entering the hackfest, since I'm familiar with processing fractals with parallel algorithms. Downsides are that I've only done it with PVM, I haven't done anything with the Cell architecture and there's all these other really cool talks to go to. That and I need to have my eyes stop glazing over when I start reading anything more detailed than the "Fire hydrant and hose reel" sign opposite me.

posted at: 11:03 | path: /tech/lca | permanent link to this entry

Tue, 29 Jan 2008

All Systems Go
After a night of continued problems with hardware in Canberra, I decided to test my mixing setup. Having borrowed a nice Edirol UA-1A USB audio input/output from my friend Mark, I wanted to test this in combination with the guitar amp from Andrew. I'd also changed my VMWare system over to use Host-only networking and convinced Samba and IPTables to talk to the VMWare client over this. So, was it going to actually work? Best not to find out on Friday Night...

After a bit of odd-hackery, I got it going - pleasingly well. The sound skips slightly when context-switching from the VMWare client, which is nothing unusual - the standard performance practice is to boot afresh and only starting those things which you absolutely need anyway. So it's all systems go for Friday night...

posted at: 23:47 | path: /tech | permanent link to this entry

All posts licensed under the CC-BY-NC license. Author Paul Wayper.