Too Busy For Words - The PaulWay Weblog

Error Message Hell

Wed, 20 Aug 2008 16:13:00 +1000

If there's one thing anyone that works with computers hates, it's an error message that is misleading or vague. "Syntax Error", "Bad Command Or File Name", "General Protection Fault", and so forth have haunted us for ages; kernel panics, strange reboots, devices that just don't seem to be recognised by the system, and programs mysteriously disappearing likewise. The trend has been to give people more information, and preferably a way to understand what they need to do to fix the problem.

I blog this because I've just been struggling with a problem in Django for the last day or so, and after much experimentation I've finally discovered what the error really means. Django, being written in Python, of course comes with huge backtraces, verbose error messages, and neat formatting of all the data in the hopes that it will give you more to work with when solving your problem. Unfortunately, this error message was both wrong - in that the error it was complaining about was not actually correct - and misleading - in that the real cause of the error was something else entirely.

Django has a urls.py file which defines a set of regular expressions for URLs, and the appropriate action to take when receiving each one. So you can set up r'/poll/(?P\d+)' as a URL, and it will call the associated view's method and pass the parameter poll_id to be whatever the URL contained. In the spirit of Don't Repeat Yourself, you can also name this URL, for example:

url(r'/poll/(?P\d+)', 'view_poll', name = 'poll_view_one')

And then in your templates you can say:

<a href="{{ url poll_view_one poll_id=poll.id }}">{{ poll.name }}</a>

Django will then find the URL with that name, feed the poll ID in at the appropriate place in the expression, and there you are - you don't have to go rewriting all your links when your site structure changes. This, to me, is a great idea.

The problem was that Django was reporting that "Reverse for 'portal.address_new_in_street' not found." when it was clearly listed in a clearly working urls.py file. Finally, I started playing around with the expression, experimenting with what would work and what wouldn't in the expression. In this case, the pattern was:

new/in/(?P\d+)/(?P[A-Za-z .'-]+)

When I changed this to:

new/in/(?P.+)/(?P.+)

It suddenly came good. And then I discovered that the the thing being fed into the 'suburb_id' was not a number, but a string. So what that error message really means is "The pattern you tried to use didn't match because of format differences between the parameters and the regular expression." Maybe it means that you can have several patterns with the same name that will try to match based on the first such pattern that does so. But until then, I'll remember this; and hopefully someone else trying to figure out this problem won't butt their head against a wall for a day like I did.

Django 101

Tue, 29 Jul 2008 13:53:00 +1000

At work I've started working on a portal written in Python using the Django framework. And I have to say I'm pretty impressed. Django does large quantities of magic to make mothe model data accessible, the templating language is pretty spiffy (it's about on a par with ClearSilver, which I'm more familiar with - each has bits that the other doesn't do), and the views and url mapping handling is nice too. I can see this as being a very attractive platform to get into in the future - I'm already considering writing my Set Dance Music Database in it just to see what it can do.

So how do I feel as a Perl programmer writing Python? Pretty good too. There are obvious differences, and traps for new players, but the fact that I can dive into something and fairly quickly be fixing bugs and implementing new features is pretty nice too. Overall, I think that once you get beyond the relatively trivial details of the structure of the code and how variables work and so on, what really makes languages strong is their libraries and interfaces, and this to me is where Perl stands out with its overwhelmingly successfull CPAN and Python, while slightly less organised from what I've seen so far, still has a similar level of power.

About the only criticism I have is the way the command line option processing is implemented - Python has tried one way (getopt) which is clearly thinking just like a C programmer, and another (optparse) which is more object oriented but is hugely cumbersome to use in its attempt to be flexible. Neither of these hold a candle to Perl's GetOpt::Long module.

No good title

Fri, 25 Jul 2008 21:22:00 +1000

History has been revised. Thank you.

The lost limericks list

Tue, 15 Jul 2008 09:51:00 +1000

After that post, I thought I'd just check which category I'd put my previous limericks in. To my horror, I discovered that I hadn't blogged them at all, but had (merely) posted them to the Linux Australia list. So I rescued them and posted them here for posterity.

That wonderful man Andrew Tridgell
Over SaMBa keeps permanent vigil.
SMB, it is said,
He decodes in his head,
And CIFS 2 will some day bear his sigil.

The great LGuest programmer Rusty,
Is virtually never seen dusty.
He eats 16K pages,
And has done so for ages,
Yet his moustache is clean and not crusty.

That marvellous girl Pia Waugh
Is certainly hard to ignore.
With her leet ninja moves,
Open Source just improves -
All Linux Australians show awe!

The Wireless Jonathan Oxer

Tue, 15 Jul 2008 09:50:00 +1000

After the three limericks I wrote about Tridge, Pia and Rusty, the conversation came up on #linux-aus about whether I could make a similar epgiram for Jon Oxer, former Linux Australia president, front-line hardware hacker and all-round good guy. It took me two months, but in an email to Jon I finally cracked it, packing much more into the rhyme than I originally thought would be possible:

The wireless Jonathan Oxer,
Waves his hand and his front door unloxer.
A remote-control loo,
And home theatre too -
If you as me, his whole house just roxor!

Who's next, I wonder?

We tune to podcasting James Purser,
Long known as a rhymer and verser.
With his darling wife Karin
They are not known as barren:
Three children now stare at their cursor.

Steve Walsh, however, is going to take a bit more thinking about.

Send your suggestions of who should be next under the pen to paulway@mabula.net

New Job, Same Old Business

Wed, 02 Jul 2008 10:29:00 +1000

I've just started work on Monday at my new employer, TransACT, a fibre and copper communications provider in (as the name implies) the Australian Capital Territory. I've read through the employee handbook, done all the financial documentation, been given a computer and installed Fedora 9 on it. The majority of the team here use Mac Minis as their desktop machines, because there's a high requirement to use Unix commands to manage the network infrastructure. Of course, we still have to hook into the Microsoft Windows support infrastructure, but that's hardly a challenge these days.

The reason I mention this is because TransACT and its parent company ActewAGL are featured on Microsoft Australia's "Get The Facts" pages as some kind of 'shining example' of a company that "Wave[d] Goodbye To Linux" and somehow saved money. The facts are radically different, even from the small sample I've seen so far. All the network infrastructure, from the set top boxes to the DHCP servers to the encryption server for the IPTV, run on some kind of Unix - Debian Linux seems to be the predominate flavour. Microsoft's "case study" is really just a small part of TransACT and ActewAGL's business, and it's hardly "waved goodbye" to Linux in the organisation.

Common code in ClearSilver 001

Sun, 15 Jun 2008 11:10:00 +1000

I've been using ClearSilver as a template language for my CGI websites in earnest for about half a year now. I decided to rewrite my Set Dance Music Database in it and it's generally been a good thing. Initially, though, I had two problems: it was hard to know exactly what data had been put into the HDF object, and it was a pain to debug template rendering problems by having to upload them to the server (surprisingly, but I think justifiably, I don't run Apache and PostgreSQL on my laptop so as to have a 'production' environment at home).

I solved this problem rather neatly by getting my code to write out the HDF object to a file, rsync'ing that file back to my own machine, and then test the template locally.

I knew that ClearSilver's Perl library had a 'readFile' method to slurp an HDF file directly into the HDF object, and a quick check of the C library said that it had an equivalent 'writeFile' call. So happily I found that they'd also provided this call in Perl. My 'site library' module provided the $hdf object and a Render function which took a template name; it was relatively simple to write to a file derived from the template name. That way I had a one-to-one correspondence between template file and data file.

Then I can run ClearSilver's cstest program to test the template - it takes two parameters, the template file and the HDF file. You either get the page rendered, or a backtrace to where the syntax error in your template occurred. I can also browse through the HDF file - which is just a text file - to work out what data is being sent to the template, which solves the problem of "why isn't that data being shown" fairly quickly.

Another possibility I haven't explored is to run a test suite against the entire site using standard HDF files each time I do a change to make sure there aren't any regressions before uploading.

Hopefully I've piqued a few people's interest in ClearSilver, because I'm going to be talking more about it in upcoming posts.

Faith in Atheism

Sun, 01 Jun 2008 21:10:00 +1000

I was in the office of a church hall on Friday, picking up the keys for the hall for our Canberra Irish Set Dance Weekend, and while waiting for the people to work out the bill a leaflet for an upcoming talk at the church caught my eye. The topic of the talk was:

Why I Don't Have The Faith To Be An Atheist.

Now, I will freely admit that that's a catchy title, because it certainly caught my eye. And while I endeavour to understand other people's points of view I am often blinded by my own ideas of what is sensible and reasonable. But I cannot fathom how that that topic can be debated seriously in the affirmative.

On the one hand, to me it requires much more faith to believe in an arbitrary, contradictory, and often non-sensical set of teachings that fly in the face of the evidence around us than to not have to believe any of that. My Australian Concise Oxford gives its first definition of "faith" as: Reliance or trust in; belief founded on authority - the other definitions are the type of religion one believes in and a promise or intent (as in in good faith). In that context I would say that all religions have some authority, be it a book or a person, that is the foundation of their belief, whereas Atheism makes no such demand. Atheism has no book which is quoted chapter and verse, no authority figure that tells people to not trust science and believe what they teach in contradiction to the evidence.

On the other hand, if this is some kind of sophistry - some kind of cunning argument or uncommon definition of "faith" or "atheism", then I think one is entitled to ask if the speaker is going to be serious at all. If it's a straw man argument, then really what's the point of it? I can respect people who stick by what they believe even as they acknowledge the flaws in their own arguments - I can't respect someone who tricks their audience with a conveniently quelled paper tiger.

I was half tempted by Kate's sensible suggestion to actually go and see this just to actually solve this logical problem before it threatened to burst my brain. But as I have this sneaking suspicion that the whole thing will be preaching, appositely, to the choir.

I must now hide myself from the metaphor police.

The conspiracy to keep children quiet

Tue, 20 May 2008 12:26:00 +1000

Thanks to Steven Hanley, I read Paul Graham's essay "Lies We Tell Kids". His basic point is that adults often don't tell children the strict truth - either by omission or by fabrication - because some questions are hard ("Is there a God?") or destroy the innocence of childhood ("What is a prostitute?"). To my mind his essay parallels Ian Stewart, Jack Cohen and Terry Pratchett's observations in the "The Science Of Discworld" series that we simplify complex stories by abstracting or leaving out details - "telling lies" by omission.

I'm lucky enough to have four nieces; since Kate and I have decided not to have children we have focussed our "raising the next generation" on these four (although I tend to be catholic, if not necessarily orthodox, in playing with any children). They are all reasonably well-adjusted, normal girls in my opinion and I think that, to varying degrees, their parents have tried to be fairly honest with them. On the Tuesday before the Armstrong family went away for a six week trip around the world, they had their family dog put down because of its extreme ill health and the likelihood that it would die while they were away. This was done by a vet in their back yard with the family and their cousins watching and supporting them through that terrible time, so Paul Graham's section on how we lie to children about death particularly resonated with me. These girls haven't suddenly become morbid, or afraid of death, or casual about it, because of that experience - they're still quite normal even after we've exposed them to something that other parents would go to great efforts to hide.

The girls know me as somewhat eccentric, partly because I play running and card games with them, partly for my collection of evil laughs, and partly because I'll bore their ears off with science and technology if they let me. I send them coded messages and make special hiding places around the house for when we play hide-and-seek. I'll tell them when I don't know something, or when I'm glossing over details in an explanation in order to make it twenty words rather than a hundred. I do think that a fair bit of my behaviour is related to keeping them behaving as children - or rather as young adults - rather than making them conform to one or the other but not both at the same time. To me, spending ten minutes talking to one of the girls when she's in trouble with her parents and explaining that I understand why she did the things she did - even though they were wrong - is far more valuable to her than being left with a sense of injustice that "you just can't win against your parents" and "no-one understands my side of the story".

Paul Graham talks at the end of his article about a sort of 'truth debt' built up by all the elisions, fabrications and contradictions the adults have told around children as they reach adulthood. "There's never a point where the adults sit you down and explain all the lies they told you," he observes. My way of dealing with this is to start early, be honest about the things you can be, and tell them when you're not being honest about the things you can't be. I hate telling lies, especially when I know that sooner or later I'm going to have to tell the truth later and then explain why I told the lie. Sure, I don't intend to freak kids out by telling them things that shatter their illusions of how the world works too quickly, but neither do I intend to shore up that illusion with even more outlandish fabrications.

I do hope that this little essay doesn't warn too many parents off from allowing me to talk to their children :-)

Stupid Error 32512

Wed, 02 Apr 2008 11:40:00 +1000

For a while my brother's been having a problem with his MythTV setup - the mythfilldatabase script won't run the associated tv_grab_au script when run automatically, but will work just fine when run manually. In the logs it says:

FAILED: xmltv returned error code 32512.

Now, after a bit of searching I have finally found that 32512 is a magic code from the C system(3) call, which basically does a "sh -c (system call and arguments)". If sh can't find the file you've specified in the system() call, it returns 127, which is shifted into the upper eight bits of a 16-bit smallint (as far as I can make out, the lower eight bits are reserved for informing the caller that the system call was aborted due to a signal - e.g. a segmentation fault).

After a lot more searching, and a good deal of abuse on the #mythtv-users channel on freenode.org, I finally found some information about shell exit codes, and it turns out that 127 is "command not found". In other words, mythfilldatabase at that point is trying to call the tv_grab_au grabber and not finding it. On my brother's machine, this is because sh under root does not get the path /usr/local/bin, which is where the grabber is stored.

(It works on my machine because I run it from a script which picks a random time, and includes /usr/local/bin in the path.

So there are two solutions, as I see it:

1) Put tv_grab_au in /usr/bin/.

2) Run mythfilldatabase from cron using a script which includes /usr/local/bin in the path.

Given the bollocking I got in #mythtv-users for suggesting something so crude and hackish (in the words of Marcus Brown, mzb_d800) as cron, I guess I'll have to go with option 1. But here's hoping that this blog entry helps someone else out there - almost every post on the mythtv-users email list that mentions 32512 never mentions a solution...

Kangaroos saved, who cares about anything else?

Wed, 02 Apr 2008 11:17:00 +1000

Once again the kangaroo cull at the old Belconnen naval signalling base has stalled again, with the animal liberation protesters calling it a victory. The pity is that these people are really protesting about the wrong thing.

On the one hand, they're protesting about the culling of 400 kangaroos. I wonder if they have realised that ACT Forestry has a license to cull 4,000 kangaroos - ten times that number - per year in the national parks. If I remember rightly, ACT Parks also has a license to cull 3,000 on other public land. So killing 400 as a once-off cull is hardly the great tragedy that it's being painted as.

On the other hand, there are animals and plants on that land that are highly endangered and under threat from the kangaroos. There's not only the Perunga Grasshopper and the Mouthless Moth in that area, but a rare species of grasslands flower that is being eaten by the kangaroos. While I'm a bit disappointed that the Liberals have chosen to make this one of those "if we were governing this wouldn't happen" issues, they're totally right - everyone's getting all upset because cute furry animals are going to die. And it looks like the Liberals are only making a fuss because Defence is flip-flopping, not because anything else might face extinction as a result of overgrazing by the kangaroos.

I got this information from a friend who wrote the policy on land care and kangaroo culling in the ACT. So I can't quote chapter and verse, because the conversation was informal and I didn't take notes. But I do think that the "animal liberationists" are way out of line. Don't get me wrong - generally I would like to avoid the kangaroos being killed at all. But to blow this out of proportion shows exactly the kind of loony-fringe unreasoning stupidity that these groups accuse their enemies of and that does their own credibility the most damage. And if you've got any species that is overrunning its ability to survive on the land, it's going to die off some way or other. Killing them quickly is far more humane than letting them starve, whether on a block in the ACT or elsewhere in NSW.

If only the human race would learn this and seriously consider population control of itself.

Solar Power For Free

Sun, 23 Mar 2008 23:12:00 +1000

I have an idea which I think would revolutionise the power industry. Of course, it requires a lot of venture capital to set up, something which I am unable to provide nor have the knowledge to find. I offer this up on my blog in the hopes that someone might read it, form a business out of it, and change the world. If I get mentioned or gain something from it, then that'd be nice, but making solar power easy for everyone to get involved in is enough reward for me.

Let's call the company Solar Sails. They offer to install a set of solar panels and the necessary equipment to feed power back into the grid for free. When you sign up with them, you pay only for the difference between the power that you use and the power your solar cells generate. If that balance is zero or negative in the billing period - say a month - then you pay nothing for your power. That's the deal for the consumer. Solar Sails itself makes its money by selling the extra power you produce (over what you consume) back to the grid.

From what I recall, the pay-off time for solar panels - the time taken for your lower power bills to recoup the cost of installing the panels - is about ten to fifteen years, and that figure is probably going to get shorter as the technology ramps up. (As Dr. Karl pointed out in a recent lecture, the power pay-off time - the time taken for the solar panels to generate more power than they took in their production - is only about two years or less). There are plenty of businesses who look at investing money in processes with longer pay-off times. Admittedly, Solar Sails is only going to get a fraction of the money those solar panels earn (since the meat of it is going into paying for your power), but on the other hand they can negotiate better deals with the grid supply companies and with the solar cell and technology providers than we as individual consumers can.

And the important thing, I feel, to get solar panel technology widely adopted is to lower that 'hump' of initial outlay. The most sensible point for that is at zero, so it costs you nothing to join the scheme. Even the most power-wasteful person has no reason not to join the scheme if it'll start saving them money without any outlay at all. Once they realise that they can save themselves more money by turning off lights and being less wasteful, you've changed their bad behaviour. And even if they don't actually change their wasteful ways, the fact that their purchase will have contributed to helping lower the price for other people (through volume purchasing by Solar Sails). So it's a pretty good proposition not just for getting more power generated by solar power and making it easier for people to adopt but also to change people's habits and reduce power consumption overall.

As usual, I'd love to hear your thoughts on this idea.

Stupid Quote of the Day

Wed, 19 Mar 2008 14:00:00 +1000

Andrew Donnellan quotes Albert Camus rehashing Pascal's Wager as if it's some kind of useful way to affirm what one believes. The certainty that the god that they believe in is the god that will be actually judging them is ... amusing.

I would counter with the Atheist's Wager: "You should live your life and try to make the world a better place for your being in it, whether or not you believe in God. If there is no God, you have lost nothing and will be remembered fondly by those you left behind. If there is a benevolent God, he may judge you on your merits coupled with your commitments, and not just on whether or not you believed in him." Perhaps a reading of the relevant chapters of Richard Dawkins' book "The God Delusion" might also useful debunking of this warped logic.

And I would also add that any God that requires my belief as a "jealous God" is a pretty poor god by even human standards. If a human required constant devotion and commitment in spite of complete and utter disdain and ignorance of the devotees, we'd call them wishy-washy or vain at best and spiteful or megalomaniac at worst. Why do so many religions then excuse their god of these emotions, coming up with ever more convoluted ineffabilities in order to justify a tyrant? I wish I could find what I thought was a Robert A. Heinlein quote on this, but it wasn't in this otherwise excellent collection.

Standard Observations

Tue, 18 Mar 2008 22:41:00 +1000

Simon Rumble mentioned Joel Spolsky's post on web standards and it really is an excellent read. The fundamental point is that as a standard grows, testing any arbitrary device's compliance with it it grows harder. Given that, for rendering HTML, not only do we have a couple of 'official' standards: HTML 4, XHTML, etc., but we also have a number of 'defacto' standards - IE 5, IE 5.5, IE 6, IE 7, Firefox, Opera, etc. etc. etc ad nauseam. For a long time, Microsoft has banked on their desktop monopoly to lever their own defacto standards onto us, but I think they never intended it to be because of bugs in their own software. And now the chickens are coming home to roost, and they're stuck with either being bug-for-bug compatible with their own software (i.e. making it more expensive to produce) or breaking all those old web pages (i.e. making it much more unpopular).

I wonder if there was anyone in Microsoft Internet Explorer development team around the time they were producing 5.0 that was saying, "No, we can't ship this until it complies with the standard; that way we know we'll have less work to do in the future." If so, I feel doubly sorry for you: you've been proved right, but you're still stuck.

However, this is not a new problem to us software engineers. We've invented various test-based coding methodologies that ensure that the software probably obeys the standard, or at least can be proven to obey some standard (as opposed to being random). We've also seen the nifty XSLT macro that takes the OpenFormula specification and produces an OpenDocument Spreadsheet that tests the formula - I can't find any live links to it but I saved a copy and put it here. So it shouldn't actually be that hard to go through and implement, if not all, then a good portion of the HTML standard as rigorous tests and then use browser scripting to test its actual output. Tell me that someone isn't doing this already.

But the problem isn't really with making software obey the standard - although obviously Microsoft has had some problem with that in the past, and therefore I don't feel we can trust them in the future. The problem is that those pieces of broken software have formed a defacto standard that isn't mapped by a document. In fact, they form several inconsistent and conflicting standards. If you want another problem, it's that people writing web site code to detect browser type in the past have written something like:

if ($browser eq 'IE') {
    if ($version <= 5.0) {
        write_IE_5_0_HTML();
    } elsif ($version <= 5.5) {
        write_IE_5_5_HTML();
    } else {
        write_IE_HTML();
    }
    ...
}

When IE 7 came along and broke new stuff, they added:

    } elsif ($version <= 6.0) {
        write_IE_6_0_HTML();

It doesn't take much of a genius to work out that you can't just assume that this current version is the last version of IE, or that new versions of IE aren't necessarily going to be bug-for-bug compatible with the last version. So really the people writing the websites are to blame.

Joel doesn't identify Microsoft's correct response in this situation. The reason for this is that we're all small coders reading Joel's blog and we just don't have the power of Microsoft. It should be relatively easy for them to write a program that goes out and checks web sites to see whether they render correctly in IE 8, and then they should work together with the web site owners whose web sites don't render correctly to fix this. Microsoft does a big publicity campaign about how it's cleaning up the web to make sure it's all standard compliant for its new standards-compliant browser, they call it a big win, everyone goes back to work without an extra headache. Instead, they're carrying on like it's not their fault that the problem exists in the first place.

Microsoft's talking big about how it's this nice friendly corporate citizen that plays nice these days - let's see it start fixing up some of its past mistakes.