Hissohathair

The "Super" Difference

2011-02-02T17:13:00.001+11:00

Megamind: You dare challenge Megamind?
Villain: This town isn't big enough for two super-villains!
Megamind: Oh, you're a villain alright. Just not a super one.
Villain: Yeah? What's the difference?
[music reaches its crescendo, the enormous flying Megamind head opens wide, spits out its tongue to reveal Megamind himself]
Megamind: Presentation!

The Real Mr His So Hat Hair

2009-09-25T17:47:00.001+10:00

Awesome.

Using CruiseControl.rb to manage a Perl Catalyst project

2009-06-03T10:58:00.001+10:00

We're working with the Catalyst framework again, porting an old Perl 5 HTML::Mason site to Catalyst and introducing some modern Perl coding standards to a fairly old stack.

One of the things we needed was a Continuous Integration tool for the project. Since we're already using CruiseControl.rb for the Rails projects I thought it should be pretty easy to incorporate the Perl project into it.

And indeed it was:

$ ./cruise add CatalystProject --url https://svn.work.com/svn/catalystproject/trunk/

CruiseControl will run a "rake" task whenever a commit is made. So we need a small Rakefile with just enough code in it to run the standard Perl tools:

$ cat Rakefile

require 'rake'

task :default => :test

desc "Runs make test"
task :test do
  t = system("eval $(perl -I$HOME/perl5/lib/perl5 -Mlocal::lib) && perl Makefile.PL && make test")
  rc = $?
  puts "\nMake finished with t=#{t} rc=#{rc.exitstatus}"
  raise "Perl make failed (rc=#{rc.exitstatus})" unless t
end

The eval line in the system call is there because we're using local::lib to manage our Perl library dependencies and we need to set the environment variables so that they can be found. To propagate any "make test" errors back out to rake we throw an exception on non-zero exit codes.

How to set your DNS search list in OSX

2009-05-12T01:36:00.003+10:00

Here's how to set your DNS search path in OSX, which is something you'll want to do if you're getting redirected to eBay when you try to go to Google, or are experiencing other odd behaviour from your browser.

Open System Preferences.
Open Network settings.
Your active network connection should be selected on the left. Select it if it is not. Click the "Advanced" button towards the bottom right hand corner.
Select the DNS tab.
On the right hand side is a column "Search Domains." It will probably already have your domain name listed, but it will be "greyed" out so that you can't remove it. That's OK. Click the "+" sign at the bottom of the column, and type the domain name again.
You should now have the domain name listed twice. One will be grey, the other black. This is good, believe it or not. :-)
Click OK.
Click Apply.

Here's a screen capture that steps through it.

The "search path fix" for ASZ.COm.Au is only a partial fix

2009-05-07T14:28:00.000+10:00

Feel I need to clarify my fix to the "eBay/ASZ redirection problem" discussed in my last post.

Adding your domain name to your DNS search path will stop your DNS resolver from asking the "au.com.au" and "com.com.au" name servers for "google.com.au.com.au". So you shouldn't get "Welcome to ASZ.COm.Au" or be redirected to eBay or other unexpected behaviour.

What you'll get instead is "site not found". That's what I mean by a partial fix. If your ISPs name server is losing DNS responses (or is too slow to return them) then you'll still have problems: namely you'll get an error message when you try and visit some sites. However clicking "Refresh" or "Reload" will generally solve that problem. Once your at the site the address will be in your resolver's cache so things will be stable for a while.

Meanwhile I've written to the auDA guys asking for their opinion on defining DNS entries for other peoples domain names as sub-domains of yours.

Welcome to ASZ.COm.Au (Or, The Resolver Library is Broken)

2009-04-30T17:43:00.087+10:00

A few days ago, I did a Google search and got the strangest error: a 404 (File Not Found) page that claimed to be from Apache with PHP and Frontpage extensions loaded. Google doesn't run Apache.

Since that was A Very Odd Thing Indeed I went to the Google home page. This time I got a new message: "Welcome to ASZ.COm.Au" (for some reason, I feel it's important to preserve the capitalisation).

My first guess was that my DNS cache had been poisoned. Or that I'd picked up a virus somehow. When I looked on Twitter I saw that I was not alone: a few people were reporting similar problems trying to access Facebook, YouTube, Google and even eBay. A discussion about the issue had started on Whirlpool. In most cases, the issue appeared to resolve itself eventually, or after a DNS cache flush.

In my case, the issue persisted for a little while and then stopped. I was able to access Google again.

It took some time for me to figure out what was going on (hey -- I'm over 27). The intermittent nature of the issue made debugging difficult. It wasn't until my partner mentioned that she'd seen the same screen when she'd tried to access the Bureau of Meteorology that I was able to make progress. Visiting http://bom.gov.au/ (but not http://www.bom.gov.au/) consistently reproduced the problem.

I chased down a few theories: malware, cache-poisoning, an "optus issue". But I finally worked out that this was the result of a documented feature of the common UNIX resolver library.

When you visit a web page your computer needs to "resolve" the host name (eg "google.com.au") to an Internet address. That's the job of the resolver, which in turn uses something called a domain name server. Your browser asks the resolver "what's the address of Google.com.au" and the resolver answers. Either the resolver already knows the answer because it's looked it up before and kept a copy (a cache), or it doesn't know, and so it asks the domain name server. The domain name server in turn may ask other domain name servers, until someone, somewhere, knows the answer. The answer is then sent back through the chain, ultimately to your resolver (called the "client").

So if the resolver doesn't know an address it asks the domain name server and waits for an answer. But it will not wait forever. In fact, it typically won't wait longer than several seconds. It's an impatient little thing and if it hasn't heard back quickly enough it assumes that maybe the hostname is wrong -- maybe the user just typed part of the hostname. Or maybe the domain name server does answer in time but the resolver cannot use the answer -- the domain name server may reply with "no one knows that hostname." Either way, the resolver will start to guess what the real (or "fully qualified") hostname might be.

There are two strategies a resolver can use when it starts searching for the fully qualified hostname.

The first is to use an explicit list of search paths. You usually provide this list yourself when configuring your network settings. If you haven't set such a list, then it will employ the second strategy. (That's going to turn out to be quite handy...)

The second thing your resolver can do is a "domain name search". It takes your domain name, prepends the hostname you're looking for and does a lookup on that. I'm with Optusnet, so my domain name is "optusnet.com.au". My reslover then might lookup "bom.gov.au.optusnet.com.au" if it doesn't get a useful answer for "bom.gov.au". If it still doesn't get a useful answer (and it this case, it won't) then it starts searching "up" the domain name -- it removes the first part of the domain name and repeats the search. So its second search is for "bom.gov.au.com.au". See what it did there? It deleted "optusnet" and tried again. (Technical note: the behaviour is documented in the resolver man page -- see RES_DNSRCH)

That right there is the flaw and the root cause of the problem.

Someone owns the domain names "au.com.au" and "com.com.au". And they have name servers. And they've set them up to answer queries for a whole host of things, among them, bom.gov.au.com.au, google.com.au.com.au and facebook.com.com.au. And our resolvers are querying them and merrily sending our browsers there if the real name servers for those domains don't get their answers back in time.

Now we know enough to say what's going on:

You try and visit Google, Facebook, Twitter (or the BoM). It's been a while so the address isn't in your resolver's cache. So it does a lookup by asking your domain name server -- this is usually provided by your ISP.
For whatever reason, your ISP's name server is either too slow to respond or the response is lost altogether. So your resolver "times out" and starts to "search". Your domain name ends in ".com.au" and so eventually, your resolver looks up "google.com.au.com.au" (or whatever site you're trying to visit, with ".com.au" added to the end). The name servers at "au.com.au" (or "com.com.au" depending on what you're looking up) do respond and do so in time.
Your resolver gives the bogus address to the browser and stores it in the DNS cache. The "TTL" (time to live) for those addresses is 4 hours, so you're going to be stuck with that address in your cache for at most 4 hours.
Eventually, your cache times out. Or maybe you know how to flush it. Either way, a second attempt by the resolver to get the right IP address works and the problem appears to be resolved.

Multiple things are going wrong here:

The ISPs name server is either too slow to respond or perhaps "dropping" packets (DNS packets are typically using UDP which is not a guaranteed delivery mechanism like TCP). I've seen this with Optusnet before but in the past I just got a "site not found". Such responses aren't cached so if you hit "Refresh" in your browser you typically find the site just fine the second time.
The name servers for "com.com.au" and "au.com.au" have records that match other peoples sites. They shouldn't. Right now, it's just confusing and annoying but its potential for phishing is obvious. It's not necessarily malicious but it should be changed.
The algorithm used by the resolver in both UNIX and Windows has a security flaw: it should not search all the way back to ".com.au".

To me the ultimate problem is that last one: UDP packets, and hence DNS responses, are not guaranteed to be delivered. It should be expected that sometimes, things will get busy and responses will be lost. The resolver is in error by searching the domain name all the way back to ".com.au" -- IMHO that's a security hole similar to the old "cookie monster" bug.

There's a fix though, which at least works on OSX (Mac). If you set an explicit search path, then the resolver won't use the second strategy described above. It will search the search path(s) and then stop there. I've set my search path to "optusnet.com.au", the same as my domain name, and can no longer reproduce the problem.

There are other things that can help:

If your ISP's domain name server is not reliable use OpenDNS. There is some anecdotal evidence that the possibility of DNS replies being late or dropped is lower. Getting the "Welcome to ASZ.COm.Au" page for Google or Facebook depended on your computer not getting the DNS response in time (or at all) so having a reliable domain name server will stop the problem happening.
Add a "." to the end of your hostnames when typing into the browser (for example, "google.com.au."). The trailing "." prevents the domain name searching from kicking in.
If you're able, configure firewalls to drop packets from the name servers at "com.com.au" and "au.com.au".

Oh, if you're wondering why "bom.gov.au" could reproduce the problem every time: Their name server does respond quickly but not with an "A" record for that domain. Which is to say, it's not a response that the resolver can use to answer the question "what's bom.gov.au's IP address". So even though the resolver gets an answer it still embarks on its domain name search until it looks up "bom.gov.au.com.au" and gets an answer (an "A" record). To reproduce the proglem with Google etc, you need to wait for the DNS response packet to go missing in order to trigger the domain search by your resolver.

Other things can be explained:

It appeared to be an "Optus problem" at one stage because their domain name servers are occasionally overloaded and therefore slow. The Optus domain ends in "com.au" and so the domain name search would go all the way back to ".com.au". TPG seems to have similar issues.
I couldn't reproduce the problem at work because my domain name there is "work.com". The resolver is smart enough not to search as far back as ".com" -- it just missed the case where a country domain has subclassifications (such as ".com.au", ".co.nz" or ".co.uk"). That's the limitation to the "counting dots" method of deciding how far to walk back.
Switching to OpenDNS would appear to solve the problem because the resolver didn't need to start a domain name search if it got the right answer right away.
Flushing the DNS cache would appear to solve the problem because it's only occasionally that DNS replies get lost. You have a good chance on your second attempt of getting the right address.

Thanks to objects, pixelgroup and 3buffalogirls for responding to my Twitter enquiries with additional information, and the posters on the Whirlpool forums.

Monitoring Rails builds with CruiseControl.rb and CCTray

2009-03-13T08:00:00.001+11:00

More for my own memory than anything else...

CruiseControl.NET comes with a tool called CCTray that gives you a handy way of monitoring the build status of multiple CruiseControl environments. It works out of the box with other CruiseControl.NET installations but needs a little trick to monitor the Ruby and Java versions (why we need the same app implemented three times is a subject for a rant one day I'm sure...).

For Ruby on Rails projects, set the monitoring URL in CCTray to this:

http://hostname.of.cruisecontrol.rb:3333/XmlStatusReport.aspx

It's not a real ASPX page but it returns XML that CCTray is expecting.

Cruise Control for Java is similar, but different ('natch):

http://hostname.of.cruisecontrol:3333/dashboard/cctray.xml

Had trouble Googling that. :-)

Keep yourself logged in to a website with anti-idle

2009-03-09T10:10:00.005+11:00

At $work I need to use a time sheet application which has a session timeout feature. I want a way to stay "logged in". So I've conceived a little plug-in for my personal web developer's proxy that will re-load certain web pages periodically in the background.

Could work like this:

Start your personal proxy with the anti-idle plug-in in the chain (below).
In your browser, go to the page you want to periodically re-load.
At the end of the URL, append a CGI argument. For example you could append "?ttt_anti_idle=300" to reload the page every 5 minutes. If there are already CGI arguments in the URL just append: "&ttt_anti_idle=300".
Load the new URL you've just typed. The anti-idle plug-in will strip out the extra argument you've appended prior to giving the URL to the "real" server.
The anti-idle plug-in monitors its stream for "ttt_anti_idle" arguments and builds a list of pages to reload at certain intervals. It discards the result of course.

Here's how I imagine I'd set up the pipeline:

$ proxy | anti_idle --use_cgi=ttt_anti_idle | respond

[...]

Initial Load Values for Nagios Load Checks (Cheat Sheet)

2009-03-06T14:51:00.000+11:00

I've put together a cheat sheet to show how you might want to initially configure your Nagios load checks. The thinking behind these initial values is set out in Tuning Nagios Load Checks.

Use	OS	Cores	Warning	Critical	Notes
CMS (Teamsite)	Solaris	1	10,7,5	20,15,10	Testing shows this app to be responsive up until these loads.
Web Server	Linux	2 x 4	16,10,4	32,24,20	Web servers are paired, so want to know if reaching 50% capacity regularly. Testing shows performance degradation from a load of 20.
DB Server	Linux	2 x 4	16,10,4	32,24,20	Same hardware, different use. Nevertheless, using same thresholds.
Nagios	Linux	1 x 2	6,4,2	12,10,7	Small box, paired with backup.

General notes:

The UNIX servers (particularly the Sun SPARC ones) seem to be able to stay up and responsive even under heavy load. And they don't count processes waiting for I/O in their load counts the way Linux does. I have no explanation for this. :-)
We track these loads over time to predict demand growth for capacity planning -- the thresholds are not a long term goal but rather a short term alert threshold.
Transaction or revenue-earning web servers might have lower thresholds because of the different commercial implications of performance degradation. YMMV.

For more information on the Nagios check_load command, see Tuning Nagios Load Checks.

No more stupid YouTube comments

2009-03-06T10:06:00.005+11:00

Prompted by Mark Damon Hughes' Stupid Comments Be Gone I wrote a small script that took YouTube HTML in on stdin, stripped out the comments, and spat the remainder out on stdout (Mark's trick uses CSS to hide them).

Now I can do this:

$ proxy | connect | kill_youtube_comments | respond

[...]

And lo! Works in all browsers. :-)

Breaking it down:

The proxy command listens on port 8080 (I configure my browser to proxy to localhost:8080). It spits all requests it sees to stdout.
The connect command reads a HTTP request on stdin, connects to the remote server, fetches the content, and spits a HTTP request on stdout.
The kill_youtube_comments command reads in HTML and strips out the div that contains YouTube comments.
The respond command reads a HTTP response and sends that (via named pipe) back to the proxy command so that it can return it to the browser.

I sometimes wonder if anyone else in the world would find a personal, hackable proxy useful.

Using Blogger's new Import Blog function to import an RSS-based blog

2009-01-16T14:31:00.010+11:00

I've been playing with Blogger's Import Blog feature , made available in Blogger in Draft last year.

Google explicitly state that only Blogger exported blogs are supported. Blogger exports its blogs in Atom format. I thought perhaps that I could convert an RSS feed to Atom and then import that into Blogger and thereby move some old non-Blogger blogs over to Blogger.

Alas, no joy! The Blogger Import tool is quite fussy about its Atom. For example, if you export a blog in Atom format, and then run that through an XML formatting tool and re-import, you'll find that Blogger complains about the uploaded file.

However, I've kept at it, and now have a simple script that can take an RSS feed and convert it to an Atom format that Blogger seems happy with. It's not quite there -- a few posts are silently dropped for reasons I haven't figured out yet. I'm toying with the idea of eventually releasing it. Of course, I'm not the only one .

via hissohathair.blogspot.com

Oh! Look! Time_t party coming!

2009-01-15T16:06:00.000+11:00

At 10:30:31 on Friday the 14th of February this year (Sydney time) the UNIX epoch time will be "1234567890".

Time for a time_t party !

via hissohathair.blogspot.com

Making it useful

2008-11-21T15:57:00.002+11:00

Played a bit more the the small web developer's toolkit in my last post. This time in a real-world production setting.

Although other tools could have given the answer as well it was kinda fun to watch this work:

$ proxy --port 8082 | log | connect | respond -e | log

proxy: Proxy started at http://macbook.local:8082/
[14:41:26] GET http://google.com/ HTTP/1.1 -> [14:41:27] HTTP/1.1 301 Moved Permanently
[14:41:27] GET http://www.google.com/ HTTP/1.1 -> [14:41:28] HTTP/1.1 302 Found
[14:41:28] GET http://www.google.com.au/ HTTP/1.1 -> [14:41:28] HTTP/1.1 200 OK
[...]

The "log" command is just printing out interesting things it sees -- by default the request and response lines and adds a timestamp. In real life I wasn't debugging Google BTW. :-)

An idea for a teeny-tiny web developer's toolkit

2008-11-17T16:33:00.020+11:00

Every now and then I need to work out what's gone wrong with a production web server. Maybe it was a deployment issue, or a configuration issue or maybe something has broken. I'm sometimes wondering "what on Earth is that server saying to this browser?"

I use Firebug, HTTP Live Headers and Tamper Data in Firefox a fair bit. But they only work in Firefox and MSIE sends different request headers which can affect the outcome. There's also the excellent Charles Web Debugging Proxy which is so awesome it lets you observe SSL encrypted traffic between client and server, serve local files to the browser instead of server files and "spoof" DNS addresses to make browsers talk to specific servers.

Some days though I find myself typing this a lot:

$ telnet app.example.com 80

Trying 127.0.0.1...
Connected to app.example.com.
Escape character is '^]'.

GET / HTTP/1.1
Host: app.example.com

HTTP/1.1 302 Found
[...]

When you're doing that over and over again you start to look for little tricks to make it go quicker. For example, I might do this so that I can use the bash command line history to replay something quicker:

$ printf "GET / HTTP/1.1\nHost: app.example.com\n\n" | telnet app.example.com 80

Trying 127.0.0.1...
Connected to app.example.com.
Escape character is '^]'.
Connection closed by foreign host.

Urgh. Except that didn't work because telnet got an interrupt when printf sent the end of file. Try this instead:

$ (printf "GET / HTTP/1.1\nHost: westfield.com\n\n" ; sleep 2) | telnet westfield.com 80

Yeah, OK. So now it's on one line and I can scroll back through the command history buffer. But what I'd really like to do is this:

$ get --be-like=firefox http://app.example.com/ | no-cache | connect | tee connect.out

HTTP/1.1 200 OK
Cache-Control: no-cache
Connection: Close
[...]

Now I don't have to remember to type the "Host" header and the "get" tool can add other useful headers (for example, to emulate Firefox or MSIE).

Or this:

$ get --be-like=msie http://app.example.com/ | remove_header accept-encoding | spoof app.example.com as 127.0.0.1 | connect | headers

Basically what you could do is build a HTTP request and response stream out of UNIX pipes. Each step in the pipe chain would modify the request (or response) before passing it on. Eventually things get passed to the command "connect" which is responsible for actually making the request from the real web server and spitting out the reply. The reply can go through a similar chain of pipes.

Kinda like Yahoo! Pipes. Only... with actual pipes.

And if I substitute the "get" command (which really is just printing out HTTP request objects as text) with a "proxy" command (which listens to port 8080 for web browser requests) then I could test Apache configurations with something like this:

$ proxy --port 8080 | spoof app.example.com as 127.0.0.1 | connect | respond -e | tee logs/respond.out

Here's the most complicated pipe chain I've gotten to work so far:

That comand:

Listens to local port 8081 for web browser proxy requests
Logs the browser requests to logs/proxy.log
Connects to the appropriate web server to get the content just requested by the browser. By default, the connect command handles any content transfer decoding
Logs the server response to logs/connect.log
Runs the server response through HTML Tidy ("tidy") which reformats the HTML with indentation, corrects any HTML errors and transforms it to XHTML if required (UTF-8 encoded of course). HTML Tidy's error messages and report is sent to /dev/null.
Save the output of HTML Tidy to logs/filter.log
Send the final response stream back to the browser, but also echo the response on stdout; and
Save the final response to logs/response.log (the contents of which should be identical to filter.log -- I was debugging).

I had initially had an idea for a much more complicated Charles-like tool with a GUI and threads and select() polling and plug-ins and the like. But this seems to capture the essence of what I was trying to do. It's the smallest implementation that will work -- and no smaller. :-)

Email Client Market Share

2008-11-03T22:25:00.002+11:00

A program called Fingerprint from Litmus is generating statistics on e-mail client market share. I was surprised to see Gmail's share so low (6% amongst business users, 4% amongst consumers) but also how old most Outlook client installs are (7% business users using Outlook 2007 versus 29% using Outlook 2003 or earlier).

At least Lotus Notes is down to 0.2% of business users.

The authors suggest running their software on your own mailing list. Presumably their analysis is a bit more sophisticated than what you'd get by looking at the Browsers dimension in Google Analytics for a specific "e-mail only" image.

More discussion on the Litmus Blog.

Reminder: The Boy Who Cried "Wolf" Got Eaten

2008-10-27T23:37:00.003+11:00

Parents and children will be familiar with the Aesop's fable, The Boy Who Cried Wolf. It's often told by parents to teach their children the importance of not raising false alarms and of telling the truth. While the moral for children is obvious there's a moral for parents as well: after all, the boy shepherd does in fact eventually confront a wolf.

Australian MPs have threatened total Internet censorship and regulation for so long now that one is tempted to assume that the latest threats of mandatory Internet censorship are just more cries of "wolf! wolf!" But Senator Conroy is not "crying wolf." Senator Conroy is the wolf.

Senator Conroy has been dishonest, aggressive and offensive in his plan for mandatory Internet censorship.

Dishonest, because he did not disclose the mandatory nature of the plan prior to the election. It is clear however that this has always been his plan.

Aggressive, because he has not consulted the Australian public nor listened to those who have raised serious and valid concerns about the fairness and practicality of his totalitarian scheme. His staff have attempted to silence critics by pressuring their employers.

And offensive, because when people have disagreed with his plan, he has accused them of being pro-child pornography.

The Senator is wrong, and his plan for mandatory Internet censorship is also wrong. It's bad policy. It's bad tech. It's security theatre. Give the $44M to law enforcement instead. And sack Senator Conroy.

Bulk Change of Passwords on Solaris (Repost)

2008-10-26T23:06:00.007+11:00

Just filing this one for future use.

I needed to change the password of a large number of accounts on a Solaris box. Unfortunately, I didn't have anything rational like Expect. Also, the Solaris passwd command won't read from stdin. Probably a good thing. :-)

Here's a script that might come in handy. Should only be used on Solaris boxes, unless you grok the Linux shadow password format. But on Linux you can probably get Expect running and that would be a less... primitive way of getting the job done. The problem with this script is that it bypasses PAM so if you're using anything other than shadow files it won't work.

The code is below. If it's getting corrupted then you can also download the script.


#!/usr/bin/perl -w
#
# change_lotsa_passwords.pl
#
# Usage: ./change_lotsa_passwords.pl [/etc/shadow [list of account names]]
#
#

use warnings;
use strict;


# List your accounts here. Or just say @accounts = @ARGV if you
# want to list them on the command line
#
my $shadow_file = shift @ARGV || '/etc/shadow';
my @accounts    = @ARGV;

# Other defaults
use constant PASSWORD_LENGTH => 8;


# Doing it this way lets us make one pass over /etc/shadow and preserve
# its line order
#
my %new_passwords = ();
foreach my $a (@accounts) {
    $new_passwords{$a} = generate_password();
}


# Backup /etc/shadow
#
my $backup_file = $shadow_file . ".BACKUP";
system("cp -p $shadow_file $backup_file");
die "cp failed to backup $shadow_file to $backup_file"
  if ( $? != 0 );


# Re-write /etc/shadow
#
open( my $backup, '<', $backup_file )
  || die "open: Unable to read $backup_file ($!)\n";
open( my $shadow, '>', $shadow_file )
  || die "open: Unable to write $shadow_file ($!)\n";

process_shadow( $backup, $shadow );

close($backup);
close($shadow);

exit(0);


# process_shadow
#   Given two file handles, read in the first file handle and copy to the second.
#   When the first file handle reads in a shadow record for a user whose password
#   we are changing, we will swap out the password and print the new password
#   on stdout.
#
sub process_shadow {
    my ( $backup, $shadow ) = @_;

    # Days since UNIX epoch, the time format used by Solaris in /etc/shadow
    my $last_changed = int( time() / ( 60 * 60 * 24 ) );

    # Read each line from backup, modify if the user is having their password
    # changed, and print the new password on stdout
    #
    my $line = 0;
    while (<$backup>) {
        $line++;
        if (/^ ([^:]+) : ([^:]{2})/x) {
            my $user = $1;
            my $salt = generate_password(2);

            if ( defined( $new_passwords{$user} ) ) {
                print "$user,$new_passwords{$user}\n";
                my $hashed = crypt( $new_passwords{$user}, $salt );
                s/^ ([^:]+:) [^:]+ : \d+:/$1$hashed:$last_changed:/x;
            }
            print $shadow $_;

        }
        else {
            warn "$shadow: Unable to parse line $line\n";
        }
    }

    return;
}

# Return a new (random) password.
# Props: http://perl.about.com/od/perltutorials/a/genpassword.htm
#
sub generate_password {
    my ($length) = @_;
    $length ||= PASSWORD_LENGTH;

    my $ALLOWED =
      'abcdefghjkmnpqrstuvwxyz23456789ABCDEFGHJKLMNPQRSTUVWXYZ!@$%^&*();.';

    my $password = '';
    while ( length($password) < $length ) {
        $password .= substr( $ALLOWED, ( int( rand( length $ALLOWED ) ) ), 1 );
    }
    return $password;
}

Here endeth the hackery. :-)

Postscript: The was originally filed as "Mass Change of Passwords on Solaris" 26 Oct 2008. It was re-posted 2 July 2009 because the code had become corrupted somehow.

Invalidating Page Caches in Mason

2008-10-06T02:48:00.009+11:00

We have a number of websites that use Mason, a (relatively) old Perl library reportedly used by Amazon and Salon among other sites. It has built-in and flexible support for caching at pretty much any level the developer needs, but often the easiest thing to do is to cache the output of an entire component, which looks like this:

<%init>
return if $m->cache_self(key => $key,
expires_in => '3 hours' );
[...]
<%/init>

What I wanted to do was invalidate the cache when the users browser had a Cache-Control or Pragma directive that indicated they did not want cached content (for example, when the user holds down the Shift key and clicks "Refresh" or "Reload").

The naive implementation does not work:

<%init>
# Somewhere, $cache_ok is set to 0 if Cache-Control
# indicates no caching wanted
if ( $cache_ok ) {
return if $m->cache_self(key => $key,
expires_in => '3 hours' );
}
[...]
</%init>

I mean, it does work for the initial request but all subsequence requests will continue to get the old cached content because the cache has not been invalidated, only skipped. The very next request for that component will get the old, cached content -- not a cached copy of the freshly calculated content.

Slightly less naive implementations didn't work either because of the way cache_self works. I was basically trying variations of "expires yesterday" but this parameter was not being consulted when Mason considered whether or not to return the cached content.

Cutting a long story short (well, not that long but kinda boring) here's how I got the page cache to be invalidated ("expired") when the user hits Shift-Refresh:

    if (  $cache_ok ) {
$m->cache_self(key => $page_cache_key, expire_if => sub { 1 }  );
} else  {
return if $m->cache_self(key =>  $page_cache_key, 
expires_in => '4 hours'  );
}
[... continue with component...]

The variable $cache_ok defaults to "1" but is set to "0" if no-cache is found in the Cache-Control request header (for completeness, also check the Pragma directive althought that might indicate the presence of a browser so old it wears flares).

What this code does is returns the cached content (provided it's younger than 4 hours) most of the time. But if the requesting client has indicated it does not want cached content then it invalidates Mason's copy of the cached output and then falls through to the bottom of the if block where normal. non-cached processing occurs.

Post-script: Since publishing this post I've changed and reformatted the source code slightly.

Tuning Nagios Load Checks

2008-07-11T09:36:00.003+10:00

[See also: check_load initial values cheat sheet].

The standard Nagios plugins include a "check_load" command which will raise a warning or error if the load averages for the target machine exceed some threshold. A little while ago the ops manager and I were discussing what those thresholds should be.

The usage for the check_load command is as follows:

Usage: check_load -w WLOAD1,WLOAD5,WLOAD15 -c CLOAD1,CLOAD5,CLOAD15

Without looking at the source I'm pretty sure that the program is either just opening a pipe to uptime or using the /dev/proc file system to read the load averages for the past 1, 5 and 15 minutes. Should be safe to assume then that Nagios' concept of load is exactly the same as uptime's and that the figures are ultimately is coming from the kernel scheduler. (Note: Yup. :-) Just checked the source.)

So the first question is: what does "load" actually measure?

Unix and Linux Load

Breifly, when Unix machines report their "load" (usually through uptime, top or who) they are reporting a weighted average of the number of processes either running or waiting for the CPU (Linux will also count processes that may be blocked waiting on I/O). This average is calculated over 1, 5 and 15 minutes (hence the three values) based on values that are sampled every 5 seconds (on Linux at least). Dr Neil Gunther has written more than you might ever want to know about how those load averages are calculated and what they mean. It's an excellent series of articles (see also the inevitable Wikipedia article).

So assuming we have a single-core CPU, a load value of "1.0" would suggest that the CPU has been 100% utilised over whatever reporting period that figure was calculated for. A load of "2.0" would mean that whenever one process had the CPU there was another that was forced to wait. However, if we have 2 cores, the same "2.0" load value would suggest that both processes got the CPU time they needed, while a load of "1.0" would suggest the CPU had only been at 50% capacity.

On a simple web server, running a single 2-core CPU a load average of "2.0, 1.0, 0.5" suggests that, over the last minute, the CPU has been 100% utilised; over the last 5 minutes it's been 50% utilised; and over the last 15 minutes, it's been 25% utilised. Halve those values if 4 cores are available and double them if only one is in the system.

You can see then that sensible threshold values for warning and critical states requires you to consider how many CPUs and CPU cores your system has. You're therefore probably going to want to set your thresholds per machine or at least set them differently for each different type of configuration.

For example, one of our Solaris boxes has 12 cores so a load of "6.0" is nothing to be concerned about. However that same load figure on another, single-core box might be worthy of a warning or even critical alert, depending on how sensitive we were to process queue lengths on that box. Except if that box is a Linux box with a lot of I/O and slow devices (like a tape drive) and is counting processes that are sitting idle and waiting for an I/O operation to finish. And what is the application running on it? Is it threaded? How is your kernel counting threads in that total -- or is it just counting processes?

Setting the Check_Load Thresholds

So determining an appropriate warning and critical set of threshold values for check_load will depend on what you think a reasonable process queue length will be; how your specific system treats threads; how your applications on that system behave (and their expected responsiveness levels); and how many CPUs / cores your system has. Oh -- and your performance targets or SLAs.

This is why experienced admins use a time honoured, complicated heuristic process to set an initial value and then continually adjust that value based on the correlation of alerts raised and actual performance and hence user impact.

In other words: we rub our bellies and take a guess and then change the values if we get too many or too few alerts. We're experienced sysadmins -- how much time do you think we have? :-)

In our case, for web servers, we decided that over 5 and 15 minute periods we expect spare capacity on the box -- but we only want to be alerted if the box is basically maxing out on CPU over a significant period. Over 1 minute we expect the occasional spike and don't really want an alert unless it's way beyond expectations. We're using Apache with no threading so 1 load point = 1 process using or waiting for CPU.

We've set warning levels for 15 minute load average at number of CPU cores times 2 (plus one!). For 5 minutes increase the threshold by 5. For one minute, increase it by 5 again. Critical threshold starts at number of CPU cores times 4 and then follows the same pattern for the 5 and 1 minute warning.

Here's a sample nrpe.cfg config file for a web server with 2 cores:

command[check_load]=/path/check_load -w 15,10,5 -c 30,25,20

It's important to actually test this set up. Use ApacheBench or JMeter or similar tool to get your load average up and test performance under those thresholds to see if it's acceptable. If your application is unacceptably slow from a user perspective at lower load values then lower your thresholds.

More Information

I've put together a little check_load cheat sheet that has some initial values for some common configurations. It might be a useful starting point if you're just starting to configure check_load in your Nagios environment.

[Note: This post has been edited since initial publication.]

Cisco VPN Software Breaks When Upgrading to Leopard

2008-07-04T21:52:00.008+10:00

I recently upgraded my MacBook to OS X 10.5 ("Leopard"). By and large it's been a pretty painless experience except for the fact that the work issued Cisco VPN client often starts up with an error. The error says, in part "Unable to communicate with the VPN subsystem."

Googling around the suggested fix is to restart the Cisco kernel extension. The following terminal command takes care of things nicely:

$ sudo /System/Library/StartupItems/CiscoVPN/CiscoVPN restart

I'm running version 4.9.01 by the way.

What the hell, here's a video of it if the command line is not where your heart truly lies.

First Look at Django

2008-05-20T00:10:00.002+10:00

Django is MVC... kinda

Django (and all of the frameworks I'm looking at) is based on the "MVC" software architecture pattern. Since these are largely personal notes (can't imagine anyone else reading this) I'm not going to cover the basics here. You can find that in the Django Book or on Wikipedia.

But here's an interesting quote from Chapter 5 of the Django Book:

Because the “C” is handled by the framework itself and most of the excitement in Django happens in models, templates, and views, Django has been referred to as an MTV framework. In the MTV development pattern,
M stands for “Model,” the data access layer. This layer contains anything and everything about the data: how to access it, how to validate it, which behaviors it has, and the relationships between the data.
T stands for “Template,” the presentation layer. This layer contains presentation-related decisions: how something should be displayed on a Web page or other type of document.
V stands for “View,” the business logic layer. This layer contains the logic that access the model and defers to the appropriate template(s). You can think of it as the bridge between models and templates.
If you’re familiar with other MVC Web-development frameworks, such as Ruby on Rails, you may consider Django views to be the “controllers” and Django templates to be the “views.” [...] Neither interpretation is more “correct” than the other. The important thing is to understand the underlying concepts.

This sounds like a subtle but possibly important distinction, particularly as I move between frameworks. I have some initial concerns about Django's interpretation of MVC but I'll keep that to myself until I've really given it time to make sense (basically, how does the split of business logic between Django "model" and "view" manifest itself in the design of your code, and can it lead to anemic domain models?).

Why Django?

Behind the 8-ball here. Not only is Django new to me but Python isn't a language I code in regularly -- in fact it's been some time since I've played with it. I like the look of Python. I've coded some trivial programs with it. The "white space thing" doesn't bother me and I don't really understand why it would bother anyone. Then again it's not the first language I've seen that does it. The first language that I saw that particular feature in was Occam, and I really liked Occam (not that I ever did anything useful with it).

So why Django?

It's based on Python and Python seems like it should be a really good language to work in. I'm a Perl refugee in some ways. I'm finding it hard to recruit really good Perl coders (as opposed to just plain old Perl coders) and I suspect that people who have taken the trouble to learn Python (or Ruby, or Scala, or LISP) are more likely to care about good code. Not because C# and Java programmers don't but ... well, in this market why would you learn Python if it wasn't because you cared about being a better programmer?

There are other Python frameworks that look very interesting: TurboGears and Pylons for example. But I have limited time and Django appears to have an edge for reasons I can't yet articulate. I'll probably drill down into TurboGears later.

Django is the first framework supported by Google App Engine.

The Django core team say sensible things about web development and the community appears active and supportive. A little smug sometimes but hey, wouldn't you be, if you were working in the One True Language too? :-)

Notes on "Mental Detox"

2008-05-16T22:24:00.002+10:00

So apparently of three that set out I was the only one to make it to Friday, which is not even a full week. For shame Kate, for shame. :-)

In the 5 days during my self imposed personal net exile:

I clocked up a modest amount of spam (219 messages), mailing list messages (184 conversations) and personal e-mail.
There were 634 RSS items in Google Reader.
There were 19 podcasts downloaded by iTunes.

And I learned:

That I don't miss, like or need Facebook. But it seems rude to delete my account...
That I follow far too many total freakin' strangers on Twitter. I need some Twitter Equilibrium :-)
I have too many RSS feeds that don't really tell me anything I need to know.

Well. Just some random notes from no-one of consequence. Seriously, why are you reading this?

Mental Detox Week (redux)

2008-05-11T21:35:00.002+10:00

Mental Detox week was 21 - 27 April but I missed it completely (despite being an Adbusters subscriber). So Kate Carruthers and some of her friends (including me) are doing a Mental Detox week from May 12 to 17. Which isn't actually a full week but baby steps people, baby steps.

Given that I'll still be working I'll still need to use a computer. But

No IM
No personal e-mail
No Facebook (that one will be easy)
No iPod (that one will be hard)
No TV, radio or Google Reader.
And no Twitter.

Don't watch TV anyway and the radio sucks but not sure what the policy is on the print version of newspapers. Not reading my RSS feeds in Google Reader leaves me with the feeling that I might "miss something." What if they announce the next Google App Engine next week? On the other hand, real news can put that kind of event into perspective.

In hindsight it was pointless to commit to this during a working week. There's still an awful lot of noise to deal with at work. Next time I'll try this kind of electronic-blackout during annual leave but I'm committed now so...

See you in a week. :-)

About the Application

2008-05-09T06:26:00.000+10:00

The application is going to be small "to do" app loosely based on David Allen's Getting Things Done. Not that the world needs another "to do" app (or even another GTD app) but it fits the basic criteria: it's small but not trivial, capable of being built in a few days and the problem is well understood enough that I can concentrate on learning the basics of the framework rather than solving some other problem.

Here's a short summary of the requirements:

Users need to be able to register with a user name and password to save and access their lists. The "stretch target" would be to allow users to start making lists anonymously, and then to save that work upon registering. If they already have an account, then the anonymous list and previously saved list should be merged sensibly.
Users will make lists of tasks. So we have the concept of a "list" (users must have at least one). A list should have a name and, when users are looking at the "list of lists" should be able to be sorted arbitrarily (in other words, users should be able to force a particular list to the "top" or above another list).
Lists contain tasks. A task needs a name, some notes, and the same sorting behaviour as lists (ie give users full control over the order of the list).
Each task can have exactly one context. A context is a GTD concept -- it's basically the things you need to get a task done. Example context might be "at work", "at home", "online" -- so that a task that has the context "at work" is something to do at work. Each user can have their own set of unique contexts but it's expected that the application will provide some sensible defaults.
Stretch target: Allow tasks to have an arbitrary number of contexts which represent the intersection of those requirements. A task that has the context "home" and "online" is one that requires an Internet connection from home to do. Not very "GTD-y" to make things overly complex though...
There is the concept of a context being "current." In other words, you tell the application "I am at work" and it will only show you the "at work" tasks.
Stretch target: When a user logs on, the default context should be the last context they were using from that particular device. For example, if they've logged on from their iPhone and have set their current context to "errands" then the next time they log on from their iPhone that should be the default context. However, if they log on from work immediately after, the default context will be "at work." It is anticipated that the browser string will be unique enough for this.
Stretch target: Mobile and Twitter support.

So, there you have it. Will not set the world on fire but I'm confident that we can get to know some of the frameworks well enough to get a "feel" for how they see the problem and how much code it takes to get it going.

I shall call it "lulztodo", because we're doing it for the lulz (thanks Nick).

Method and Approach to the Evaluation

2008-05-08T06:07:00.001+10:00

I think we learn best by doing. I learn best by doing. So I'm going to attempt to build an application in some of the frameworks I'm looking at.

The application will need to be

Small: Nothing too fancy -- the point is to learn a little about web development with a given framework, not solve an actual real problem. That will come later.
Well understood: In order to concentrate on learning the framework, I'm not going to try and solve the Travelling Salesman problem in O(1) time.
Non-trivial: Despite the above, it will need to be a bit more fancy than "Hello World." It should require persistent data storage, authentication and do something vaguely useful.

So, if I develop the same application over and over again in different frameworks each time I hope to get roughly comparable results.

Searching for the One True Framework

2008-05-07T23:20:00.005+10:00

No, I don't really believe there is One True Framework. But I am thinking about future web architecture directions for our group.

I'm going to write a series of posts comparing and contrasting a few different languages and framework combinations. These frameworks seem to be emerging as leaders in their language / platform niche:

Django (Python)
Catalyst (Perl)
CakePHP (PHP)
Rails (Ruby)
Wicket (Java)
Grails (Groovy)

That list in no particular order by the way. I don't think I could sensibly look at all of them or do them justice. And it might be smarter to pick a language first and then evaluate the frameworks available for it. But at this stage I'm going for "breadth" first and then I'll drill down into more detail.

Purpose and Function

2008-05-04T23:58:00.000+10:00

To aid memory.

Who are you?

2008-05-03T16:30:00.003+10:00

Who are you?

No-one of consequence.

I must know!

Get used to disappointment.

;-)