The usability of usability statistics

I have always loved statistics as a subject - chi square tests, confidence levels, Weibull distribution et al...this an entire discipline which enables me to say with an exact confidence level on how much what I know about a small group of people is applicable to humankind as a whole.

Statistics can reveal some quite interesting things - as Mr. Levitt has pointed out in his albeit pompously titled but thoroughly entertaining book Freakonomics: A Rogue Economist Explores the Hidden Side of Everything.

Onto statistics on usability. There are innumerable. And I don't know what to do with them. For an exhaustive list of what you can derive with just 4 pieces of data:
1. Task completion
2. Completion time
3. Satisfaction ratings
4. Errors encountered
go here.

I would rather use SUS or System Usabilty Scale, a very simple measure of overall usability which was invented more than 20 years ago. For a very short document on its use, go here. In a study conducted on the various usabilty satisfaction survey questionnaires and their efficacy presented at the UPA Conference in 2004 available here, the SUS was found to give the most reliable results.
Excerpt:
One of the simplest questionnaires studied, SUS (with only 10 rating scales), yielded among the most reliable results across sample sizes.
–Also the only one whose questions all address different aspects of the user’s reaction to the website as a whole.

Indicentally, Morae - one of the most popular usabilty testing tools, comes pre-packaged with SUS.

So what to do with the rest of the statistics? I personally don't use them and not sure of how I could. I think conducting usability tests is an explicit recognition of the possibility that there might be usability flaws, and given the subjective nature of usability, it is better to gather them from a whole bunch of representative users than a few experts. Usability statistics just give you numbers which reinforce what you learn from the qualitative feedback. Analyzing the qualitative feedback gives more actionable information than statistics.

But if you really need an easy comparison point between an earlier and redesigned UX, I would use the SUS.