The usability of usability statistics

I have always loved statistics as a subject - chi square tests, confidence levels, Weibull distribution et al...this an entire discipline which enables me to say with an exact confidence level on how much what I know about a small group of people is applicable to humankind as a whole.

Statistics can reveal some quite interesting things - as Mr. Levitt has pointed out in his albeit pompously titled but thoroughly entertaining book Freakonomics: A Rogue Economist Explores the Hidden Side of Everything.

Onto statistics on usability. There are innumerable. And I don't know what to do with them. For an exhaustive list of what you can derive with just 4 pieces of data:
1. Task completion
2. Completion time
3. Satisfaction ratings
4. Errors encountered
go here.

I would rather use SUS or System Usabilty Scale, a very simple measure of overall usability which was invented more than 20 years ago. For a very short document on its use, go here. In a study conducted on the various usabilty satisfaction survey questionnaires and their efficacy presented at the UPA Conference in 2004 available here, the SUS was found to give the most reliable results.
Excerpt:
One of the simplest questionnaires studied, SUS (with only 10 rating scales), yielded among the most reliable results across sample sizes.
–Also the only one whose questions all address different aspects of the user’s reaction to the website as a whole.

Indicentally, Morae - one of the most popular usabilty testing tools, comes pre-packaged with SUS.

So what to do with the rest of the statistics? I personally don't use them and not sure of how I could. I think conducting usability tests is an explicit recognition of the possibility that there might be usability flaws, and given the subjective nature of usability, it is better to gather them from a whole bunch of representative users than a few experts. Usability statistics just give you numbers which reinforce what you learn from the qualitative feedback. Analyzing the qualitative feedback gives more actionable information than statistics.

But if you really need an easy comparison point between an earlier and redesigned UX, I would use the SUS.

The technology in UX

I'm currently wrapping up a UX consulting assignment for a financial services ISV in the east coast. Quite a rewarding experience, and a test of the process I posted on this blog here.

An interesting problem to solve was this:

How do you make it transparent to the user that the huge amount of data being manipulated in a desktop application comes from a server?

To give more context, imagine you had around 5-10K rows of data, some 30-50 columns in each row in an Excel spreadsheet. This data keeps changing as there are other users updating, as well as the system. New rows also keep getting added.

Your user is someone who monitors this data, making sure the system is running as expected, making changes as required. To enable this, the application allows slicing and dicing, custom views, conditional formatting etc.

The problem restated - as the user is using the various features, how does the application make the user feel the data is on his desktop rather than on the server? Also, now that you've provided these features, how do you reduce the server load when each client submits a query every few seconds?

The solution - load the huge amount of data on the user's machine, and sync this with the server every few seconds.

However, this doesn't solve the problem of the server load. So instead of executing the query on the database every few seconds, the query could be executed every few seconds and kept in a cache on the server. This is still inefficient, so each time the application receives data, it also gets when it should make the next request, allowing you to distribute the load on the servers.

What does this allow? The user now doesn't feel the data being updated. It happens in the background, and any changes to the data currently being viewed is done unobtrusively. There is no longer the need to click a "Refresh" button to get the latest data. The response to any filtering or custom view being applied is instant. For all the user knows, the data is on their desktop.

Firefox 3: First Impressions

Today is Firefox 3 download day. Although FF3 beta 3's been available a few weeks, I religiously download and install Firefox 3 today to do my bit towards setting the record. Download Firefox 3 from here, if you haven't already.

Before I give you my comments on its usability, a little aside on the browsers I use, just to set context.

Firefox has been my default browser since version 1.2 (or thereabouts). Over 90% of my desktop browsing is via Firefox. I also use Opera at times - I think it is a pretty nifty browser and really the forerunner for feature introductions. They came up with tabbed browsing, for example. I use Internet Explorer (now IE 8 Beta) as well - Sharepoint sites render better on IE. I like Flock too, but it plays second fiddle to Firefox only because I use so many plug-ins/add-ons on Firefox.

Enough digression.

Here are my first impressions, usability wise. For an application that gets used so much, usability is also a function of time - a novelty today may become an irritant tomorrow.

1. "What was that site again?". How many of us have tried rummaging browser history to find that great blog post we read, or event we heard about? Address (URL) matching is passe. Firefox 3 matches words within the page title and tags on that page. Also sorts itself by time (how recently you viewed that page) and frequency. Neat. This is a type of feature that gets refined over time.

This feature could have been made a little smarter, though. Try typing 'com' and you'll see what I mean. I get google.com as the first choice. It also looks inside ID fields that form part of the URL - that makes it more confusing. The transition (appearance) of this drop down could be more subtle - it is a bit of a distraction when it appears, so I can see some users getting put off by this (they have the option to turn it off, btw).

2. Enough has been said about Firefox performance woes on Vista - and I have been at the receiving end of it more than I'd like to. Performance and Security have been a focus area this time round, with over 15,000 fixes made. Although it is difficult to tell right away, I hope this means less trouble with Firefox on Vista for me.

3. I usually have dozens of tabs open as I browse, so I'm pleased with the simple animated transition on the tabs - a little carousel like. I really wish they'd made an improvement on where the a "new tab" shows up. I always think that clicking on "Open in New Tab" should create a tab to the immediate right of the current tab - so you know it is right there. Firefox opens new tabs at the rightmost end of the list of tabs (not a 3.0 feature, its always been so) - painful when you have 10+ tabs already open. You either a) have to scroll scroll till you get there or b) forget that you opened the new tab or c) lose context of what you were reading earlier.

4. They've visually integrated the browser well with Vista (and Linux and Mac, apparently - I've only seen screenshots of those). Here's a couple of examples where the icons have more Vista-ish look and feel.

Why does this matter?

When you are trying to eat into Internet Explorer's market share, and move beyond techno-savvy users, you'd like to get the users "comfortable" with your application, and the more it fits in to the OS the faster you get comfortable with it. Take iTunes for example. The look and feel is (and has always been) Mac OS-like. Takes a while for users to figure their way out and get used to.

Here's a quote from Alex Faaborg's blog about firefox 3, describing its usability goals

Why do We Believe Visual Integration is Important?

We decided to focus heavily on visual integration with the platform for the following reasons:

-Cross platform applications that use a consistent appearance across different operating systems (like RealPlayer, or applications developed using Java Swing) feel foreign and strange
-The Web browser is a central part of the user’s computing experience
-We want the user’s first impression to be feeling comfortable with the UI
-If the transition in and out of Firefox is jarring, the user won’t achieve flow when completing tasks (like when you are driving a car and you realize you haven’t been thinking about driving for quite awhile, or when you are reading an interesting book and you turn pages without conscious thought).
-We want Firefox to feel like the browser your computer should have shipped with

Those of you who have worked with cross platform consistency will appreciate the ambition and rigor required to achieve this.

The sorry part about Firefox 3 is not all the Add-ons I use have been upgraded to be compatible yet, I guess they wwill be, soon enough.

What are your impressions of Firefox 3? What's your favorite Firefox feature, regardless of the version? Time to spread some Firefox love.

What do you think?