Current Issue
Top reasons why usability doesn't pay - 1 June 2006
by Gary Bunker
Read this article in Chinese (Translated by Binghua xu, Proofread by Sean Liu, Christina Li)
Courtesy of Usability by Design
I’d love to say that usability investment pays back every time.
Unfortunately that’s just not the case; often a usability project will completely fail to deliver measurable benefits, and this can have a devastating effect on future organizational commitment to the user-centered cause.
The good news though is that this is easily avoidable. So, in reverse order here are the top six reasons for failing to see a good return on investment from usability, and how to avoid them.
6. Not measuring it
We are always astounded when customers invest huge sums of money into development, commit to usability testing the result – and then fail to measure the ‘before and after’ figures.
We often receive notification that the new software or web sites have been launched, thanking us for our input – but when we ask whether they have seen measurable improvements yet, the answer is often ‘oh we don’t know, we don’t really have the figures from before’.
It’s impossible to tell if you have delivered benefits from your usability improvements unless you measure what happened before and what happens after. Take key metrics from core processes such as sign-up, basket/checkout and payment. For software you can record key timing and task data from the application itself, but also support call ratios and performance support usage.
Once you know how the figures have improved, it’s easy enough to convert that into a dollar figure and understand just how much you’ve saved/made.
5. Over-spending
I was training some guys from a major insurance company a year or so ago, in what basically amounted to ‘do-it-yourself’ usability. During one of the breaks they asked me a number of questions about a specific interface and some major problems they had with it. I did the best I could without seeing it, but in the end I had to tell them that I really needed to be able to see the thing in order to analyze what might be causing the problems – and to do that we’d have to talk about paying for a minor amount of my time as there wasn’t time during the course.
To put this into perspective I was talking about a couple of hundred dollars, and this was an International insurer with a national, mission-critical system.
To my surprise, the guys told me that there was no way they’d be able to gain approval for any spend at all. When I asked why, they told me that a large consultancy had recently come into the company to run a major usability overhaul of a different application. They had delivered measurable results – but at a cost of several hundred thousand dollars, far greater than any savings delivered. The company had had its fingers burned, and didn’t want to touch usability projects again.
In another project the client was not convinced that a usability test of just 12 people would tell them all of the critical issues they needed to know; they felt they needed to speak to at least 20 or 30 people to have any kind of statistical basis. We argued that they didn’t need that sample size and even showed them evidence, but they were unswayed and we dutifully set up a test of 25 people over five days.
By the end of the second day it was a bit like the movie Groundhog day, watching the same results come out time and time and time again.
After the project, the usability advocate who’d gained budget approval glumly told us that further testing was looking unlikely; the same marketing team who’d pushed for 30 usability tests had worked out that the results couldn’t be cost justified.
Usability has to fit the budget and the project. Many times when we’re approached to run usability tests we end up proposing cheaper alternatives such as expert reviews; that way the customer can see a greater return for their investment.
4. Not listening
Not so long ago I had to report back to an e-commerce site on a usability test for their current interface. One area that had particularly faired poorly was the registration, which was three pages long and asked for a lot of personal data. We’d identified many particular problems as well as an overall sense of frustration and intrusion from the users; they simply hated being asked for so much that they didn’t feel this site needed to know. As we knew there were problems with the conversion rates for this process I didn’t expect any major resistance, but I was wrong.
When I explained the overall user perception, the team began making excuses; excuses such as:
- We ‘need’ that level of data, so we can service them better later on.
- Marketing need to know more about them.
- Only the ones who didn’t really mean to buy are dropping out.
- and my personal favorite, “these users must be stupid”.
In another project for an industry-specific web service, potential users (who would be paying for the service once launched) showed a clear preference for navigation-based access to several different types of information. They wanted to drill down, but have a search for backup as well.
When we presented that to the design team, they looked pretty unimpressed; they’d spent weeks developing a smart search system and thought that search was the answer to every question.
In the end they developed the service based around search, with almost no drill-down abilities. The service did okay, but not as well as they’d hoped.
There’s very little point talking to your users and following a user-centered design process if you’re not going to listen to what they say.
3. Not doing
We often contact our customers a few weeks or months after large projects, to see how they are going and to capture positive metric figures they may have seen.
It’s disheartening how often we find that nothing has changed.
Often usability tests are run almost with an expectation of receiving a big green tick; they hope that we’ll tell them everything is rosy in the garden and nothing could possibly be improved.
of course it’s never like that, and even the best interfaces are going to get a list of places where they can still improve; often, that’s a bit of a shock to teams not quite ready to act. Testing has been run, results have come in and a report has clearly shown how the interface needs to improve. But months go by, and without the management commitment nothing actually changes.
It’s vital that any usability project, even if it’s just a quick safety check, take the long view and plan to implement changes if changes are needed, otherwise there’s little point and nothing is going to improve.
2. Doing it wrong
I’ve been involved in usability now since 1997, almost a decade, and in that time I’ve come across three projects where the usability component was a complete and utter failure.
In all three cases it was because the work was, to put it simply, done wrong.
Usability and user-centered design are skill sets that anyone can learn, but they need to be learned and applied in the right way. In the same way that asking your plumber to wire your house for you is asking for trouble, getting your developer to run usability tests or a project manager to facilitate user-centric design sessions is going to lead to the wrong data being captured. The term ‘Garbage-in, Garbage-out’ relates to usability as well as to computing in general; if you capture the wrong data, you’re not going to design the right system.
In one of these three projects, the participants had been over-incentivised and felt pressured into giving positive results. They were paid well over market rates (which left them feeling they owed something) and were being interviewed by the very same company who designed the software (though not one of the design team). Unsurprisingly they gave glowing reviews of pretty much everything they were shown.
It was only after market sales slowed and support costs rocketed that a problem was suspected, and a second, neutral and correctly run test showed a completely different picture – numerous major design flaws came out.
You need to ensure you have:
- The right usability skill set.
- The right people doing the right jobs (plumbers plumbing).
- The right tools (test equipment, analysis tools, logging software, audit sheets).
- The right users (absolutely paramount).
1. Focusing on the end-game
Usability is like the words in a good stick of rock – it should run all the way through.
To ensure you have a user-centered design you need to focus on the users and their requirements right from the start, from the requirements stage. With too many projects we see, usability is a tacked-on effort, used merely as a check at the end.
Several years ago we were asked to usability test a service-based website for professionals in the health care industry. The service was already on it’s third incarnation, after failing miserably the first two times. When we asked why it failed, nobody seemed too sure – they only knew that take-up had been abysmally low.
In this case usability was tacked-on, and tacked-on after two failed attempts at that. All we could do was to take the third design into testing and see what happened.
The results were terrible; so terrible in fact that for the only time ever I had to tell the client to cancel the tests. Not only were we simply seeing the same catastrophic failure to meet any of the user needs time and time again as we ran each session, but we were actively alienating the audience – as each participant saw just how little improvement had been made in this version, they were becoming increasingly despondent and negative towards the brand.
An analogy I used in a training course once was that usability was like a periscope on a submarine when you were trying to find your way to a target. Without the periscope you were heading out blind towards your goal – you might get to exactly where you wanted to be, but it would almost be more luck than judgment. But with the periscope you could pop up and check where you are along the route, adjust course accordingly and ensure you made it to the right spot every time. Okay, I didn’t say it was a good analogy..!
The point is that if you stop and check the route as you progress, using evaluations, collaborative design, user interviews, user testing, then you can see how you move closer or further from the target. That way when you arrive at the end game and launch time looms, you can be pretty certain that you’re almost exactly where you need to be. Focus just on the end game, and you could be anywhere.
Comments made
Possible Related Articles:
Latest Comments