Too many surveys!
If you visit enterprise hardware and software vendor websites frequently, there’s a good chance that you’ve been sent a survey. They often appear as a pop-behind window that detects when you leave the main website and asks you a few dozen vague questions. These questions give you a 1-10 point scale to grade them on such topics as “is this site easy to use?” or “how does this site match your expectations?”
Perhaps there is something else going on behind the scenes, but these questions don’t seem useful for improving a website. What change would you make if your site scored a 6.3 average on “matches expectations” that would be different than the one you would make if you scored 8.2?
This is the a similar mistake to that of Microsoft’s statistics focused design of Windows 8. Statistics and design don’t interact in the obvious way. You don’t survey all the people who walk in your company’s front door, ask them their favorite color, and paint the door the average of all the colors everyone picked. Instead, try picking a philosophy first, then using statistics to gauge whether you met it. Decide on a reason why the door should be blue and use a solid metric that is objectively correct to measure it. Perhaps the door to your company is meant to make the club feel exclusive to insiders, or to invite passerby to walk in. Then you can measure the number of new or returning customers to determine whether your ideas are right.
The best example of this is the wrongheaded way Microsoft is thinking about the “Task Manager” in Windows 8. They surveyed everyone who opened task manager and figured out what they did. This number is useless. Let’s examine why:
Looking at the data and talking with customers, we determined that the most common usage of the tool was to simply end or “kill” an application or a process.
The reason the most common usage of the tool is this is because, historically, this has been the key purpose of Task Manager, a feature that was not provided anywhere else in Windows. They had a few other buttons: ”Switch To”, ”Arrange Windows”, “Cascade”, “Tile”, “New Task”, etc. But during the Windows 95 switch, cascading, tiling, and arranging windows became superfluous since it was now better handled via the Windows taskbar. Starting new tasks became the function of the Start menu. There was no need for the Task Manager to do anything but end a task that had become stuck.
So if you survey people to figure out “What do they do here?” you will get an answer that is dependent on what you trained people to do in the past. That’s bad statistics. That results in lame design like Windows 8’s “Fewer Details View” - I have never seen a more useless dialog box in my life. Why should it exist?
If you instead hire a good UI designer, the designer will ask you “What feature is this dialog intended to accomplish?” You then realize that Task Manager exists to find a problem application - one that is stuck or consuming resources - and to allow the user to kill it; this is a function that no other part of Windows performs. The UI designer will next ask you, “What information is needed to allow the user to make that determination?”
With this task in mind, you can research the list of information that’s helpful to the user, and you can then evaluate other feature proposals by how well they match the goals set forth by the UI designer. Should we add a new button? Does that new button complement the purpose of this dialog, or does it just add needless complexity?
When all is complete, you can design a survey to discover if customers understand the goal of the dialog, if they found the information they needed, and if they were able to kill the task that was misbehaving.
