Moving into our new office gave us the opportunity to design and build the ideal usability testing lab. Combining the Flow team’s years of testing experience with my work as a sound engineer, we came up with a plan that incorporated everything we could ever want.
We have two ways for clients to view usability tests from outside the room. The first is from a room right next-door, looking through a one-way mirror. We have aptly called this room the Bat Cave as it is small, dark, and due to our fantastically unequal air-conditioning, always a bit cold.
This room is mainly for behind-the-glass testing, where a moderator can view and talk to the recipient through the one way mirror. This is great when testing live websites or very high fidelity prototypes. When a recipient is alone in the testing room they tend to be more willing to work out any problems may have with a site. We have found that it is very easy for a recipient to ask, “Am I doing it right?” when we are sitting next to them, but if we are in the other room giving them instructions every so often though a speaker, we tend to get more realistic results.
The room works well as a viewing space for small groups that can keep quiet, but as anyone that has watched a usability test will know, this is harder than it seems. Screams like “HOW COULD THEY NOT SEE THAT BUTTON?!” slip out by mistake and can usually be heard through the glass. So we set up the boardroom as the second viewing room. This is further away from the testing room so clients can go crazy with yells, comments and discussions without disturbing the test.
This led to our first technical decision: how to send the video and audio from the testing room to the boardroom. It could be done digitally over the network using something like Morae, or it could be done with a separate camera and mic in the testing room connected to a projector and speakers in the boardroom. We settled on the latter, mainly because of the flexibility. Along with the computer in the lab, the digital method would require another computer in the boardroom. It also would have made it difficult to stream the audio into the Bat Cave.
In the end we settled on:
This way if you sat in either the Bat Cave or boardroom you where able to see and hear the recipient and see their screen.
With this analog setup we have not only been able to run website and app usability tests but also one-on-one interviews, mobile device and world-wide remote testing. All recorded and all viewed live the comfort of a boardroom. It gives us the flexibility to adapt the setup as we need it and as new testing challenges arise.
I recently gave a talk about mobile UX for SUGSA.
I touched on two issues that I think are causing trouble at the moment:
Why does your boss always want an iPhone app – even when it’s not necessarily appropriate for your business? And what can you do about it?
Why do apps seem to work better when you remove all the navigational furniture? How come people don’t get lost?
Point one spends some time pondering the topic of responsive design. I think responsive web apps will happen. But there’s a journey of several years before we get there. The technology needs to improve a little more, and many more sites need to get basic responsive design working nicely. Then smartphone users will start to trust the web a bit more, and slowly get used to the idea that they can do a lot via their mobile web browser.
The ideal situation for any qualitative research project is for the facilitator to rely on someone else to take notes. That way the facilitator can focus all their attention on the participant. This holds true for contextual inquiries, in-depth interviews (IDIs), as well as usability tests. However, sometimes it’s just not possible to have a separate note-taker on a project. In those cases the interviewer has to take their own notes — but that can be distracting and terribly inefficient.
So, what’s the best way to be your own note-taker?
I’ve seen interviewers taking their own notes in a variety of ways, but the inherent flaws in each approach has always made me uncomfortable. Some interviewers use their laptops to make notes while the interview is going on. This is very efficient (there’s no transcribing afterwards), but the clicking of the keyboard can be distracting and off-putting to the participant.
Others print out their interview scripts, and leave blank spaces for writing notes about each question. The problem here is that scripts are fluid. You sometimes skip over questions, while other times you go off on an important tangent that isn’t covered in the script. So you tend to end up with empty spaces and cramped notes, all spanning multiple pages. Not ideal.
I am currently working on a project to make it easier for talk radio producers to do their jobs. As a first phase we’re doing a bunch of IDIs with producers — and I have to take my own notes. So I decided to try a new approach, and I like it so far.
I started the project with a long, free-form interview with one of the project leads to develop a generic user journey for producers. I looked for common elements that remain constant regardless of the process each producer might use to perform their tasks, and used that to build a basic journey model for talk radio production. It’s not a full-on journey map, just a list of steps that all producers have to complete when they put on a show.
I then created an A3 sheet (6.5 x 11.7 in) for each interview, consisting of the participant’s name, time slot, and the headings for each of the steps in the journey. While conducting the interview I filled out any insights that came out for each of the steps — as we worked our way through the script. Here’s an example of what a sheet looked like after an interview:
I discovered that this approach has several advantages over other note-taking methods I’ve tried:
I’m sure this method of note-taking isn’t perfect, but I’m quite happy with how it turned out. Please let me know if you have any suggestions to improved this process — or if you have a different method that works well for you.
The critical thing about the design process is to identify your scarcest resource. Despite what you may think, that very often is not money. For example, in a NASA moon shot, money is abundant but lightness is scarce; every ounce of weight requires tons of material below. On the design of a beach vacation home, the limitation may be your ocean-front footage. You have to make sure your whole team understands what scarce resource you’re optimizing.
I’ve been thinking about this in the context of web design over the past couple of days. What is the scarcest resource in web design projects — assuming it’s not money? I’d say that in most cases, the user’s attention is the scarcest resource online. There are just so many distractions, interruptions, and other things vying for attention (I mean, there are goat videos to watch, after all). What does this mean for the design process? Here are some examples:
The examples above are pretty obvious, but each design problem will have its own unique set of challenges, so I think this should be an important part of any process. When starting a new project, try to work with the team to agree on what the user’s scarcest resources are, and how you plan to optimize for that.
(link via @retinart)
Here’s some great advice from Joshua Porter: when you’re in a design meeting (or any other meeting, for that matter), Always be capturing:
“Always be capturing” is about the habit of continuously recording the value from your conversation. For example: If you’re talking about a new concept, you should be sketching it as you talk so your team has a shared understanding and an artifact of the conversation.
Joshua gives some great tips, including taking photos of your sketches and uploading them to Dropbox immediately. We do this as well, except instead of Dropbox, we add the photos directly to a Trello card related to the project at hand (using the iOS/Android app). Those photos are then immediately accessible to everyone who works on the project:
We’re increasingly using Trello not just to schedule our work, but also to make our thoughts and sketches immediately available to the team within the context of the task at hand. That’s something Dropbox can’t do.
Cennydd Bowles casts a critical eye on the UCD process in Looking Beyond User-Centered Design. He lists several weaknesses of UCD, including the negation of style — the idea that “Good design is invisible”:
By negating the idea of a designer’s influence, we also negate the idea of style within the UX discipline. We’re saying that, done properly, it should be impossible to tell one UX designer’s work from another’s. There should be no signature elements, no philosophical movements, no overarching tenets except that of transparency. […]
Of course our designs must put users first. But there is never just a single way to meet user goals. Instead of trying to deprecate style, we should embrace it as a way to drive our practice forward and lend personality to the things we make. In a marketplace of bewildering clutter, products with a damn opinion are by far the most interesting.
Cennydd’s main point is not that UCD is a bad methodology and that we should stop using it, but that its dominance might be a problem. Designers should be familiar with many modes of design, so that we can draw on a variety of techniques, not just our favorite one. It’s a good article, well worth your time.
One of the alternative approaches that Cennydd touches on is Activity-Centered Design (ACD). I want to focus on that because Mike Long wrote a great article about ACD recently, cheekily called Stop Designing for “Users”. He discusses some limitations of personas before introducing Activity Theory, and a technique they use often called the Human Activity System Diagram.
As an example, Mike shows how he used the diagram to understand behaviors and actions in a local carpool:
Image source: Stop Designing for “Users”
Mike explains how ACD resulted in a better understanding of the problem (and possible solutions) than UCD would have:
By analyzing the whole system, rather than individuals, I came to the realization that the objective for drivers and passengers to carpool with minimal coordination was already happening. In the casual carpool pickup spots rituals and norms naturally emerge with readily available tools and artifacts. Namely, vehicles and queues. What about other issues like lost-and-found items? The system has already compensated: If someone forgets to take a purse or a phone when they get out of a vehicle, the driver can simply post “found” signs at the head of the passenger queue. Any other solution I can think of would require another layer of coordination between drivers and passengers.
Both these articles challenged me to think broader than UCD and explore other design systems — even ones I’ve been skeptical about in the past, like Genius Design. I realised that even though UCD is a proven way to build great user experiences, it would be foolish to ignore the good lessons we can learn from other methodologies.
As an industry, we sell websites like paintings. Instead, we should be selling beautiful and easy access to content, agnostic of device, screen size, or context. If you can get your client to believe in the sales process that you’ll do that for them, they won’t care what the site looks like.
I agree with Dan’s viewpoint on what we should sell to clients, but not that they won’t care what it looks like. They always care. A little earlier in the post Dan says this:
I don’t think we’re in a post-PSD era, but I do think we’re moving towards a post-“full-comp” era.
Now that I agree with completely. At Flow our deliverables to clients have increasingly moved away from a PSD for every page on the site, to a combination of clickable prototypes, style tiles, and PSDs for key pages. Developers and Product Managers love this because they can play around with the prototype to see how the site/app works, not just what it looks like. Other decision-makers love the style tiles in particular, because it allows them to guide what the site looks like without getting distracted by the intricacies of the interaction design (which requires a different type of discussion/feedback cycle).
In other words: we should sell clients on our design process, agree upfront on what deliverables will help them accomplish their goals, and make the appropriate amount of PSDs part of those deliverables as needed.
In his introduction to a very interesting user research case study on the MailChimp blog, Gregg Bernstein writes:
Here at MailChimp, we’re realists—as much as we love email and all the things you can do with it, we understand that building a campaign is a task, not a life event. You want to get in, get done, and get on with things. Duly noted.
The post goes on to explain how they managed to shave an average of 32 seconds off a core email campaign creation task at MailChimp. But it’s that opening sentence I’d like to dwell on for a bit.
I wish more companies understood this crucial point. As designers and product managers we obsess over every detail of our product, but it most likely makes up a minuscule part of our users’ days. Unless you work for Facebook or Twitter users don’t wake up wondering what new features you’ve released, how your conversion rate has changed over time, or what awards you’ve won. They care about getting a task done, and they care about nothing getting in their way — they care about getting on with their lives.
Yesterday The Onion published an article that is such a spot-on commentary on how we’re mostly ignoring this reality. From Internet Users Demand Less Interactivity:
Tired of being bombarded with constant requests to share content on social media, bestow ratings, leave comments, and generally “join in on the discussion,” the nation’s Internet users demanded substantially less interactivity this week.
Speaking with reporters, web users expressed a near unanimous desire to visit a website and simply look at it, for once, without having every aspect of the user interface tailored to a set of demographic information culled from their previous browsing history.
Exactly. We’re in an environment where too often products and functionality are shaped by who we are and what’s technically possible, not by what user needs call for. XKCD called us on it years ago, but we’re just not listening:
The solution to this problem is to get out into the world and understand how our products and services fit into the lives of our users. How we can help them accomplish their tasks more effectively. Mark Hurst summed this up well in his post What is a career in user experience really about?:
Good user research isn’t a matter of learning the steps of some trendy methods, as though one were just following a cookbook. Instead, good UX work requires a genuine interest in observing, listening to, and learning from other people: primarily the customers themselves, but also the organization that owns the product. That observation, and that listening, must stem from a genuine human interest in people.
We can all do with a shot of humility about our products. We might think what we’re making is a gift to humankind that deserves proper respect — and we absolutely should be proud of our work. But a bit of human empathy will show us that most users have only a passing fly-by relationship with our products. That’s ok though. Understanding how our products fit into people’s lives realistically will help us to improve that fit and (hopefully) become indispensable to them. That’s our job as User Experience Designers.
We often see marketing teams asking for heaps of information during signup. “How did you hear about us?” “What is your title?” (I’ve seen a title list that included not just Mr, Mrs and Ms, but His Royal Highness and Admiral. I kid you not.)
I appreciate that marketers want to know these things to help them in their jobs. But adding fields will cost you users. So here’s a handy guide to deciding which fields you need to get rid of…
I did this talk for a meeting of the SA UX forum. It looks at three patterns of irrationality: loss aversion, the Ikea effect and the identifiable victim effect, and how to work around them to get the best from yourself and your team.