72/100SDT The Ins and Outs of User Research. Lessons Learned at UX Poland 2015

72/100SDT The Ins and Outs of User Research. Lessons Learned at UX Poland 2015

It’s been a month since UX Poland 2015 conference but I’ve finally managed to summarize my learnings from the workshop with Matt Lee (Booking.com) and Dan Maccarone (Charming Robot). It was a 2 days workshop full of theory and practice on doing user research. This time my notes are in English, presented in no specific order:

1. You are not the user! Do not test the product in-house only.
2. You should only research on the people who use the product. Do not make a mistake of “researching in Starbucks”, by showing the designs to random people and asking “is it pretty?”
3. Sometimes research highlights the core problems and that is not easy for the management. It can make a mess in someone’s vision, and that’s the reason people avoid it – just to avoid finding out that there are problems.
4. There’s always the next version – the product is never finished.
5. Lab tests are often too unnatural.
6. The more relaxed and natural environment of the user, the better results research gives.
7. Recording the mobile session: best done with an external camera from the top, or only by observing.
8. Cultural differences: at booking.com they localize not only the text but also the design itself.
9. Language: do the research only in the mother tongue of the interviewee, with native speaker researcher.
10. Usually best in the research team: 1 male and 1 female. 1 mainly doing notes, 1 asking questions.
11. About UX: image sliders are usually a bad solution.
12. A/B tests should be stretched for a week or two, as there are various users at various times of the day and week. Do so even if you got 1000 answers in 2h – better to stretch.
13. At a certain stage you have to make a decision that this amount of research is enough. 12 in depth context interviews are more than enough.
14. Doing reseach with teenagers: always divide the group into guys and girls. This way they won’t try to impress the other ones.
15. About recruitment via Facebook: sometimes you have to work with what is available. So if the budget is low you should try out if social media gives the right research participants.
16. 2 types of users: loyal ones (recruit these e.g. via a newsletter) and others (recruitment via external professional firm)
17. Remote research (e.g. via Skype): if needed then it’s OK to do it, but you should be aware of limitations and not rely only on this type of research.
18. Do not interview techies, developers, UX designers. They are too biased.
19. Start the research by saying something about yourself, something human, to make people trust you more.
20. It happens often that the interviewee doesn’t show up. Avoid this by good preparation, plan B, and if it’s already too late then improvise – get someone else.
21. Always be familiar with the interview guideline. Remember main topics to discuss and specific questions. Idea: how about a visual (drawn) interview guideline?
22. Remind yourself somehow to take photos of the interviewed person.
23. Early stage, with a prototype or a concept only: do a wide scope of the research, as well as task testing (ask the user to follow specific steps). With a live product: do natural tests, in the right context.
24. Pay for the time of the interviewees. Pay them their normal hourly rate. With specialists such as doctors simply get a private appointment.
25. Do blind tests – do not say what is the company you are working for. Hide your product within other competitors’: e.g. test booking.com, hotels.com and HSR.
26. About analysis: personas are not always the best way to go. Rather try thinking in terms of specific behaviours.
27. Do a recap right after the research. Go for a drink in your team and note down the most pressing points. It’s a simple way to find most common insights.
28. More tips about analysis can be found in the presentation below, from the slide 42 onwards.


Q and A session:

How to convince the management to do user research?
– Try to quantify and look at the return of investment from the research. Research findings turn into real changes in the product, which increase the revenue.

Tips on low budget research?
– Research doesn’t have to be expensive – with a single laptop and with users from your target group, you can do great research. Record your sessions.
– Do round table research. Invite experts for a beer and let them discuss the topic.
– Street research. Just don’t do the mistake of showing stuff and asking “do you like this?”. Guerilla research works as long as you can validate the findings with some other method.

Length of interviews?
– Try to limit the discussion to one hour. Otherwise no one can keep up the energy.

Should you do research with an in-house team or with an external company?
– It’s great to have one person with expertise and the other one learning. So you can get an external consultant and assign to him one person from the company to conduct research and also learn.
– Outsourcing research has one huge advantage: the researcher is not biased and is not involved in the politics of the company. He can find out and deliver the hard truth without the fear of anyone in the company. The hallmark of a good researcher is being unbiased.

What do you do when the manager doesn’t want to listen to the researcher’s findings?
– Nothing, your job is to do the research as good as possible, deliver it and then say that it’s their responsibility to take the suggestions or not.

What are good deliverables for the management?
– Give them 3 pages report summary with a couple of top highlights and top issues. Have a longer version prepared if they want it. Also do a video with edited user recording sessions, they work very well to present the issues. Short ones are good for the top management, but even an hour long video highlight is something really interesting for the team as it shows the real users.

Doing the product research on companies’ employees?
– One way to do it correctly is to divide the data into 2 categories: a) coming from the product representatives and b) the rest of employees such as call center, support, etc. It allows to limit the result only to the second group which knows the user the best and is not biased by building the product.

Is it a good idea to take developers, who are not experienced in conducting research, to a user interview session?
– Yes, as long as they know the rules what they can and can’t do. Everybody who touches the product – developers, designers, product managers, should be able to at least observe users interacting with the product. At first they should have a passive role (note taking) but they can learn.

Quantitative vs qualitative?
– You should do both. Basically user sessions allow you to put a user “face” on the hard data.
– No new feature is released on booking.com without going through an A/B test. Such tests require quite a lot of traffic but still even small startups should try it out.


UX Poland 2015 was a good event! If you want to find out more about all 4 days of the conference, check out the post by Adrian Dampc at UX Watch.
I’m looking forward to the conference next year.

Andrzej jest założycielem TAK Innovation Agency, regularnie współpracuje z Creation Center przy Deutsche Telekom w Berlinie. Jest absolwentem Advanced Track w HPI School of Design Thinking (D-School). Prowadził warsztaty, projekty lub badania użytkowników m.in. dla takich organizacji jak Kloeckner, Nike, Centrum Nauki Kopernik, UCANDO.pl, Stowarzyszenie TOP 500 Innovators oraz Škoda Polska.
Organizuje warszawskie spotkania Design Thinking Meetups. Ciągle w trasie między Berlinem, Warszawą a Katowicami.