The role of Games User Researchers is to help the team design and build a game with the best possible user experience given the resources available.
However, working with people is hard – incredibly hard and the chance of committing an error or making the wrong conclusion is high even for experienced user researchers.
It is possible to do user research right. To avoid any flaws of specific methods and approaches. But it would be prohibitively expensive to do so. In reality, we are faced with limitations such as resourcing and deadlines, and this means that we have to deal with inadequacies in our research methods and limitations in the results we can derive from them. Not even university-based research can be done without compromises and assumptions. Being aware of these limitations, flaws, and problems is vital to inform the kinds of conclusions we can draw from user research.
Challenges of Games User Research
User research in games is typically done on unfinished products and outside the usage situations that end users will actually play these games in. This introduces a number of challenges, not limited to:
An error is when something is not done correctly. We test a limited build but pretend it is the final one. We use only a sub-section of our target audience in tests. There is a lot of potential for errors in user research. Errors come in two types:
- Bias is the introduction of a systematic source of error. This can be underlying errors or errors of measurement. This could for example be an error introduced, which systematically affects a part of the participant group. For example, having very small font size in-game menus and testing with participants with and without bad eyesight, without accounting for this in your analysis. In this case, participants with bad eyesight would be systematically affected by the menus, which can lead to bias in the test results.
- Noise is the same thing as bias but is a random or unsystematic error in your experimental setup. Noise can be deeply problematic as it can hide signals in data – signals we want to find and learn from. For example, running performance tests at different times of the day (morning, evening, night …) and not accounting for this, means introducing a variance in the performance of the players that we cannot control for, and introducing noise.
Validity of User Research
Validity refers to our ability to avoid an error. It is the ability of a user test to actually measure what it is supposed to measure. The archetypical example in games user research is when playtesters are asked to rate how “fun” an experience was. There are several problems here, for example, how we understand “fun” as a concept varies across people and cultures. Additionally, fun is ill-defined and thus difficult to define a measurement scale for. Another typical issue is the assumption that playing in a user testing lab will see the same behaviours and experiences as playing in the natural environment the game is intended for (e.g. at home).
There are many threats to the validity of human-focused experiments. For example, learning effects. If we are testing the controls of a game with a group of testers, then reinviting them to test the controls again after some tweaking, they will have learned and remembered something from the first test, which influences their ability to use the controls and thus lead to erroneous conclusions about how easy-to-use the controls are. Another common validity threat in games user research is fatigue – at some point testers will get tired and this will affect their user experience, performance, etc., and this, in turn, affects the validity of your measures. Having a list of validity threats handy is a useful planning tool.
Generalizability of Your Research
Generalizability (also called ecological validity) is the degree to which we can assume that results from a piece of research will characterize not just the sample we are working with, but the population we are interested in. For example, to what degree would we expect the experiences of a 12-person participant sample to generalize to the population of players for our game? We may have excellent validity in an experiment without being able to generalize the results.
Within the confines of generalizability, false positives and negatives are important concepts. These are associated with test conditions. A false positive is when we observe something which is tied to the condition and context of a user test and would not exist in a real-world use case. For example, if we are testing with a limited number of weapons versus the full complement that would be available in the finished game. complaints about lack of weapon diversity in a test would not translate into the real-world situation. A false negative is when we miss something in a user testing situation, which would have happened in the real world. For example, testing a game’s controls only with expert players may lead to the impression that all players will be able to handle the controls.
Beware of Assumptions
Another important concept is assumptions. Assumptions are the theories you make about your sample, method, context etc. without knowing if this is actually true. For example, you might make the assumption that only male teenagers will want to play your game, and therefore only user experience test with this demographic. Assumptions could also relate to methods, e.g. assuming that a think-aloud usability test will capture any serious usability problems. A typical challenge here is the inherent assumption in much human-centred statistics, where the use of specific probability cut-off values are assumed to be evidence of actual relationships between variables. Assumptions are rarely something that is explicitly discussed in books on empirical research methods, but being unaware of your assumptions is the same as introducing errors you are not aware of.
In sum total, assumptions, generalizability, validity and bias are key concepts for Games User Research and should be kept front and center when planning user research.
Many of our colleagues told us their number one UX issue is that studies introduce biases. That’s a huge problem. Let’s fix it. Learn more about biases in UX research in our upcoming webinar. Tickets on early-bird sale until February 10.