Learning and Improving

Rose Lazuli

Last devlog, I talked about how I created a playtest protocol for my team to test a few metrics. Unfortunately, the actual playtest itself didn't live up to the vision, in no small part due to errors I made over the course of developing the protocol and the survey for the test. With a chance in front of me to improve, I wanted to walk down the steps I took to make sure the upcoming playtest will work better.

The Problems

The playtest had a number of issues, which can be categorized into three particular categories:

1. Lack of communication
2. Lack of relevance
3. Lack of data

Lack of communication

While I did create an in-depth playtesting protocol document, this contained a great deal of information, only some of which was necessary to moderators during tests. To try and help offset the issue, I created a "cheat sheet" of sorts, which streamlined information to Setup, Moderator Script, Observation Prompts, and Communication Slips. Each of these was meant to be a short refresher of what to do, and not a comprehensive instruction set. However, two problems occurred with this strategy: moderators used the cheat sheet only, as that was the primary document given to them, a major error on my behalf, and additionally, the document was not clearly communicated to some moderators, leading to issues with what steps were followed correctly.

Lack of relevance

The second major issue was with the survey. While goals had been clearly defined as part of the protocol, the survey undergoing revisions to the last possible moment. As such, some survey questions were not relevant to the goals outlined in the playtest, and distracted from other opportunities the team could have benefitted from. This did overall result in some data the team received not being useful during the analysis for the playtest report.

Lack of data

As a direct result of the two aforementioned issues, the team was not briefed clearly on expectations for the playtest. This was a direct result of my actions and failure to consider certain areas of the playtest. This resulted in data loss, as two moderator observation documents were not submitted, as well as two playtest surveys and four playtest videos. As such, with limited data, only cursory analysis and broad conclusions could be drawn, limiting the efficacy of the playtest overall.

The Lessons

From the outcome of playtest one and the observations of the flaws within it, I learned several key lessons that I used in the preparation of the second playtest:

• The document made available to the team needs to contain exactly as much information as the team needs to execute the playtest fully.
• The document should be placed in a singular ZIP-file package alongside the playtest build, and sent to all teammates with guidance to use the document directly.
• The survey's questions should all relate directly to at least one of the playtest's goals.
• Survey questions should try to gather qualitative data as best as possible by clearly communicating the expectations to testers in the survey.
• Tester demographics data should be collected in the survey and not in moderator notes.
• Moderators should be provided clear expectations as to what quality of submission is required of them.

While a lengthy list, these lessons were all taken into account when making the new playtest protocol.

The Solutions

Lack of Communication

To begin, I reformatted the protocol to remove the goals from being outlined. While documented in other formats, this is not necessary to the completion of the playtest. Instead the document makes clear it should be read and understood, then transitions directly into setup steps to make clear to moderators where to begin:

-----------------------------------------------------------------------------------------------------------------------------------------------------

-----------------------------------------------------------------------------------------------------------------------------------------------------

Additionally, clear expectations were set around note-taking, expanding the observation prompts from the prior protocol and outlining the why of what should be recorded, not just the what, while also providing a quick reference list for mid-test.

-----------------------------------------------------------------------------------------------------------------------------------------------------

-----------------------------------------------------------------------------------------------------------------------------------------------------

Lack of Relevance

To9 assist in making sure every question was related to the playtest goals, temporary headers were included while making the survey. Each header outlines what the goal is, and each question underneath relates to the goal. This should help ensure specific answers relevant to the playtest can be obtained.

-----------------------------------------------------------------------------------------------------------------------------------------------------

-----------------------------------------------------------------------------------------------------------------------------------------------------

Lack of data

To help make sure relevant data is gathered and available for analysis, the survey now includes self-reported demographics data to help identify key audiences and their reactions to specific elements of the test, instead of relying on moderator notes to contain this information.

-----------------------------------------------------------------------------------------------------------------------------------------------------

-----------------------------------------------------------------------------------------------------------------------------------------------------

Finally, clear step instructions were provided to ensure testers knew exactly what data to submit, as well and the location to put them to make them accessible. This should help reduce data loss and improve the outcome of the playtest overall.

-----------------------------------------------------------------------------------------------------------------------------------------------------