{"id":110245,"date":"2020-03-13T07:12:00","date_gmt":"2020-03-13T07:12:00","guid":{"rendered":"http:\/\/www.gamasutra.com\/view\/news\/359240"},"modified":"2020-03-13T07:12:00","modified_gmt":"2020-03-13T07:12:00","slug":"a-look-at-how-different-sized-studios-approach-the-challenges-of-qa","status":"publish","type":"post","link":"https:\/\/sickgaming.net\/blog\/2020\/03\/13\/a-look-at-how-different-sized-studios-approach-the-challenges-of-qa\/","title":{"rendered":"A look at how different-sized studios approach the challenges of QA"},"content":{"rendered":"

Quality assurance is an often underappreciated and misunderstood part of the game development process. Depending on the studio, team sizes can vary from a single person to a whole team of individuals, responsible for testing, tracking, and communicating any bugs they may find.<\/span><\/p>\n

Because of this, the workflow can sometimes be markedly different between studios, with teams preferring certain software or processes over others. <\/span><\/p>\n

Lesleyann White is the principal QA specialist at Failbetter Games<\/a>,<\/span> a studio specializing in interactive fiction games like Sunless Sea<\/em><\/a> and Sunless Skies<\/em><\/a>. Before that, she worked on Runescape<\/em> at Jagex where she worked alongside a QA department of roughly 30 people. But at Failbetter she is the only person responsible for quality assurance.<\/p>\n

\u201cThe hierarchy is so different,\u201d explains White. \u201cI remember before I arrived, Failbetter didn\u2019t have a QA department, so it is kind of an all-encompassing role, working across all the projects, doing all of the work, all of the planning, whereas in a larger company \u2013 for example, when I worked at Jagex\u2026we had junior testers who would be able to be promoted up to a tester, a senior, a lead. Then we had the QA manager who was responsible for the whole of the QA department, all the personnel side as well, such as people management<\/span> <\/span><\/p>\n

Failbetter Games’<\/em> Sunless Skies<\/center> <\/p>\n

Team17<\/a> <\/span>is a Wakefield-based developer and publisher. The QA department is responsible for polishing their own games, such as Worms W.M.D.<\/em><\/a>, but also helping out smaller teams QA test their own projects as well, including hit games like Overcooked<\/em><\/a> and Overcooked 2<\/em><\/a>. The QA department at the publisher is currently 43 people strong, with a range of roles within that team. <\/p>\n

Chloe Crookes is a senior QA analyst at the studio. During a recent talk at Yorkshire Games Festival, she explained her role as follows: \u201cThe difference between standard QA and senior, in my experience, is generally senior QA have their own team of testers and will oversee their work as well as completing their own. You might also complete test plans and organize test cases.\u201d Crookes’ current project is <\/span>Moving Out<\/em><\/a>, a new co-operative game developed by Devm Games and SMG Studio, and published by Team17.<\/p>\n

<\/p>\n

SMG Studio and Devm Games’<\/em> Moving Out<\/center> <\/p>\n

Cambridge-based studio Frontier Developments<\/a><\/span> is another larger development studio, which primarily develops simulation games such as Jurassic World Evolution<\/em><\/a> and Planet Zoo<\/em><\/a>.<\/p>\n

Much like Team17, they also have a range of different roles within their QA department. Colin Davis, the head of QA at the studio, explains the advantage of this: \u201cOur size allows us to offer a large variety of testing. We have a number of specialized teams working alongside the more typical functionality test approach. Our Functionality testers are responsible for testing all gameplay and features that form the game they are assigned to, while our specialized teams are focused on whichever game requires their particular type of testing at that given time.\u201d<\/span><\/p>\n

As well as different roles, there are also a number of different tools available to those in QA and these can differ depending on the studio and their own internal workflow.<\/span><\/p>\n

\u201cWe pretty much perform all types of testing at Failbetter,\u201d White explains. \u201cSystem, integration, smoke, regression, performance, compatibility, compliance, etc. It\u2019s easier to say what types of testing we don\u2019t do at Failbetter: localization and unit testing.\u201d White explains that the former is due to the large word count within Failbetter\u2019s games that would make the process costly and inefficient, while the latter is something that she would like to introduce in the future. <\/span><\/p>\n

For both bug reporting and project management, White uses Jira<\/span> as her preferred tool, or has done for the last year or so. As a system administrator, she enjoys how configurable it can be and how flexible it is for both smaller and larger projects — an opinion that\u2019s supported by Frontier also adopting the same issue tracking software at times for their own needs. <\/p>\n

As for other tools, White uses software such as Unity for testing and bug fixing and various other bespoke testing tools, including macros like AutoHotkey<\/a><\/span> and Failbetter\u2019s in-house CMS: Storynexus. That\u2019s in addition to Confluence or Google Docs for writing technical documents. Given the small size of the studio the team like to keep things simple and cost-effective to avoid creating more issues for themselves or overcomplicating their workflow.<\/p>\n

Team17, meanwhile uses Excel for test plans and Redmine<\/a> <\/span>as their database system of choice. Redmine allows them to give a summary of each issue they encounter, a description of that issue, and enter custom fields to elaborate on which format or platform they encountered the problem. They can also add any relevant screenshots or videos to give developers more information to go on. <\/p>\n

\u201cGenerally, we\u2019ll always attempt to reproduce issues as many times as we can to get a solid reproduction for it before we give it to the developers to fix,\u201d Crookes elaborates in our interview following her talk. \u201cMore often than not if a bug is happening, it will be obvious why it is happening so usually that\u2019s just a very easy thing to do\u2026But sometimes you have edge case issues where you\u2019re like I\u2019m not quite sure why this is happening…In times like that, we just have to provide the developers with as much as we can give them and then kind of see if we encounter it again.\u201d <\/span><\/p>\n

<\/span><\/p>\n

Frontier Developments’<\/em> Planet Zoo<\/center> <\/p>\n

According to Davis, Frontier has gone a step further to create their own bespoke tool that can interface with several bug databases. This way Frontier developers can template much of their bug writing, making everything more efficient and compatible overall. On projects, they apply a variety of testing based on the requirements of a particular game. This includes everything from certification to compatibility to localization, user experience, destructive, and automation.<\/span><\/p>\n

Davis explains, \u201cAutomation is an important part of what we do, as it allows us to automate some of the more repetitive work like placing every single asset down in a game like <\/span>Planet Coaster<\/em> and have the machine check that they can be placed in the game. This frees us up to look a little more closely at what the game experience is like.<\/p>\n

\u201cIn addition to this, we will also use automated testing to create a range of tests to help us with each game we work on. These tests can range from simple things like ensuring you can start the game on any given platform on a newly compiled build, to more advanced things like flying a ship around in <\/span>Elite<\/em>, landing at various stations and then working out what the best trade route might be from the information presented. This type of automation is closer to what a player might be doing in the game and can aid in catching issues towards the more experiential side.\u201d<\/p>\n

Both White and Crookes also elaborated on the importance of \u2018regression testing\u2019 or \u2018halo testing\u2019 as it is sometimes called. That is, looking around a fix to make sure a change hasn\u2019t impacted the build elsewhere. <\/span><\/p>\n

\u201cWhat we would do is directly test the issue at hand and then, for example, if they\u2019ve changed the way an achievement unlocks you would then shift your focus to a broader look at that achievement,\u201d Crookes states. \u201cSo, for example, if you need to complete a certain level to unlock it, maybe you test failing the level and make sure that doesn\u2019t unlock it. So, it is kind of blind testing around it, but you also have to pull things from past experience, so if you know something similar has happened in another game you can test around that.\u201d<\/span><\/p>\n

One thing the teams all attest to is the value of putting a game in front of a players. This is especially true in relation to the issues of accessibility and balancing.<\/span><\/p>\n

White explains, \u201cWith regards to accessibility testing, it\u2019s important for QA teams to perform it, but I feel there\u2019s no substitute for putting the product in front of the target audience. Having specific test groups evaluate a feature often raises aspects about the design\u2019s accessibility that you hadn\u2019t considered. It\u2019s one of the reasons we put our games in Early Access. It allows us to get the game in front of players with a wide range of accessibility concerns and gather their feedback.\u201d<\/span><\/p>\n

Recently, for instance, Failbetter ran a new beta for an interactive map (embedded below) for <\/span>Fallen London<\/a>.<\/em> As part of this process, they reached out to those within their community who use a screen reader to hear their input about this new addition.  <\/p>\n

<\/p>\n

Team17 meanwhile perform regular usability tests<\/a><\/span>, where they invite members of the public to get hands-on with their games and test how they are balanced. This allows them to get feedback from individuals unfamiliar with the game being played, which has proven an invaluable asset.<\/p>\n

\u201cWe get people from the general public to come and test the games and provide feedback,\u201d says Crookes. \u201cI think it\u2019s really useful for us, because the QA teams can be on projects for like sixteen months and sometimes because we\u2019re like so good at the game sometimes we can make it a bit difficult as well. And it\u2019s hard for us to take a step back and go \u2018Right, this doesn\u2019t actually play very well.\u2019\u201d<\/span><\/p>\n

For those thinking of getting into QA, I asked what steps people can take to make themselves stand out and what studios are looking for in terms of a desirable applicant for a QA position. This is what they had to say:<\/span><\/p>\n

\u201cWhen it comes to starting out, try not to treat QA as a foot in the door for the games industry<\/strong>,\u201d argues Davis. \u201cQuality Assurance is an integral part of the development process, it\u2019s a great career choice for anyone who\u2019s interested in games and wants to make a difference.\u201d He gives some examples of how to develop your skills. \u201cIf you\u2019ve ever tried speed running a game, try enhancing these skills without relying on information you find online. The skills and abilities you pick up from doing this can be extremely useful in QA!\u201d<\/span><\/p>\n

\u201cThe best thing is to sell the fact you want to be in the industry, not the fact that you want to play games,\u201d answers Crookes. \u201cPeople will apply thinking it\u2019s just playing games all day and they won\u2019t last very long, because it is quite a tough job when you have submissions coming up and deadlines<\/strong>. I think being able to prove that you want to be there because you have a passion for the industry is probably the main thing. If you\u2019ve made games at home, show them, and say, \u2018Hey, look at all this effort I\u2019ve been putting in. This is where I want to be. This is what I want to do\u2019. I think you just really need to be able to sell it.\u201d<\/span><\/p>\n","protected":false},"excerpt":{"rendered":"

Quality assurance is an often underappreciated and misunderstood part of the game development process. Depending on the studio, team sizes can vary from a single person to a whole team of individuals, responsible for testing, tracking, and communicating any bugs they may find. Because of this, the workflow can sometimes be markedly different between studios, […]<\/p>\n","protected":false},"author":2,"featured_media":110246,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[20],"tags":[],"class_list":["post-110245","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-news"],"_links":{"self":[{"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/posts\/110245","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/comments?post=110245"}],"version-history":[{"count":0,"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/posts\/110245\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/media\/110246"}],"wp:attachment":[{"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/media?parent=110245"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/categories?post=110245"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/tags?post=110245"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}