{"id":114570,"date":"2020-06-24T17:28:39","date_gmt":"2020-06-24T17:28:39","guid":{"rendered":"http:\/\/BungieNet_ContentItem_49132"},"modified":"2020-06-24T17:28:39","modified_gmt":"2020-06-24T17:28:39","slug":"playtesting-in-the-cloud","status":"publish","type":"post","link":"https:\/\/sickgaming.net\/blog\/2020\/06\/24\/playtesting-in-the-cloud\/","title":{"rendered":"Playtesting in the Cloud"},"content":{"rendered":"
<\/div>\n

Tyler Duncan and Jeff Fox are on the front lines of what\u2019s
\nhappening with Destiny development. As members of the Bungie Test team, they help to
\norganize the testing of the game across many different groups within the
\nstudio. They\u2019re used to seeing the latest builds of the game and helping to coordinate
\ntesting \u2013 from playing through scenarios and activities to organizing mass PVP sessions
\ninvolving dozens of Bungie employees. <\/p>\n

Testing happens in every videogame studio
\nand the more testing you can do, the better. The function is so essential to a
\nstudio\u2019s day-to-day operations that it\u2019s easy to take it for granted. From
\ndesigners to developers, engineers to marketing folks, Bungie team members are
\noften invited to playtest sessions. You show up to the playtest lab, put on the
\nheadphones, play for a while, and then share your experiences with others. That
\nfundamental process of play, feedback, and communication is part of the
\nlifeblood of a game studio. <\/div>\n
\nIt’s so essential, in fact, that the alternative \u2013
\ndeveloping games without that rigorous testing \u2013 is unthinkable. So what
\nhappens when all of these time-tested and finely tuned processes get upended by
\nsomething like the COVID-19 pandemic? How do you continue to carry on such an important
\ndevelopment practice when everyone is working remotely?<\/div>\n
\nAs Fox and Duncan will tell you, you get creative. <\/div>\n
\nBefore we talk about adapting to new circumstances, let\u2019s
\npaint a picture of a typical test at Bungie. Like most game studios, Bungie has
\nplaytest labs where developers can gather on regular basis to test a build of a
\ngame. As you might expect, these labs are constantly busy, with lots of teams
\nlooking to book time to get a session in and to share feedback with each other
\nabout how a particular area of development is going. Maybe it\u2019s the scenario
\nteam who is testing out the difficulty of a particular mission, or the audio
\nteam making sure that the weapon sounds are where they want them to be. <\/div>\n
\nIn general terms, testing can be broken up between traditional
\nQA (quality assurance) testing and playtesting. Traditional testing is about analyzing
\nand validating design and engineering implementations. Are there bugs or
\nglitches that need to logged and fixed? Are things working as intended? On the
\nother hand, playtesting is about the intended experience for players. Does an
\nevent have the desired effect on the player? Does it convey the right mood? Is
\nit fun to play?<\/div>\n
\nAs playtest coordinator at Bungie, Duncan\u2019s role is focused
\non organizing playtest sessions across the studio. \u201c[It\u2019s] a very hybrid
\nrole,\u201d said Duncan, who has been with the company for almost three years. \u201cIt\u2019s
\nmore akin to a lab manager. We have to do a lot of similar to IT work, in terms
\nof troubleshooting [things] like hardware and software issues. We have the test
\nbackground, so we know how to debug through things, how we get audio up and
\nworking.\u201d There\u2019s also the
\norganizational aspects, working with teams to decide which content is going to
\nbe tested, which team members will attend the tests, and more. <\/div>\n
\nA typical busy week at Bungie HQ will see the playtest labs
\nbeing used every day, with multiple sessions per week, scheduled and set up in
\nadvance by Duncan and his team. It\u2019s no wonder then, that the playtest labs are
\nsome of the busiest parts of the studio during a normal week.<\/div>\n
\nEnter the COVID-19 crisis and the idea of \u201cnormal\u201d has been
\nthrown out the window. In late February, Bungie began a massive effort underway
\nacross all parts of the studio to gear up for an extended period of remote
\nworking. That meant everyone would be leaving the studio\u2026 including testing. No
\nmore playtest labs, no more in-person sessions to play and discuss. In short,
\nthings were changing. \u201cIt was the whole stages of grief,\u201d said
\nDuncan, when asked about his initial reaction to the news that they weren\u2019t going
\nto be allowed in the studio any longer. \u201cThere was definitely some denial at
\nfirst. My team specifically was like, \u2018Oh, we don’t need to be home. There’s a
\nlot we can do in the studio.\u2019\u201d <\/div>\n
Initially the team talked about using the
\npotential down time as an opportunity to tidy up and make some improvements
\naround the labs. <\/div>\n
\n\u201c[We thought] we could do things like
\nimprove the hardware, do a lot of the manual labor that needs to be done to get
\nre-organized,\u201d said Duncan. <\/div>\n
\nAt the same time, the team was already
\naware that a pause in playtesting wasn\u2019t going to be acceptable. \u201cWe [were]
\nthinking about how do we help teams playtest from home,\u201d he said. \u201cWe were
\ndoing an exploratory process [asking], \u2018What are our different options? What is
\nviable and not viable?\u2019\u201d<\/div>\n
\nThe complications of remote testing while
\nworking from home pile up quickly. There\u2019s the basics of making sure that
\neveryone has a powerful enough machine to play on (in the case of PC testing).
\nThen there are the inherent security risks with removing expensive development
\nkits from the studio. And with internet connections being what they are, it\u2019s
\nunreasonable to expect remote testers to use their home bandwidth to download a
\nnew build of the game remotely each time they wanted to hold a testing session.<\/div>\n
\nWith these restrictions in mind, how do
\nyou continue the rigorous testing schedule that is one of the keys to shipping
\nnew Destiny experiences on time?<\/div>\n

\nThe answer, it turns out, was in the
\ncloud.<\/p>\n

Jeff Fox has been with Bungie for seven
\nyears, working as a test lead. Alongside Duncan and others on the Test team,
\nFox has helped test a huge variety of Destiny gameplay, including activities,
\nmatchmaking, networking, and more. His most recent project was launching
\nDestiny 2 on Google\u2019s then-new Stadia streaming service.<\/div>\n
\n\u201cIt was definitely unique because we\u2019d
\nbeen historically on traditional platforms \u2013 Sony PlayStation, Xbox, PC,\u201d Fox
\nsaid. \u201cGoing into this weird new streaming platform in the cloud was a unique
\nchallenge. It was also really exciting from that perspective as well; once
\nyou\u2019ve been in QA long enough, everything is kind of very similar. So getting a
\nfresh new perspective on a different platform opens up new challenges [and]
\nit\u2019s always great to move into that.\u201d <\/div>\n
\nBy the launch of Destiny 2 on Stadia,
\nBungie developers and testers were becoming more familiar with the platform. That
\nexperience, combined with a workflow that was designed for ease of use, and it
\nwasn\u2019t long after the studio-wide \u201cwork from home\u201d orders were issued that the
\nidea of shifting a chunk of all-up testing onto the Stadia platform came up.<\/div>\n
\n\u201cUsing Stadia in the \u2018work from home\u2019
\ntransfer seemed like the easiest thing we could have done, and the fact that we
\nalready had our game stood up on that platform made it kind of a no-brainer to
\nstart looking into that,\u201d Fox said.<\/div>\n
\nWhereas a traditional test session is
\npreceded by a relatively lengthy process of propping a build onto multiple
\nconsoles or PCs in the testing lab and working through any technical snafus
\nthat may crop up, testing on Stadia was a relative breeze. <\/div>\n

\n\u201cOn
\nStadia we can publish a build in a way that all of the instances we use
\nautomatically get the build distributed to them at the same time,\u201d said Fox. \u201cWe’re
\nable to very<\/span> easily get a pool of up to 300 instances or so with the game ready
\nto play at a click of a button, which is fantastic. You can’t do that any other
\nway when we’re running a big studio playtest like that.\u201d<\/p>\n

\nTesting also places a high level of
\nimportance on uniformity of setup \u2013 in a playtesting lab, everyone is using the
\nsame equipment as much as possible. As Fox pointed out, this is even easier
\nwith Stadia. \u201cIt’s all cloud-based,\u201d he said. \u201cThere is no physical hardware in
\nstudio. You can use a variety of compatible controllers. The best thing about
\ndeveloping Stadia [is that] there’s literally no hardware at all on the desk.
\nIt’s all in the cloud so we didn’t have to worry about that at all.\u201d<\/div>\n
\nWhile playtesting with Stadia has its
\ndistinct advantages \u2013 ease of setup for both players and coordinators, hardware
\nuniformity \u2013 it\u2019s taken some work to get there. That\u2019s in part because
\ntraditional playtesting success is often the result of long-standing rituals
\nand routines. The teams schedule a test session, the coordinators work to get
\nthe lab set up with the correct build, and everyone knows the process of where
\nand when they need to be at the lab. It\u2019s a scripted routine that is the result
\nof a lot of learning over time. <\/div>\n
\nMoving to a new system requires new
\ncommunication paths, and new rituals to form, not to mention the very real
\nchanges that have come when an entire studio is learning to function remotely.
\nIt\u2019s something that Bungie\u2019s Test team is still focusing on as the weeks go on.\n<\/div>\n
\n\u201cIn terms of stabilizing and having a
\nnormal day to day, we’ve just about got there,\u201d Duncan said. \u201c[The] first thing
\nwas, let’s get teams playtesting again. Then we started to transition to
\nwrapping our old team rituals or processes back into the Stadia thing.\u201d<\/div>\n
\nWhile there\u2019s still more work to go, the
\nteam is feeling good about the progress made so far and the potential this kind
\nof cloud-based testing has for Bungie. That\u2019s in no small part due to Bungie\u2019s
\nwillingness to adapt, said Fox.<\/div>\n
\n\u201cI was surprised at how quickly people
\nwere able to pick up on the process,\u201d Fox said. \u201cGenerally when you say, \u2018Oh
\nyeah, we’ll stream it over the network. It will be fine!\u2019 people are [going to
\nbe] pretty skeptical. But the overall feedback has been really good. This is
\nworking for us now during work from home.\u201d<\/div>\n
\nDuncan is even more bullish on the
\nfuture.<\/div>\n

\n\u201cThis is going to change everything,\u201d said Duncan. \u201cWe are in a new tradition. Just because we used to have every team
\nplaytest in labs [in the past], it doesn’t mean we stop doing that. But the
\nprocess is evolving and changing. And we need to continue to be flexible.

<\/span><\/p>\n

\u201cThe future is now. What we thought was
\nimpossible is definitely not the case.\u201d<\/span><\/p>\n","protected":false},"excerpt":{"rendered":"

Tyler Duncan and Jeff Fox are on the front lines of what\u2019s happening with Destiny development. As members of the Bungie Test team, they help to organize the testing of the game across many different groups within the studio. They\u2019re used to seeing the latest builds of the game and helping to coordinate testing \u2013 […]<\/p>\n","protected":false},"author":2,"featured_media":114571,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[34],"tags":[],"class_list":["post-114570","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-bungie-news"],"_links":{"self":[{"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/posts\/114570","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/comments?post=114570"}],"version-history":[{"count":0,"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/posts\/114570\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/media\/114571"}],"wp:attachment":[{"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/media?parent=114570"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/categories?post=114570"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/tags?post=114570"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}