06-25-2020, 08:10 PM
Playtesting in the Cloud
<div style="margin: 5px 5% 10px 5%;"><img src="https://www.sickgaming.net/blog/wp-content/uploads/2020/06/playtesting-in-the-cloud.jpg" width="1920" height="590" title="" alt="" /></div><div><div><img src="https://www.bungie.net/pubassets/pkgs/137/137780/Stadia_Playtesting_Blog_1920x590.jpg" class="ff-og-image-inserted"></div>
<p> Tyler Duncan and Jeff Fox are on the front lines of what’s<br />
happening with Destiny development. As members of the Bungie Test team, they help to<br />
organize the testing of the game across many different groups within the<br />
studio. They’re used to seeing the latest builds of the game and helping to coordinate<br />
testing – from playing through scenarios and activities to organizing mass PVP sessions<br />
involving dozens of Bungie employees. </p>
<div>Testing happens in every videogame studio<br />
and the more testing you can do, the better. The function is so essential to a<br />
studio’s day-to-day operations that it’s easy to take it for granted. From<br />
designers to developers, engineers to marketing folks, Bungie team members are<br />
often invited to playtest sessions. You show up to the playtest lab, put on the<br />
headphones, play for a while, and then share your experiences with others. That<br />
fundamental process of play, feedback, and communication is part of the<br />
lifeblood of a game studio. </div>
<div>
It’s so essential, in fact, that the alternative –<br />
developing games without that rigorous testing – is unthinkable. So what<br />
happens when all of these time-tested and finely tuned processes get upended by<br />
something like the COVID-19 pandemic? How do you continue to carry on such an important<br />
development practice when everyone is working remotely?</div>
<div>
As Fox and Duncan will tell you, you get creative. </div>
<div>
Before we talk about adapting to new circumstances, let’s<br />
paint a picture of a typical test at Bungie. Like most game studios, Bungie has<br />
playtest labs where developers can gather on regular basis to test a build of a<br />
game. As you might expect, these labs are constantly busy, with lots of teams<br />
looking to book time to get a session in and to share feedback with each other<br />
about how a particular area of development is going. Maybe it’s the scenario<br />
team who is testing out the difficulty of a particular mission, or the audio<br />
team making sure that the weapon sounds are where they want them to be. </div>
<div>
In general terms, testing can be broken up between traditional<br />
QA (quality assurance) testing and playtesting. Traditional testing is about analyzing<br />
and validating design and engineering implementations. Are there bugs or<br />
glitches that need to logged and fixed? Are things working as intended? On the<br />
other hand, playtesting is about the intended experience for players. Does an<br />
event have the desired effect on the player? Does it convey the right mood? Is<br />
it fun to play?</div>
<div>
As playtest coordinator at Bungie, Duncan’s role is focused<br />
on organizing playtest sessions across the studio. “[It’s] a very hybrid<br />
role,” said Duncan, who has been with the company for almost three years. “It’s<br />
more akin to a lab manager. We have to do a lot of similar to IT work, in terms<br />
of troubleshooting [things] like hardware and software issues. We have the test<br />
background, so we know how to debug through things, how we get audio up and<br />
working.” There’s also the<br />
organizational aspects, working with teams to decide which content is going to<br />
be tested, which team members will attend the tests, and more. </div>
<div>
A typical busy week at Bungie HQ will see the playtest labs<br />
being used every day, with multiple sessions per week, scheduled and set up in<br />
advance by Duncan and his team. It’s no wonder then, that the playtest labs are<br />
some of the busiest parts of the studio during a normal week.</div>
<div>
Enter the COVID-19 crisis and the idea of “normal” has been<br />
thrown out the window. In late February, Bungie began a massive effort underway<br />
across all parts of the studio to gear up for an extended period of remote<br />
working. That meant everyone would be leaving the studio… including testing. No<br />
more playtest labs, no more in-person sessions to play and discuss. In short,<br />
things were changing. “It was the whole stages of grief,” said<br />
Duncan, when asked about his initial reaction to the news that they weren’t going<br />
to be allowed in the studio any longer. “There was definitely some denial at<br />
first. My team specifically was like, ‘Oh, we don’t need to be home. There’s a<br />
lot we can do in the studio.’” </div>
<div>Initially the team talked about using the<br />
potential down time as an opportunity to tidy up and make some improvements<br />
around the labs. </div>
<div>
“[We thought] we could do things like<br />
improve the hardware, do a lot of the manual labor that needs to be done to get<br />
re-organized,” said Duncan. </div>
<div>
At the same time, the team was already<br />
aware that a pause in playtesting wasn’t going to be acceptable. “We [were]<br />
thinking about how do we help teams playtest from home,” he said. “We were<br />
doing an exploratory process [asking], ‘What are our different options? What is<br />
viable and not viable?’”</div>
<div>
The complications of remote testing while<br />
working from home pile up quickly. There’s the basics of making sure that<br />
everyone has a powerful enough machine to play on (in the case of PC testing).<br />
Then there are the inherent security risks with removing expensive development<br />
kits from the studio. And with internet connections being what they are, it’s<br />
unreasonable to expect remote testers to use their home bandwidth to download a<br />
new build of the game remotely each time they wanted to hold a testing session.</div>
<div>
With these restrictions in mind, how do<br />
you continue the rigorous testing schedule that is one of the keys to shipping<br />
new Destiny experiences on time?</div>
<p>
The answer, it turns out, was in the<br />
cloud.</p>
<div>Jeff Fox has been with Bungie for seven<br />
years, working as a test lead. Alongside Duncan and others on the Test team,<br />
Fox has helped test a huge variety of Destiny gameplay, including activities,<br />
matchmaking, networking, and more. His most recent project was launching<br />
Destiny 2 on Google’s then-new Stadia streaming service.</div>
<div>
“It was definitely unique because we’d<br />
been historically on traditional platforms – Sony PlayStation, Xbox, PC,” Fox<br />
said. “Going into this weird new streaming platform in the cloud was a unique<br />
challenge. It was also really exciting from that perspective as well; once<br />
you’ve been in QA long enough, everything is kind of very similar. So getting a<br />
fresh new perspective on a different platform opens up new challenges [and]<br />
it’s always great to move into that.” </div>
<div>
By the launch of Destiny 2 on Stadia,<br />
Bungie developers and testers were becoming more familiar with the platform. That<br />
experience, combined with a workflow that was designed for ease of use, and it<br />
wasn’t long after the studio-wide “work from home” orders were issued that the<br />
idea of shifting a chunk of all-up testing onto the Stadia platform came up.</div>
<div>
“Using Stadia in the ‘work from home’<br />
transfer seemed like the easiest thing we could have done, and the fact that we<br />
already had our game stood up on that platform made it kind of a no-brainer to<br />
start looking into that,” Fox said.</div>
<div>
Whereas a traditional test session is<br />
preceded by a relatively lengthy process of propping a build onto multiple<br />
consoles or PCs in the testing lab and working through any technical snafus<br />
that may crop up, testing on Stadia was a relative breeze. </div>
<p>
“<span>On<br />
Stadia we can publish a build in a way that all of the instances we use<br />
automatically get the build distributed to them at the same time,” said Fox. “We’re<br />
able to very</span> easily get a pool of up to 300 instances or so with the game ready<br />
to play at a click of a button, which is fantastic. You can’t do that any other<br />
way when we’re running a big studio playtest like that.”</p>
<div>
Testing also places a high level of<br />
importance on uniformity of setup – in a playtesting lab, everyone is using the<br />
same equipment as much as possible. As Fox pointed out, this is even easier<br />
with Stadia. “It’s all cloud-based,” he said. “There is no physical hardware in<br />
studio. You can use a variety of compatible controllers. The best thing about<br />
developing Stadia [is that] there’s literally no hardware at all on the desk.<br />
It’s all in the cloud so we didn’t have to worry about that at all.”</div>
<div>
While playtesting with Stadia has its<br />
distinct advantages – ease of setup for both players and coordinators, hardware<br />
uniformity – it’s taken some work to get there. That’s in part because<br />
traditional playtesting success is often the result of long-standing rituals<br />
and routines. The teams schedule a test session, the coordinators work to get<br />
the lab set up with the correct build, and everyone knows the process of where<br />
and when they need to be at the lab. It’s a scripted routine that is the result<br />
of a lot of learning over time. </div>
<div>
Moving to a new system requires new<br />
communication paths, and new rituals to form, not to mention the very real<br />
changes that have come when an entire studio is learning to function remotely.<br />
It’s something that Bungie’s Test team is still focusing on as the weeks go on.
</div>
<div>
“In terms of stabilizing and having a<br />
normal day to day, we’ve just about got there,” Duncan said. “[The] first thing<br />
was, let’s get teams playtesting again. Then we started to transition to<br />
wrapping our old team rituals or processes back into the Stadia thing.”</div>
<div>
While there’s still more work to go, the<br />
team is feeling good about the progress made so far and the potential this kind<br />
of cloud-based testing has for Bungie. That’s in no small part due to Bungie’s<br />
willingness to adapt, said Fox.</div>
<div>
“I was surprised at how quickly people<br />
were able to pick up on the process,” Fox said. “Generally when you say, ‘Oh<br />
yeah, we’ll stream it over the network. It will be fine!’ people are [going to<br />
be] pretty skeptical. But the overall feedback has been really good. This is<br />
working for us now during work from home.”</div>
<div>
Duncan is even more bullish on the<br />
future.</div>
<p>
“This is going to change everything,” said Duncan. “We are in a new tradition. Just because we used to have every team<br />
playtest in labs [in the past], it doesn’t mean we stop doing that. But the<br />
process is evolving and changing. And we need to continue to be flexible.<br /><span><br /></span></p>
<p><span>“The future is now. What we thought was<br />
impossible is definitely not the case.”</span></p>
</div>
https://www.sickgaming.net/blog/2020/06/...the-cloud/
<div style="margin: 5px 5% 10px 5%;"><img src="https://www.sickgaming.net/blog/wp-content/uploads/2020/06/playtesting-in-the-cloud.jpg" width="1920" height="590" title="" alt="" /></div><div><div><img src="https://www.bungie.net/pubassets/pkgs/137/137780/Stadia_Playtesting_Blog_1920x590.jpg" class="ff-og-image-inserted"></div>
<p> Tyler Duncan and Jeff Fox are on the front lines of what’s<br />
happening with Destiny development. As members of the Bungie Test team, they help to<br />
organize the testing of the game across many different groups within the<br />
studio. They’re used to seeing the latest builds of the game and helping to coordinate<br />
testing – from playing through scenarios and activities to organizing mass PVP sessions<br />
involving dozens of Bungie employees. </p>
<div>Testing happens in every videogame studio<br />
and the more testing you can do, the better. The function is so essential to a<br />
studio’s day-to-day operations that it’s easy to take it for granted. From<br />
designers to developers, engineers to marketing folks, Bungie team members are<br />
often invited to playtest sessions. You show up to the playtest lab, put on the<br />
headphones, play for a while, and then share your experiences with others. That<br />
fundamental process of play, feedback, and communication is part of the<br />
lifeblood of a game studio. </div>
<div>
It’s so essential, in fact, that the alternative –<br />
developing games without that rigorous testing – is unthinkable. So what<br />
happens when all of these time-tested and finely tuned processes get upended by<br />
something like the COVID-19 pandemic? How do you continue to carry on such an important<br />
development practice when everyone is working remotely?</div>
<div>
As Fox and Duncan will tell you, you get creative. </div>
<div>
Before we talk about adapting to new circumstances, let’s<br />
paint a picture of a typical test at Bungie. Like most game studios, Bungie has<br />
playtest labs where developers can gather on regular basis to test a build of a<br />
game. As you might expect, these labs are constantly busy, with lots of teams<br />
looking to book time to get a session in and to share feedback with each other<br />
about how a particular area of development is going. Maybe it’s the scenario<br />
team who is testing out the difficulty of a particular mission, or the audio<br />
team making sure that the weapon sounds are where they want them to be. </div>
<div>
In general terms, testing can be broken up between traditional<br />
QA (quality assurance) testing and playtesting. Traditional testing is about analyzing<br />
and validating design and engineering implementations. Are there bugs or<br />
glitches that need to logged and fixed? Are things working as intended? On the<br />
other hand, playtesting is about the intended experience for players. Does an<br />
event have the desired effect on the player? Does it convey the right mood? Is<br />
it fun to play?</div>
<div>
As playtest coordinator at Bungie, Duncan’s role is focused<br />
on organizing playtest sessions across the studio. “[It’s] a very hybrid<br />
role,” said Duncan, who has been with the company for almost three years. “It’s<br />
more akin to a lab manager. We have to do a lot of similar to IT work, in terms<br />
of troubleshooting [things] like hardware and software issues. We have the test<br />
background, so we know how to debug through things, how we get audio up and<br />
working.” There’s also the<br />
organizational aspects, working with teams to decide which content is going to<br />
be tested, which team members will attend the tests, and more. </div>
<div>
A typical busy week at Bungie HQ will see the playtest labs<br />
being used every day, with multiple sessions per week, scheduled and set up in<br />
advance by Duncan and his team. It’s no wonder then, that the playtest labs are<br />
some of the busiest parts of the studio during a normal week.</div>
<div>
Enter the COVID-19 crisis and the idea of “normal” has been<br />
thrown out the window. In late February, Bungie began a massive effort underway<br />
across all parts of the studio to gear up for an extended period of remote<br />
working. That meant everyone would be leaving the studio… including testing. No<br />
more playtest labs, no more in-person sessions to play and discuss. In short,<br />
things were changing. “It was the whole stages of grief,” said<br />
Duncan, when asked about his initial reaction to the news that they weren’t going<br />
to be allowed in the studio any longer. “There was definitely some denial at<br />
first. My team specifically was like, ‘Oh, we don’t need to be home. There’s a<br />
lot we can do in the studio.’” </div>
<div>Initially the team talked about using the<br />
potential down time as an opportunity to tidy up and make some improvements<br />
around the labs. </div>
<div>
“[We thought] we could do things like<br />
improve the hardware, do a lot of the manual labor that needs to be done to get<br />
re-organized,” said Duncan. </div>
<div>
At the same time, the team was already<br />
aware that a pause in playtesting wasn’t going to be acceptable. “We [were]<br />
thinking about how do we help teams playtest from home,” he said. “We were<br />
doing an exploratory process [asking], ‘What are our different options? What is<br />
viable and not viable?’”</div>
<div>
The complications of remote testing while<br />
working from home pile up quickly. There’s the basics of making sure that<br />
everyone has a powerful enough machine to play on (in the case of PC testing).<br />
Then there are the inherent security risks with removing expensive development<br />
kits from the studio. And with internet connections being what they are, it’s<br />
unreasonable to expect remote testers to use their home bandwidth to download a<br />
new build of the game remotely each time they wanted to hold a testing session.</div>
<div>
With these restrictions in mind, how do<br />
you continue the rigorous testing schedule that is one of the keys to shipping<br />
new Destiny experiences on time?</div>
<p>
The answer, it turns out, was in the<br />
cloud.</p>
<div>Jeff Fox has been with Bungie for seven<br />
years, working as a test lead. Alongside Duncan and others on the Test team,<br />
Fox has helped test a huge variety of Destiny gameplay, including activities,<br />
matchmaking, networking, and more. His most recent project was launching<br />
Destiny 2 on Google’s then-new Stadia streaming service.</div>
<div>
“It was definitely unique because we’d<br />
been historically on traditional platforms – Sony PlayStation, Xbox, PC,” Fox<br />
said. “Going into this weird new streaming platform in the cloud was a unique<br />
challenge. It was also really exciting from that perspective as well; once<br />
you’ve been in QA long enough, everything is kind of very similar. So getting a<br />
fresh new perspective on a different platform opens up new challenges [and]<br />
it’s always great to move into that.” </div>
<div>
By the launch of Destiny 2 on Stadia,<br />
Bungie developers and testers were becoming more familiar with the platform. That<br />
experience, combined with a workflow that was designed for ease of use, and it<br />
wasn’t long after the studio-wide “work from home” orders were issued that the<br />
idea of shifting a chunk of all-up testing onto the Stadia platform came up.</div>
<div>
“Using Stadia in the ‘work from home’<br />
transfer seemed like the easiest thing we could have done, and the fact that we<br />
already had our game stood up on that platform made it kind of a no-brainer to<br />
start looking into that,” Fox said.</div>
<div>
Whereas a traditional test session is<br />
preceded by a relatively lengthy process of propping a build onto multiple<br />
consoles or PCs in the testing lab and working through any technical snafus<br />
that may crop up, testing on Stadia was a relative breeze. </div>
<p>
“<span>On<br />
Stadia we can publish a build in a way that all of the instances we use<br />
automatically get the build distributed to them at the same time,” said Fox. “We’re<br />
able to very</span> easily get a pool of up to 300 instances or so with the game ready<br />
to play at a click of a button, which is fantastic. You can’t do that any other<br />
way when we’re running a big studio playtest like that.”</p>
<div>
Testing also places a high level of<br />
importance on uniformity of setup – in a playtesting lab, everyone is using the<br />
same equipment as much as possible. As Fox pointed out, this is even easier<br />
with Stadia. “It’s all cloud-based,” he said. “There is no physical hardware in<br />
studio. You can use a variety of compatible controllers. The best thing about<br />
developing Stadia [is that] there’s literally no hardware at all on the desk.<br />
It’s all in the cloud so we didn’t have to worry about that at all.”</div>
<div>
While playtesting with Stadia has its<br />
distinct advantages – ease of setup for both players and coordinators, hardware<br />
uniformity – it’s taken some work to get there. That’s in part because<br />
traditional playtesting success is often the result of long-standing rituals<br />
and routines. The teams schedule a test session, the coordinators work to get<br />
the lab set up with the correct build, and everyone knows the process of where<br />
and when they need to be at the lab. It’s a scripted routine that is the result<br />
of a lot of learning over time. </div>
<div>
Moving to a new system requires new<br />
communication paths, and new rituals to form, not to mention the very real<br />
changes that have come when an entire studio is learning to function remotely.<br />
It’s something that Bungie’s Test team is still focusing on as the weeks go on.
</div>
<div>
“In terms of stabilizing and having a<br />
normal day to day, we’ve just about got there,” Duncan said. “[The] first thing<br />
was, let’s get teams playtesting again. Then we started to transition to<br />
wrapping our old team rituals or processes back into the Stadia thing.”</div>
<div>
While there’s still more work to go, the<br />
team is feeling good about the progress made so far and the potential this kind<br />
of cloud-based testing has for Bungie. That’s in no small part due to Bungie’s<br />
willingness to adapt, said Fox.</div>
<div>
“I was surprised at how quickly people<br />
were able to pick up on the process,” Fox said. “Generally when you say, ‘Oh<br />
yeah, we’ll stream it over the network. It will be fine!’ people are [going to<br />
be] pretty skeptical. But the overall feedback has been really good. This is<br />
working for us now during work from home.”</div>
<div>
Duncan is even more bullish on the<br />
future.</div>
<p>
“This is going to change everything,” said Duncan. “We are in a new tradition. Just because we used to have every team<br />
playtest in labs [in the past], it doesn’t mean we stop doing that. But the<br />
process is evolving and changing. And we need to continue to be flexible.<br /><span><br /></span></p>
<p><span>“The future is now. What we thought was<br />
impossible is definitely not the case.”</span></p>
</div>
https://www.sickgaming.net/blog/2020/06/...the-cloud/