06-25-2020, 08:10 PM
Playtesting in the Cloud
![[Image: Stadia_Playtesting_Blog_1920x590.jpg]](https://www.bungie.net/pubassets/pkgs/137/137780/Stadia_Playtesting_Blog_1920x590.jpg)
Tyler Duncan and Jeff Fox are on the front lines of what’s
happening with Destiny development. As members of the Bungie Test team, they help to
organize the testing of the game across many different groups within the
studio. They’re used to seeing the latest builds of the game and helping to coordinate
testing – from playing through scenarios and activities to organizing mass PVP sessions
involving dozens of Bungie employees.
Testing happens in every videogame studio
and the more testing you can do, the better. The function is so essential to a
studio’s day-to-day operations that it’s easy to take it for granted. From
designers to developers, engineers to marketing folks, Bungie team members are
often invited to playtest sessions. You show up to the playtest lab, put on the
headphones, play for a while, and then share your experiences with others. That
fundamental process of play, feedback, and communication is part of the
lifeblood of a game studio.
It’s so essential, in fact, that the alternative –
developing games without that rigorous testing – is unthinkable. So what
happens when all of these time-tested and finely tuned processes get upended by
something like the COVID-19 pandemic? How do you continue to carry on such an important
development practice when everyone is working remotely?
As Fox and Duncan will tell you, you get creative.
Before we talk about adapting to new circumstances, let’s
paint a picture of a typical test at Bungie. Like most game studios, Bungie has
playtest labs where developers can gather on regular basis to test a build of a
game. As you might expect, these labs are constantly busy, with lots of teams
looking to book time to get a session in and to share feedback with each other
about how a particular area of development is going. Maybe it’s the scenario
team who is testing out the difficulty of a particular mission, or the audio
team making sure that the weapon sounds are where they want them to be.
In general terms, testing can be broken up between traditional
QA (quality assurance) testing and playtesting. Traditional testing is about analyzing
and validating design and engineering implementations. Are there bugs or
glitches that need to logged and fixed? Are things working as intended? On the
other hand, playtesting is about the intended experience for players. Does an
event have the desired effect on the player? Does it convey the right mood? Is
it fun to play?
As playtest coordinator at Bungie, Duncan’s role is focused
on organizing playtest sessions across the studio. “[It’s] a very hybrid
role,” said Duncan, who has been with the company for almost three years. “It’s
more akin to a lab manager. We have to do a lot of similar to IT work, in terms
of troubleshooting [things] like hardware and software issues. We have the test
background, so we know how to debug through things, how we get audio up and
working.” There’s also the
organizational aspects, working with teams to decide which content is going to
be tested, which team members will attend the tests, and more.
A typical busy week at Bungie HQ will see the playtest labs
being used every day, with multiple sessions per week, scheduled and set up in
advance by Duncan and his team. It’s no wonder then, that the playtest labs are
some of the busiest parts of the studio during a normal week.
Enter the COVID-19 crisis and the idea of “normal” has been
thrown out the window. In late February, Bungie began a massive effort underway
across all parts of the studio to gear up for an extended period of remote
working. That meant everyone would be leaving the studio… including testing. No
more playtest labs, no more in-person sessions to play and discuss. In short,
things were changing. “It was the whole stages of grief,” said
Duncan, when asked about his initial reaction to the news that they weren’t going
to be allowed in the studio any longer. “There was definitely some denial at
first. My team specifically was like, ‘Oh, we don’t need to be home. There’s a
lot we can do in the studio.’”
Initially the team talked about using the
potential down time as an opportunity to tidy up and make some improvements
around the labs.
“[We thought] we could do things like
improve the hardware, do a lot of the manual labor that needs to be done to get
re-organized,” said Duncan.
At the same time, the team was already
aware that a pause in playtesting wasn’t going to be acceptable. “We [were]
thinking about how do we help teams playtest from home,” he said. “We were
doing an exploratory process [asking], ‘What are our different options? What is
viable and not viable?’”
The complications of remote testing while
working from home pile up quickly. There’s the basics of making sure that
everyone has a powerful enough machine to play on (in the case of PC testing).
Then there are the inherent security risks with removing expensive development
kits from the studio. And with internet connections being what they are, it’s
unreasonable to expect remote testers to use their home bandwidth to download a
new build of the game remotely each time they wanted to hold a testing session.
With these restrictions in mind, how do
you continue the rigorous testing schedule that is one of the keys to shipping
new Destiny experiences on time?
The answer, it turns out, was in the
cloud.
Jeff Fox has been with Bungie for seven
years, working as a test lead. Alongside Duncan and others on the Test team,
Fox has helped test a huge variety of Destiny gameplay, including activities,
matchmaking, networking, and more. His most recent project was launching
Destiny 2 on Google’s then-new Stadia streaming service.
“It was definitely unique because we’d
been historically on traditional platforms – Sony PlayStation, Xbox, PC,” Fox
said. “Going into this weird new streaming platform in the cloud was a unique
challenge. It was also really exciting from that perspective as well; once
you’ve been in QA long enough, everything is kind of very similar. So getting a
fresh new perspective on a different platform opens up new challenges [and]
it’s always great to move into that.”
By the launch of Destiny 2 on Stadia,
Bungie developers and testers were becoming more familiar with the platform. That
experience, combined with a workflow that was designed for ease of use, and it
wasn’t long after the studio-wide “work from home” orders were issued that the
idea of shifting a chunk of all-up testing onto the Stadia platform came up.
“Using Stadia in the ‘work from home’
transfer seemed like the easiest thing we could have done, and the fact that we
already had our game stood up on that platform made it kind of a no-brainer to
start looking into that,” Fox said.
Whereas a traditional test session is
preceded by a relatively lengthy process of propping a build onto multiple
consoles or PCs in the testing lab and working through any technical snafus
that may crop up, testing on Stadia was a relative breeze.
“On
Stadia we can publish a build in a way that all of the instances we use
automatically get the build distributed to them at the same time,” said Fox. “We’re
able to very easily get a pool of up to 300 instances or so with the game ready
to play at a click of a button, which is fantastic. You can’t do that any other
way when we’re running a big studio playtest like that.”
Testing also places a high level of
importance on uniformity of setup – in a playtesting lab, everyone is using the
same equipment as much as possible. As Fox pointed out, this is even easier
with Stadia. “It’s all cloud-based,” he said. “There is no physical hardware in
studio. You can use a variety of compatible controllers. The best thing about
developing Stadia [is that] there’s literally no hardware at all on the desk.
It’s all in the cloud so we didn’t have to worry about that at all.”
While playtesting with Stadia has its
distinct advantages – ease of setup for both players and coordinators, hardware
uniformity – it’s taken some work to get there. That’s in part because
traditional playtesting success is often the result of long-standing rituals
and routines. The teams schedule a test session, the coordinators work to get
the lab set up with the correct build, and everyone knows the process of where
and when they need to be at the lab. It’s a scripted routine that is the result
of a lot of learning over time.
Moving to a new system requires new
communication paths, and new rituals to form, not to mention the very real
changes that have come when an entire studio is learning to function remotely.
It’s something that Bungie’s Test team is still focusing on as the weeks go on.
“In terms of stabilizing and having a
normal day to day, we’ve just about got there,” Duncan said. “[The] first thing
was, let’s get teams playtesting again. Then we started to transition to
wrapping our old team rituals or processes back into the Stadia thing.”
While there’s still more work to go, the
team is feeling good about the progress made so far and the potential this kind
of cloud-based testing has for Bungie. That’s in no small part due to Bungie’s
willingness to adapt, said Fox.
“I was surprised at how quickly people
were able to pick up on the process,” Fox said. “Generally when you say, ‘Oh
yeah, we’ll stream it over the network. It will be fine!’ people are [going to
be] pretty skeptical. But the overall feedback has been really good. This is
working for us now during work from home.”
Duncan is even more bullish on the
future.
“This is going to change everything,” said Duncan. “We are in a new tradition. Just because we used to have every team
playtest in labs [in the past], it doesn’t mean we stop doing that. But the
process is evolving and changing. And we need to continue to be flexible.
“The future is now. What we thought was
impossible is definitely not the case.”
Tyler Duncan and Jeff Fox are on the front lines of what’s
happening with Destiny development. As members of the Bungie Test team, they help to
organize the testing of the game across many different groups within the
studio. They’re used to seeing the latest builds of the game and helping to coordinate
testing – from playing through scenarios and activities to organizing mass PVP sessions
involving dozens of Bungie employees.
Testing happens in every videogame studio
and the more testing you can do, the better. The function is so essential to a
studio’s day-to-day operations that it’s easy to take it for granted. From
designers to developers, engineers to marketing folks, Bungie team members are
often invited to playtest sessions. You show up to the playtest lab, put on the
headphones, play for a while, and then share your experiences with others. That
fundamental process of play, feedback, and communication is part of the
lifeblood of a game studio.
It’s so essential, in fact, that the alternative –
developing games without that rigorous testing – is unthinkable. So what
happens when all of these time-tested and finely tuned processes get upended by
something like the COVID-19 pandemic? How do you continue to carry on such an important
development practice when everyone is working remotely?
As Fox and Duncan will tell you, you get creative.
Before we talk about adapting to new circumstances, let’s
paint a picture of a typical test at Bungie. Like most game studios, Bungie has
playtest labs where developers can gather on regular basis to test a build of a
game. As you might expect, these labs are constantly busy, with lots of teams
looking to book time to get a session in and to share feedback with each other
about how a particular area of development is going. Maybe it’s the scenario
team who is testing out the difficulty of a particular mission, or the audio
team making sure that the weapon sounds are where they want them to be.
In general terms, testing can be broken up between traditional
QA (quality assurance) testing and playtesting. Traditional testing is about analyzing
and validating design and engineering implementations. Are there bugs or
glitches that need to logged and fixed? Are things working as intended? On the
other hand, playtesting is about the intended experience for players. Does an
event have the desired effect on the player? Does it convey the right mood? Is
it fun to play?
As playtest coordinator at Bungie, Duncan’s role is focused
on organizing playtest sessions across the studio. “[It’s] a very hybrid
role,” said Duncan, who has been with the company for almost three years. “It’s
more akin to a lab manager. We have to do a lot of similar to IT work, in terms
of troubleshooting [things] like hardware and software issues. We have the test
background, so we know how to debug through things, how we get audio up and
working.” There’s also the
organizational aspects, working with teams to decide which content is going to
be tested, which team members will attend the tests, and more.
A typical busy week at Bungie HQ will see the playtest labs
being used every day, with multiple sessions per week, scheduled and set up in
advance by Duncan and his team. It’s no wonder then, that the playtest labs are
some of the busiest parts of the studio during a normal week.
Enter the COVID-19 crisis and the idea of “normal” has been
thrown out the window. In late February, Bungie began a massive effort underway
across all parts of the studio to gear up for an extended period of remote
working. That meant everyone would be leaving the studio… including testing. No
more playtest labs, no more in-person sessions to play and discuss. In short,
things were changing. “It was the whole stages of grief,” said
Duncan, when asked about his initial reaction to the news that they weren’t going
to be allowed in the studio any longer. “There was definitely some denial at
first. My team specifically was like, ‘Oh, we don’t need to be home. There’s a
lot we can do in the studio.’”
Initially the team talked about using the
potential down time as an opportunity to tidy up and make some improvements
around the labs.
“[We thought] we could do things like
improve the hardware, do a lot of the manual labor that needs to be done to get
re-organized,” said Duncan.
At the same time, the team was already
aware that a pause in playtesting wasn’t going to be acceptable. “We [were]
thinking about how do we help teams playtest from home,” he said. “We were
doing an exploratory process [asking], ‘What are our different options? What is
viable and not viable?’”
The complications of remote testing while
working from home pile up quickly. There’s the basics of making sure that
everyone has a powerful enough machine to play on (in the case of PC testing).
Then there are the inherent security risks with removing expensive development
kits from the studio. And with internet connections being what they are, it’s
unreasonable to expect remote testers to use their home bandwidth to download a
new build of the game remotely each time they wanted to hold a testing session.
With these restrictions in mind, how do
you continue the rigorous testing schedule that is one of the keys to shipping
new Destiny experiences on time?
The answer, it turns out, was in the
cloud.
Jeff Fox has been with Bungie for seven
years, working as a test lead. Alongside Duncan and others on the Test team,
Fox has helped test a huge variety of Destiny gameplay, including activities,
matchmaking, networking, and more. His most recent project was launching
Destiny 2 on Google’s then-new Stadia streaming service.
“It was definitely unique because we’d
been historically on traditional platforms – Sony PlayStation, Xbox, PC,” Fox
said. “Going into this weird new streaming platform in the cloud was a unique
challenge. It was also really exciting from that perspective as well; once
you’ve been in QA long enough, everything is kind of very similar. So getting a
fresh new perspective on a different platform opens up new challenges [and]
it’s always great to move into that.”
By the launch of Destiny 2 on Stadia,
Bungie developers and testers were becoming more familiar with the platform. That
experience, combined with a workflow that was designed for ease of use, and it
wasn’t long after the studio-wide “work from home” orders were issued that the
idea of shifting a chunk of all-up testing onto the Stadia platform came up.
“Using Stadia in the ‘work from home’
transfer seemed like the easiest thing we could have done, and the fact that we
already had our game stood up on that platform made it kind of a no-brainer to
start looking into that,” Fox said.
Whereas a traditional test session is
preceded by a relatively lengthy process of propping a build onto multiple
consoles or PCs in the testing lab and working through any technical snafus
that may crop up, testing on Stadia was a relative breeze.
“On
Stadia we can publish a build in a way that all of the instances we use
automatically get the build distributed to them at the same time,” said Fox. “We’re
able to very easily get a pool of up to 300 instances or so with the game ready
to play at a click of a button, which is fantastic. You can’t do that any other
way when we’re running a big studio playtest like that.”
Testing also places a high level of
importance on uniformity of setup – in a playtesting lab, everyone is using the
same equipment as much as possible. As Fox pointed out, this is even easier
with Stadia. “It’s all cloud-based,” he said. “There is no physical hardware in
studio. You can use a variety of compatible controllers. The best thing about
developing Stadia [is that] there’s literally no hardware at all on the desk.
It’s all in the cloud so we didn’t have to worry about that at all.”
While playtesting with Stadia has its
distinct advantages – ease of setup for both players and coordinators, hardware
uniformity – it’s taken some work to get there. That’s in part because
traditional playtesting success is often the result of long-standing rituals
and routines. The teams schedule a test session, the coordinators work to get
the lab set up with the correct build, and everyone knows the process of where
and when they need to be at the lab. It’s a scripted routine that is the result
of a lot of learning over time.
Moving to a new system requires new
communication paths, and new rituals to form, not to mention the very real
changes that have come when an entire studio is learning to function remotely.
It’s something that Bungie’s Test team is still focusing on as the weeks go on.
“In terms of stabilizing and having a
normal day to day, we’ve just about got there,” Duncan said. “[The] first thing
was, let’s get teams playtesting again. Then we started to transition to
wrapping our old team rituals or processes back into the Stadia thing.”
While there’s still more work to go, the
team is feeling good about the progress made so far and the potential this kind
of cloud-based testing has for Bungie. That’s in no small part due to Bungie’s
willingness to adapt, said Fox.
“I was surprised at how quickly people
were able to pick up on the process,” Fox said. “Generally when you say, ‘Oh
yeah, we’ll stream it over the network. It will be fine!’ people are [going to
be] pretty skeptical. But the overall feedback has been really good. This is
working for us now during work from home.”
Duncan is even more bullish on the
future.
“This is going to change everything,” said Duncan. “We are in a new tradition. Just because we used to have every team
playtest in labs [in the past], it doesn’t mean we stop doing that. But the
process is evolving and changing. And we need to continue to be flexible.
“The future is now. What we thought was
impossible is definitely not the case.”